opendal

Apache OpenDAL™ Python binding

Installation

pip install opendal

Usage

import opendal

op = opendal.Operator("fs", root="/tmp")
op.write("test.txt", b"Hello World")
print(op.read("test.txt"))
print(op.stat("test.txt").content_length)

Or using the async API:

import asyncio

async def main():
op = opendal.AsyncOperator("fs", root="/tmp")
await op.write("test.txt", b"Hello World")
print(await op.read("test.txt"))

asyncio.run(main())

 1# Licensed to the Apache Software Foundation (ASF) under one
 2# or more contributor license agreements.  See the NOTICE file
 3# distributed with this work for additional information
 4# regarding copyright ownership.  The ASF licenses this file
 5# to you under the Apache License, Version 2.0 (the
 6# "License"); you may not use this file except in compliance
 7# with the License.  You may obtain a copy of the License at
 8#
 9#   http://www.apache.org/licenses/LICENSE-2.0
10#
11# Unless required by applicable law or agreed to in writing,
12# software distributed under the License is distributed on an
13# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14# KIND, either express or implied.  See the License for the
15# specific language governing permissions and limitations
16# under the License.
17
18from ._opendal import *
19
20__doc__ = _opendal.__doc__
21__all__ = _opendal.__all__
class Operator:

Operator is the entry for all public blocking APIs

Create a new blocking Operator with the given scheme and options(**kwargs).

def layer(self, /, layer):

Add new layers upon existing operator

def open(self, /, path, mode):

Open a file-like reader for the given path.

def read(self, /, path):

Read the whole path into bytes.

def write(self, /, path, bs, **kwargs):

Write bytes into given path.

def stat(self, /, path):

Get current path's metadata without cache directly.

def copy(self, /, source, target):

Copy source to target.

def rename(self, /, source, target):

Rename filename.

def remove_all(self, /, path):

Remove all file

def create_dir(self, /, path):

Create a dir at given path.

Notes

To indicate that a path is a directory, it is compulsory to include a trailing / in the path. Failure to do so may result in NotADirectory error being returned by OpenDAL.

Behavior

  • Create on existing dir will succeed.
  • Create dir is always recursive, works like mkdir -p
def delete(self, /, path):

Delete given path.

Notes

  • Delete not existing error won't return errors.
def list(self, /, path):

List current dir path.

def scan(self, /, path):

List dir in flat way.

def capability(self, /):
def to_async_operator(self, /):
class AsyncOperator:

AsyncOperator is the entry for all public async APIs

Create a new AsyncOperator with the given scheme and options(**kwargs).

def layer(self, /, layer):

Add new layers upon existing operator

def open(self, /, path, mode):

Open a file-like reader for the given path.

def read(self, /, path):

Read the whole path into bytes.

def write(self, /, path, bs, **kwargs):

Write bytes into given path.

def stat(self, /, path):

Get current path's metadata without cache directly.

def copy(self, /, source, target):

Copy source to target.``

def rename(self, /, source, target):

Rename filename

def remove_all(self, /, path):

Remove all file

def create_dir(self, /, path):

Create a dir at given path.

Notes

To indicate that a path is a directory, it is compulsory to include a trailing / in the path. Failure to do so may result in NotADirectory error being returned by OpenDAL.

Behavior

  • Create on existing dir will succeed.
  • Create dir is always recursive, works like mkdir -p
def delete(self, /, path):

Delete given path.

Notes

  • Delete not existing error won't return errors.
def list(self, /, path):

List current dir path.

def scan(self, /, path):

List dir in flat way.

def presign_stat(self, /, path, expire_second):

Presign an operation for stat(head) which expires after expire_second seconds.

def presign_read(self, /, path, expire_second):

Presign an operation for read which expires after expire_second seconds.

def presign_write(self, /, path, expire_second):

Presign an operation for write which expires after expire_second seconds.

def capability(self, /):
def to_operator(self, /):
class File:

A file-like object. Can be used as a context manager.

def read(self, /, size=None):

Read and return at most size bytes, or if size is not given, until EOF.

def readline(self, /, size=None):

Read a single line from the file. A newline character (\n) is left at the end of the string, and is only omitted on the last line of the file if the file doesn’t end in a newline. If size is specified, at most size bytes will be read.

def readinto(self, /, buffer):

Read bytes into a pre-allocated, writable buffer

def write(self, /, bs):

Write bytes into the file.

def seek(self, /, pos, whence=0):

Change the stream position to the given byte offset. Offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are:

  • SEEK_SET or 0 – start of the stream (the default); offset should be zero or positive
  • SEEK_CUR or 1 – current stream position; offset may be negative
  • SEEK_END or 2 – end of the stream; offset is usually negative

Return the new absolute position.

def tell(self, /):

Return the current stream position.

def close(self, /):
def flush(self, /):

Flush the underlying writer. Is a no-op if the file is opened in reading mode.

def readable(self, /):

Return True if the stream can be read from.

def writable(self, /):

Return True if the stream can be written to.

def seekable(self, /):

Return True if the stream can be repositioned.

In OpenDAL this is limited to only readable streams.

closed

Return True if the stream is closed.

class AsyncFile:

A file-like async reader. Can be used as an async context manager.

def read(self, /, size=None):

Read and return at most size bytes, or if size is not given, until EOF.

def write(self, /, bs):

Write bytes into the file.

def seek(self, /, pos, whence=0):

Change the stream position to the given byte offset. offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are:

  • SEEK_SET or 0 – start of the stream (the default); offset should be zero or positive
  • SEEK_CUR or 1 – current stream position; offset may be negative
  • SEEK_END or 2 – end of the stream; offset is usually negative

Return the new absolute position.

def tell(self, /):

Return the current stream position.

def close(self, /):
def readable(self, /):

Check if the stream may be read from.

def writable(self, /):

Check if the stream may be written to.

def seekable(self, /):

Check if the stream reader may be re-located.

closed

Check if the stream is closed.

class Entry:
path

Path of entry. Path is relative to operator's root.

class EntryMode:
def is_file(self, /):

Returns True if this is a file.

def is_dir(self, /):

Returns True if this is a directory.

class Metadata:
mode

mode represent this entry's mode.

content_type

Content Type of this entry.

content_disposition
content_md5

Content MD5 of this entry.

etag

ETag of this entry.

content_length

Content length of this entry.

class PresignedRequest:
headers

Return the HTTP headers of this request.

method

Return the HTTP method of this request.

url

Return the URL of this request.

class Capability:

Capability is used to describe what operations are supported by current Operator.

read_with_override_cache_control

if operator supports read with override cache control.

write_can_multi

If operator supports write can be called in multi times.

list_with_recursive

If backend supports list without delimiter.

presign

If operator supports presign.

presign_read

If operator supports presign read.

write_multi_max_size

write_multi_max_size is the max size that services support in write_multi.

For example, AWS S3 supports 5GiB as max in write_multi.

shared

If operator supports shared.

list_with_limit

If backend supports list with limit.

batch_max_operations

The max operations that operator supports in batch.

write_with_content_disposition

If operator supports write with content disposition.

read_with_override_content_type

if operator supports read with override content type.

write

If operator supports write.

stat_with_if_none_match

If operator supports stat with if none match.

write_multi_min_size

write_multi_min_size is the min size that services support in write_multi.

For example, AWS S3 requires at least 5MiB in write_multi expect the last one.

list

If operator supports list.

list_with_start_after

If backend supports list with start after.

write_with_cache_control

If operator supports write with cache control.

read_with_if_none_match

If operator supports read with if none match.

batch_delete

If operator supports batch delete.

read

If operator supports read.

presign_stat

If operator supports presign stat.

read_with_if_match

If operator supports read with if match.

write_with_content_type

If operator supports write with content type.

stat

If operator supports stat.

create_dir

If operator supports create dir.

presign_write

If operator supports presign write.

batch

If operator supports batch.

stat_with_if_match

If operator supports stat with if match.

delete

If operator supports delete.

write_total_max_size

write_total_max_size is the max size that services support in write_total.

For example, Cloudflare D1 supports 1MB as max in write_total.

write_can_empty

If operator supports write with empty content.

read_with_override_content_disposition

if operator supports read with override content disposition.

blocking

If operator supports blocking.

rename

If operator supports rename.

copy

If operator supports copy.

write_can_append

If operator supports write by append.

class WriteOptions: