Skip to main content

Python

Apache OpenDAL™ Python Binding

Status PyPI Website

This package provides a native Python binding for Apache OpenDAL™, a data access layer that allows you to access various storage services in a unified way.

OpenDAL Python Usage Demo

Features​

  • Unified API: Access S3, GCS, Azure Blob, HDFS, FTP, and more with the same set of commands.
  • Native Performance: Built in Rust for high performance and safety.
  • Async Support: First-class async API for modern Python applications.
  • Easy to Use: Simple and intuitive API design.

Installation​

Install the package directly from PyPI:

pip install opendal

Usage​

Here are a few examples of how to use OpenDAL with different storage backends.

Local Filesystem (fs)​

import opendal

# Initialize the operator for the local filesystem
op = opendal.Operator("fs", root="/tmp")

# Write data to a file
op.write("test.txt", b"Hello World")

# Read data from the file
content = op.read("test.txt")
print(op.read("test.txt"))

# Get metadata
metadata = op.stat("test.txt")
print(f"Content length: {metadata.content_length}") # Output: 11

Amazon S3​

The API remains the same—just change the scheme and credentials.

import opendal

# Initialize the operator for S3
op = opendal.Operator(
    "s3",
    bucket="your_bucket_name",
    region="your_region",
    root="/path/to/root"
)

op.write("test.txt", b"Hello World")
print(op.read("test.txt"))
print(op.stat("test.txt").content_length)

Async Usage (s3)​

OpenDAL also provides a fully asynchronous API.

import asyncio
import opendal

async def main():
    # Use AsyncOperator for async operations
    op = opendal.AsyncOperator("s3", root="/tmp", bucket="your_bucket_name", region="your_region")

    await op.write("test.txt", b"Hello World")
    print(await op.read("test.txt"))

asyncio.run(main())

Development​

This project uses just as a command runner to simplify the development workflow.

  1. Clone the repository and set up the environment:

    # This will create a virtual environment and install all dependencies
    just setup

  2. Run tests:

    # Example: Run tests for the 'fs' operator
    OPENDAL_TEST=fs OPENDAL_FS_ROOT=/tmp just test

For a complete guide on building, testing, and contributing, please see our CONTRIBUTING.md file.

Used By​

Check out the users list for more details on who is using OpenDAL.

License and Trademarks​

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.