Crate opendal

Expand description

Apache OpenDAL™ is an Open Data Access Layer that enables seamless interaction with diverse storage services.

OpenDAL’s development is guided by its vision of One Layer, All Storage and its core principles: Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture. Read the explained vision at OpenDAL Vision.

§Quick Start

OpenDAL’s API entry points are Operator and blocking::Operator. All public APIs are accessible through the operator. To utilize OpenDAL, you need to:

Init a service
Compose layers
Use operator

§Init a service

The first step is to pick a service and init it with a builder. All supported services could be found at services.

Let’s take services::S3 as an example:

use opendal::services;
use opendal::Operator;
use opendal::Result;

fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default().bucket("test");

    // Init an operator
    let op = Operator::new(builder)?.finish();
    Ok(())
}

§Compose layers

The next setup is to compose layers. Layers are modules that provide extra features for every operation. All builtin layers could be found at layers.

Let’s use layers::LoggingLayer as an example; this layer adds logging to every operation that OpenDAL performs.

use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;
use opendal::Result;

#[tokio::main]
async fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default().bucket("test");

    // Init an operator
    let op = Operator::new(builder)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        .finish();

    Ok(())
}

§Use operator

The final step is to use the operator. OpenDAL supports both async Operator and blocking blocking::Operator. Please pick the one that fits your use case.

Every Operator API follows a consistent pattern. For example, consider the read operation:

Operator::read: Executes a read operation.
Operator::read_with: Executes a read operation with additional options using the builder pattern.
Operator::read_options: Executes a read operation with extra options provided via a options::ReadOptions struct.
Operator::reader: Creates a reader for streaming data, allowing for flexible access.
Operator::reader_with: Creates a reader with advanced options using the builder pattern.
Operator::reader_options: Creates a reader with extra options provided via a options::ReadOptions struct.

The Reader created by Operator supports custom read control methods and can be converted into [futures::AsyncRead] or [futures::Stream] for broader ecosystem compatibility.

use opendal::layers::LoggingLayer;
use opendal::options;
use opendal::services;
use opendal::Operator;
use opendal::Result;

#[tokio::main]
async fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default().bucket("test");

    // Init an operator
    let op = Operator::new(builder)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        .finish();

    // Fetch this file's metadata
    let meta = op.stat("hello.txt").await?;
    let length = meta.content_length();

    // Read data from `hello.txt` with options.
    let bs = op
        .read_with("hello.txt")
        .range(0..8 * 1024 * 1024)
        .chunk(1024 * 1024)
        .concurrent(4)
        .await?;

    // The same to:
    let bs = op
        .read_options("hello.txt", options::ReadOptions {
            range: (0..8 * 1024 * 1024).into(),
            chunk: Some(1024 * 1024),
            concurrent: 4,
            ..Default::default()
        })
        .await?;

    Ok(())
}

§Useful Links

Modules§

blockingblocking: blocking module provides blocking APIs for OpenDAL.
docsdocsrs: This module holds documentation for OpenDAL.
executors: executors module provides implementations for the Execute trait for widely used runtimes.
layers: Layer is the mechanism to intercept operations.
operator_futures: Futures provides the futures generated by Operator
options: Options module provides options definitions for operations.
raw: Raw modules provide raw APIs that used by underlying services
services: Services will provide builders to build underlying backends.

Structs§

Buffer: Buffer is a wrapper of contiguous Bytes and non-contiguous [Bytes].
BufferSink: BufferSink is the adapter of [futures::Sink] generated by Writer::into_sink
BufferStream: BufferStream is a stream of buffers, created by Reader::into_stream
Capability: Capability defines the supported operations and their constraints for a storage Operator.
DeleteInput: DeleteInput is the input for delete operations.
Deleter: Deleter is designed to continuously remove content from storage.
Entry: Entry returned by Lister or [BlockingLister] to represent a path and it’s relative metadata.
Error: Error is the error struct returned by all opendal functions.
Executor: Executor that runs futures in background.
FuturesAsyncReader: FuturesAsyncReader is the adapter of [AsyncRead], [AsyncBufRead] and [AsyncSeek] generated by Reader::into_futures_async_read.
FuturesAsyncWriter: FuturesIoAsyncWriter is the adapter of [AsyncWrite] for Writer.
FuturesBytesSink: FuturesBytesSink is the adapter of [futures::Sink] generated by Writer::into_bytes_sink.
FuturesBytesStream: FuturesBytesStream is the adapter of [Stream] generated by Reader::into_bytes_stream.
FuturesDeleteSink: FuturesDeleteSink is a sink that generated by Deleter
Lister: Lister is designed to list entries at given path in an asynchronous manner.
Metadata: Metadata contains all the information related to a specific path.
Operator: The Operator serves as the entry point for all public asynchronous APIs.
OperatorBuilder: OperatorBuilder is a typed builder to build an Operator.
OperatorInfo: Metadata for operator, users can use this metadata to get information of operator.
OperatorRegistry: Global registry that maps schemes to OperatorFactory functions.
OperatorUri: Parsed representation of an operator URI with normalized components.
Reader: Reader is designed to read data from given path in an asynchronous manner.
Writer: Writer is designed to write data into given path in an asynchronous manner.

Enums§

EntryMode: EntryMode represents the mode.
ErrorKind: ErrorKind is all kinds of Error of opendal.
Scheme: Services that OpenDAL supports

Statics§

DEFAULT_OPERATOR_REGISTRY: Default registry initialized with builtin services.

Traits§

Builder: Builder is used to set up underlying services.
Configurator: Configurator is used to configure the underlying service.
Execute: Execute trait is used to execute task in background.
IntoDeleteInput: IntoDeleteInput is a helper trait that makes it easier for users to play with Deleter.
IntoOperatorUri: Conversion trait that builds OperatorUri from various inputs.

Type Aliases§

OperatorFactory: Factory signature used to construct Operator from a URI and extra options.
Result: Result that is a wrapper of Result<T, opendal::Error>

Crate opendal

Crate opendal Copy item path

§Quick Start

§Init a service

§Compose layers

§Use operator

§Useful Links

Modules§

Structs§

Enums§

Statics§

Traits§

Type Aliases§

Crate opendal