Module concurrent_write 

Concurrent Writes

OpenDAL writes parts sequentially by default. This keeps memory and request pressure low, but one in-flight request often cannot fill the available bandwidth on a high-latency path.

let mut writer = op.writer("large-object.bin").await?;
writer.write(first).await?;
writer.write(second).await?;
writer.close().await?;

Set concurrent above 1 to let services with multipart or block-based write support upload independent parts in parallel:

let mut writer = op
    .writer_with("large-object.bin")
    .concurrent(8)
    .await?;

writer.write(first).await?;
writer.write(second).await?;
writer.close().await?;

For multipart and block writers, OpenDAL schedules part uploads through its Executor and tracks their results in a task queue. A write call can return while earlier parts are still in flight when the queue has capacity. Writer::close drains the queue and performs the final commit, so applications must always check the result from close.

The storage service determines the native mechanism. S3 uses multipart upload, while Azure Blob uses block upload. Services without a concurrent write path continue to execute sequentially even when the option is present. Check Capability::write_can_multi before tuning multipart or block writes.

Tuning

Concurrent writes have two controls:

• concurrent is the maximum number of part uploads OpenDAL attempts to keep in flight. The default is 1, and values below 1 are normalized to 1.
• chunk is the target size of each part. Its default and valid range depend on the service.

concurrent

concurrent is an upper bound, not a guaranteed level. Setting it to 8 does not create eight active requests unless the input arrives quickly enough and the object contains enough parts. Object size, request latency, and service support can all reduce the observed concurrency.

Start with 2 or 4, then try 8 while measuring the real workload. Higher concurrency can improve bandwidth utilization, but it also increases in-flight requests, buffered data, and pressure on service request limits. Once the network or service is saturated, increasing it further can reduce throughput by causing throttling and retries.

chunk

OpenDAL buffers small writes until it can form a part of the configured size. This avoids issuing one storage request for every small input buffer. The final part may be smaller.

Larger chunks reduce request count and per-request overhead, but use more memory per in-flight part and require more data to be retried after a failed upload. Smaller chunks begin parallel work sooner, but increase request rate and may be invalid for the service. For example, S3 multipart upload has a 5 MiB minimum part size; OpenDAL raises smaller configured values to the service minimum.

Inspect Capability::write_multi_min_size and Capability::write_multi_max_size before overriding the service default. For large object-storage uploads, 8 MiB is a useful starting point, followed by measurements at the expected object sizes and concurrency.

One-shot and adapter APIs

write_with can split one in-memory buffer into concurrent parts:

let _metadata = op
    .write_with("large-object.bin", data)
    .chunk(8 * 1024 * 1024)
    .concurrent(4)
    .await?;

A configured writer keeps the same chunking and concurrency behavior after it is converted with Writer::into_sink, Writer::into_bytes_sink, or Writer::into_futures_async_write.

use std::io;

use futures::SinkExt;
use opendal_core::{Buffer, Operator};

async fn upload_with_sink(op: Operator, data: Buffer) -> io::Result<()> {
    let mut sink = op
        .writer_with("large-object.bin")
        .chunk(8 * 1024 * 1024)
        .concurrent(4)
        .await?
        .into_sink();

    sink.send(data).await?;
    sink.close().await?;
    Ok(())
}

Measure end-to-end throughput and completion latency together with peak memory, request count, retries, throttling, and the result from the final close.