docs
only.Expand description
Upgrade and migrate procedures while OpenDAL meets breaking changes.
§Upgrade to v0.46
§Public API
§MSRV Changed to 1.75
Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT
and AFIT
.
§Services Feature Flag
Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.
Additionally, we have removed all reqwest
-related feature flags:
- Users must now directly use
reqwest
’s feature flags for options likerustls
,native-tls
, etc. - The
rustls
feature is no longer enabled by default; it must be activated manually. - OpenDAL no longer offers the
trust-dns
option; users should configure the client builder directly.
§Range Based Read
Since v0.46, OpenDAL transformed it’s Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.
opendal::Reader
now have APIs like:
let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;
§Buffer Based IO
Since version 0.46, OpenDAL features a native Buffer
struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.
OpenDAL’s Reader
will return Buffer
and Writer
will accept Buffer
as input. Users who have implemented their own IO traits should update their code to use the new Buffer
struct.
let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;
let w = op.writer("test.txt").await?;
// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;
// Make sure file has been written completely.
w.close().await?;
To enhance usability, we’ve integrated bridges into bytes::Buf
and bytes::BufMut
, allowing users to directly interact with the bytes API.
let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;
let w = op.writer("test.txt").await?;
// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;
// Make sure file has been written completely.
w.close().await?;
§Bridge API
OpenDAL’s Reader
and Writer
previously implemented APIs such as AsyncRead
and AsyncWrite
directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.
Since v0.46, OpenDAL provides bridge APIs for Reader
and Writer
instead.
let r = op.reader("test.txt").await?;
// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);
let w = op.writer("test.txt").await?;
// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();
§Raw API
§Async in IO trait
Since version 0.46, OpenDAL has adopted Rust’s native async_in_trait
for our core IO traits, including oio::Read
, oio::Write
, and oio::List
.
This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.
Users who have implemented their own IO traits should update their code to use the new async trait syntax.
§Upgrade to v0.45
§Public API
§BlockingLayer is not enabled by default
To further enhance the optionality of tokio
, we have introduced a new feature called layers-blocking
. The default usage of the blocking layer has been disabled. To utilize the BlockingLayer
, please enable the layers-blocking
feature.
§TimeoutLayer deprecated with_speed
The with_speed
API has been deprecated. Please use with_io_timeout
instead.
§Raw API
No raw API changes.
§Upgrade to v0.44
§Public API
§Moka Service Configuration
- The
thread_pool_enabled
option has been removed.
§List Prefix Supported
After RFC: List Prefix landed, we have changed the behavior of list
a path without /
. OpenDAL used to return NotADirectory
error, but now we will return the list of entries that start with given prefix instead.
§Upgrade to v0.43
§Public API
§List Recursive
After RFC-3526: List Recursive landed, we have changed the list
API to accept recursive
instead of delimiter
:
Users will need to change the following usage:
op.list_with(path).delimiter("")
->op.list_with(path).recursive(true)
op.list_with(path).delimiter("/")
->op.list_with(path).recursive(false)
delimiter
other than ""
and "/"
is not supported anymore.
§Stat a dir path
After RFC: List Prefix landed, we have changed the behavior of stat
a dir path:
Here are the behavior list:
Case | Path | Result |
---|---|---|
stat existing dir | abc/ | Metadata with dir mode |
stat existing file | abc/def_file | Metadata with file mode |
stat dir without / | abc/def_dir | Error NotFound or metadata with dir mode |
stat file with / | abc/def_file/ | Error NotFound |
stat not existing path | xyz | Error NotFound |
Services like s3, azblob can handle stat("abc/")
correctly by check if there are objects with prefix abc/
.
§Raw API
§Lister Align
We changed our internal lister
implementation to align with the list
public API for better performance and readability.
- trait
Page
=>List
- struct
Pager
=>Lister
- trait
BlockingPage
=>BlockingList
- struct
BlockingPager
=>BlockingLister
Every call to next
will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of async_trait
.
§Upgrade to v0.42
§Public API
§MSRV Changed
OpenDAL bumps it’s MSRV to 1.67.0.
§S3 Service Configuration
- The
enable_exact_buf_write
option has been deprecated and is superseded byBufferedWriter
, introduced in version 0.40.
§Oss Service Configuration
- The
write_min_size
option has been deprecated and replaced byBufferedWriter
, also introduced in version 0.40. - A new setting,
allow_anonymous
, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enablingallow_anonymous
to fallback to request without credentials.
§Ghac Service Configuration
- The
enable_create_simulation
option has been removed. We add this option to allow ghac simulate create empty file, but it could result in unexpected behavior when users create a file with content length1
. So we remove it.
§Wasabi Service Removed
wasabi
service native support has been removed. Users who want to access wasabi can use our s3
service instead.
§Upgrade to v0.41
There is no public API and raw API changes.
§Upgrade to v0.40
§Public API
§RFC-2578 Merge Append Into Write
RFC-2578 merges append
into write
and removes append
API.
- For writing a file at once, please use
op.write()
for convenience. - For appending a file, please use
op.write_with().append(true)
instead ofop.append()
.
The same rule applies to writer()
and writer_with()
.
§RFC-2774 Lister API
RFC-2774 proposes a new lister
API to replace current list
and scan
. And we add a new API list
to return entries directly.
- For listing a directory at once, please use
list()
for convenience. - For listing a directory recursively, please use
list_with().delimiter("")
orlister_with().delimiter("")
instead ofscan()
. - For listing in streaming, please use
lister()
orlister_with()
instead.
§RFC-2779 List With Metakey
RFC-2779 proposes a new op.list_with().metakey()
API to allow list with metakey and removes op.metadata(&entry)
API.
Please use op.list_with().metakey()
instead of op.metadata(&entry)
, for example:
// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entris {
let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
println!("{} {}", entry.name(), entry.metadata().content_length());
}
// After
let entries: Vec<Entry> = op
.list_with("dir/")
.metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entris {
println!("{} {}", entry.name(), entry.metadata().content_length());
}
§RFC-2852: Native Capability
RFC-2852 proposes new native_capability
and full_capability
API to allow users to check if the underlying service supports a capability natively.
native_capability
returnstrue
if the capability is supported natively.full_capability
returnstrue
if the capability is supported, maybe via a layer.
Most of time, you can use full_capability
to replace capability
call. But to check if the capability is supported natively for better performance design, please use native_capability
instead.
§Buffered Writer
OpenDAL v0.40 added buffered writer support!
Users don’t need to specify the content_length()
for writer anymore!
- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;
Users can specify the buffer()
to control the size we call underlying storage:
let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;
If buffer is not specified, we will call underlying storage everytime we call write
. Otherwise, we will make sure to call underlying storage when buffer is full or close
is called.
§RangeRead and RangeReader
OpenDAL v0.40 removed the origin range_read
and range_reader
interfaces, please use read_with().range()
or reader_with().range()
.
- op.range_read(path, range_start..range_end).await?;
+ op.read_with(path).range(range_start..range_end).await?;
- let reader = op.range_reader(path, range_start..range_end).await?;
+ let reader = op.reader_with(path).range(range_start..range_end).await?;
§Raw API
§RFC-3017 Remove Write Copy From
RFC-3017 removes copy_from
API from the oio::Write
trait. Users who implements services and layers by hand should remove this API.
§Upgrade to v0.39
§Public API
§Service S3 Role Arn Behavior
In PR #2687, OpenDAL changed the behavior when role_arn
has been specified.
OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified role_arn
and external_id
(if supplied).
§RetryLayer supports RetryInterceptor
In PR #2666, RetryLayer
supports RetryInterceptor
. To implement this change, RetryLayer
changed it’s in-memory layout by adding a new generic parameter I
to RetryLayer<I>
.
Users who stores RetryLayer
in struct or enum will need to change the type if they don’t want to use default behavior.
§Raw API
In PR #2698, OpenDAL re-org the internal structure of opendal::raw::oio
and changed some APIs name.
§Upgrade to v0.38
There are no public API changes.
§Raw API
OpenDAL add the Write::sink
API to enable streaming writing. This is a breaking change for users who depend on the raw API.
For a quick fix, users who have implemented opendal::raw::oio::Write
can return an Unsupported
error for Write::sink()
.
More details could be found at RFC: Writer sink
API.
§Upgrade to v0.37
In v0.37.0, OpenDAL bump the version of reqsign
to v0.13.0.
There are no public API and raw API changes.
§Upgrade to v0.36
§Public API
In v0.36, OpenDAL improving the xxx_with
API by allow it to be called in chain:
After this change, all xxx_with
alike call will be changed from
let bs = op.read_with(
"path/to/file",
OpRead::new()
.with_range(0..=1024)
.with_if_match("<etag>")
.with_if_none_match("<etag>")
.with_override_cache_control("<cache_control>")
.with_override_content_disposition("<content_disposition>")
).await?;
to
let bs = op.read_with("path/to/file")
.range(0..=1024)
.if_match("<etag>")
.if_none_match("<etag>")
.override_cache_control("<cache_control>")
.override_content_disposition("<content_disposition>")
.await?;
For blocking API calls, we will need a call()
at the end:
let bs = bop.read_with("path/to/file")
.range(0..=1024)
.if_match("<etag>")
.if_none_match("<etag>")
.override_cache_control("<cache_control>")
.override_content_disposition("<content_disposition>")
.call()?;
Along with this change, users don’t need to call OpXxx
anymore so we moved it to raw
API.
More details could be found at RFC: Chain Based Operator API.
§Raw API
Migrated opendal::ops
to opendal::raw::ops
.
§Upgrade to v0.35
§Public API
- OpenDAL removes rarely used
Operator::from_env
andOperator::from_iter
APIs- Users can use
Operator::via_map
instead.
- Users can use
§Raw API
- OpenDAL adds
append
support with could break existing layers. Please make sureappend
requests have been forward correctly. - After the merging of
scan
andlist
, OpenDAL removes thescan
from raw API. Please uselist_without_delimiter
instead.
§Upgrade to v0.34
§Public API
- OpenDAL raises it’s MSRV to 1.65 for dependencies changes
OperatorInfo::can_scan
has been removed, to check if underlying services support scan a dir natively, please useCapability::list_without_delimiter
instead.
§Raw API
§Merged scan
into list
After Capability
introduced, we have added delimiter
in OpList
. Users can specify the delimiter to ""
or "/"
to control the list behavior.
Along with this change, Operator::scan()
becomes a short alias of Operator::list_with(OpList::new().with_delimiter(""))
.
§Typed Kv Adapter
In v0.34, OpenDAL adds a typed kv adapter for zero-copy read and write. If you are implemented kv adapter for a rust in-memory data struct, please consider migrate.
§Upgrade to v0.33
§Public API
OpenDAL 0.33 has redesigned the Writer
API, replacing all instances of writer.append()
with writer.write()
. For more information, please refer to Writer
.
§Raw API
In addition to the redesign of the Writer
API, we have removed append
from oio::Write
. Therefore, users who implement services and layers should also remove it.
After v0.33 landing, services should handle OpWrite::content_length
correctly by following these guidelines:
- If the writer does not support uploading unsized data, return a response of
NotSupported
if content length isNone
. - Otherwise, continue writing data until either
close
orabort
has been called.
Furthermore, OpenDAL 0.33 introduces a new concept called Capability
which replaces AccessorCapability
. Services must adapt to this change.
§Upgrade to v0.32
OpenDAL 0.32 doesn’t have much breaking changes.
We changed Accessor::create
into Accessor::create_dir
. Only users who implement Layer
need to change.
§Upgrade to v0.31
In version v0.31 of OpenDAL, we made some internal refactoring to improve its compatibility with the ecosystem.
§MSRV Bump
We increased the MSRV to 1.64 from v0.31 onwards. Although it is still possible to build OpenDAL under older rustc versions, we cannot guarantee that any issues related to them will be fixed.
§Accept std::time::Duration
instead
Previously, OpenDAL accepted time::Duration
as input for presign_xxx
. However, since v0.31, we have changed this to accept std::time::Duration
so that users do not need to depend on time
. Internally, we migrated from time
to chrono
for better integration with other parts of the ecosystem.
§disable_ec2_metadata
for services s3
We have added a new configuration option called disable_ec2_metadata
for the S3 service in response to a mistake where it was mixed up with another option called disable_config_load
. Users who want to disable loading credentials from EC2 metadata should set this option instead.
§Services Feature Flag
Starting from v0.31, all services in OpenDAL are split into different feature flags. To enable only S3 support, use the following TOML configuration:
opendal = {
version = "0.31",
default-features = false,
features = ["services-s3"]
}
§Upgrade to v0.30
In version 0.30, we made significant breaking changes by removing objects. Our goal in doing so was to provide our users with APIs that are easier to understand and maintain.
More details could be found at RFC: Remove Object Concept.
To upgrade to OpenDAL v0.30, users need to make the following changes:
- regex replace
object\((.*)\).reader\(\)
toreader($1)
- replace the function on your case, it’s recommended to do it one by one
- rename
ObjectMetakey
=>Metakey
- rename
ObjectMode
=>EntryMode
- replace
ErrorKind::ObjectXxx
toErrorKind::Xxx
- rename
AccessorMetadata
=>AccessorInfo
- rename
ObjectMetadata
=>Metadata
- replace
operator.metadata()
=>operator.info()
§Upgrade to v0.29
In v0.29, we introduced Object Writer to replace existing Multipart related APIs.
Users can now append multiparts bytes into object via:
let mut w = o.writer().await?;
w.write(bs1).await?;
w.write(bs2).await?;
w.close()
Along with this change, we cleaned up a lot of internal structs and traits. Users who used to depend on opendal::raw::io::{input,output}
should use opendal::raw::oio
instead.
Also, decompress related feature also removed. Users can use async-compression
with ObjectReader
directly.
§Upgrade to v0.28
In v0.28, we introduced Query Based Metadata. Users can query cached metadata with ObjectMetakey
to make sure that OpenDAL always makes the best decision.
- pub async fn metadata(&self) -> Result<ObjectMetadata>;
+ pub async fn metadata(
+ &self,
+ flags: impl Into<FlagSet<ObjectMetakey>>,
+ ) -> Result<Arc<ObjectMetadata>>;
Please visit Object::metadata()
’s example for more details.
§Upgrade to v0.27
In v0.27, we refactored our list
related logic and added scan
support. So make Pager
and BlockingPager
associated types in Accessor
too!
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
+ type Pager: output::Page;
+ type BlockingPager: output::BlockingPage;
}
§User defined layers
Due to this change, all layers implementation should be changed. If there is not changed over pager, they can be changed like the following:
impl<A: Accessor> LayeredAccessor for MyAccessor<A> {
type Inner = A;
type Reader = MyReader<A::Reader>;
type BlockingReader = MyReader<A::BlockingReader>;
+ type Pager = A::Pager;
+ type BlockingPager = A::BlockingPager;
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> {
+ self.inner.list(path, args).await
+ }
+ async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> {
+ self.inner.scan(path, args).await
+ }
+ fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> {
+ self.inner.blocking_list(path, args)
+ }
+ fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> {
+ self.inner.blocking_scan(path, args)
+ }
}
§Usage of ops
To reduce the understanding overhead, we move all OpXxx
into opendal::ops
now. User may need to change:
- use opendal::OpWrite;
+ use opendal::ops::OpWrite;
§Usage of RetryLayer
backon
is the implementation detail of our RetryLayer
, so we hide it from our public API. Users of RetryLayer
need to change the code like:
- RetryLayer::new(backon::ExponentialBackoff::default())
+ RetryLayer::new()
§Upgrade to v0.26
In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost.
Due to this change, we have to refactor the logic of Operator
’s init logic. In v0.26, we added opendal::Builder
trait and opendal::OperatorBuilder
. For the first glance, the only change to existing code will be like:
- let op = Operator::new(builder.build()?);
+ let op = Operator::new(builder.build()?).finish();
By adding a finish()
call, we will erase all generic types so that Operator
can still be easily used everywhere as before.
§Accessor
In v0.26, Accessor
has been changed into trait with associated types.
All services need to declare the types returned as Reader
or BlockingReader
:
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}
If your service doesn’t support read
or blocking_read
, we can use ()
to represent a dummy reader:
impl Accessor for MyDummyAccessor {
type Reader = ();
type BlockingReader = ();
}
§Layer
As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new Layer
and LayeredAccessor
trait:
pub trait Layer<A: Accessor> {
type LayeredAccessor: Accessor;
fn layer(&self, inner: A) -> Self::LayeredAccessor;
}
#[async_trait]
pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static {
type Inner: Accessor;
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}
LayeredAccessor
is a wrapper of Accessor
with the typed Innder
. All methods that not implemented will be forward to inner instead.
§Builder
Since v0.26, we implement opendal::Builder
for all services, and services’ mod will not be exported.
- use opendal::services::s3::Builder;
+ use opendal::services::S3;
§Conclusion
Sorry again for the big changes in this release. It’s a big step for OpenDAL to work in more critical systems.
§Upgrade to v0.25
In v0.25, we bring the same feature sets from ObjectReader
to BlockingObjectReader
.
Due to this change, all code that depends on BlockingBytesReader
should be refactored.
BlockingBytesReader
=>input::BlockingReader
BlockingOutputBytesReader
=>output::BlockingReader
Most changes only happen inside. Users not using opendal::raw::*
will not be affected.
Apart from this change, we refactored s3 credential loading logic. After this change, we can disable the config load instead of the credential methods.
builder.disable_credential_loader
=>builder.disable_config_load
§Upgrade to v0.24
In v0.24, we made a big refactor on our internal IO-related traits. In this version, we split our IO traits into input
and output
versions:
Take Reader
as an example:
input::Reader
is the user input reader, which only requires futures::AsyncRead + Send
.
output::Reader
is the reader returned by OpenDAL
, which implements futures::AsyncRead
, futures::AsyncSeek
, and futures::Stream<Item=io::Result<Bytes>>
. Besides, output::Reader
also implements Send + Sync
, which makes it useful for users.
Due to this change, all code that depends on BytesReader
should be refactored.
BytesReader
=>input::Reader
OutputBytesReader
=>output::Reader
Thanks to the change of IO trait split, we make ObjectReader
implements all needed traits:
futures::AsyncRead
futures::AsyncSeek
futures::Stream<Item=io::Result<Bytes>>
Thus, we removed the seekable_reader
API. They can be replaced by range_reader
:
o.seekable_reader
=>o.range_reader
Most changes only happen inside. Users not using opendal::raw::*
will not be affected.
Sorry for the inconvenience. I think those changes are required and make OpenDAL better! Welcome any comments at Discussion.
§Upgrade to v0.21
v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor
APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.
Let’s start with our errors.
§Error Handling
As described in RFC-0977: Refactor Error, we refactor opendal error by a new error
called opendal::Error
.
This change will affect all APIs that are used to return io::Error
.
To migrate this, please replace std::io::Error
with opendal::Error
:
- use std::io::Result;
+ use opendal::Result;
And the following error kinds should be updated:
std::io::ErrorKind::NotFound
=>opendal::ErrorKind::ObjectNotFound
std::io::ErrorKind::PermissionDenied
=>opendal::ErrorKind::ObjectPermissionDenied
And since v0.21, we will return errors ObjectIsADirectory
and ObjectNotADirectory
instead of anyhow::Error
.
§Accessor API
In v0.21, we refactor the whole Accessor
’s API:
- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
+ async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>
Since v0.21, we will return a reply struct for different operations called RpWrite
instead of an exact type. We can split OpenDAL’s public API and raw API with this change.
§ObjectList and Page
Since v0.21, Accessor
will return Pager
for List
:
- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, output::Pager)>
And Object
will return an ObjectLister
which is built upon Page
:
pub async fn list(&self) -> Result<ObjectLister> { ... }
ObjectLister
can be used as an object stream as before. It also provides the function next_page
to get the underlying pages directly:
impl ObjectLister {
pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
}
§Code Layout
Since v0.21, we have categorized all APIs into public
and raw
.
Public APIs are exposed under opendal::Xxx
; they are user-face APIs that are easy to use and understand.
Raw APIs are exposed under opendal::raw::Xxx
; they are implementation details for underlying services and layers.
Please replace all usage of opendal::io_util::*
and opendal::http_util::*
to opendal::raw::*
instead.
With this change, new users of OpenDAL maybe be it easier to get started.
§Summary
Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.
§Upgrade to v0.20
v0.20 is a big release that we introduce a lot of performance related changes.
To make the best of information from read
operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader
now!
pub struct ObjectReader {
inner: BytesReader
meta: ObjectMetadata,
}
impl ObjectReader {
pub fn content_length(&self) -> u64 {}
pub fn last_modified(&self) -> Option<OffsetDateTime> {}
pub fn etag(&self) -> Option<String> {}
}
To make this happen, we changed our Accessor
API:
- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}
All layers should be updated to meet this change. Also, it’s required to return content_length
while building ObjectReader
. Please make sure the returning ObjectMetadata
is used correctly.
§Upgrade to v0.19
OpenDAL deprecate some features:
serde
: We will enable it by default.layers-retry
: We will enable retry support by default.layers-metadata-cache
: We will enable it by default.
Deprecated types like DirEntry
has been removed.
§Upgrade to v0.18
OpenDAL v0.18 introduces the following breaking changes:
- Deprecated feature flag
services-http
has been removed. - All
DirXxx
items have been renamed toObjectXxx
to make them more consistent.DirEntry
->Entry
DirStream
->ObjectStream
DirStreamer
->ObjectStream
DirIterate
->ObjectIterate
DirIterator
->ObjectIterator
Besides, we also make a big change to our Entry
API. Since v0.18, we can fully reuse the metadata that fetched during list
. Take entry.content_length()
for example:
- If
content_length
is already known, we will return directly. - If not, we will check if the object entry is
complete
:- If
complete
, the entry already fetched all metadata that it could have, return directly. - If not, we will send a
stat
call to get themetadata
and refresh our cache.
- If
This change means:
- All API like
content_length
will be changed into async functions. metadata
andblocking_metadata
will not return errors anymore.- To retrieve the latest meta, please use
entry.into_object().metadata()
instead.
§Upgrade to v0.17
OpenDAL v0.17 refactor the Accessor
to make space for future features.
We move path String
out of the OpXxx
to function args so that we don’t need to clone twice.
- async fn read(&self, args: OpRead) -> Result<BytesReader>
+ async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>
For more information about this change, please refer to RFC-0661: Path In Accessor.
And since OpenDAL v0.17, we will use rustls
as default tls engine for our underlying http client. Since this release, we will not depend on openssl
anymore.
§Upgrade to v0.16
OpenDAL v0.16 refactor the internal implementation of http
service. Since v0.16, http service can be used directly without enabling services-http
feature. Accompany by these changes, http service has the following breaking changes:
services-http
feature has been deprecated. Enablingservices-http
is a no-op now.- http service is read only services and can’t be used to
list
orwrite
.
OpenDAL introduces a new layer ImmutableIndexLayer
that can add list
capability for services:
use opendal::layers::ImmutableIndexLayer;
use opendal::Operator;
use opendal::Scheme;
async fn main() {
let mut iil = ImmutableIndexLayer::default();
for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] {
iil.insert(i.to_string())
}
let op = Operator::from_env(Scheme::Http)?.layer(iil);
}
For more information about this change, please refer to RFC-0627: Split Capabilities.
§Upgrade to v0.14
OpenDAL v0.14 removed all deprecated APIs in previous versions, including:
Operator::with_backoff
in v0.13- All services
Builder::finish()
in v0.12 - All services
Backend::build()
in v0.12
Please visit related version’s upgrade guide for migration.
And in OpenDAL v0.14, we introduce a break change for write
operations.
pub trait Accessor {
- async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
+ async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
}
The following APIs have affected by this change:
Object::write
now acceptimpl Into<Vec<u8>>
instead ofAsRef<&[u8]>
Object::writer
has been removed.Object::write_from
has been added to support write from a reader.- All layers should be refactored to adapt new
Accessor
trait.
For more information about this change, please refer to RFC-0554: Write Refactor.
§Upgrade to v0.13
OpenDAL deprecate Operator::with_backoff
since v0.13.
Please use RetryLayer
instead:
use anyhow::Result;
use backon::ExponentialBackoff;
use opendal::layers::RetryLayer;
use opendal::Operator;
use opendal::Scheme;
let _ = Operator::from_env(Scheme::Fs)
.expect("must init")
.layer(RetryLayer::new(ExponentialBackoff::default()));
§Upgrade to v0.12
OpenDAL introduces breaking changes for services initiation.
Since v0.12, Operator::new
will accept impl Accessor + 'static
instead of Arc<dyn Accessor>
:
impl Operator {
pub fn new(accessor: impl Accessor + 'static) -> Self { .. }
}
Every service’s Builder
now have a build()
API which can be run without async:
let mut builder = fs::Builder::default();
let op: Operator = Operator::new(builder.build()?);
Along with these changes, Operator::from_iter
and Operator::from_env
now is a blocking API too.
For more information about this change, please refer to RFC-0501: New Builder.
The following APIs have been deprecated:
- All services
Builder::finish()
(replaced byBuilder::build()
) - All services
Backend::build()
(replace byBuilder::default()
)
The following APIs have been removed:
- public struct
Metadata
(deprecated in v0.8, replaced byObjectMetadata
)
§Upgrade to v0.8
OpenDAL introduces a breaking change of list
related operations in v0.8.
Since v0.8, list
will return DirStreamer
instead:
pub trait Accessor: Send + Sync + Debug {
async fn list(&self, args: &OpList) -> Result<DirStreamer> {}
}
DirStreamer
streams DirEntry
which carries ObjectMode
, so that we don’t need an extra call to get object mode:
impl DirEntry {
pub fn mode(&self) -> ObjectMode {
self.mode
}
}
And DirEntry
can be converted into Object
without overhead:
let o: Object = de.into()
Since v0.8
, opendal::Metadata
has been deprecated by opendal::ObjectMetadata
.
§Upgrade to v0.7
OpenDAL introduces a breaking change of decompress_read
related in v0.7.
Since v0.7, decompress_read
and decompress_reader
will return Ok(None)
while OpenDAL can’t detect the correct compress algorithm.
impl Object {
pub async fn decompress_read(&self) -> Result<Option<Vec<u8>>> {}
pub async fn decompress_reader(&self) -> Result<Option<impl BytesRead>> {}
}
So users should match and check the None
case:
let bs = o.decompress_read().await?.expect("must have valid compress algorithm");
§Upgrade to v0.4
OpenDAL introduces many breaking changes in v0.4.
§Object::reader() is not AsyncSeek
anymore
Since v0.4, Object::reader()
will return impl BytesRead
instead of Reader
that implements AsyncRead
and AsyncSeek
. Users who want AsyncSeek
please wrapped with opendal::io_util::seekable_read
:
use opendal::io_util::seekable_read;
let o = op.object("test");
let mut r = seekable_read(&o, 10..);
r.seek(SeekFrom::Current(10)).await?;
let mut bs = vec![0;10];
r.read(&mut bs).await?;
§Use RangeBounds instead
Since v0.4, the following APIs will be removed.
Object::limited_reader(size: u64)
Object::offset_reader(offset: u64)
Object::range_reader(offset: u64, size: u64)
Instead, OpenDAL is providing a more general range_reader
powered by RangeBounds
:
pub async fn range_reader(&self, range: impl RangeBounds<u64>) -> Result<impl BytesRead>
Users can use their familiar rust range syntax:
let r = o.range_reader(1024..2048).await?;
§Return io::Result instead
Since v0.4, all functions in OpenDAL will return std::io::Result
instead.
Please check via std::io::ErrorKind
directly:
use std::io::ErrorKind;
if let Err(e) = op.object("test_file").metadata().await {
if e.kind() == ErrorKind::NotFound {
println!("object not exist")
}
}
§Removing Credential
Since v0.4, Credential
has been removed, please use the API provided by Builder
directly.
builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");
§Write returns BytesWriter
instead
Since v0.4, Accessor::write
will return a BytesWriter
instead accepting a BoxedAsyncReader
.
Along with this change, the old Writer
has been replaced by a new set of write functions:
pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}
Users can write into an object more easily:
let _ = op.object("path/to/file").write("Hello, World!").await?;
§io_util
replaces readers
Since v0.4, mod io_util
will replace readers
. In io_utils
, OpenDAL provides helpful functions like:
into_reader
: ConvertBytesStream
intoBytesRead
into_sink
: ConvertBytesWrite
intoBytesSink
into_stream
: ConvertBytesRead
intoBytesStream
into_writer
: ConvertBytesSink
intoBytesWrite
observe_read
: Add callback forBytesReader
observe_write
: Add callback forBytesWrite
§New type alias
For better naming, types that OpenDAL returns have been renamed:
AsyncRead + Unpin + Send
=>BytesRead
BoxedAsyncReader
=>BytesReader
AsyncWrite + Unpin + Send
=>BytesWrite
BoxedAsyncWriter
=>BytesWriter
ObjectStream
=>ObjectStreamer