Skip to main content

Apache OpenDAL™ is now Graduated

· 6 min read
Xuanwo
Xuanwo

Hello, everyone! I'm happy to announce that Apache OpenDAL™ has graduated from the Apache Incubator to become a Top-Level Project of the Apache Software Foundation.

What's Apache OpenDAL?​

Apache OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. Our VISION is access data freely.

OpenDAL could be used as a better SDK for your storage services: A SDK with native integration of retry, logging, metrics, tracing, timeout, throttle, and more.

OpenDAL could be used as a super connector for your storage services: A connector that supports all kinds of storage services from Object Storage (s3, gcs, azblob), File Storage (fs, azdls, hdfs), Consumer Cloud Storage (gdrive, onedrive), Key-Value Storage (rocksdb, sled) to Cache Storage (memcached, moka).

OpenDAL could be used as an elegant client for your storage services: A client with well designed API and many language bindings: Rust, C, Cpp, Dotnet, Go, Haskell, Java, Lua, Node.js, Ocaml, Php, Python, Ruby, Swift and Zig.

Need to access data? Give OpenDAL a try!

async fn main() -> Result<()> {
    // Init s3 service.
    let mut builder = services::S3::default();
    builder.bucket("test");

    // Init an operator
    let op = Operator::via_map(builder)?
        // Add logging
        .layer(LoggingLayer::default())
        .finish();

    // Write data
    op.write("hello.txt", "Hello, World!").await?;

    // Read data
    let bs = op.read("hello.txt").await?;

    // Fetch metadata
    let meta = op.stat("hello.txt").await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    op.delete("hello.txt").await?;

    Ok(())
}

What's the ASF?​

The Apache Software Foundation (ASF) is a nonprofit corporation to support a number of open-source software projects. The Apache Software Foundation exists to provide software for the public good. We believe in the power of community over code, known as The Apache Way. Thousands of people around the world contribute to ASF open source projects every day.

The OpenDAL Community believes the apache way that:

  • Earned Authority: all individuals are given the opportunity to participate, but their influence is based on publicly earned merit – what they contribute to the community.
  • Community of Peers: individuals participate at the ASF, not organizations.
  • Open Communications: as a virtual organization, the ASF requires all communications related to code and decision-making to be publicly accessible to ensure asynchronous collaboration, as necessitated by a globally-distributed community.
  • Consensus Decision Making: Apache Projects are overseen by a self-selected team of active volunteers who are contributing to their respective projects.
  • Responsible Oversight: The ASF governance model is based on trust and delegated oversight.

The original creators Databend chosen to contribute OpenDAL to the ASF, embracing the Apache way through joining the incubator program.

What's graduation?​

In the Apache Incubator, the OpenDAL community is learning the Apache Way through daily development activities, growing its community and producing Apache releases.

During the incubation, we:

  • Consist of 19 committers, including mentors, with 12 serving as PPMC members.
  • Boast 164 contributors.
  • Made 9 releases—averaging at least one per month.
  • Had 7 different release managers to date.
  • Used by 10 known entities and is a dependency for 263 GitHub projects and 18 crates.io packages.
  • Opened 1,200+ issues with 1,100+ successfully resolved.
  • Submitted a total of 2,400+ PRs, most of them have been merged or closed.

The graduation signifies that the OpenDAL Community is recognized as a mature community, which entails:

  • CODE: OpenDAL is an Apache 2.0 licensed open-source project with accessible, buildable code on GitHub, featuring a traceable history and authenticated code provenance.
  • LICENSE: OpenDAL maintains open-source compliance for all code and dependencies, requires contributor agreements, and clearly documents copyright ownership.
  • Releases: OpenDAL offers standardized, committee-approved source code releases with secure signatures, provides convenience binaries, and has a well-documented, repeatable release process.
  • Quality: OpenDAL is committed to code quality transparency, prioritizes security with quick issue responses, ensures backward compatibility with clear documentation, and actively addresses bug reports in a timely manner.
  • Community: OpenDAL offers a comprehensive homepage, welcomes diverse contributions, promotes a meritocratic approach for active contributors, operates on community consensus, and ensures timely responses to user queries through various channels.
  • Consensus: OpenDAL has a public list of key decision-makers and uses a consensus approach for decisions, documented on its main communication channel. It follows standard voting rules and records all important discussions in writing.
  • Independence: OpenDAL is independent, with contributors from various companies acting on their own, not as representatives of any organization.

What's next?​

After graduation, OpenDAL Community will continue to focus on the following aspects under the VISION: access data freely.

More Stable Services​

OpenDAL now supports 59 services, although only some of them are stable.

stable for OpenDAL means that

  • Have integration tests covered.
  • Have at least one production user.

The stable service established a feedback loop between the OpenDAL community and its users. Users can submit bug reports or feature requests to the OpenDAL community, which in turn can enhance the service using this feedback while ensuring existing features remain intact.

After graduation, the OpenDAL community will focus on improving the stability of current services instead of just expanding our offerings.

We plan to:

More Useful Documents​

OpenDAL have good docs for its rust core, but not for other language bindings.

The lack of comprehensive documentation makes OpenDAL challenging for users to operate in Java or Python. Without user feedback, the community is unable to enhance this documentation, leading to a detrimental cycle that must be broken.

After graduation, the OpenDAL community will improve the documentation of other language bindings.

We plan to:

  • Introduce code generation to automatically create documentation for the service builder due to its numerous configurations.
  • Add more API Docs and examples for other language bindings.

OpenDAL have good docs for its public API, but not for its internal design.

OpenDAL is proud of its elegant design, but it is not well documented. This makes it difficult for new contributors to understand the codebase and make contributions.

After graduation, the OpenDAL community will improve the documentation of its internal design.

We plan to:

  • Optimize the codebase to make it easier to understand.
  • Add more blog posts to explain the design of OpenDAL.

More Production Users​

OpenDAL requires more production users, as they are vital to the success of our project. Increased user production leads to more valuable feedback, a more engaged contributor base, and a stronger community. We've started the initial loop; let's expand it!

After graduation, the OpenDAL community will focus on attracting more production users.

We plan to:

Conclusion​

The OpenDAL Community aims to create a world where users can freely access data across any storage service in any manner they choose. Graduation is just the beginning—let's work together to make our VISION a reality!