Struct opendal::services::Webhdfs

pub struct Webhdfs {
    pub atomic_write_dir: Option<String>,
    /* private fields */
}

Available on crate feature services-webhdfs only.

WebHDFS’s REST API support. There two implementations of WebHDFS REST API:

• Native via HDFS Namenode and Datanode, data are transferred between nodes directly.
• HttpFS is a gateway before hdfs nodes, data are proxied.

Capabilities

This service can be used to:

• stat
• read
• write
• create_dir
• delete
• copy
• rename
• list
• scan
• presign
• blocking

Differences with HDFS

Hdfs is powered by HDFS’s native java client. Users need to set up the HDFS services correctly. But webhdfs can access from HTTP API and no extra setup needed.

WebHDFS Compatibility Guidelines

File Creation and Write

For File creation and write operations, OpenDAL WebHDFS is optimized for Hadoop Distributed File System (HDFS) versions 2.9 and later. This involves two API calls in webhdfs, where the initial put call to the namenode is redirected to the datanode handling the file data. The optional noredirect flag can be set to prevent redirection. If used, the API response body contains the datanode URL, which is then utilized for the subsequent put call with the actual file data. OpenDAL automatically sets the noredirect flag with the first put call. This feature is supported starting from HDFS version 2.9.

Multi-Write Support

OpenDAL WebHDFS supports multi-write operations by creating temporary files in the specified atomic_write_dir. The final concatenation of these temporary files occurs when the writer is closed. However, it’s essential to be aware of HDFS concat restrictions for earlier versions, where the target file must not be empty, and its last block must be full. Due to these constraints, the concat operation might fail for HDFS 2.6. This issue, identified as HDFS-6641, has been addressed in later versions of HDFS.

In summary, OpenDAL WebHDFS is designed for optimal compatibility with HDFS, specifically versions 2.9 and later.

Configurations

• root: The root path of the WebHDFS service.
• endpoint: The endpoint of the WebHDFS service.
• delegation: The delegation token for WebHDFS.
• atomic_write_dir: The tmp write dir of multi write for WebHDFS.Needs to be configured for multi write support.

Refer to Builder’s public API docs for more information.

Examples

Via Builder

use std::sync::Arc;

use anyhow::Result;
use opendal::services::Webhdfs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = Webhdfs::default();
    // set the root for WebHDFS, all operations will happen under this root
    //
    // Note:
    // if the root is not exists, the builder will automatically create the
    // root directory for you
    // if the root exists and is a directory, the builder will continue working
    // if the root exists and is a folder, the builder will fail on building backend
    builder.root("/path/to/dir");
    // set the endpoint of webhdfs namenode, controlled by dfs.namenode.http-address
    // default is http://127.0.0.1:9870
    builder.endpoint("http://127.0.0.1:9870");
    // set the delegation_token for builder
    builder.delegation("delegation_token");
    // set atomic_write_dir for builder
    builder.atomic_write_dir(".opendal_tmp/");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

Fields

atomic_write_dir: Option<String>

atomic_write_dir of this backend

Implementations

impl WebhdfsBuilder

pub fn root(&mut self, root: &str) -> &mut Self

Set the working directory of this backend

All operations will happen under this root

Note

The root will be automatically created if not exists.

pub fn endpoint(&mut self, endpoint: &str) -> &mut Self

Set the remote address of this backend default to http://127.0.0.1:9870

Endpoints should be full uri, e.g.

• https://webhdfs.example.com:9870
• http://192.168.66.88:9870

If user inputs endpoint without scheme, we will prepend http:// to it.

pub fn delegation(&mut self, delegation: &str) -> &mut Self

Set the delegation token of this backend, used for authentication

Note

The builder prefers using delegation token over username. If both are set, delegation token will be used.

pub fn disable_list_batch(&mut self) -> &mut Self

Disable batch listing

Note

When listing a directory, the backend will default to use batch listing. If disable, the backend will list all files/directories in one request.

pub fn atomic_write_dir(&mut self, dir: &str) -> &mut Self

Set temp dir for atomic write.

Notes

If not set, write multi not support, eg: .opendal_tmp/.

Trait Implementations

impl Builder for WebhdfsBuilder

fn build(&mut self) -> Result<Self::Accessor>

build the backend

Note

when building backend, the built backend will check if the root directory exits. if the directory does not exits, the directory will be automatically created

const SCHEME: Scheme = Scheme::Webhdfs

Associated scheme for this builder. It indicates what underlying service is.

type Accessor = WebhdfsBackend

The accessor that built by this builder.

fn from_map(map: HashMap<String, String>) -> Self

Construct a builder from given map which contains several parameters needed by underlying service.

impl Clone for WebhdfsBuilder

fn clone(&self) -> WebhdfsBuilder

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WebhdfsBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for WebhdfsBuilder

fn default() -> WebhdfsBuilder

Returns the “default value” for a type.