Skip to main content

Hugging Face

Hugging Face services support.

This service will visit the Huggingface API to access the Huggingface File System. Currently, we only support the model and dataset types of repositories, and operations are limited to reading and listing/stating.

Huggingface doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the huggingface_hub Source Code.

Capabilities​

This service can be used to:

  • stat
  • read
  • write
  • create_dir
  • delete
  • copy
  • rename
  • list
  • scan
  • presign
  • blocking

Configurations​

  • repo_type: The type of the repository.
  • repo_id: The id of the repository.
  • revision: The revision of the repository.
  • root: Set the work directory for backend.
  • token: The token for accessing the repository.

Refer to [HuggingfaceBuilder]'s public API docs for more information.

Examples​

Via Builder​

use std::sync::Arc;

use anyhow::Result;
use opendal::services::Huggingface;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Create Huggingface backend builder
    let mut builder = Huggingface::default();
    
    // set the type of Huggingface repository
    builder.repo_type("dataset");
    // set the id of Huggingface repository
    builder.repo_id("databricks/databricks-dolly-15k");
    // set the revision of Huggingface repository
    builder.revision("main");
    // set the root for Huggingface, all operations will happen under this root
    builder.root("/path/to/dir");
    // set the token for accessing the repository
    builder.token("access_token");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

Via Config​

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<()> {
    let mut map = HashMap::new();

    map.insert("repo_type".to_string(), "dataset".to_string());
    map.insert("repo_id".to_string(), "databricks/databricks-dolly-15k".to_string());
    map.insert("revision".to_string(), "main".to_string());
    map.insert("root".to_string(), "/path/to/dir".to_string());
    map.insert("token".to_string(), "access_token".to_string());

    let op: Operator = Operator::via_map(Scheme::Huggingface, map)?;

    Ok(())
}