Struct tvix_castore::blobservice::object_store::ObjectStoreBlobService

pub struct ObjectStoreBlobService {
    object_store: Arc<dyn ObjectStore>,
    base_path: Path,
    avg_chunk_size: u32,
}

Uses any object storage supported by the object_store crate to provide a tvix-castore BlobService.

Data format

Data is organized in “blobs” and “chunks”. Blobs don’t hold the actual data, but instead contain a list of more granular chunks that assemble to the contents requested. This allows clients to seek, and not download chunks they already have locally, as it’s referred to from other files. Check rpc_blobstore and more general BlobStore docs on that.

Blobs

Stored at ${base_path}/blobs/b3/$digest_key. They contains the serialized StatBlobResponse for the blob with the digest.

Chunks

Chunks are stored at ${base_path}/chunks/b3/$digest_key. They contain the literal contents of the chunk, but are zstd-compressed.

Digest key sharding

The blake3 digest encoded in lower hex, and sharded after the second character. The blob for “Hello World” is stored at ${base_path}/blobs/b3/41/41f8394111eb713a22165c46c90ab8f0fd9399c92028fd6d288944b23ff5bf76.

This reduces the number of files in the same directory, which would be a problem at least when using object_store::local::LocalFileSystem.

Future changes

There’s no guarantees about this being a final format yet. Once object_store gets support for additional metadata / content-types, we can eliminate some requests (small blobs only consisting of a single chunk can be stored as-is, without the blob index file). It also allows signalling any compression of chunks in the content-type. Migration should be possible by simply adding the right content-types to all keys stored so far, but no promises ;-)

Fields

object_store: Arc<dyn ObjectStore>

base_path: Path

avg_chunk_size: u32

Average chunk size for FastCDC, in bytes. min value is half, max value double of that number.

Trait Implementations

impl BlobService for ObjectStoreBlobService

fn has<'life0, 'life1, 'async_trait>( &'life0 self, digest: &'life1 B3Digest, ) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Check if the service has the blob, by its content hash. On implementations returning chunks, this must also work for chunks.

fn open_read<'life0, 'life1, 'async_trait>( &'life0 self, digest: &'life1 B3Digest, ) -> Pin<Box<dyn Future<Output = Result<Option<Box<dyn BlobReader>>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Request a blob from the store, by its content hash. On implementations returning chunks, this must also work for chunks.

fn open_write<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Box<dyn BlobWriter>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Insert a new blob into the store. Returns a BlobWriter, which implements tokio::io::AsyncWrite and a BlobWriter::close to finalize the blob and get its digest.

fn chunks<'life0, 'life1, 'async_trait>( &'life0 self, digest: &'life1 B3Digest, ) -> Pin<Box<dyn Future<Output = Result<Option<Vec<ChunkMeta>>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return a list of chunks for a given blob. There’s a distinction between returning Ok(None) and Ok(Some(vec![])). The former return value is sent in case the blob is not present at all, while the second one is sent in case there’s no more granular chunks (or the backend does not support chunking). A default implementation checking for existence and then returning it does not have more granular chunks available is provided.

impl Clone for ObjectStoreBlobService

fn clone(&self) -> ObjectStoreBlobService

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.