Struct tvix_castore::blobservice::ObjectStoreBlobService
source · pub struct ObjectStoreBlobService {
instance_name: String,
object_store: Arc<dyn ObjectStore>,
base_path: Path,
avg_chunk_size: u32,
}
Expand description
Uses any object storage supported by the object_store crate to provide a tvix-castore BlobService.
§Data format
Data is organized in “blobs” and “chunks”.
Blobs don’t hold the actual data, but instead contain a list of more
granular chunks that assemble to the contents requested.
This allows clients to seek, and not download chunks they already have
locally, as it’s referred to from other files.
Check rpc_blobstore
and more general BlobStore docs on that.
§Blobs
Stored at ${base_path}/blobs/b3/$digest_key
. They contains the serialized
StatBlobResponse for the blob with the digest.
§Chunks
Chunks are stored at ${base_path}/chunks/b3/$digest_key
. They contain
the literal contents of the chunk, but are zstd-compressed.
§Digest key sharding
The blake3 digest encoded in lower hex, and sharded after the second
character.
The blob for “Hello World” is stored at
${base_path}/blobs/b3/41/41f8394111eb713a22165c46c90ab8f0fd9399c92028fd6d288944b23ff5bf76
.
This reduces the number of files in the same directory, which would be a problem at least when using object_store::local::LocalFileSystem.
§Future changes
There’s no guarantees about this being a final format yet. Once object_store gets support for additional metadata / content-types, we can eliminate some requests (small blobs only consisting of a single chunk can be stored as-is, without the blob index file). It also allows signalling any compression of chunks in the content-type. Migration should be possible by simply adding the right content-types to all keys stored so far, but no promises ;-)
Fields§
§instance_name: String
§object_store: Arc<dyn ObjectStore>
§base_path: Path
§avg_chunk_size: u32
Average chunk size for FastCDC, in bytes. min value is half, max value double of that number.
Trait Implementations§
source§impl BlobService for ObjectStoreBlobService
impl BlobService for ObjectStoreBlobService
source§fn has<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn has<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
source§fn open_read<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<Option<Box<dyn BlobReader>>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn open_read<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<Option<Box<dyn BlobReader>>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
source§fn open_write<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Box<dyn BlobWriter>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
fn open_write<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Box<dyn BlobWriter>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
source§fn chunks<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<Option<Vec<ChunkMeta>>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn chunks<'life0, 'life1, 'async_trait>(
&'life0 self,
digest: &'life1 B3Digest,
) -> Pin<Box<dyn Future<Output = Result<Option<Vec<ChunkMeta>>>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
source§impl Clone for ObjectStoreBlobService
impl Clone for ObjectStoreBlobService
source§fn clone(&self) -> ObjectStoreBlobService
fn clone(&self) -> ObjectStoreBlobService
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moreAuto Trait Implementations§
impl Freeze for ObjectStoreBlobService
impl !RefUnwindSafe for ObjectStoreBlobService
impl Send for ObjectStoreBlobService
impl Sync for ObjectStoreBlobService
impl Unpin for ObjectStoreBlobService
impl !UnwindSafe for ObjectStoreBlobService
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
source§unsafe fn clone_to_uninit(&self, dst: *mut T)
unsafe fn clone_to_uninit(&self, dst: *mut T)
clone_to_uninit
)source§impl<T> FutureExt for T
impl<T> FutureExt for T
source§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
source§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
source§impl<T> Instrument for T
impl<T> Instrument for T
source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T
in a tonic::Request