Struct tvix_castore::import::blobs::ConcurrentBlobUploader

pub struct ConcurrentBlobUploader<BS> {
    blob_service: BS,
    upload_tasks: JoinSet<Result<(), Error>>,
    upload_semaphore: Arc<Semaphore>,
}

The concurrent blob uploader provides a mechanism for concurrently uploading small blobs. This is useful when ingesting from sources like tarballs and archives which each blob entry must be read sequentially. Ingesting many small blobs sequentially becomes slow due to round trip time with the blob service. The concurrent blob uploader will buffer small blobs in memory and upload them to the blob service in the background.

Once all blobs have been uploaded, make sure to call ConcurrentBlobUploader::join to wait for all background jobs to complete and check for any errors.

Fields

blob_service: BS

upload_tasks: JoinSet<Result<(), Error>>

upload_semaphore: Arc<Semaphore>

Implementations

impl<BS> ConcurrentBlobUploader<BS>

where BS: BlobService + Clone + 'static,

pub fn new(blob_service: BS) -> Self

Creates a new concurrent blob uploader which uploads blobs to the provided blob service.

pub async fn upload<R>( &mut self, path: &Path, expected_size: u64, r: R, ) -> Result<B3Digest, Error>

where R: AsyncRead + Unpin,

Uploads a blob to the blob service. If the blob is small enough it will be read to a buffer and uploaded in the background. This will read the entirety of the provided reader unless an error occurs, even if blobs are uploaded in the background..

pub async fn join(self) -> Result<(), Error>

Waits for all background upload jobs to complete, returning any upload errors.