Struct object_store::WriteMultipart

pub struct WriteMultipart { /* private fields */ }

A synchronous write API for uploading data in parallel in fixed size chunks

Uses multiple tokio tasks in a JoinSet to multiplex upload tasks in parallel

The design also takes inspiration from Sink with WriteMultipart::wait_for_capacity allowing back pressure on producers, prior to buffering the next part. However, unlike Sink this back pressure is optional, allowing integration with synchronous producers

Implementations

impl WriteMultipart

pub fn new(upload: Box<dyn MultipartUpload>) -> Self

Create a new WriteMultipart that will upload using 5MB chunks

pub fn new_with_chunk_size( upload: Box<dyn MultipartUpload>, chunk_size: usize, ) -> Self

Create a new WriteMultipart that will upload in fixed chunk_size sized chunks

pub fn poll_for_capacity( &mut self, cx: &mut Context<'_>, max_concurrency: usize, ) -> Poll<Result<()>>

Polls for there to be less than max_concurrency UploadPart in progress

See Self::wait_for_capacity for an async version of this function

pub async fn wait_for_capacity(&mut self, max_concurrency: usize) -> Result<()>

Wait until there are less than max_concurrency UploadPart in progress

See Self::poll_for_capacity for a Poll version of this function

pub fn write(&mut self, buf: &[u8])

Write data to this WriteMultipart

Data is buffered using PutPayloadMut::extend_from_slice. Implementations looking to write data from owned buffers may prefer Self::put as this avoids copying.

Note this method is synchronous (not async) and will immediately start new uploads as soon as the internal chunk_size is hit, regardless of how many outstanding uploads are already in progress.

Back pressure can optionally be applied to producers by calling Self::wait_for_capacity prior to calling this method

pub fn put(&mut self, bytes: Bytes)

Put a chunk of data into this WriteMultipart without copying

Data is buffered using PutPayloadMut::push. Implementations looking to perform writes from non-owned buffers should prefer Self::write as this will allow multiple calls to share the same underlying allocation.

See Self::write for information on backpressure

pub async fn abort(self) -> Result<()>

Abort this upload, attempting to clean up any successfully uploaded parts

pub async fn finish(self) -> Result<PutResult>

Flush final chunk, and await completion of all in-flight requests

Trait Implementations

impl Debug for WriteMultipart

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.