Struct object_store::local::LocalFileSystem

pub struct LocalFileSystem { /* private fields */ }

Local filesystem storage providing an ObjectStore interface to files on local disk. Can optionally be created with a directory prefix

Path Semantics

This implementation follows the file URI scheme outlined in RFC 3986. In particular paths are delimited by /

Path Semantics

LocalFileSystem will expose the path semantics of the underlying filesystem, which may have additional restrictions beyond those enforced by Path.

For example:

• Windows forbids certain filenames, e.g. COM0,
• Windows forbids folders with trailing .
• Windows forbids certain ASCII characters, e.g. < or |
• OS X forbids filenames containing :
• Leading - are discouraged on Unix systems where they may be interpreted as CLI flags
• Filesystems may have restrictions on the maximum path or path segment length
• Filesystem support for non-ASCII characters is inconsistent

Additionally some filesystems, such as NTFS, are case-insensitive, whilst others like FAT don’t preserve case at all. Further some filesystems support non-unicode character sequences, such as unpaired UTF-16 surrogates, and LocalFileSystem will error on encountering such sequences.

Finally, filenames matching the regex /.*#\d+/, e.g. foo.parquet#123, are not supported by LocalFileSystem as they are used to provide atomic writes. Such files will be ignored for listing operations, and attempting to address such a file will error.

Tokio Compatibility

Tokio discourages performing blocking IO on a tokio worker thread, however, no major operating systems have stable async file APIs. Therefore if called from a tokio context, this will use tokio::runtime::Handle::spawn_blocking to dispatch IO to a blocking thread pool, much like tokio::fs does under-the-hood.

If not called from a tokio context, this will perform IO on the current thread with no additional complexity or overheads

Symlinks

LocalFileSystem will follow symlinks as normal, however, it is worth noting:

• Broken symlinks will be silently ignored by listing operations
• No effort is made to prevent breaking symlinks when deleting files
• Symlinks that resolve to paths outside the root will be followed
• Mutating a file through one or more symlinks will mutate the underlying file
• Deleting a path that resolves to a symlink will only delete the symlink

Cross-Filesystem Copy

LocalFileSystem::copy is implemented using std::fs::hard_link, and therefore does not support copying across filesystem boundaries.

Implementations

impl LocalFileSystem

pub fn new() -> Self

Create new filesystem storage with no prefix

pub fn new_with_prefix(prefix: impl AsRef<Path>) -> Result<Self>

Create new filesystem storage with prefix applied to all paths

Returns an error if the path does not exist

pub fn path_to_filesystem(&self, location: &Path) -> Result<PathBuf>

Return an absolute filesystem path of the given file location

pub fn with_automatic_cleanup(self, automatic_cleanup: bool) -> Self

Enable automatic cleanup of empty directories when deleting files

Trait Implementations

impl Debug for LocalFileSystem

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LocalFileSystem

fn default() -> Self

Returns the “default value” for a type.

impl Display for LocalFileSystem

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ObjectStore for LocalFileSystem

fn put_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload, opts: PutOptions, ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided payload to location with the given options

fn put_multipart_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, opts: PutMultipartOpts, ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload with options

fn get_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, options: GetOptions, ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a get request with options

fn get_range<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, range: Range<usize>, ) -> Pin<Box<dyn Future<Output = Result<Bytes>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the bytes that are stored at the specified location in the given byte range.

fn get_ranges<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, location: &'life1 Path, ranges: &'life2 [Range<usize>], ) -> Pin<Box<dyn Future<Output = Result<Vec<Bytes>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Return the bytes that are stored at the specified location in the given byte ranges

fn delete<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Delete the object at the specified location.

fn list(&self, prefix: Option<&Path>) -> BoxStream<'_, Result<ObjectMeta>>

List all the objects with the given prefix.

fn list_with_delimiter<'life0, 'life1, 'async_trait>( &'life0 self, prefix: Option<&'life1 Path>, ) -> Pin<Box<dyn Future<Output = Result<ListResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

List objects with the given prefix and an implementation specific delimiter. Returns common prefixes (directories) in addition to object metadata.

fn copy<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another in the same object store.

fn rename<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store.

fn copy_if_not_exists<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another, only if destination is empty.

fn put<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload, ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided bytes to the specified location

fn put_multipart<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload

fn get<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the bytes that are stored at the specified location.

fn head<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<ObjectMeta>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Return the metadata for the specified location

fn delete_stream<'a>( &'a self, locations: BoxStream<'a, Result<Path>>, ) -> BoxStream<'a, Result<Path>>

Delete all the objects at the specified locations

fn list_with_offset( &self, prefix: Option<&Path>, offset: &Path, ) -> BoxStream<'_, Result<ObjectMeta>>

List all the objects with the given prefix and a location greater than offset

fn rename_if_not_exists<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, from: &'life1 Path, to: &'life2 Path, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store.