Skip to content

"FileSystem" DataStore #1723

@rfecher

Description

@rfecher

While rocksdb is a great (and fast) option for direct access to store geowave data on a local filesystem, there is a growing need to have an additional file system datastore for a couple important reasons:

  1. By using Java NIO for all filesystem operations, the underlying file system is completely extensible using FileSystemProvider. GeoWave is included with S3 and HDFS FileSystemProviders on the classpath so geowave stores directly using s3:// and hdfs:// protocols will inherently be supported in addition to of course local paths.
  2. RocksDB stores data in Static Sorted Table (SST) files which while offering very good performance, may not always be optimal for a filesystem geowave store in which users want to interact with the actual files on the filesystem. SST files are tightly compressed and very specific, but for example there are use cases where users want to be able to see rows of GeoWave data as JSON files and perhaps even directly edit a file. While this isn't a traditional use of GeoWave, there is a need. It may even be found useful for debugging purposes as well. Because there are a variety of potential formats a user might want to use, this need will be addressed by a "format" datastore option. Available format options will be discovered at runtime based on SPI implementations of org.locationtech.geowave.datastore.filesystem.FileSystemDataFormatterSpi. By default there will be a "binary" format that stores rows in geowave's default row serialization (a compact binary representation). The plugin also can provide an index/type directory structure perhaps to add to user friendliness although there is a default implementation for all plugins to inherit that provides the directory structure. GeoWave metadata will still be serialized in a "metadata" directory in a binary format that is not intended to be user friendly, but the rows can be customized to be any representative format that is appropriate.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions