I am trying to come up with a Kubernetes StatefulSet to have a 3 node cluster with a custom arcadedb image.
We will need to run the server containers (in the pods) in what Kube calls as readOnlyRootFilesystem. Turning on that essentially makes the entire filesystem as read only (no writes allowed) and only the mounted directories via PVCs or ephemeral <emptyDir> directories can be written to.

To the extent I have assessed till now, there is one issue that I have come across where the RAFT data is being persisted to a folder like raft-storage-<hostname>_<raftport> relative to what Arcadedb refers to as server root path (configurable via arcadedb.server.rootPath).

By default arcadedb.server.rootPath resolves to working directory and is kind of set the root of the filsystem where arcadedb files can be found like lib, config etc.,

The content in /bin, /lib I guess is the only static content from the image which runtime needs. Rest of it is written to at runtime to preserve data, security related info (in config), logs, backups etc.,

If I set arcadedb.server.rootPath to say /var/lib/arcadedb, and then mount directories like /var/lib/arcadedb/databases, /var/lib/arcadedb/config (into durable PVC storage) and /var/lib/arcadedb/logs (into ephemeral storage), that works fine. And also server can create a directory for RAFT storage in /var/lib/arcadedb/raft-storage-<hostname>_<raftport>.

The problem with setting arcadedb.server.rootPath to say /var/lib/arcadedb is that, the server looks for the runtime looks for plugins in ${arcadedb.server.rootPath}/lib/plugins ... This would break if the lib/plugins folder from the image is not made available at /var/lib/arcadedb/lib, which essentially means having to copy the entire lib directory onto an ephemeral storage directory (probably as part of an init container or during container startup)

One more problem is the directory name /var/lib/arcadedb/raft-storage-<hostname>_<raftport> is kind of dynamic and it is not a straight forward to mount that directory in a Kube StatefulSet. The hostname (or rather pod name) is dynamic like -<pod number 0, 1,2 etc.,>
If the directory can have a fixed name and the path to that is configurable, the problem would get solved.

Somehow I get a feeling that each and every runtime written directory path should be configurable...

1. databases - Already configurable via arcadedb.server.databaseDirectory
2. config - Not configurable ... it is relative to arcadedb.server.rootPath
3. backups - Already configurable via arcadedb.server.backupDirectory
4. log - Not configurable ... it is relative to arcadedb.server.rootPath
5. raft-storage-* - Not configurable ... it is relative to arcadedb.server.rootPath

In the sense, the writeable directories should be decoupled from the server binary content coming from the image (~/lib, ~/bin etc.,). The defaults can be relative to server home directory for ease of adoption like the way it is.

I have a workaround for now to copy the entire image contents to say /var/lib/arcadedb (on ephemeral storage), but then when server creates the raft-storage-* directory there, I cannot have a large storage on ephemeral storage.

The simplest immediate solution I can think of is like this ...

1. Instead of RAFT storage being ${arcadedb.server.rootPath}/raft-storage-<peer-id>, make it ${arcadedb.server.rootPath}/raft/raft-storage-<peer-id>
2. Allow the path ${arcadedb.server.rootPath}/raft as configurable.

Essentially the parent directory (for the dynamically created directory raft-storage-<peer-id>) should be configurable.