Time-series data does not appear to replicate across HA/Raft nodes

Hi, I’m testing ArcadeDB time-series support in a 3-node HA/Raft setup and I’m seeing behaviour that suggests time-series samples are being written only to the node that receives the write, while schema/document/graph data replicates correctly.

I wanted to post this as a discussion first in case I’m missing configuration or intended behaviour.

Setup

• ArcadeDB version: 26.5.1
• 3-node HA/Raft cluster on separate EC2 instances
• HA configured with:
  • arcadedb.ha.enabled=true
  • arcadedb.ha.clusterName=<cluster>
  • arcadedb.ha.clusterToken=<token>
  • arcadedb.ha.raftPort=2434
  • arcadedb.ha.serverList=<node1-private-ip>:2434:2480,<node2-private-ip>:2434:2480,<node3-private-ip>:2434:2480
• Normal document/vertex/edge writes replicate across the cluster.
• Schema changes also replicate across the cluster.

Scenario Tested

I created a minimal time-series type on the current leader, based on the docs, for example:

CREATE TIMESERIES TYPE TestReading
TIMESTAMP ts
TAGS (deviceId STRING)
FIELDS (value DOUBLE)

The type definition replicated to the followers within a short period, so schema replication appears to be working.

Observed Behaviour

1. ILP write to leader

I wrote a sample via ILP to the leader.

Result:

• The sample appeared immediately on the leader.
• I could query it from the leader.
• The sample did not appear on followers.
• Querying the follower returned no samples.

2. ILP write to follower

I then wrote a sample via ILP to a follower.

Result:

• The sample appeared immediately on that follower.
• I could query it from that follower.
• The sample did not appear on the leader.
• The sample did not appear on the other follower.

This suggests ILP writes are accepted and stored locally on whichever node receives the request.

3. SQL INSERT sent to follower

I then sent a single time-series insert via SQL to a follower.

Result:

• The sample did not appear locally on the follower.
• The sample appeared immediately on the leader.

This suggests normal /command SQL writes on followers are being forwarded to the leader, which is expected for HA writes.

4. SQL INSERT sent to leader

SQL insert sent directly to leader writes on the leader, but again the sample does not appear on followers.

File-Level Symptom

The time-series files also differ between nodes.

For example, after audit/login history writes:

Follower:

AuditHistory_shard_0.463.65536.v0.tstb  65536 bytes
AuditHistory_shard_0.ts.sealed          27 bytes

Leader:

AuditHistory_shard_0.463.65536.v0.tstb  196608 bytes
AuditHistory_shard_0.ts.sealed          29771 bytes

So this does not appear to be only a query/read consistency issue; the local files differ.

Codebase Investigation / Hypothesis

From reading the ArcadeDB code, it looks like normal HA replication depends on RaftReplicatedDatabase intercepting commit() and submitting WAL changes through Raft.

Relevant path:

• ha-raft/src/main/java/com/arcadedb/server/ha/raft/RaftReplicatedDatabase.java
  • class comment says it wraps DatabaseInternal and intercepts commit() for Raft consensus.
  • follower /command writes are analysed and forwarded to the leader.

For time-series writes, there seems to be a different path:

• server/src/main/java/com/arcadedb/server/http/handler/PostTimeSeriesWriteHandler.java

  • parses ILP
  • gets the TimeSeriesEngine
  • calls engine.appendSamples(...) directly

• engine/src/main/java/com/arcadedb/query/sql/executor/SaveElementStep.java

  • SQL insert into a time-series type is routed to TimeSeriesEngine.appendSamples(...)

• engine/src/main/java/com/arcadedb/engine/timeseries/TimeSeriesShard.java

  • appendSamples(...) opens and commits its own nested transaction:
    • database.begin()
    • mutableBucket.appendSamples(...)
    • database.commit()

• engine/src/main/java/com/arcadedb/schema/LocalTimeSeriesType.java

  • initEngine() constructs the engine from schema.getDatabase():
    • new TimeSeriesEngine((DatabaseInternal) schema.getDatabase(), ...)

My suspicion is that the time-series engine may be holding or using the underlying local database rather than the HA/Raft wrapper, so the nested transaction commit does not go through RaftReplicatedDatabase.commit().

That would explain the observed behaviour:

• ILP to leader writes locally on leader only.
• ILP to follower writes locally on follower only.
• SQL insert to follower is forwarded to leader first, then writes locally on leader only.
• Schema changes replicate because they use the normal DDL/HA path.
• Normal document/graph writes replicate because they use the normal wrapped transaction path.

There is also this comment in TimeSeriesSealedStore:

compacted time-series data is derived from the replicated mutable TimeSeriesBucket pages

That seems to imply the mutable .tstb bucket pages are expected to replicate through the normal paginated component path, but in our test they do not appear to do so.

Questions

1. Is time-series sample data currently expected to replicate in HA/Raft mode?
2. Should ILP writes to a follower be forwarded to the leader, rejected, or written locally?
3. Are mutable time-series bucket pages expected to be replicated via Raft/WAL?
4. Is there a required HA/time-series configuration option we may be missing?
5. Does the TimeSeriesEngine need to be initialised with the HA-wrapped DatabaseInternal rather than the underlying local database?

Happy to turn this into a minimal reproducible issue if this behaviour is unexpected.