How to replicate databases when new cluster nodes are added in Kubernetes ? #4522

venkataramanam · 2026-06-08T12:36:29Z

venkataramanam
Jun 8, 2026

Hi,

I have initially started with a 3 node cluster. And the cluster has two databases (one initialized via arcadedb.server.defaultDatabases and another one imported from Studio).

If I scale up the Kube cluster (a StatefulSet) from 3 to 5, I do not see the database that has been imported.
Do we need to turn on any server flag to help with ?

Here are the relevant settings for the server ...

arcadedb.ha.enabled=true
arcadedb.ha.k8s=true
arcadedb.ha.k8sSuffix=.wxg-arcadedb-svc.cpd-instance.svc.cluster.local

arcadedb.ha.serverList=wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:{raft:2434,http:2480,https:2490},wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:{raft:2434,http:2480,https:2490},wxg-arcadedb-2.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:{raft:2434,http:2480,https:2490},wxg-arcadedb-3.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:{raft:2434,http:2480,https:2490},wxg-arcadedb-4.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:{raft:2434,http:2480,https:2490}

The list of databases inside the databases directory for each node...

> oc exec wxg-arcadedb-0 -- ls /var/lib/arcadedb/databases
OpenBeer
dbinst1


> oc exec wxg-arcadedb-1 -- ls /var/lib/arcadedb/databases
OpenBeer
dbinst1

> oc exec wxg-arcadedb-2 -- ls /var/lib/arcadedb/databases
OpenBeer
dbinst1

> oc exec wxg-arcadedb-3 -- ls /var/lib/arcadedb/databases
dbinst1

> oc exec wxg-arcadedb-4 -- ls /var/lib/arcadedb/databases
dbinst1

From the above info, it is clear that the newly added nodes wxg-arcadedb-3 and wxg-arcadedb-4 post StatefulSet scale-up only have a single database (which was initialized from arcadedb.server.defaultDatabases), but are missing the OpenBeer database.

Note that I have first scaled down the cluster to zero from 3 and then scaled up to 5. This is typically a normal practice (it has some disruption though).

Here is the output of the api/v1/server?mode=cluster API ...

{
    "user": "root",
    "version": "26.6.1 (build 9f775953dfe7a9013cca4072c42b63d3eb4abb00/1780517499650/main)",
    "serverName": "wxg-arcadedb-1",
    "languages":
    [
        "js",
        "java",
        "sql",
        "sqlscript",
        "opencypher",
        "graphql",
        "redis",
        "cypher"
    ],
    "ha":
    {
        "clusterName": "arcadedb",
        "leader": "wxg-arcadedb-4 (wxg-arcadedb-4.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480)",
        "electionStatus": "DONE",
        "network":
        {
            "isLeader": false,
            "replicas":
            [
                {
                    "address": "wxg-arcadedb-2.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2434",
                    "httpAddress": "wxg-arcadedb-2.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480",
                    "id": "wxg-arcadedb-2.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434"
                },
                {
                    "address": "wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2434",
                    "httpAddress": "wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480",
                    "id": "wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434"
                },
                {
                    "address": "wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2434",
                    "httpAddress": "wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480",
                    "id": "wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434"
                },
                {
                    "address": "wxg-arcadedb-3.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2434",
                    "httpAddress": "wxg-arcadedb-3.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480",
                    "id": "wxg-arcadedb-3.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434"
                }
            ],
            "localPeerId": "wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434",
            "configuredServers": 5
        },
        "databases":
        [
            {
                "name": ".raft",
                "quorum": "MAJORITY"
            },
            {
                "name": "OpenBeer",
                "quorum": "MAJORITY"
            },
            {
                "name": "dbinst1",
                "quorum": "MAJORITY"
            }
        ],
        "leaderAddress": "wxg-arcadedb-4.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480",
        "replicaAddresses": "wxg-arcadedb-2.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480,wxg-arcadedb-1.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480,wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480,wxg-arcadedb-3.wxg-arcadedb-svc.cpd-instance.svc.cluster.local:2480"
    }
}

One particular observation from the above cluster layout is that a new node wxg-arcadedb-4 post scale up has been elected as LEADER.

Answered by lvca

Jun 24, 2026

Hi @venkataramanam,

Thanks for the detailed write-up. The cluster dump and the per-node databases listing make the situation clear.

What's happening

With the Raft-based HA, databases reach a node in two ways:

Databases created/imported while the cluster is online are committed to the Raft log and shipped to the peers that are members at that moment. That is why OpenBeer is on -0/-1/-2.
A node that joins later with an empty data dir only auto-installs snapshots for databases it already has on disk. The follower snapshot-install path iterates over the databases present locally and refreshes those from the leader; it does not pull a database it has never seen. So -3/-4 come up with only dbi…

View full answer

venkataramanam · 2026-06-08T16:55:20Z

venkataramanam
Jun 8, 2026
Author

If I scale up the Kube cluster (a StatefulSet) from 3 to 5, I do not see the database that has been imported.

I get a feeling this is likely by design. It probably is too costly to replicate the entire database over http.
https://docs.arcadedb.com/arcadedb/how-to/operations/kubernetes#kubernetes-ha-bootstrap

Trying to understand what steps would be needed to replicate the databases on new nodes.
The database files in ~/databases directory for a specific database, are they exactly the same content in all nodes ? Or any node specific imprint is left in any files. Trying to see, if I were to copy content from one of the existing member PVCs onto the PVCs of the new nodes, would that work ? Lets say, this copy is done offline by scaling down the cluster completely.

0 replies

lvca · 2026-06-24T04:56:52Z

lvca
Jun 24, 2026
Maintainer

Hi @venkataramanam,

Thanks for the detailed write-up. The cluster dump and the per-node databases listing make the situation clear.

What's happening

With the Raft-based HA, databases reach a node in two ways:

Databases created/imported while the cluster is online are committed to the Raft log and shipped to the peers that are members at that moment. That is why OpenBeer is on -0/-1/-2.
A node that joins later with an empty data dir only auto-installs snapshots for databases it already has on disk. The follower snapshot-install path iterates over the databases present locally and refreshes those from the leader; it does not pull a database it has never seen. So -3/-4 come up with only dbinst1 (created from defaultDatabases at boot) and never acquire OpenBeer. Your observation is correct, this is the current behavior, not a transient.

Two things made it worse in your run:

You scaled to 0 and then up to 5. Stopping the whole cluster forces a fresh bootstrap election on restart, and the brand-new empty pods (-3/-4) take part in it.
A node that does not hold OpenBeer (-4) got elected leader. Since both resync and the follower snapshot install download from the leader, the only authoritative copies of OpenBeer are now on followers, which blocks the easy recovery until you move leadership.

How to recover now

Move leadership to a node that actually has OpenBeer (any of -0/-1/-2):
```
POST /api/v1/cluster/leader
{ "peerId": "wxg-arcadedb-0.wxg-arcadedb-svc.cpd-instance.svc.cluster.local_2434" }
```
peerId is the id field from the cluster API. An empty body lets Ratis pick a candidate.
On each node missing the DB (-3 and -4), pull it from the leader. Run this on that node (the endpoint always acts on the node receiving the request):
```
POST /api/v1/cluster/resync/OpenBeer
```
It downloads a full snapshot from the leader and atomically swaps it in, creating the database locally even if it was not there before.
Verify consistency across the cluster:
```
POST /api/v1/cluster/verify/OpenBeer
```

Your offline-copy question

Yes, that works too. A user database directory (e.g. databases/OpenBeer) is portable across nodes, there is no node identity baked into the user data. With the cluster fully stopped you can copy the whole databases/OpenBeer folder from an existing PVC onto the new PVCs, then start up. The resync API is cleaner because it cannot race a running server, but the offline copy is a valid fallback. Copy the entire folder, do not try to merge individual files.

Recommended scale-up procedure

Avoid scaling to 0. Scale the StatefulSet up incrementally while the cluster stays online (3 -> 4 -> 5). Keeping a quorum of nodes-with-data online means leadership stays on an authoritative node, and you can resync each new pod right after it joins. If you must do a cold restart, make sure the first pod(s) back are ones that hold all databases so the leader is authoritative.

Going forward

I agree the friction is real: a freshly added empty node should be able to acquire databases it has never seen without a manual resync. I'll track auto-distribution of missing databases to joining nodes as an improvement. Thanks for the clear report, it helps a lot.

0 replies

lvca · 2026-06-24T11:14:06Z

lvca
Jun 24, 2026
Maintainer

I opened #4727 to track the improvement (a freshly added empty node should auto-acquire databases it has never seen, instead of only refreshing the ones it already has). Follow there for progress.

0 replies

venkataramanam · 2026-06-24T13:31:12Z

venkataramanam
Jun 24, 2026
Author

@lvca

Thanks very much for the elaborate response.

Was busy with some other commitments.
Let me go through your response and respond back by tomorrow.

I was trying to relate to my experience of working with Etcd and Clickhouse. While Etcd didn't have anything auto discovery of new members and member addition had to be manual, it did seamlessly replicate data across the new members. While in the case of Clickhouse, it was a case of manually having to create the data schema before the Clickhouse would start replicating.

0 replies

lvca · 2026-06-26T00:24:39Z

lvca
Jun 26, 2026
Maintainer

The issue has been closed, everything has been implemented and tested, please @venkataramanam let me know if you can retry with the latest snapshot.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

How to replicate databases when new cluster nodes are added in Kubernetes ? #4522

Uh oh!

{{title}}

Uh oh!

Replies: 6 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Uh oh!

How to replicate databases when new cluster nodes are added in Kubernetes ? #4522

Uh oh!

venkataramanam Jun 8, 2026

What's happening

Replies: 6 comments

Uh oh!

venkataramanam Jun 8, 2026 Author

Uh oh!

lvca Jun 24, 2026 Maintainer

What's happening

How to recover now

Your offline-copy question

Recommended scale-up procedure

Going forward

Uh oh!

lvca Jun 24, 2026 Maintainer

Uh oh!

venkataramanam Jun 24, 2026 Author

Uh oh!

lvca Jun 26, 2026 Maintainer

venkataramanam
Jun 8, 2026

venkataramanam
Jun 8, 2026
Author

lvca
Jun 24, 2026
Maintainer

lvca
Jun 24, 2026
Maintainer

venkataramanam
Jun 24, 2026
Author

lvca
Jun 26, 2026
Maintainer