Skip to content

Feature : Auto-tune maxPageRAM based on detected container memory limit at startup #3580

@KwonSunJae

Description

@KwonSunJae

Summary

While running a comparative benchmark between ArcadeDB and Neo4j under identical Docker resource constraints, I discovered that ArcadeDB's page cache defaults can cause silent OOM kills in memory-constrained containers.

ArcadeDB already detects available memory at startup (MAXRAM), but memory-related settings like maxPageRAM are not automatically adjusted to fit within that limit. It would be great if ArcadeDB could auto-tune page cache settings during initialization based on the detected MAXRAM, similar to how Neo4j automatically scales its page cache relative to available memory.

How I Found This

I was building a graph database benchmark suite to compare Neo4j and ArcadeDB (both Cypher/Bolt and SQL/HTTP modes) under identical conditions — same Docker resource limits, same JVM heap, same dataset size. During the benchmark, Neo4j completed all operations without issues, but ArcadeDB was silently killed by the Docker daemon mid-insert. After investigating with docker inspect (OOMKilled: true) and reviewing ArcadeDB's startup logs, I traced the root cause to the default maxPageRAM=4096MB exceeding the container's memory limit.

Context

At startup, ArcadeDB correctly detected the container memory:

ArcadeDB Server started in 'development' mode (CPUs=2 MAXRAM=2.00GB)

However, maxPageRAM remained at its default of 4096MB — double the available memory. During bulk insert (200K nodes with 1024-dim vector embeddings + LSM_VECTOR index), the page cache grew beyond the container limit, and the process was OOM-killed:

$ docker inspect bench-arcadedb --format='{{.State.OOMKilled}}'
true

No error logs were produced — the process was killed with SIGKILL during vector sub-index creation. The last log entry:

[ArcadeDB] Creating sub-index 'Person_0_65702909910379(10)'

Benchmark Environment

OS | Debian (Docker)
Nodes | 200,000 (each with 1024-dim float vector)
Edges | 500,000 (random connections)
Docker limits | 2 CPU cores, 2GB RAM
JVM | -Xms512m -Xmx1536m
Vector index | ArcadeDB: LSM_VECTOR (COSINE), Neo4j: VECTOR INDEX (COSINE)
ArcadeDB crash point | ~155,000 / 200,000 nodes
Neo4j | Completed all operations successfully

Proposal

Since ArcadeDB already reads MAXRAM at startup, it could use this value to auto-tune page cache defaults when they are not explicitly set by the user. For example:

if maxPageRAM not explicitly configured:
    maxPageRAM = MAXRAM * 0.25   # 25-30% of detected memory

This would make ArcadeDB safe by default in memory-constrained environments (Docker, Kubernetes, cloud PaaS) without requiring users to manually discover and configure maxPageRAM.

Optionally, a startup warning when maxPageRAM > MAXRAM would also help users catch misconfigurations early.

Current Workaround

Manually setting page cache limits via JAVA_OPTS:

-Darcadedb.maxPageRAM=512 -Darcadedb.initialPageCacheSize=16384

This works, but requires users to know about these settings beforehand — which is difficult to discover when the only symptom is a silent OOM kill with no logs.


If you're interested, I'd be happy to share the full benchmark results and reproduction setup (Docker Compose + benchmark scripts) that led to this finding.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions