Conversation
… algos can use now graph olap
|
Warning Gemini is experiencing higher than usual traffic and was unable to create the summary. Please try again in a few hours by commenting |
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences |
|
Code Review: Graph Analytical View (GAV) Impressive and ambitious feature. The SPI abstraction is clean, the delta overlay design is clever, and the test coverage is thorough. Issues found, by severity: HIGH SEVERITY 1. compacting flag never reset on RejectedExecutionException - GraphAnalyticalView.applyDelta() After compacting.compareAndSet(false, true), if the executor submit throws RejectedExecutionException (executor shut down concurrently), the exception is likely swallowed in the post-commit callback. The compacting flag stays true permanently, silently disabling all future compaction for this view instance. Fix: wrap the submit in try/catch and call compacting.set(false) in the catch. 2. O(N^2) compaction cost per commit - DeltaOverlay.merge() merge() deep-copies the entire overlay state on every SYNCHRONOUS commit, even for single-edge transactions. For N commits before the 10000-edge compaction threshold, total work is O(N^2). Under sustained write rates this causes progressive latency and GC pressure. Consider a lazy-fold structure or a lower compaction threshold. MEDIUM SEVERITY 3. closeExecutor() holds class lock for 30 seconds - GraphAnalyticalView The synchronized(GraphAnalyticalView.class) block includes awaitTermination(30, SECONDS), blocking buildAsync() and onRelevantCommit() callers for up to 30 seconds during shutdown. Null the static field inside the lock and drain outside it. 4. GAV disabled for all multi-hop anonymous patterns - MatchRelationshipStep.canUseFastPath() Removing isInternalAnonymousVar blocks GAV for MATCH (a)-[*2..4]->(b) patterns. Space-prefixed anonymous variables do not capture edge data; they only track uniqueness. Consider whether uniqueness tracking can use internal dense IDs instead of full edge objects to re-enable CSR acceleration. 5. HashMap autoboxes int on every edge in NodeIdMapping.getGlobalId() bucketIdToIdx.get(rid.getBucketId()) autoboxes int on every call - O(E) boxing during CSRBuilder Phase B. Since ArcadeDB bucket IDs are small contiguous integers, a plain int array lookup table would eliminate all boxing and align with the zero-GC goal. 6. One RID allocation per neighbor in the hot query loop - NodeIdMapping.getRID() provider.getRID(neighbors) allocates one RID per result row. For analytics returning millions of neighbors this contradicts the zero-GC design goal. Lazy materialization (allocate RID only when it escapes the step) would help significantly. LOW SEVERITY 7. propertySampleSize not volatile but read from background threads Safe today (written before async work starts) but JMM does not formally guarantee visibility. Making it final (injected via constructor) would make the guarantee explicit. 8. Subtle JMM ordering not documented - MatchRelationshipStep.resolveGavProvider() The two separate volatile writes (gavProvider then gavProviderResolved) are correct per JMM: a reader seeing gavProviderResolved == true is guaranteed to see the preceding write to gavProvider. This reasoning is non-obvious; a comment explaining the happens-before chain would prevent a future regression. 9. ConcurrentHashMap inside GraphAnalyticalViewRegistry adds no value All access is guarded by synchronized(REGISTRY). The inner ConcurrentHashMap could be a plain HashMap. 10. Self-loop dedup relies on undocumented OLTP invariant - MatchRelationshipStep.getVertices() The invariant that each self-loop produces exactly 2 entries is not asserted or referenced. If the OLTP storage layer changes, selfLoopSeen % 2 == 0 silently produces wrong results. 11. WeakHashMap safety-net claim misleading - GraphTraversalProviderRegistry Providers hold strong references back to the database key so the WeakHashMap behaves as a regular HashMap during normal operation. The Javadoc should clarify the safety net only activates after all providers are unregistered. MINOR
Overall this is high-quality work. Main concerns before merging: compaction flag stuck-state bug (1), quadratic merge cost (2), and per-neighbor RID allocation (6) which causes GC pressure contradicting the zero-GC design principle. |
PR Review: Graph Analytical View (GAV) — CSR-based OLAP accelerationThis is an impressive and well-designed feature. The CSR-backed OLAP overlay is a solid approach to bridging OLTP and analytical workloads. The architecture, documentation, and test coverage are all notably thorough. Below are observations across correctness, performance, and style. ✅ Strengths
🐛 Potential Bugs1.
private boolean useWhenStale;This field is read by the query planner thread via 2. Import ordering in Lines 56–59 insert GAV-related imports out of alphabetical order within the import com.arcadedb.graph.Vertex;
import com.arcadedb.graph.VertexInternal;
import com.arcadedb.graph.GraphTraversalProviderRegistry; // ← out of order
import com.arcadedb.graph.olap.GraphAnalyticalView;
...These should be sorted alphabetically with the other 3.
|
|
GAV Code Review test |
Graph Analytical View (GAV) - Code ReviewThis is a substantial and well-engineered feature addition. The architecture is sound and the implementation shows deep understanding of both graph OLAP patterns and ArcadeDB internals. Note: Replacing the test comment above with this full review. |
|
Graph Analytical View (GAV) Code Review (1/4 - Architecture) STRENGTHS: CSR double-indexed layout ideal for OLAP. Snapshot atomic-swap gives lock-free reads. hasAnyProviders fast-path avoids sync cost. DeltaCollector thread-ID-keyed ConcurrentHashMap avoids ThreadLocal leaks. Schema extension persistence is clean. CONCERNS: |
|
Graph Analytical View (GAV) Code Review (2/4 - Thread Safety and Performance) THREAD SAFETY: PERFORMANCE: |
|
Graph Analytical View (GAV) Code Review (3/4 - Correctness and Code Style) CORRECTNESS: CODE STYLE (per CLAUDE.md): |
|
Graph Analytical View (GAV) Code Review (4/4 - Tests, Security, Summary) TEST COVERAGE: SECURITY: No concerns identified. HTTP endpoints for GAV CRUD will be protected by the existing DatabaseAbstractHandler authentication/authorization layer. SUMMARY - Key action items before merge:
Overall: very well done. The CSR + delta-overlay design and query planner integration are particularly impressive pieces of work. |
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences |
Code Review — Graph Analytical View (GAV) CSR-based OLAP AccelerationThis is a substantial and architecturally sound feature. The core design — immutable 🔴 Bug —
|
Code Review: Graph Analytical View (GAV) — CSR-based OLAP accelerationThis is a large and impressive feature adding a CSR-based OLAP overlay to ArcadeDB. The architecture is well-thought-out, with clean separation between the OLTP graph, the analytical layer, and the query planner integration. Below are observations grouped by severity. Critical / Correctness Issues1. Silent delta loss on buffer overflow (GraphAnalyticalView.applyDelta()) When pendingDeltas exceeds MAX_PENDING_DELTAS, the buffer is set to null. Subsequent calls to applyDelta() skip buffering entirely due to the if (pendingDeltas != null) guard, silently discarding committed transaction changes without any warning. A stale view is expected in OFF mode, but in SYNCHRONOUS mode this means the overlay diverges silently from reality. Consider logging a warning when deltas are dropped, and explicitly transitioning to STALE state in this scenario. 2. Mutable collections in DeltaOverlay private constructor The private constructor accepts HashMap, HashSet, ArrayList etc. and stores them without defensive copying. The merge() method constructs new collections from the old ones — if the caller retains a reference and mutates after construction, overlay state can be corrupted. Since the class is package-private this is manageable, but worth an explicit comment or assertion that the inputs are not retained by the caller post-construction. 3. NodeIdMapping.addNode() vs compact() race addNode() checks if (positionsBuilder == null) without synchronization. If compact() is called concurrently from another thread, positionsBuilder can become null between the null-check and the array access, causing a NullPointerException or data corruption. This is only relevant during the build phase if builds can run concurrently — likely safe today, but fragile for future changes. 4. CSRAdjacencyIndex — no bounds validation on degree/offset accessors outDegree(nodeId), inDegree(nodeId), outOffset(nodeId), etc. access fwdOffsets[nodeId + 1] and bwdOffsets[nodeId + 1] without checking that nodeId < nodeCount. An invalid nodeId will throw ArrayIndexOutOfBoundsException with no useful message. A precondition check would improve debuggability significantly. Concurrency / Threading Issues5. GraphAnalyticalView.setUpdateMode() — listener registration gap setUpdateMode() synchronizes on this, but the interleaved sequence of unregisterChangeListeners() -> async build -> registerChangeListeners() creates a window where committed transactions are not captured. Transactions during the build period could produce TxDeltas that arrive after unregister but before register, causing those changes to be missed in SYNCHRONOUS mode. 6. Global executor shutdown timeout closeExecutor() uses a 30-second drain timeout that applies across all databases. If one database closes with a long-running build in progress, another database closing simultaneously may have its builds forcibly terminated. Consider per-database executor instances, or at minimum document this shared-pool behavior. 7. hasAnyProviders flag in GraphTraversalProviderRegistry The fast-path volatile flag is read without holding synchronized(REGISTRY). While the volatile guarantees visibility, the flag can be transiently stale. This will not cause correctness issues (an empty list just means no CSR acceleration), but a comment explaining why the TOCTOU is acceptable would help future maintainers. Performance Considerations8. NodeIdMapping — binary search for bucket index on every lookup bucketIdxForNaturalId() does a binary search on bucketBase for every call to getGlobalId(). For queries doing millions of vertex lookups this adds O(log m) per lookup where m is the number of buckets. The bucketIdToIdx[] array already exists for O(1) lookup — verify it is used on the hot path and the binary search is only a fallback. 9. Unbounded bucketIdToIdx growth for sparse bucket IDs If a database uses sparse bucket IDs (e.g., bucket ID = 1,000,000), bucketIdToIdx is allocated as a 1M-entry array of mostly -1 values (~4 MB per view). With multiple GAVs this could add up. A HashMap fallback for sparse bucket ranges would be safer. 10. DeltaOverlay.buildNeighborIndex() — unbounded array growth Neighbor arrays in the delta index start at size 5 and double on overflow. For a node with many new edges in a single transaction, repeated doubling can cause significant temporary allocations. Consider using an initial capacity hint based on the degree observed during the TxDelta scan. Design / Code Quality11. GraphAnalyticalView.closeExecutor() is static with global state The shared executor EXECUTOR is a static field. If users embed multiple ArcadeDB instances in the same JVM, all databases share one executor pool, and closeExecutor() shuts it down for everyone. A reference-counted or per-isolate design would be safer. 12. DeltaOverlay — packEdge() magic constant ((long) src << 32) | (tgt & 0xFFFFFFFFL) — the 0xFFFFFFFFL mask is correct but unexplained. A named constant like UNSIGNED_INT_MASK with a comment explaining why it is needed (tgt can be negative as a signed int) would improve maintainability. 13. NodeIdMapping.applyReordering() — no permutation validation If oldToNewMapping has duplicates or is otherwise invalid, newToOld[oldToNewMapping[i]] = i silently produces an inconsistent state. An assertion in debug/test builds would catch bugs early. 14. DB_AFTER_OPEN callback error handling In LocalDatabase.open(), the DB_AFTER_OPEN callback failure is caught and logged at SEVERE, but execution continues. For a failure during open, this could leave the database in an inconsistent state. Document why continuing is safe, or propagate the exception to abort the open. 15. Grammar: new keywords not added to identifier rule VERTEX, EDGE, REBUILD, and ANALYTICAL are used in the new DDL grammar but not added to the identifier fallback rule (only MODE, GRAPH, AUTO, PROPERTIES, COMPACTION, THRESHOLD were added). If any of these strings appear as user identifiers in queries (e.g., a property or type named vertex or edge), the parser will fail on existing queries. Each new keyword should either be added to identifier or confirmed safe to not appear in identifier positions. Test CoverageThe test suite is thorough — GraphAnalyticalViewTest at 2,445 lines is excellent. A few additional scenarios worth covering:
SummaryHigh-quality, well-architected feature with good test coverage. Priority items to address:
The SPI design via GraphTraversalProvider, the builder pattern for GraphAnalyticalView, and the query planner integration (both optimizer and traditional paths) are all well-structured. The 50-70x traversal speedup claim is credible given the CSR architecture. |
Issue #3617
Introduces Graph Analytical View (GAV), a CSR (Compressed Sparse Row) based analytical overlay on top of the OLTP graph engine. GAV provides 50-70x speedup for graph traversals by maintaining a columnar,
cache-friendly adjacency index in memory alongside the transactional graph.
Key Features
Core GAV Engine (com.arcadedb.graph.olap)
Update Modes
Query Planner Integration
DELETE)
SQL Commands
Graph Algorithms (50+ procedures refactored to use GAV)
Studio UI
Server Integration
Tests