Skip to content

Tx explicit lock#2292

Merged
lvca merged 7 commits intomainfrom
tx-explicit-lock
Jun 21, 2025
Merged

Tx explicit lock#2292
lvca merged 7 commits intomainfrom
tx-explicit-lock

Conversation

@lvca
Copy link
Contributor

@lvca lvca commented Jun 21, 2025

This pull request introduces explicit locking mechanisms for transactions, adds a new SQL grammar feature for LOCK statements, and includes minor refactoring and cleanup across the codebase. The changes enhance transaction management by allowing explicit locks on database buckets and types, while also improving code maintainability and readability.

Transaction Management Enhancements:

  • Introduced TransactionExplicitLock interface and its implementation LocalTransactionExplicitLock to allow explicit locking of buckets and types in transactions, avoiding the retry mechanism of implicit locking. (engine/src/main/java/com/arcadedb/database/TransactionExplicitLock.java - [1] engine/src/main/java/com/arcadedb/database/LocalTransactionExplicitLock.java - [2]
  • Updated TransactionContext to support explicit locking, including methods for acquiring and managing these locks, and ensuring proper cleanup during transaction reset or kill operations. (engine/src/main/java/com/arcadedb/database/TransactionContext.java - [1] [2] [3] [4] [5] [6]
  • Modified TransactionManager to support locking files with a generic requester object instead of just threads, enabling compatibility with explicit locking. (engine/src/main/java/com/arcadedb/engine/TransactionManager.java - [1] [2]

SQL Grammar Updates:

  • Added support for a new LOCK SQL statement in the grammar definition, including syntax for locking types and buckets. (engine/src/main/grammar/SQLGrammar.jjt - [1] [2] [3]

Code Refactoring and Cleanup:

These changes collectively improve the robustness of transaction handling, extend the SQL feature set, and enhance the maintainability of the codebase.

lvca added 6 commits June 20, 2025 16:24
Still remote database missing and more checks on calling lock after a page was loaded/modified in a tx
LockManager requester now is configurable, so in case of remote call, it's the session id instead of the Thread object. This allow to unlock resources from a different thread.
@lvca lvca requested a review from Copilot June 21, 2025 05:51
@lvca lvca self-assigned this Jun 21, 2025
@lvca lvca added the enhancement New feature or request label Jun 21, 2025
@lvca lvca added this to the 25.6.1 milestone Jun 21, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @lvca, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances transaction management by introducing explicit locking capabilities for database buckets and types. This new mechanism allows transactions to acquire pessimistic locks, thereby avoiding the default retry behavior of implicit locking. Additionally, a new LOCK SQL statement has been added to provide a direct way for users to manage these explicit locks. The changes also include foundational updates to the transaction manager to support more flexible locking requesters and general code refactoring for improved maintainability.

Highlights

  • Explicit Transaction Locking: Introduced TransactionExplicitLock interface and LocalTransactionExplicitLock implementation to allow explicit, pessimistic locking of database buckets and types within transactions. This aims to prevent transaction retries caused by implicit locking.
  • New SQL LOCK Statement: Added a new LOCK SQL command to the grammar, enabling users to explicitly lock types and buckets directly via SQL.
  • Transaction Manager Enhancements: Modified the TransactionManager to support generic requester objects (instead of just Thread instances) for file locking, improving flexibility for explicit locking scenarios (e.g., HTTP sessions).
  • Transaction Context Updates: Enhanced TransactionContext to manage explicit locks, ensuring proper acquisition, release, and cleanup during transaction lifecycle events (commit, rollback, kill).
  • Remote Database Support: Extended remote database operations to support explicit locking, allowing remote clients to utilize the new locking mechanisms.
  • Code Refactoring and Cleanup: Performed various code cleanups, including import consolidation, minor formatting adjustments, and removal of unused code, improving overall code maintainability.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly enhances transaction management by introducing explicit locking mechanisms, allowing users to acquire pessimistic locks on database buckets and types. The new LOCK SQL statement provides a clear interface for this feature. The changes also include necessary refactoring in the TransactionManager to support generic requesters for locks, which is a robust improvement for multi-threaded environments. Comprehensive test cases have been added to validate the new functionality and its error handling. Overall, the changes improve the robustness and control over transactions.

Comment on lines +813 to +824
protected void explicitLock(final Set<Integer> filesToLock) {
if (explicitLockedFiles != null)
throw new TransactionException("Explicit lock already acquired");

if (!indexChanges.isEmpty() || !immutablePages.isEmpty() || //
(modifiedPages != null && !modifiedPages.isEmpty()) || //
(newPages != null && !newPages.isEmpty()) //
)
throw new TransactionException("Explicit lock must be acquired before any modification");

explicitLockedFiles = lockFilesInOrder(filesToLock);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The explicitLock method includes crucial checks to ensure the integrity of the explicit locking mechanism. Specifically, it prevents acquiring an explicit lock if one is already active or if any modifications (index changes, modified pages, new pages) have already occurred within the transaction. This strict enforcement ensures that pessimistic locks are established on a clean state, preventing potential race conditions or inconsistent states. This is a strong and correct design choice for pessimistic locking.

Comment on lines +588 to +595
if (explicitLockedFiles != null) {
// CHECK THE LOCKED FILES ARE ALL LOCKED ALREADY
if (!explicitLockedFiles.containsAll(modifiedFiles)) {
final HashSet<Integer> left = new HashSet<>(modifiedFiles);
left.removeAll(explicitLockedFiles);
throw new TransactionException(
"Cannot commit transaction because not all the modified resources were locked: " + left);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The check if (!explicitLockedFiles.containsAll(modifiedFiles)) within commit1stPhase is vital for correctness when explicit locks are in use. It ensures that all files that were actually modified during the transaction were also explicitly locked. If a transaction attempts to modify a resource that was not part of its explicit lock set, it correctly throws a TransactionException. This prevents accidental modifications to unlocked resources, which is a core tenet of pessimistic locking.

case "name":
final String newTypeName = identifierValue.getStringValue();
final int bucketSize = type.getInvolvedBuckets().size();
final int bucketSize = type.getBuckets(false).size();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The change from type.getInvolvedBuckets().size() to type.getBuckets(false).size() for calculating bucketSize in the ALTER TYPE NAME statement has a functional implication. getInvolvedBuckets() would include buckets from polymorphic subtypes, while getBuckets(false) only includes buckets directly associated with the type itself. If the subsequent logic for renaming a type relies on the total number of buckets within the type's hierarchy (including inherited ones), this change could lead to incorrect behavior or allow operations that should be restricted. For example, a type might have no direct buckets but its subtypes do, and this change would make bucketSize zero, potentially bypassing checks. Please ensure this change aligns with the intended behavior for ALTER TYPE NAME across the entire type hierarchy.

private final AtomicLong transactionIds = new AtomicLong();
private final AtomicLong logFileCounter = new AtomicLong();
private final LockManager<Integer, Thread> fileIdsLockManager = new LockManager<>();
private final LockManager<Integer, Object> fileIdsLockManager = new LockManager<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Changing the generic type of fileIdsLockManager from Thread to Object is a necessary and correct adjustment. This allows the LockManager to handle a more generalized requester object, which is crucial for supporting explicit locking in scenarios where the lock initiator might not always be the current thread (e.g., an HTTP session ID). This enhances the flexibility and correctness of the locking mechanism.

* the new acquired locks are released.
*/
public List<Integer> tryLockFiles(final Collection<Integer> fileIds, final long timeout) {
public List<Integer> tryLockFiles(final Collection<Integer> fileIds, final long timeout, final Object requester) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The addition of the requester parameter to tryLockFiles and its subsequent propagation to tryLockFile and unlockFilesInOrder is a significant improvement. This ensures that locks are correctly associated with the specific entity (e.g., a user session) that initiated the transaction, rather than just the executing thread. This is vital for accurate lock management in concurrent and distributed environments, preventing issues like incorrect unlocking or deadlocks when multiple sessions might share threads.

Comment on lines +49 to +50
if (requester == null)
throw new IllegalArgumentException("Requester is null");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Adding a null check for the requester parameter in the tryLock method is a good defensive programming practice. Since the requester type has been generalized to Object, it's important to explicitly validate that it's not null before proceeding, preventing potential NullPointerExceptions in downstream operations that rely on the requester object.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Introduces explicit transaction locks, extends SQL grammar for LOCK statements, and updates related handlers and tests.

  • Adds TransactionExplicitLock interface and implementations for explicit pessimistic locking
  • Updates SQL parser and grammar to recognize and execute LOCK statements
  • Enhances server and network layers to propagate a generic requester for file locks
  • Adds integration and unit tests for explicit lock behavior

Reviewed Changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
server/src/test/java/com/arcadedb/remote/RemoteDatabaseJavaApiIT.java Adds an end-to-end test for remote explicit locking
engine/src/test/java/com/arcadedb/query/sql/method/collection/SQLMethodTransformTest.java Stubs out acquireLock in a test harness (returns null)
engine/src/main/java/com/arcadedb/database/TransactionContext.java Implements new explicit lock state, requester tracking, and cleanup
engine/src/main/java/com/arcadedb/database/LocalTransactionExplicitLock.java Implements bucket/type-to-file mapping and delegation to TransactionContext
engine/src/main/java/com/arcadedb/query/sql/parser/LockStatement.java Adds AST node and execution logic for the LOCK SQL statement
Comments suppressed due to low confidence (2)

engine/src/main/java/com/arcadedb/database/TransactionContext.java:138

  • [nitpick] This public lock() method lacks Javadoc. Consider adding a comment explaining its role in initiating an explicit lock within the transaction lifecycle.
  public LocalTransactionExplicitLock lock() {

engine/src/main/java/com/arcadedb/database/LocalTransactionExplicitLock.java:19

  • The generated-file header references OAlterClassStatement.java, which is unrelated to this class. Please update or remove the incorrect header to prevent confusion.
/* Generated By:JJTree: Do not edit this line. OAlterClassStatement.java Version 4.3 */


@Override
public LocalTransactionExplicitLock acquireLock() {
return null;
Copy link

Copilot AI Jun 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning null from acquireLock will lead to NullPointerExceptions when the test invokes locking logic. Consider returning a valid LocalTransactionExplicitLock or a mock implementation to avoid NPEs and ensure the test runs as intended.

Suggested change
return null;
return new LocalTransactionExplicitLock() {
@Override
public void lock() {
// Mock lock behavior
}
@Override
public void unlock() {
// Mock unlock behavior
}
};

Copilot uses AI. Check for mistakes.

} catch (Exception e) {
caughtExceptions.incrementAndGet();
e.printStackTrace();
Copy link

Copilot AI Jun 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Avoid using e.printStackTrace() in tests. Instead, let the exception propagate or use a logger/assertion to capture failures so the test framework reports errors cleanly.

Copilot uses AI. Check for mistakes.
assertThat(rid[0].asVertex().getInteger("id")).isEqualTo(CONCURRENT_THREADS * TOT);
assertThat(committed.get()).isEqualTo(CONCURRENT_THREADS * TOT);
assertThat(caughtExceptions.get()).isEqualTo(0);
assertThat(committed.get() + caughtExceptions.get()).isEqualTo(TOT * CONCURRENT_THREADS);
Copy link

Copilot AI Jun 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The test opens a RemoteDatabase instance at the start but never closes it. Consider adding database.close() at the end of the test to properly clean up resources.

Copilot uses AI. Check for mistakes.
@codacy-production
Copy link

codacy-production bot commented Jun 21, 2025

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
-0.21% 76.45%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (b997918) 70259 45155 64.27%
Head commit (1931f1f) 70439 (+180) 45125 (-30) 64.06% (-0.21%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#2292) 1792 1370 76.45%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

@lvca lvca merged commit 3e03494 into main Jun 21, 2025
5 of 7 checks passed
@lvca lvca deleted the tx-explicit-lock branch June 21, 2025 06:47
robfrank pushed a commit that referenced this pull request Jun 23, 2025
* First version of explicit tx lock.

Still remote database missing and more checks on calling lock after a page was loaded/modified in a tx

* test: fixed test case

* test: refactoring test case

* test: added more test cases to implicit tx locking

* Extended SQL BEGIN adding LOCK TYPE|BUCKET

* Supported SQL `lock` command to be used also by RemoteDatabase API

LockManager requester now is configurable, so in case of remote call, it's the session id instead of the Thread object. This allow to unlock resources from a different thread.

* test: added missing db.close()

(cherry picked from commit 3e03494)
@lvca lvca added the fixed label Jul 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants