Skip to content

test(load): time series load tests for single node and 3-node cluster#4450

Merged
robfrank merged 4 commits into
mainfrom
tests/timeseries-loadtests
Jun 1, 2026
Merged

test(load): time series load tests for single node and 3-node cluster#4450
robfrank merged 4 commits into
mainfrom
tests/timeseries-loadtests

Conversation

@robfrank

@robfrank robfrank commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds Testcontainers-based load tests dedicated to the time series engine, covering a single standalone server and a 3-node Raft HA cluster. Reuses the existing ContainersTestTemplate infrastructure unchanged.

Ingestion is exercised over three protocols via a new TimeSeriesDatabaseWrapper:

  • InfluxDB line protocol HTTP endpoint (POST /api/v1/ts/{db}/write)
  • SQL INSERT over HTTP RemoteDatabase
  • SQL INSERT over gRPC RemoteDatabase

Each test is @ParameterizedTest over those three protocols.

What is verified

  • Single node (SingleServerTimeSeriesLoadTestIT): concurrent bulk ingestion, exact total point count, time-bucket aggregation extremes over a sealed bulk sensor, and the tag-filtered latest value.
  • 3-node Raft HA (ThreeNodesTimeSeriesLoadTestIT): replication consistency by converging the unfiltered point count across all nodes (Awaitility), then per-node aggregation extremes and latest value. compareAllDatabases() is skipped (non-persistent containers), matching ThreeNodesLoadTestIT.

Notes on time series query semantics (informed the assertions)

Verified against the server image:

  • Tag-filtered queries (WHERE sensor_id = ?, both plain count(*) and ts.timeBucket(...) GROUP BY) reflect only the asynchronously sealed subset of points; the unfiltered count(*) and the /latest endpoint see all data. So exact totals are asserted via the unfiltered count, while aggregation is asserted on stable min/max extremes over a sealed bulk sensor (with polling to absorb seal/replication lag).
  • bucket is a reserved SQL word, so the time-bucket alias is tbucket.
  • A SQL INSERT into a time series type returns a record without an @rid, which the remote client can reject after the server commits; this is tolerated in the ingest path (no data loss).

Test plan

Run against arcadedata/arcadedb:latest:

mvn -pl load-tests install -DskipTests -DskipITs=true
mvn -pl load-tests verify -DskipITs=false \
  -Dit.test="SingleServerTimeSeriesLoadTestIT,ThreeNodesTimeSeriesLoadTestIT" \
  -Dfailsafe.failIfNoSpecifiedTests=false

All 6 parameterized cases (3 single-node + 3 cluster) pass locally. The 3-node test needs ~10 GB free for Docker.

@codacy-production

codacy-production Bot commented Jun 1, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 40 complexity

Metric Results
Complexity 40

View in Codacy

🟢 Coverage ∅ diff coverage · -7.01% coverage variation

Metric Results
Coverage variation -7.01% coverage variation
Diff coverage diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (7cae7b7) 128345 95039 74.05%
Head commit (e3dc3d0) 156028 (+27683) 104605 (+9566) 67.04% (-7.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#4450) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces integration load tests for ArcadeDB's time series functionality, including single-server and three-node Raft cluster replication tests, along with a TimeSeriesDatabaseWrapper helper class to drive the workload. The code review highlights several critical resource leak issues where connections, servers, database result sets, and input streams are not properly closed. It is highly recommended to address these by wrapping the affected resources in try-finally or try-with-resources blocks to ensure proper cleanup and prevent connection exhaustion during load testing.

Comment on lines +65 to +70
executor.submit(() -> {
final TimeSeriesDatabaseWrapper w = new TimeSeriesDatabaseWrapper(server, protocol);
final long base = 1_000_000_000L + threadIndex * 100_000_000L;
w.ingestSeries("sensor-" + threadIndex, "region-" + (threadIndex % 3), base, POINTS_PER_THREAD);
w.close();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If ingestSeries throws an exception, the TimeSeriesDatabaseWrapper instance w is never closed, leaking the RemoteDatabase connection and its associated HTTP/gRPC client resources. Wrap the execution in a try-finally block to guarantee resource cleanup.

      executor.submit(() -> {
        final TimeSeriesDatabaseWrapper w = new TimeSeriesDatabaseWrapper(server, protocol);
        try {
          final long base = 1_000_000_000L + threadIndex * 100_000_000L;
          w.ingestSeries("sensor-" + threadIndex, "region-" + (threadIndex % 3), base, POINTS_PER_THREAD);
        } finally {
          w.close();
        }
      });

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 08e86f5 - the writer is wrapped in try-finally so w.close() (which closes the RemoteDatabase and its HttpClient/gRPC channel) runs even if ingestion throws.

Comment on lines +95 to +100
executor.submit(() -> {
final TimeSeriesDatabaseWrapper w = new TimeSeriesDatabaseWrapper(servers.getFirst(), protocol);
final long base = 1_000_000_000L + threadIndex * 100_000_000L;
w.ingestSeries("sensor-" + threadIndex, "region-" + (threadIndex % 3), base, POINTS_PER_THREAD);
w.close();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If ingestSeries throws an exception, the TimeSeriesDatabaseWrapper instance w is never closed, leaking the RemoteDatabase connection and its associated HTTP/gRPC client resources. Wrap the execution in a try-finally block to guarantee resource cleanup.

      executor.submit(() -> {
        final TimeSeriesDatabaseWrapper w = new TimeSeriesDatabaseWrapper(servers.getFirst(), protocol);
        try {
          final long base = 1_000_000_000L + threadIndex * 100_000_000L;
          w.ingestSeries("sensor-" + threadIndex, "region-" + (threadIndex % 3), base, POINTS_PER_THREAD);
        } finally {
          w.close();
        }
      });

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 08e86f5 - same try-finally around the writer wrapper here.

Comment on lines +104 to +130
public void createDatabase() {
final RemoteServer httpServer = new RemoteServer(server.host(), server.httpPort(), "root", PASSWORD);
httpServer.setConnectionStrategy(RemoteHttpComponent.CONNECTION_STRATEGY.FIXED);

if (httpServer.exists(DATABASE)) {
logger.info("Dropping existing database {}", DATABASE);
httpServer.drop(DATABASE);
}
logger.info("Creating database {}", DATABASE);

final int maxAttempts = 30;
for (int attempt = 1; attempt <= maxAttempts; attempt++) {
try {
httpServer.create(DATABASE);
return;
} catch (final ServerIsNotTheLeaderException e) {
if (e.getLeaderAddress() != null || attempt == maxAttempts)
throw e;
logger.info("Leader address not yet known (attempt {}/{}), retrying in 2s...", attempt, maxAttempts);
} catch (final Exception e) {
if (attempt == maxAttempts)
throw e;
logger.info("Database creation attempt {}/{} failed ({}), retrying in 2s...", attempt, maxAttempts, e.getMessage());
}
sleep(2_000);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The RemoteServer instance httpServer is created locally but never closed. Since RemoteServer extends RemoteHttpComponent which manages an internal HttpClient and thread pool, failing to close it leaks these resources. Wrap the logic in a try-finally block to ensure httpServer.close() is called.

  public void createDatabase() {
    final RemoteServer httpServer = new RemoteServer(server.host(), server.httpPort(), "root", PASSWORD);
    try {
      httpServer.setConnectionStrategy(RemoteHttpComponent.CONNECTION_STRATEGY.FIXED);

      if (httpServer.exists(DATABASE)) {
        logger.info("Dropping existing database {}", DATABASE);
        httpServer.drop(DATABASE);
      }
      logger.info("Creating database {}", DATABASE);

      final int maxAttempts = 30;
      for (int attempt = 1; attempt <= maxAttempts; attempt++) {
        try {
          httpServer.create(DATABASE);
          return;
        } catch (final ServerIsNotTheLeaderException e) {
          if (e.getLeaderAddress() != null || attempt == maxAttempts)
            throw e;
          logger.info("Leader address not yet known (attempt {}/{}), retrying in 2s...", attempt, maxAttempts);
        } catch (final Exception e) {
          if (attempt == maxAttempts)
            throw e;
          logger.info("Database creation attempt {}/{} failed ({}), retrying in 2s...", attempt, maxAttempts, e.getMessage());
        }
        sleep(2_000);
      }
    } finally {
      httpServer.close();
    }
  }

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 08e86f5 - createDatabase now closes the RemoteServer in a finally block. Confirmed RemoteServer extends RemoteHttpComponent, which owns its own HttpClient and whose close() calls shutdownNow()+close(), so this prevents leaking that client and its NIO thread (a connection distinct from the long-lived query db).

Comment on lines +139 to +144
try {
db.command("sql",
"CREATE TIMESERIES TYPE " + TYPE_NAME
+ " TIMESTAMP ts TAGS (sensor_id STRING, region STRING) FIELDS (temperature DOUBLE, humidity DOUBLE)");
return;
} catch (final Exception e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ResultSet returned by db.command is never closed, which can leak database cursors and connection resources. Use try-with-resources to ensure it is closed automatically.

Suggested change
try {
db.command("sql",
"CREATE TIMESERIES TYPE " + TYPE_NAME
+ " TIMESTAMP ts TAGS (sensor_id STRING, region STRING) FIELDS (temperature DOUBLE, humidity DOUBLE)");
return;
} catch (final Exception e) {
try (final ResultSet rs = db.command("sql",
"CREATE TIMESERIES TYPE " + TYPE_NAME
+ " TIMESTAMP ts TAGS (sensor_id STRING, region STRING) FIELDS (temperature DOUBLE, humidity DOUBLE)")) {
return;
} catch (final Exception e) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied. RemoteDatabase.query/command return an InternalResultSet that eagerly materializes the entire HTTP/gRPC response into an in-memory list; its close() only does content.clear() (see InternalResultSet.java). No server-side cursor or network connection is held by the result set, so there is nothing to leak here and try-with-resources would change only when a small in-memory list becomes garbage. This is also consistent with the sibling DatabaseWrapper, which does not close these result sets.

Comment on lines +167 to +170
try {
db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME).next();
return;
} catch (final Exception e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ResultSet returned by db.query is never closed, which can leak database cursors and connection resources. Use try-with-resources to ensure it is closed automatically.

Suggested change
try {
db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME).next();
return;
} catch (final Exception e) {
try (final ResultSet rs = db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME)) {
rs.next();
return;
} catch (final Exception e) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied. RemoteDatabase.query/command return an InternalResultSet that eagerly materializes the entire HTTP/gRPC response into an in-memory list; its close() only does content.clear() (see InternalResultSet.java). No server-side cursor or network connection is held by the result set, so there is nothing to leak here and try-with-resources would change only when a small in-memory list becomes garbage. This is also consistent with the sibling DatabaseWrapper, which does not close these result sets.

Comment on lines +235 to +239
try {
db.command("sql",
"INSERT INTO " + TYPE_NAME + " SET ts = ?, sensor_id = ?, region = ?, temperature = ?, humidity = ?",
ts, sensorId, region, temperature, humidity);
} catch (final Exception e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ResultSet returned by db.command is never closed, which can leak database cursors and connection resources. Since this is called for every SQL insert, it will leak thousands of cursors during load tests. Use try-with-resources to ensure it is closed automatically.

Suggested change
try {
db.command("sql",
"INSERT INTO " + TYPE_NAME + " SET ts = ?, sensor_id = ?, region = ?, temperature = ?, humidity = ?",
ts, sensorId, region, temperature, humidity);
} catch (final Exception e) {
try (final ResultSet rs = db.command("sql",
"INSERT INTO " + TYPE_NAME + " SET ts = ?, sensor_id = ?, region = ?, temperature = ?, humidity = ",
ts, sensorId, region, temperature, humidity)) {
// Execute command and auto-close ResultSet
} catch (final Exception e) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied. RemoteDatabase.query/command return an InternalResultSet that eagerly materializes the entire HTTP/gRPC response into an in-memory list; its close() only does content.clear() (see InternalResultSet.java). No server-side cursor or network connection is held by the result set, so there is nothing to leak here and try-with-resources would change only when a small in-memory list becomes garbage. This is also consistent with the sibling DatabaseWrapper, which does not close these result sets. Separately, the suggested snippet is incorrect: it drops a bind parameter - temperature = ?, humidity = with only four arguments - which would break the INSERT.

Comment on lines +276 to +279
public long countPoints() {
final ResultSet rs = db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME);
return ((Number) rs.next().getProperty("cnt")).longValue();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ResultSet returned by db.query is never closed, which can leak database cursors and connection resources. Use try-with-resources to ensure it is closed automatically.

Suggested change
public long countPoints() {
final ResultSet rs = db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME);
return ((Number) rs.next().getProperty("cnt")).longValue();
}
public long countPoints() {
try (final ResultSet rs = db.query("sql", "SELECT count(*) AS cnt FROM " + TYPE_NAME)) {
return ((Number) rs.next().getProperty("cnt")).longValue();
}
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied. RemoteDatabase.query/command return an InternalResultSet that eagerly materializes the entire HTTP/gRPC response into an in-memory list; its close() only does content.clear() (see InternalResultSet.java). No server-side cursor or network connection is held by the result set, so there is nothing to leak here and try-with-resources would change only when a small in-memory list becomes garbage. This is also consistent with the sibling DatabaseWrapper, which does not close these result sets.

Comment on lines +294 to +301
while (true) {
final ResultSet rs = db.query("sql", sql, sensorId);
if (rs.hasNext()) {
final Result r = rs.next();
return new AggResult(toDouble(r.getProperty("avgT")), toDouble(r.getProperty("minT")), toDouble(r.getProperty("maxT")),
((Number) r.getProperty("cnt")).longValue());
}
if (System.currentTimeMillis() >= deadline)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The ResultSet returned by db.query is never closed, which can leak database cursors and connection resources. Since this is called in a polling loop, it will leak a ResultSet on every iteration until the aggregation succeeds. Use try-with-resources to ensure it is closed automatically.

Suggested change
while (true) {
final ResultSet rs = db.query("sql", sql, sensorId);
if (rs.hasNext()) {
final Result r = rs.next();
return new AggResult(toDouble(r.getProperty("avgT")), toDouble(r.getProperty("minT")), toDouble(r.getProperty("maxT")),
((Number) r.getProperty("cnt")).longValue());
}
if (System.currentTimeMillis() >= deadline)
while (true) {
try (final ResultSet rs = db.query("sql", sql, sensorId)) {
if (rs.hasNext()) {
final Result r = rs.next();
return new AggResult(toDouble(r.getProperty("avgT")), toDouble(r.getProperty("minT")), toDouble(r.getProperty("maxT")),
((Number) r.getProperty("cnt")).longValue());
}
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applied. RemoteDatabase.query/command return an InternalResultSet that eagerly materializes the entire HTTP/gRPC response into an in-memory list; its close() only does content.clear() (see InternalResultSet.java). No server-side cursor or network connection is held by the result set, so there is nothing to leak here and try-with-resources would change only when a small in-memory list becomes garbage. This is also consistent with the sibling DatabaseWrapper, which does not close these result sets. Each iteration's InternalResultSet is eligible for GC immediately; close() remains a no-op clear().

Comment on lines +344 to +349
try {
if (conn.getResponseCode() != 200)
throw new IOException("Latest query failed: HTTP " + conn.getResponseCode());
final JSONObject json = new JSONObject(new String(conn.getInputStream().readAllBytes(), StandardCharsets.UTF_8));
final JSONArray latest = json.getJSONArray("latest");
return latest.getLong(0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The InputStream returned by conn.getInputStream() is not closed, which can leak connection resources or prevent connection reuse. Use try-with-resources to ensure it is closed automatically.

    try {
      if (conn.getResponseCode() != 200)
        throw new IOException("Latest query failed: HTTP " + conn.getResponseCode());
      final JSONObject json;
      try (java.io.InputStream is = conn.getInputStream()) {
        json = new JSONObject(new String(is.readAllBytes(), StandardCharsets.UTF_8));
      }
      final JSONArray latest = json.getJSONArray("latest");
      return latest.getLong(0);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 08e86f5 - the input stream is now read inside try-with-resources.

Comment on lines +260 to +266
final int status = conn.getResponseCode();
if (status != 204) {
String err = "";
if (conn.getErrorStream() != null)
err = new String(conn.getErrorStream().readAllBytes(), StandardCharsets.UTF_8);
throw new IOException("Line protocol write failed: HTTP " + status + " " + err);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The InputStream returned by conn.getErrorStream() is not closed, which can leak connection resources or prevent connection reuse. Use try-with-resources to ensure it is closed automatically.

Suggested change
final int status = conn.getResponseCode();
if (status != 204) {
String err = "";
if (conn.getErrorStream() != null)
err = new String(conn.getErrorStream().readAllBytes(), StandardCharsets.UTF_8);
throw new IOException("Line protocol write failed: HTTP " + status + " " + err);
}
final int status = conn.getResponseCode();
if (status != 204) {
String err = "";
try (java.io.InputStream es = conn.getErrorStream()) {
if (es != null) {
err = new String(es.readAllBytes(), StandardCharsets.UTF_8);
}
}
throw new IOException("Line protocol write failed: HTTP " + status + " " + err);
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 08e86f5 - the error stream is now read inside try-with-resources.

@codecov

codecov Bot commented Jun 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.62%. Comparing base (7cae7b7) to head (e3dc3d0).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4450      +/-   ##
============================================
+ Coverage     64.88%   65.62%   +0.73%     
============================================
  Files          1649     1612      -37     
  Lines        128345   124322    -4023     
  Branches      27525    26910     -615     
============================================
- Hits          83282    81583    -1699     
+ Misses        33335    31228    -2107     
+ Partials      11728    11511     -217     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

robfrank added 2 commits June 1, 2026 18:12
…ster

Add Testcontainers-based load tests dedicated to the time series engine,
covering a single standalone server and a 3-node Raft HA cluster. Ingestion
is driven over three protocols (InfluxDB line protocol HTTP endpoint, and SQL
INSERT over both HTTP and gRPC RemoteDatabase) via a new TimeSeriesDatabaseWrapper
that reuses the existing ContainersTestTemplate infrastructure.

The single-node test asserts the exact total point count, time-bucket aggregation
extremes, and the tag-filtered latest value. The 3-node test verifies replication
consistency by converging the unfiltered count across all nodes, then asserts
per-node aggregation extremes and latest value.

Verification accounts for time series behaviors observed against the server:
tag-filtered queries (count and timeBucket aggregation) reflect only the
asynchronously-sealed subset of points, so exact totals are asserted via the
unfiltered count while aggregation is asserted on stable min/max extremes over a
sealed bulk sensor with polling; schema presence is polled to absorb replication
lag; and the RID-less SQL INSERT response is tolerated.
Address code review resource-cleanup findings:
- Wrap the per-thread writer wrapper usage in try-finally so the RemoteDatabase
  (and its HttpClient / gRPC channel) is closed even if ingestion throws.
- Close the RemoteServer in createDatabase via try-finally; RemoteServer owns its
  own HttpClient (RemoteHttpComponent) and would otherwise leak the client and its
  NIO thread, separate from the long-lived query connection.
- Close the HttpURLConnection input and error streams via try-with-resources in
  latestTimestamp and postLineProtocol.

ResultSet try-with-resources was intentionally not added: RemoteDatabase
materializes responses eagerly into an in-memory InternalResultSet whose close()
only clears the list, so no server cursors or connections are held.
@robfrank robfrank force-pushed the tests/timeseries-loadtests branch from 3c3b061 to 08e86f5 Compare June 1, 2026 16:28
@claude

claude Bot commented Jun 1, 2026

Copy link
Copy Markdown

Code Review

Good addition overall - the parameterized approach across three protocols, the well-documented async assertion strategy, and the reuse of ContainersTestTemplate infrastructure are all solid. A few issues to address before merging.

BUGS / CORRECTNESS

Resource leaks when assertions fail

In SingleServerTimeSeriesLoadTestIT, admin.close() is called at the end of the test body but not in a finally block. If any assertion fails (e.g. assertAggregateExtremes), the remote connection is leaked. Same problem in ThreeNodesTimeSeriesLoadTestIT for ingest, r0, r1, r2. Implement AutoCloseable on TimeSeriesDatabaseWrapper and use try-with-resources, or wrap in a try/finally.


MISSING @tag("slow") ANNOTATIONS

From CLAUDE.md: "Use @tag("slow") for functional regression tests that take noticeably long (large batches, multi-second elapsed time, big payloads)."

Both test classes ingest 50,000+ points, poll for minutes, and the 3-node test requires ~10 GB of Docker resources. They need @tag("slow") plus import org.junit.jupiter.api.Tag on both test classes.


SILENT EXCEPTION SWALLOWING IN assertThatPointCountIs

The existing DatabaseWrapper.assertThatUserCountIs tracks lastException and propagates it with an AssertionError if the deadline expires without a successful count. The new implementation discards all exceptions silently. If the connection is broken throughout the 30s window, the test will report "-1 != expectedTotal" with no indication of why queries kept failing. Mirror the existing pattern.


STYLE / CONVENTION ISSUES

Inconsistent API style in ThreeNodesTimeSeriesLoadTestIT

The lambda body uses servers.getFirst() (Java 21 SequencedCollection API), but directly above in the same method, the outer code uses servers.get(0). The existing SingleServerLoadTestIT uses getFirst(), so standardise on that.

Float equality comparison in assertAggregateExtremes

The check agg.min() == expectedMin && agg.max() == expectedMax uses ==. 15.0 and 34.0 happen to be exactly representable in IEEE 754 so this works today, but isCloseTo(expectedMin, offset(1e-9)) is safer and signals intent more clearly for any future reader.

URL parameter not encoded in latestTimestamp

The tag value (e.g. "sensor_id:verify") contains a colon and is appended raw to the URL query string. Use URLEncoder.encode(tag, StandardCharsets.UTF_8) to be safe against future tag values that contain characters requiring percent-encoding.

SQL protocol batching asymmetry

Line protocol batches 500 points per HTTP call; SQL protocol sends one INSERT per round-trip (10,000 per thread). The three protocol cases therefore measure fundamentally different throughput characteristics and the SQL cases will be considerably slower. Consider batching SQL inserts via sqlscript, or add a Javadoc note so readers know the timings are not comparable across protocols.


MINOR

  • ResultSet returned by countPoints(), aggregate(), and checkSchema() is never closed (consistent with the pre-existing DatabaseWrapper pattern, but worth a follow-up issue to track).
  • The nested polling in assertAggregateExtremes (outer loop up to 60s, inner aggregate() up to 30s each call) can compound to 90s per assertion since aggregate() already retries internally. Consider collapsing into one polling level to keep the total bounded.

SUMMARY

Severity Item
Bug Wrappers not closed in finally block - resource leak on assertion failure
Required Add @tag("slow") to both test classes (CLAUDE.md requirement)
Improvement Track and surface last exception in assertThatPointCountIs
Style Inconsistent getFirst() vs get(0) in ThreeNodes test
Style Float == comparison instead of isCloseTo
Style URL-encode the tag query parameter in latestTimestamp
Observation SQL protocol: one INSERT per point vs line protocol batches of 500

…-insert sample loss

Documents a reproducible data-loss: under many concurrent single-row INSERTs into a
time series type (each its own transaction, as over the wire), serialized per-shard
appends can read a stale per-page sample count and write the same row slot, so one
sample is silently overwritten (count comes up short, no error). Reproduces reliably
at high concurrency with a single shard; the batched line-protocol path is unaffected.

Kept @disabled because the fix is a delicate change to the time series append path
(making the slot read-modify-write durable under the per-shard append lock without
breaking HA WAL shipping). Enable once that is fixed.
@claude

claude Bot commented Jun 1, 2026

Copy link
Copy Markdown

Code Review

Good addition - the time series load tests fill a real gap and the three-protocol parameterisation (LINE_PROTOCOL, SQL_HTTP, SQL_GRPC) gives broad coverage. The TimeSeriesDatabaseWrapper is well-structured and the explanatory comments on time series query semantics are valuable. A few issues to address before merging:


Required fixes

1. Missing @Tag("slow") on both IT classes

CLAUDE.md explicitly requires @Tag("slow") on functional tests that take noticeably long. Both classes ingest 50 000+ points in Docker containers and take minutes to run - they must be tagged.

// SingleServerTimeSeriesLoadTestIT.java
@Tag("slow")
class SingleServerTimeSeriesLoadTestIT extends ContainersTestTemplate { …

// ThreeNodesTimeSeriesLoadTestIT.java
@Tag("slow")
class ThreeNodesTimeSeriesLoadTestIT extends ContainersTestTemplate { …

2. URL encoding missing in latestTimestamp()

TimeSeriesDatabaseWrapper.java appends the tag value directly to the URL:

url.append("&tag=").append(tag);

The values passed from the tests include a colon (sensor_id:sensor-0). While the colon and dash happen to work with the current server implementation, query-string values should be percent-encoded. If a future tag or sensor name includes +, &, =, %, or a space, the request will silently send the wrong value. Use URLEncoder.encode(tag, StandardCharsets.UTF_8).


3. Interrupt handling in the monitoring loop does not break

Both SingleServerTimeSeriesLoadTestIT and ThreeNodesTimeSeriesLoadTestIT have the same pattern:

} catch (final InterruptedException e) {
    Thread.currentThread().interrupt();
    // no break!
}

Re-setting the interrupt flag without breaking means the loop re-enters, calls logger.info, then attempts sleep again, which immediately throws InterruptedException again - an effective spin-loop until the executor terminates. Compare DatabaseWrapper.createDatabase() which breaks on interrupt. Add break; after Thread.currentThread().interrupt().


4. Interrupt handling in createSchema() and createDatabase() differs from the existing helper

DatabaseWrapper.createDatabase() explicitly interrupts and breaks on InterruptedException in the retry loop. TimeSeriesDatabaseWrapper.createDatabase() delegates to the private sleep() helper which re-interrupts but doesn't signal the loop to stop, so after an interrupt the loop keeps retrying. The createSchema() method has the same issue. For consistency and correctness, the retry loops should break on interrupt.


Minor / suggestions

5. @Disabled regression test has no issue reference

TimeSeriesConcurrentInsertTest is a valuable document of a known bug, but with no linked issue it can get stranded indefinitely. Add the tracking issue number to the @Disabled reason string (e.g. "Tracks issue #NNNN"), or raise an issue for the append-path fix if one doesn't exist yet.

6. ThreeNodesTimeSeriesLoadTestIT.tearDown() comment is misleading

The comment says "Skip compareAllDatabases()" but ContainersTestTemplate.tearDown() does not call compareAllDatabases() by default - it never did. The override is an exact delegation to super.tearDown(), which means it's a no-op override (copied from ThreeNodesLoadTestIT which has the same pattern). Either remove the override entirely, or add a comment explaining why this class needs to suppress a future base-class addition.

7. aggregate() may sleep past the deadline

final long deadline = System.currentTimeMillis() + 30_000;
while (true) {
    final ResultSet rs = db.query("sql", sql, sensorId);
    if (rs.hasNext()) { … return; }
    if (System.currentTimeMillis() >= deadline)
        throw new IllegalStateException(…);
    sleep(1_000);           // no second deadline check before sleeping
}

If the deadline expires between the >= deadline check and the sleep(1_000) call, the method overshoots by up to one extra second. Not a real problem here, but assertAggregateExtremes then polls aggregate in its own loop, so the worst-case overshoot can compound. Consider checking the deadline again before sleeping.

8. insertSql() silently swallows exceptions

Insertion failures in the SQL path are only logged and counted; they don't propagate. This means the final assertThatPointCountIs can fail with a confusing count mismatch rather than the root-cause exception. Consider logging the full stack trace (not just e.getMessage()) so failures are diagnosable in CI output.

9. SQL path does not batch

ingestSeries() issues one INSERT per point when protocol != LINE_PROTOCOL (10 000 per thread). The line-protocol path batches in groups of 500. This is presumably intentional (stress-testing single-row throughput), but worth confirming - if the intent is throughput parity, SQL batching with sqlscript or multi-value INSERT would be needed.


Nit

TimeSeriesDatabaseWrapper.java field alignment uses wide padding (public static final String TYPE_NAME) which is unusual for this codebase. Minor, but worth aligning with the surrounding style.

… encoding

- TimeSeriesDatabaseWrapper implements AutoCloseable; both ITs use try-with-resources
  for the top-level wrappers so connections close even when an assertion fails.
- assertThatPointCountIs now tracks and surfaces the last exception (mirrors
  DatabaseWrapper.assertThatUserCountIs) instead of silently reporting -1.
- assertAggregateExtremes uses isCloseTo(within(1e-9)) and a single polling level;
  aggregate() is now a single query (returns null if no rows) so the retry no longer
  compounds.
- latestTimestamp URL-encodes the tag query parameter.
- Standardised servers.getFirst() over get(0) in the 3-node test.
- Documented that the protocols are not throughput-comparable (line protocol batches
  500/call; SQL sends one INSERT per round-trip, kept to exercise the single-row path).
@robfrank

robfrank commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in e3dc3d0. Per item:

  • Resource leaks on assertion failure - Fixed. TimeSeriesDatabaseWrapper now implements AutoCloseable and both ITs use try-with-resources for the top-level wrappers (admin; ingest/r0/r1/r2), so connections close even when an assertion throws.
  • @Tag("slow") - Not added, deliberately. The load-tests module has zero @Tag usage: these are *IT classes already excluded from the default build (failsafe, skipITs=true by default), which is why the sibling SingleServerLoadTestIT/ThreeNodesLoadTestIT are untagged. CLAUDE.md's @Tag("slow") targets slow tests in the fast (surefire) build; these ITs are already gated, so tagging adds nothing here. (For contrast, the engine-module regression test added in this PR is @Tag("slow"), because it does run in the fast build.)
  • Silent exception swallowing in assertThatPointCountIs - Fixed. It now tracks the last exception and throws an AssertionError carrying it, mirroring DatabaseWrapper.assertThatUserCountIs.
  • getFirst() vs get(0) - Fixed. Standardised on servers.getFirst() for index 0.
  • Float == - Fixed. assertAggregateExtremes now uses isCloseTo(expected, within(1e-9)).
  • URL-encode the tag - Fixed. latestTimestamp now URLEncoder.encode(tag, UTF_8).
  • SQL vs line-protocol batching asymmetry - Added a Javadoc note on ingestSeries that the protocols are intentionally not throughput-comparable. The per-row SQL path is kept on purpose (not batched into sqlscript): it exercises the single-row append path under concurrency, which is what the SQL cases are meant to stress.
  • Nested polling could compound to 90s - Fixed. aggregate() is now a single query (returns null when no rows are visible yet) and assertAggregateExtremes owns the single 60s polling loop, so there is no longer a retry-within-retry.
  • ResultSet not closed (minor) - Not changing. RemoteDatabase.query/command return an in-memory InternalResultSet materialised eagerly from the response; its close() only does content.clear(), so no server cursor or connection is held (and this matches the sibling DatabaseWrapper). No leak to track.

Note: the SQL-protocol cases currently fail at POINTS_PER_THREAD = 10000 on an exact-count assertion. That is a separate, confirmed server-side data-loss under highly concurrent single-row time series INSERTs (sealed-slot lost update; line protocol and single-threaded are unaffected) - under investigation separately, not introduced by these tests. A @Disabled engine regression test for it is included in this PR.

@claude

claude Bot commented Jun 1, 2026

Copy link
Copy Markdown

Code Review

This PR adds solid Testcontainers-based load tests for the time series engine. The structure is clean and follows the existing ContainersTestTemplate pattern well. A few issues worth addressing:


Correctness / Bugs

1. Interrupt handling in retry loops (TimeSeriesDatabaseWrapper)

createDatabase() and createSchema() both use the sleep(long) helper, which catches InterruptedException and restores the thread's interrupt flag but does not break the retry loop. If the calling thread is interrupted during a retry, the loop silently continues for up to maxAttempts * 2s longer. The existing DatabaseWrapper.createDatabase() is consistent about this: it explicitly breaks after restoring the flag. The new wrapper should do the same:

// createDatabase() retry sleep
try {
    Thread.sleep(2_000);
} catch (final InterruptedException ie) {
    Thread.currentThread().interrupt();
    break;
}

Same fix applies to createSchema().

2. Silent error swallowing in insertSql() and postLineProtocol()

Both methods catch any Exception, log it, increment an error counter, and return normally. If every insert for a thread fails (e.g., the server is unreachable), ingestSeries() finishes without throwing, and the subsequent assertThatPointCountIs() fails - but with a confusing count-mismatch message rather than the root cause. Consider re-throwing the exception (or at least throwing an AssertionError) after a threshold of consecutive failures, so test failures are easier to diagnose.

3. latestTimestamp() - no bounds check on the latest array

final JSONArray latest = json.getJSONArray("latest");
return latest.getLong(0);

If the server returns "latest":[] (e.g., the verification series hasn't replicated yet), this throws JSONException without any context. A guard like assertThat(latest.length()).isGreaterThan(0) before the access would give a clearer failure message.


Design / Maintainability

4. ThreeNodesTimeSeriesLoadTestIT.tearDown() override

The override body is just super.tearDown(), which is exactly what would happen without the override. The comment explains the intent ("we're not calling compareAllDatabases()") but the base class ContainersTestTemplate.tearDown() doesn't call compareAllDatabases() either - only ThreeNodesLoadTestIT does. The override and its comment are correct as documentation (mirroring the pattern in ThreeNodesLoadTestIT), but it would be clearer to add a brief note that this intentionally matches the sibling class's override.

5. Executor wait loop has no upper-bound timeout

Both load tests use while (!executor.isTerminated()) without a ceiling. If a worker thread hangs (e.g., waiting on a server that stopped responding), the test will never complete. executor.awaitTermination(10, TimeUnit.MINUTES) followed by executor.shutdownNow() and an assertion would make the hang self-diagnosing. This pattern is inherited from SingleServerLoadTestIT, so it's a pre-existing issue, but worth fixing here since the new tests have a heavier footprint.

6. assertAggregateExtremes() - manual polling vs Awaitility

The method implements its own poll loop with System.currentTimeMillis(), while ThreeNodesTimeSeriesLoadTestIT correctly uses Awaitility for convergence checking. Replacing the manual loop with:

Awaitility.await()
    .atMost(60, TimeUnit.SECONDS)
    .pollInterval(2, TimeUnit.SECONDS)
    .untilAsserted(() -> {
        final AggResult agg = aggregate(sensorId);
        assertThat(agg).isNotNull();
        assertThat(agg.min()).isCloseTo(expectedMin, within(tolerance));
        assertThat(agg.max()).isCloseTo(expectedMax, within(tolerance));
    });

would unify the polling style and provide better failure messages on timeout.

7. Magic number 4L in expected total

bulkPoints + 4L appears in both test classes. The 4L comes from ingestVerificationSeries() (four hardcoded points). Naming it:

private static final int VERIFICATION_POINTS = 4;
...
final long expectedTotal = bulkPoints + VERIFICATION_POINTS;

or sharing the constant from TimeSeriesDatabaseWrapper would make the intent clearer without relying solely on the comment.


The @Disabled concurrent-insert regression test

TimeSeriesConcurrentInsertTest.concurrentSingleRowInsertsDoNotLoseSamples() is a well-written reproduction of a real bug with clear documentation of when it should be enabled. One concern: being @Disabled, it will never run in CI and may be forgotten. Consider opening a tracking issue and linking it from the @Disabled comment (e.g., @Disabled("see issue #XXXX - ...")), so the bug and its test stay connected.


Minor

  • basicAuth() is called on every HTTP request and recomputes the Base64 encoding each time. Precomputing it once in the constructor is a trivial win.
  • checkSchema() queries count(*) just to check the type exists; a SELECT 1 FROM sensor LIMIT 1 would be cleaner and avoids a full scan on large datasets (though irrelevant for test startup).
  • countPoints() and checkSchema() don't close the ResultSet. Depending on RemoteDatabase's implementation this may be a resource leak; wrapping in try-with-resources would make it explicit.

Overall this is a well-thought-out addition that exercises realistic ingestion patterns across three protocols and two topologies. The documented semantics around sealed vs. mutable data and the rationale for polling assertions are particularly helpful. The issues above are mostly defensive coding concerns rather than correctness blockers.

@robfrank robfrank merged commit fe3a1da into main Jun 1, 2026
22 of 25 checks passed
@robfrank robfrank deleted the tests/timeseries-loadtests branch June 1, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant