The top-level project ties all the other projects together for convenience whilst working within this repo. All of the below commands should be run in the root of the XT2 repo.

• Kanban board

• Github Actions (CI)

• Git practices in XTDB

• Developer documentation

You will need at least JDK 21 installed.

XT2 uses Gradle - a JVM build tool that supports multi-module, polyglot projects. You do not need to install Gradle to develop XT2 - there is a Gradle wrapper in the repo.

• Java classes are (re-)compiled automatically when you (re-)start a REPL or run tests

• Start a REPL with ./gradlew :clojureRepl (potentially requiring ./gradlew clean :clojureRepl)

  • -PreplPort=7888 if you need the REPL on a specific port (e.g. for Clojure MCP)

• Once you’ve connected to the REPL, in the user namespace, run:

  • (dev) to require and go to the dev namespace.

  • (go) to start up the dev node

  • (halt) to stop it

  • (reset) to stop it, reload changed namespaces, and restart it

  • if you’re using Emacs/CIDER, cider-ns-refresh will do all this for you - C-c M-n M-r, , s x in Spacemacs, , r r in Doom.

  • Conjure users can use ConjureRefresh, see the docs for bindings

  • see Integrant REPL for more details.

• You should now have a running XTDB node under dev/node - you can verify this by calling (xt/status node) (in the dev namespace).

• Most of the time, you shouldn’t need to bounce the REPL, but:

  • if you add a module, or change any of the dependencies of any of the modules, that’ll require a REPL bounce.

  • if you change any of the Java classes, that’ll require a REPL bounce

  • otherwise, (dev/reset) (or just (reset) if you’re already in the dev ns) should be sufficient.

  • Please don’t put any more side-effecting top-level code in dev namespaces - you’ll break this reload ability and make me sad.

  • Run git config core.hooksPath .githooks to add xtdb hooks to your local repo.

XTDB builds a custom minimal JRE via jlink to slim down Docker images and local task execution. All Test, JavaExec, and clojureRepl tasks automatically use this custom JRE.

If you hit missing-module errors (e.g. ClassNotFoundException for something in the JDK):

• Escape hatch: add -PfullJdk to your Gradle command — e.g. ./gradlew :clojureRepl -PfullJdk

• Fix forward: add the missing module to the jlinkModules list in build-logic/jlink/build.gradle.kts and rebuild.

XTDB nodes can be started in 'playground' mode - either through ./gradlew :run, clj -m xtdb.main playground or the standalone Docker image, or by running the playground-config ir/set-prep in the dev namespace.

In playground mode, to begin with, we only start the pgwire server, without an associated XTDB node. Then, whenever a new connection is initiated, we create a new isolated in-memory node for every distinct 'database' specified in the connection parameters.

This is particularly useful for non-JVM testing - see the /lang README for more details.

1. Add -PdebugJvm to the Gradle command - e.g. ./gradlew :clojureRepl -PdebugJvm - you should see Listening for transport dt_socket at address: 5005.

   1. Optionally, add -PnoLocalsClearing if Clojure’s locals-clearing is getting in the way of you debugging (i.e. you’re seeing a lot of null local variables in the debugger)

2. Connect the debugger - in IntelliJ, 'Run' → 'Attach to Process' (or Ctrl-Alt-5).

• Test all with ./gradlew test; ./gradlew integration-test for longer tests

• Property-based testing with ./gradlew property-test

  • Run with custom iterations: ./gradlew property-test -Piterations=500 (defaults to 100)

  • Uses test.check generators to test with randomized data including composite types

• Some tests have external dependencies which require docker-compose:

  • docker-compose up (docker-compose up <kafka> etc for individual containers),

  • ./gradlew kafka-test

  • docker-compose down

To attach YourKit:

• Install YourKit (it’s on the AUR, for Arch folks)

• ./gradlew :clojureRepl -Pyourkit

• You might also want -ParrowUnsafeMemoryAccess which turns off bounds checking.

  This assumes YourKit is installed under /opt/yourkit (as it does from the AUR) - feel free to adapt the property (or even use its value) if you have it installed elsewhere.

The monitoring/ directory contains a Docker Compose stack for local observability:

• Prometheus - metrics collection (scrapes :8081/metrics)

• Grafana - visualization with preloaded dashboards (http://localhost:3000, login admin/admin)

• Tempo - distributed tracing backend for OpenTelemetry traces

Starting it from root:

The dev node includes a tracer (disabled by default). To enable tracing, set :tracer {:enabled? true} in your node config - traces will be sent to Tempo and viewable in Grafana’s Explore view.

See ../monitoring/README.adoc for full configuration details, including how to export dashboards and scrape multiple nodes.

Metabase is included in the root docker-compose.yml for local development.

1. Start Metabase:

   docker-compose up metabase

2. Open http://localhost:3001 and complete the initial setup (create your admin username and password).

3. Add XTDB as a database connection:

   • Database type: PostgreSQL

   • Display name: XTDB (or your preference)

   • Host: 172.17.0.1 (Docker bridge network, to reach your host machine)

   • Port: 5432 (or whichever port your dev node’s pgwire server is on)

   • Database name: xtdb

   • Username: xtdb

   • Password: (leave blank)

4. Click "Connect database" - you should now be able to query XTDB from Metabase.

See RELEASING.adoc.

A couple of ./gradlew tools:

Checking for dependency updates

./gradlew dependencyUpdates reports available updates for all dependencies in the version catalog. Uses the ben-manes versions plugin.

Reading an Arrow file

These tools output an Arrow file in EDN format

Reading a hash trie file

• ./gradlew -q :readHashTrieFile -Pfile=<file>

• Pipe to a file: ./gradlew -q :readHashTrieFile -Pfile=<file> > output.edn

Reading a table-block file

• ./gradlew -q :readTableBlockFile -Pfile=<file>

• Pipe to a file: ./gradlew -q :readTableBlockFile -Pfile=<file> > output.edn

Reading crash logs

Process crash logs into EDN format:

1. Run: ./dev/read-crash-log-folder.sh <crash-log-dir>

2. EDN files will be output to <crash-log-dir>/edn/

Example: ./dev/read-crash-log-folder.sh /path/to/my-crash-logs outputs to /path/to/my-crash-logs/edn/

We aspire to use Clojure-inspired simplicity throughout (even when writing Kotlin). XTDB is an essentially complex codebase (cf. Out of the Tar Pit), so we must wherever possible avoid adding additional accidental complexity on top of that. This means that we deliberately choose to take longer with code changes in order to do the Right Thing™, rather than the quick thing.

Specifically, this means:

Make illegal states unrepresentable

If your types can only represent valid states, you get correctness by construction instead of by validation (cf. Yaron Minsky’s principle).

• If two fields are always set together, they belong in a single composite type.

• If a field only makes sense in some contexts, use sealed types to separate those contexts rather than making it nullable everywhere.

• If a value is always the same in a given context, the data structure is wrong for that context — change the structure rather than passing a constant.

  The corollary: don’t pass hack values. If you find yourself passing true, null, or emptyMap() to satisfy a field that doesn’t apply, that’s a sign the type is complecting two concerns.

Tidy first

We develop using tidy-first methodology — separate equivalence changes (refactorings that don’t affect runtime behaviour, changes that increase our options) from changes that advance behaviour. Even on feature branches, a tidying change gets its own commit and is often cherry-picked to main independently, so that the resulting PR is easier to review. See GIT.adoc for more on how this works in practice.

Intentional defaults

Kotlin parameter defaults are appropriate when the default is the overwhelming norm and only tests or unusual configurations would override it — e.g. HashTrie page sizes, flush timeouts.

For data structures where the caller SHOULD care about every field (wire format types, domain records), don’t use defaults — force every construction site to be explicit. Defaults hide intent; a caller can accidentally omit a field and get a value that compiles but doesn’t mean what they think.