Configuring
Modules
A XTDB node consists of a number of modules, which can be independently configured and augmented.
Once you have an in-memory XTDB node set up, you can then start to configure the various modules. Configure modules using a JSON config file, EDN config file, or programmatically:
On the command line, you can supply a JSON/EDN configuration file using -f <file>.
For a Java in-process node, the modules are configured using the supplied Configurator, a file, or a classpath resource:
import xtdb.api.IXtdb;
IXtdb xtdbNode = IXtdb.startNode(new File("resources/config.json"));
IXtdb xtdbNode = IXtdb.startNode(MyApp.class.getResource("config.json"));
IXtdb xtdbNode = IXtdb.startNode(n -> {
// ...
});For a Kotlin in-process node, the modules are configured using the supplied Configurator, a file, or a classpath resource:
import xtdb.api.IXtdb
val xtdbNode: IXtdb = IXtdb.startNode(File("config.json"))
val xtdbNode = IXtdb.startNode(MyApp::class.java.getResource("config.json"))
val xtdbNode = IXtdb.startNode { n ->
// ...
}For a Clojure in-process node, the start-node function accepts a module tree, a file, or a resource.
(require '[xtdb.api :as xt]
'[clojure.java.io :as io])
(xt/start-node (io/file "resources/config.json"))
(xt/start-node (io/resource "config.json"))
(xt/start-node {
;; Configuration Map
})Without any explicit configuration, XTDB will start an in-memory node.
At this point, you can start submitting transactions and running queries!
Storage
XTDB has three main pluggable persistence stores: the transaction log, the document store, and the query index store. All three are backed by local KV stores by default, but they can be independently configured and overridden - you might choose to host the transaction log in Kafka, the document store in AWS’s S3, and the query indices in RocksDB.
Transaction Log |
Document Store |
Index Store |
|
✓ |
|||
✓ |
|||
DLT - Corda [1] |
✓ |
✓ |
|
✓ |
|||
✓ |
✓ |
||
✓ |
✓ |
||
In-memory KV |
✓ |
✓ |
✓ |
LMDB (KV) |
✓ |
✓ |
✓ |
RocksDB (KV) |
✓ |
✓ |
✓ |
✓ |
✓ |
✓ |
For specific details and examples of how to configure each of these modules, see their individual sections.
Each module has both an underlying implementation and overridable parameters - for each module, you can choose to keep the implementation and override its parameters, or you can choose to override the implementation entirely.
To add the HTTP server module, and specify its port:
IXtdb xtdbNode = IXtdb.startNode(n -> {
n.with("xtdb.http-server/server", http -> {
http.set("port", 3000);
});
});val xtdbNode = IXtdb.startNode { n ->
n.with("xtdb.http-server/server") { http ->
http["port"] = 3000
}
}{
"xtdb.http-server/server": {
"port": 3000
}
}(xt/start-node {:xtdb.http-server/server {:port 3000}}){:xtdb.http-server/server {:port 3000}}Overriding the module implementation
To override the underlying implementation, specify the factory function of the new implementation.
For example, using S3’s xtdb.s3/->document-store factory:
IXtdb xtdbNode = IXtdb.startNode(n -> {
n.with("xtdb/document-store", docStore -> {
docStore.module("xtdb.s3/->document-store");
docStore.set("bucket", "my-bucket");
docStore.set("prefix", "my-prefix");
});
});val xtdbNode = IXtdb.startNode { n ->
n.with("xtdb/document-store") { docStore ->
docStore.module("xtdb.s3/->document-store")
docStore["bucket"] = "my-bucket"
docStore["prefix"] = "my-prefix"
}
}{
"xtdb/document-store": {
"xtdb/module": "xtdb.s3/->document-store",
"bucket": "my-bucket",
"prefix": "my-prefix"
}
}(xt/start-node {:xtdb/document-store {:xtdb/module 'xtdb.s3/->document-store
:bucket "my-bucket"
:prefix "my-prefix"}}){:xtdb/document-store {:xtdb/module xtdb.s3/->document-store
:bucket "my-bucket"
:prefix "my-prefix"}}Nested modules
Modules in XTDB form an arbitrarily nested tree - parent modules depend on child modules.
For example, the default implementations of the three main XTDB modules are KV store backed implementations - the KV transaction log, the KV document store and the KV index store.
Each of these implementations depends on being given a concrete KV store implementation - by default, an in-memory KV store.
To override the implementation and parameters of this KV store (for example, to replace it with RocksDB), we override its kv-store dependency, replacing the implementation of the nested module:
IXtdb xtdbNode = IXtdb.startNode(n -> {
n.with("xtdb/tx-log", txLog -> {
txLog.with("kv-store", kv -> {
kv.module("xtdb.rocksdb/->kv-store");
kv.set("db-dir", new File("/tmp/rocksdb"));
});
});
n.with("xtdb/document-store", docStore -> { ... });
n.with("xtdb/index-store", indexStore -> { ... });
});val xtdbNode = IXtdb.startNode{ n ->
n.with("xtdb/tx-log") { txLog ->
txLog.with("kv-store") { kv ->
kv.module("xtdb.rocksdb/->kv-store")
kv["db-dir"] = File("/tmp/rocksdb")
}
}
n.with("xtdb/document-store") { docStore -> ... }
n.with("xtdb/index-store") { indexStore -> ... }
}{
"xtdb/tx-log": {
"kv-store": {
"xtdb/module": "xtdb.rocksdb/->kv-store",
"db-dir": "/tmp/txs"
}
},
"xtdb/document-store": { ... },
"xtdb/index-store": { ... }
}(xt/start-node {:xtdb/tx-log {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
:db-dir (io/file "/tmp/txs")}}
:xtdb/document-store { }
:xtdb/index-store { }
}){:xtdb/tx-log {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
:db-dir "/tmp/txs"}}
:xtdb/document-store {...}
:xtdb/index-store {...}}|
The tx-log and document-store are considered 'golden stores'. The query indices can, should you wish to, be thrown away and rebuilt from these golden stores. Ensure that you either persist both or neither of these golden stores. If not, XTDB will work fine until you restart the node, at which point some will evaporate, but others will remain. XTDB tends to get rather confused in this situation! Likewise, if you persist the query indices, you’ll need to persist both the golden stores. |
Sharing modules - references
When two modules depend on a similar type of module, by default, they get an instance each. For example, if we were to write the following, the transaction log and the document store would get their own RocksDB instance:
{
"xtdb/tx-log": {
"kv-store": {
"xtdb/module": "xtdb.rocksdb/->kv-store",
"db-dir": "/tmp/txs"
}
},
"xtdb/document-store": {
"kv-store": {
"xtdb/module": "xtdb.rocksdb/->kv-store",
"db-dir": "/tmp/docs"
}
}
}We can store both the transaction log and the document store in the same KV store, to save ourselves some hassle. We specify a new top-level module, and then refer to it by name where required:
IXtdb xtdbNode = IXtdb.startNode(n -> {
n.with("my-rocksdb", rocks -> {
rocks.module("xtdb.rocksdb/->kv-store");
rocks.set("db-dir", new File("/tmp/rocksdb"));
});
n.with("xtdb/document-store", docStore -> {
docStore.with("kv-store", "my-rocksdb");
});
n.with("xtdb/tx-log", txLog -> {
txLog.with("kv-store", "my-rocksdb");
});
});val xtdbNode = IXtdb.startNode { n ->
n.with("my-rocksdb") { rocks ->
rocks.module("xtdb.rocksdb/->kv-store")
rocks["db-dir"] = File("/tmp/rocksdb")
}
n.with("xtdb/document-store") { docStore ->
docStore["kv-store"] = "my-rocksdb"
}
n.with("xtdb/tx-log") { txLog ->
txLog["kv-store"] = "my-rocksdb"
}
}{
"my-rocksdb": {
"xtdb/module": "xtdb.rocksdb/->kv-store",
"db-dir": "/tmp/txs"
},
"xtdb/tx-log": {
"kv-store": "my-rocksdb"
},
"xtdb/document-store": {
"kv-store": "my-rocksdb"
}
}(xt/start-node {:my-rocksdb {:xtdb/module 'xtdb.rocksdb/->kv-store
:db-dir (io/file "/tmp/rocksdb")}
:xtdb/tx-log {:kv-store :my-rocksdb}
:xtdb/document-store {:kv-store :my-rocksdb}}){:my-rocksdb {:xtdb/module xtdb.rocksdb/->kv-store
:db-dir "/tmp/rocksdb"}
:xtdb/tx-log {:kv-store :my-rocksdb}
:xtdb/document-store {:kv-store :my-rocksdb}}Query Engine
There are a number of different defaults that can be overridden within the XTDB query engine.
{
"xtdb/query-engine": {
"fn-allow-list": ["foo/bar"]
}
}{:xtdb/query-engine {:fn-allow-list '[foo/bar]}}{:xtdb/query-engine {:fn-allow-list [foo/bar]}Parameters
-
entity-cache-size(int, default 32768): query entity cache size (number of entries, not bytes). -
query-timeout(int, default 30000): query timeout in milliseconds. -
batch-size(int, default 100): batch size of results. -
fn-allow-list(Predicate Allowlist, default nil): list of allowed namespaces/functions in predicate functions.
Predicate Allowlist
By default, users can invoke any predicate function in their queries.
To restrict what functions a user can invoke, you can specify a :fn-allow-list of safe functions/namespaces.
We include a set of pure functions from clojure.core, for convenience.
Allowlists can be a list of the following:
-
Unqualified symbols/strings, for example,
clojure.set. These are assumed to be namespaces, and all functions within that namespace are allowed. -
Fully qualified symbols/strings, for example,
clojure.string/capitalize. These are assumed to be a permitted function.
Upgrading
Break Versioning
XTDB follows a Break Versioning scheme.
This versioning emphasizes the maximum amount of impact an XTDB version update could have on users, and recognises that there are only two types of version bumps: those that are safe, and those that require you to carefully read the changelog.
It is intented to be comfortable to follow strictly and be reliable in practice.
<major>.<minor>.<non-breaking>[-<optional-qualifier>]: ------------------------------------------------------ <major> - Major breaking changes [or discretionary "major non-breaking changes"] <minor> - Minor breaking changes [or discretionary "minor non-breaking changes"] <non-breaking> - Strictly no breaking changes <optional-qualifier> - Tag-style qualifiers: -alpha1, -RC2, etc.
Index Rebuilding
Upgrading a node from XTDB version 1.X.n to 1.Y.n (a minor bump) often requires rebuilding the node’s local KV index-store from the golden stores, i.e. the transaction log and the document store.
However, it isn’t always the case that minor bumps require an index rebuild and this distinction is made clear in the release notes for a given version.
You can perform the rebuild process by simply shutting down the node and removing the index-store’s db-dir (and similarly for Lucene’s db-dir indexes where xtdb-lucene is enabled).
XTDB will then rebuild the indices when the node is restarted.
During development and other manual interactions, it is strongly suggested to rename or move the index directories rather than delete them, in case you need to rollback.
Writing Custom Modules
XTDB modules are (currently) vanilla 1-arg Clojure functions with some optional metadata to specify dependencies and arguments.
By convention, these are named ->your-component, to signify that it’s returning an instance of your component.
If the value returned implements AutoCloseable/Closeable, the module will be closed when the XTDB node is stopped.
The most basic component would be just a Clojure function, returning the started module:
(defn ->server [opts]
;; start your server
)You can specify arguments using the :xtdb.system/args metadata key - this example declares a required :port option, checked against the given spec, defaulting to 3000:
(require '[xtdb.system :as sys])
(defn ->server {::sys/args {:port {:spec ::sys/int
:doc "Port to start the server on"
:required? true
:default 3000}}}
[{:keys [port] :as options}]
;; start your server
)You can specify dependencies using :xtdb.system/deps - a map of the dependency key to its options.
The options takes the same form as the end-user options - you can specify :xtdb/module for the default implementation, as well as any parameters.
The started dependencies are passed to you as part of the function’s parameter, with the args.
Bear in mind that any options you do specify can be overridden by end-users!
(defn ->server {::sys/deps {:other-module {:xtdb/module `->other-module
:param "value"}
...}}
[{:keys [other-module]}]
;; start your server
)You can also use refs - for example, to depend on the XTDB node:
(defn ->server {::sys/deps {:xtdb-node :xtdb/node}
::sys/args {:spec ::sys/int
:doc "Port to start the server on"
:required? true
:default 3000}}
[{:keys [xtdb-node] :as options}]
;; start your server
)Environment Variables
| Environment variable | JVM property | Description |
|---|---|---|
|
|
Adds |
|
|
Adds |
|
If you’re using RocksDB and seeing out-of-memory issues, we recommend setting the environment variable |
|
|
Set to use XT’s Java SHA1 implementation rather than system libraries. |
|
|
Set to disable use of libcrypto (OpenSSL) for SHA1 hashing |
|
|
Set to disable use of libgcrypt for SHA1 hashing. |