Datalog Transactions

Transactions are how you insert and modify data within XTDB.

Transactions are atomic, and comprised of a sequence of operations to be performed.

If the transaction contains pre-conditions, all pre-conditions must pass, or the entire transaction is aborted. This processing happens at each node during indexing, and not when submitting the transaction.

Java
Clojure

A transaction is performed by calling .submitTx on an IXtdbSubmitAPI with a Transaction object.

The Transaction object can be created by either using Transaction.Builder directly or using a Consumer.

If using the Consumer approach, we recommend importing xtdb.api.tx.Transaction.buildTx statically for brevity.

import static xtdb.api.tx.Transaction.buildTx;

Transaction.builder()
        .put(document)
        .build();

buildTx(tx -> {
    tx.put(document);
});

// To run the transaction:
node.submitTx(transaction);

// To run a transaction directly:
node.submitTx(buildTx(tx -> {
    tx.put(document);
}));

A transaction is performed by calling xtdb.api/submit-tx on a node with a list of transaction operations.

(xt/submit-tx node
                [
                ;; Operations
                ])

Operations

There are five transaction (write) operations:

Table 1. Write Operations
Operation	Purpose	Pre-condition?
Put	Write a version of a document
Delete	Deletes a specific document
Match	Check the document state against the given document	✓
Evict	Evicts a document entirely, including all historical versions
Function	Runs a transaction function	✓

Java
Clojure

You can add individual operations to the Transaction.Builder instance with their respective methods.

Transaction Operations are vectors which have their associated keyword as their first value.

Put

Puts a Document into XTDB. If a document already exists with the same id, a new version of this document will be created at the supplied valid time. See Valid Time for more details.

Java
Clojure

node.submitTx(buildTx(tx -> {
    tx.put(document1); (1)
    tx.put(document2, validTime1); (2)
    tx.put(document3, validTime2, endValidTime); (3)
}));

1	Putting a document as of now.
2	Putting a document with a specific valid time
3	Putting a document with a valid time and end valid time

(xt/submit-tx node [[::xt/put
                     {:xt/id :dbpedia.resource/Pablo-Picasso :first-name :Pablo} (1)
                     #inst "2018-05-18T09:20:27.966-00:00" (2)
                     #inst "2018-05-19T08:31:15.966-00:00"]] ) (3)

1	Document to add
2	(optional) `valid time`
3	(optional) `end valid time`

Delete

Deletes a Document . See Valid Time for details on how this interacts with multiple versions of the document.

Java
Clojure

node.submitTx(buildTx(tx -> {
    tx.delete(documentId1); (1)
    tx.delete(documentId2, validTime1); (2)
    tx.delete(documentId3, validTime2, endValidTime); (3)
}));

1	Deletes as of now
2	Deleting with a specific valid time
3	Deleting with a valid time and end valid time

(xt/submit-tx node [[::xt/delete
                     :dbpedia.resource/Pablo-Picasso  (1)
                     #inst "2018-05-18T09:20:27.966-00:00" (2)
                     #inst "2018-05-19T08:31:15.966-00:00"]]) (3)

1	Document ID to be deleted
2	(optional) `valid time`
3	(optional) `end valid time`

Match

Match checks the state of an entity - if the entity doesn’t match the provided document, the transaction will not continue.

Use the hasTxCommitted (tx-committed?) API to check whether the transaction was successfully committed or not due to a failed match operation.

Java
Clojure

node.submitTx(buildTx(tx -> {
    tx.match(document1); (1)
    tx.match(document2, validTime1); (2)

    tx.matchNotExists(documentId1); (3)
    tx.matchNotExists(documentId2, validTime2); (4)

    tx.put(document3); (5)
}));

1	Passes if `document1` is exactly present now
2	Passes if `document2` is exactly present at `validTime1`
3	Passes if there is no document with the id `documentId1` present now
4	Passes if there is no document with the id `documentId2` present at `validTime2`
5	Operation(s) to apply if all preconditions are met

(xt/submit-tx node [[::xt/match
                     :dbpedia.resource/Pablo-Picasso (1)
                     {:xt/id :dbpedia.resource/Pablo-Picasso :first-name :Pablo} (2)
                     #inst "2018-05-18T09:21:31.846-00:00"] (3)
                    [::xt/delete :dbpedia.resource/Pablo-Picasso]]) (4)

1	ID to be matched (for an entity which may or may not exist)
2	A specific document revision (or `nil`)
3	(optional) `valid time`
4	Operation(s) to perform if the document is matched

If the document supplied is nil, the match only passes if there does not exist a document with the given ID.

Evict

Evicts a document from XTDB. Historical versions of the document will no longer be available.

Java
Clojure

node.submitTx(buildTx(tx -> {
    tx.evict(documentId);
}));

(xt/submit-tx node [[::xt/evict :dbpedia.resource/Pablo-Picasso]])

Evict is primarily used for GDPR Right to Erasure compliance.

It is important to note that Evict is the only operation which will have effects on the results returned when querying against an earlier Transaction Time.

Transaction Functions

Transaction functions are user-supplied functions, defined using Clojure, that will run on each individual node where a transaction is being ingested and processed after having been read from the transaction log. They are conceptually similar to "stored procedures" in traditional databases, except they can only return new transaction operations.

Transaction functions can be used, for example, to safely check the current database state before applying a transaction, for integrity checks, or to patch an entity.

It is critical that transaction functions are "pure" - as in they only use the information passed to them via explicit arguments. This is essential for deterministic operation. Actions like generating UUIDs or accessing the system clock would be examples of impurity that can easily lead to non-determinism and break the transactional guarantees that XTDB offers. XTDB will not attempt to analyse a transaction function for purity, and instead relies on users to take sufficient care to avoid all forms of impurity. Side-effects should also be avoided unless strictly needed (e.g. debug logging).

Anatomy

(fn [ctx eid]  (1)
  (let [db (xtdb.api/db ctx) (2)
        entity (xtdb.api/entity db eid)]
    [[::xt/put (update entity :age inc)]])) (3)

1	Transaction functions are passed a `context` parameter and any number of other parameters.
2	The `context` parameter can be used to obtain a database value using `db` or `open-db`.
3	Transaction functions should return a list of transaction operations or `false`

If a list of transaction operations is returned, these are indexed as part of the transaction.

If false is returned, or an exception is thrown, the whole transaction will roll back.

The context reflects the speculative accumulation of all transaction operations applied up to the current point in the processing.

There is a xtdb.api/indexing-tx API available which can be passed the context to return information about the current, in-flight transaction (such as the new tx-time and tx-id).

Creating / Updating

Transaction functions are created/updated by submitting a document to XTDB with the desired function.

Java
Clojure

You create a function document with XtdbDocument.createFunction.

It takes the ID for the function as well as a string consisting of the Clojure function to run.

TransactionInstant ti = node.submitTx(buildTx(tx -> {
    tx.put(XtdbDocument.createFunction("incAge",
    "(fn [ctx eid] (let [db (xtdb.api/db ctx) entity (xtdb.api/entity db eid)] [[:xtdb.api/put (update entity :age inc)]]))"));
}));

The document should use the :xt/fn key (note, not ::xt/fn).

(xt/submit-tx node [[::xt/put {:xt/id :increment-age
                               :xt/fn '(fn [ctx eid] (1)
                                          (let [db (xtdb.api/db ctx)
                                                entity (xtdb.api/entity db eid)]
                                            [[::xt/put (update entity :age inc)]]))}]])

1	Note that the function itself is quoted

Usage

When invoking a transaction function, you specify its ID and (optionally) other arguments

Java
Clojure

node.submitTx(buildTx(tx -> {
    tx.invokeFunction("incAge", "ivan");
}));

Note that the transaction function operation ::xt/fn is Clojure shorthand for :xtdb.api/fn (assuming the xt require alias is used) - be careful not to confuse this keyword with the :xt/fn keyword used inside the transaction function document.

(xt/submit-tx node [[::xt/fn
                     :increment-age (1)
                     :ivan]]) (2)

1	Function ID
2	Parameter(s)

Transaction functions may return further transaction function invocation operations, which will in turn expand recursively until there are only primitive transaction operations remaining for the given transaction.

Documents

Java
Clojure

A XtdbDocument is created with an ID that must be of a valid type.

The instance itself is immutable and plusing/minusing data yields a new instance of XtdbDocument.

Similarly to Transactions, you can use XtdbDocument.Builder directly or use the Consumer approach.

XtdbDocument.builder("pablo-picasso")
    .put("name", "Pablo")
    .put("lastName", "Picasso")
    .build();

build("pablo-picasso", doc -> {
    doc.put("name", "Pablo");
    doc.put("lastName", "Picasso");
});

// You can also chain creating new instances of the document,
// but this will be slow for larger documents.
XtdbDocument.create("pablo-picasso")
    .plus("name", "Pablo")
    .plus("lastName", "Picasso");

A document is a map from keywords to values.

{:xt/id :dbpedia.resource/Pablo-Picasso
 :name "Pablo"
 :last-name "Picasso"}

All documents must contain the :xt/id key.

Persistence of Clojure metadata is not supported (although is not currently rejected either) - we strongly recommend that all metadata should be removed from each document prior to submission to XTDB, to avoid potential equality checking issues (see here for more details).

For operations containing documents, the id and the document are hashed, and the operation and hash is submitted to the tx-topic in the event log. The document itself is submitted to the doc-topic, using its content hash as key. In Kafka, the doc-topic is compacted, which enables later eviction of documents.

Valid IDs

Java
Clojure

The following types of document IDs are allowed:

Table 2. Valid ID types (Java)
Type	Example
Keyword	`Keyword.intern("my-id")`
String	`"my-id"`
Long	`42L`
UUID	`UUID.randomUUID()`
URI	`URI.create("mailto:hello@xtdb.com")`
IPersistentMap	`PersistentArrayMap.EMPTY.assoc("foo", "bar")`

URLs are valid IDs for historical reasons, but discouraged due to their hashCode depending on a DNS lookup.

The following types of :xt/id are allowed:

Table 3. Valid ID types (Clojure)
Type	Example
Keyword	`:my-id` or `:dbpedia.resource/Pablo-Picasso`
String	`"my-id"`
Integers/Longs	`42`
UUID	`#uuid "6f0232d0-f3f9-4020-a75f-17b067f41203"`
URI	`(URI. "mailto:hello@xtdb.com")`
Maps	`{:this :id-field}`

URLs are valid IDs for historical reasons, but discouraged due to their hashCode depending on a DNS lookup.

Indexing

XTDB automatically indexes the top-level fields across all documents as Entity-Attribute-Value triples. Vector and Set values are indexed as cardinality-many values, i.e. producing multiple EAVs.

Transaction Time

When you submit a transaction, the current time will be the Transaction Time.

You can override the transaction time for a transaction (e.g. for importing data into XTDB from another bitemporal database) so long as:

the transaction time provided is no earlier than any other transaction currently in the system - i.e. transaction times must be increasing.
the transaction time is no later than the clock on the transaction log (e.g. Kafka) - i.e. transactions cannot be inserted into the future.

Java
Clojure

buildTx(tx -> {
  tx.put(document);
  tx.withTxTime(txTime);
});

(xt/submit-tx node
              [[::xt/put {:xt/id :foo}]]
              {::xt/tx-time #inst "2020"})

Valid Times

When an optional valid time is omitted from a transaction operation, the Transaction Time will be used as valid time.

Only Valid Time Provided

When valid time is used as a parameter for Put or Delete, it specifies the time from which the change is valid.

If there is a document, A, already present at the given valid time, the operation’s effect is valid from valid time to the end of the current validity of A.

End Valid Time

When both valid time and end valid time are provided for Put or Delete, the operation’s effect is valid from valid time (inclusive) to end valid time (exclusive).

Awaiting Transactions

After a transaction is submitted, it needs to be indexed before it is visible in XTDB DB snapshots.

The return value from submitting the transaction can be used to wait for the transaction to be indexed.

This return value holds both the ID of the transaction, and the Transaction Time

Java
Clojure

In Java, you receive a TransactionInstant from the submitTx call.

TransactionInstant ti = node.submitTx(buildTx(tx -> {
    tx.put(XtdbDocument.create("Ivan"));
}));

// This will be null because the transaction won't have been indexed yet
assertNull(node.db().entity("Ivan"));

// Here we will wait until it has been indexed
node.awaitTx(ti, Duration.ofSeconds(5));

// And now our new document will be in the DB snapshot
assertNotNull(node.db().entity("Ivan"));

In Clojure, you receive a map from submit-tx containing ::xt/tx-id and ::xt/tx-time

(let [tx (xt/submit-tx node [[::xt/put {:xt/id :ivan}]])]
  ;; The transaction won't have indexed yet so :ivan won't exist in a snapshot
  (xt/entity (xt/db node) :ivan) ;; => nil

  ;; Wait for the transaction to be indexed
  (xt/await-tx node tx)

  ;; Now :ivan will exist in a snapshot
  (xt/entity (xt/db node) :ivan)) ;; => {:xt/id :ivan}

Speculative transactions

You can submit speculative transactions to XTDB, to see what the results of your queries would be if a new transaction were to be applied. This is particularly useful for forecasting/projections or further integrity checks, without persisting the changes or affecting other users of the database.

You’ll receive a new database value, against which you can make queries and entity requests as you would any normal database value. Only you will see the effect of these transactions - they’re not submitted to the cluster, and they’re not visible to any other database value in your application.

Java
Clojure

You submit these transactions to an instance of IXtdbDatasource using withTx:

TransactionInstant ti = node.submitTx(buildTx(tx -> {
    tx.put(XtdbDocument.create("Ivan"));
}));

awaitTx(node, ti);

IXtdbDatasource db = node.db();

assertNotNull(db.entity("Ivan"));
assertNull(db.entity("Petr"));

IXtdbDatasource speculativeDb = db.withTx(buildTx(tx -> {
    tx.put(XtdbDocument.create("Petr"));
}));

// Petr is in our speculative db
assertNotNull(speculativeDb.entity("Ivan"));
assertNotNull(speculativeDb.entity("Petr"));

// We haven't impacted our original db
assertNotNull(db.entity("Ivan"));
assertNull(db.entity("Petr"));

// Nor have we impacted our node
assertNotNull(node.db().entity("Ivan"));
assertNull(node.db().entity("Petr"));

You submit these transactions to a database value using with-tx:

(let [real-tx (xt/submit-tx node [[::xt/put {:xt/id :ivan, :name "Ivan"}]])
      _ (xt/await-tx node real-tx)
      all-names '{:find [?name], :where [[?e :name ?name]]}
      db (xt/db node)]

  (xt/q db all-names) ; => #{["Ivan"]}
  (let [speculative-db (xt/with-tx db
                         [[::xt/put {:xt/id :petr, :name "Petr"}]])]
    (xt/q speculative-db all-names) ; => #{["Petr"] ["Ivan"]}
    )

  ;; we haven't impacted the original db value, nor the node
  (xt/q db all-names) ; => #{["Ivan"]}
  (xt/q (xt/db node) all-names) ; => #{["Ivan"]}
  )

The entities submitted by the speculative Put take their valid time (if not explicitly specified) from the valid time of the db from which they were forked.

Dropping the database

XTDB does not currently provide any API to perform a "drop" operation (physical deletion) of the entire database. To drop the database, you must first manually call the close method on the node(s), and then delete both the index-store directory and the relevant data (or directories) for the backend modules used for the transaction log and document store components (as per your configuration).

For testing purposes you can use Files/createTempDirectory or equivalent.