RocksDB

RocksDB is often used as the data store for XTDB’s query indices, but can also be used as a transaction log and/or document store in single node clusters.

Project Dependency

In order to use RocksDB within XTDB, you must first add RocksDB as a project dependency:

  • deps.edn

  • pom.xml

com.xtdb/xtdb-rocksdb {:mvn/version "1.23.0"}
<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-rocksdb</artifactId>
    <version>1.23.0</version>
</dependency>

If you’re using RocksDB and seeing out-of-memory issues, we recommend setting the environment variable MALLOC_ARENA_MAX=2 - see this issue for more details. You will likely also want to configure the JVM parameter MaxDirectMemorySize (e.g. -XX:MaxDirectMemorySize=1024m), because the default value varies by JVM and could be unbounded (e.g. in the case of openjdk-11, confirmed using -XX:+PrintFinalFlags). Ensure that the sum of MaxDirectMemorySize and Xmx memory is available from the system, and be aware that consumption of direct memory may be shared with any other processes running on the system.

Using RocksDB

Replace the implementation of the desired component with xtdb.rocksdb/->kv-store

  • JSON

  • Clojure

  • EDN

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "db-dir": "/tmp/rocksdb"
    }
  },

  "xtdb/document-store": { ... },
  "xtdb/tx-log": { ... }
}
{:xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :db-dir (io/file "/tmp/rocksdb")}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}
{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :db-dir "/tmp/rocksdb"}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

It is generally advised to use independent RocksDB instances for each component, although using a single instance for the transaction log and document store is possible. Do not share the RocksDB instance used for the index store with other components as you cannot then perform XTDB version upgrades.

Dependencies

Parameters

  • db-dir (required, string/File/Path): path to RocksDB data directory

  • sync? (boolean, default false): sync to disk after every write

  • disable-wal? (boolean): disables the write-ahead log

  • db-options (RocksDB Options object): extra options to pass directly to RocksDB.

Monitoring RocksDB

To include RocksDB metrics in monitoring, override the metrics dependency:

  • JSON

  • Clojure

  • EDN

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "metrics": {
        "xtdb/module": "xtdb.rocksdb.metrics/->metrics"
      }
      ...
    }
  },

  "xtdb/document-store": { ... },
  "xtdb/tx-log": { ... }
}
{:xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :metrics {:xtdb/module 'xtdb.rocksdb.metrics/->metrics}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}
{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :metrics {:xtdb/module xtdb.rocksdb.metrics/->metrics}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

Parameters

  • instance-name (string, default "rocksdb"): unique name for this instance of RocksDB, used in metrics domains

  • sample-window (duration, default 3s): sample window of statistics collector

Configuring the Block Cache

To configure the block cache used by the RocksDB instance, override the block-cache dependency. In the example below, there is a single shared cache between multiple kv-stores:

  • JSON

  • Clojure

  • EDN

{
  "xtdb.rocksdb/block-cache": {
    "xtdb/module": "xtdb.rocksdb/>lru-block-cache",
    "cache-size":536870912
  },
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
      ...
    }
  },
  "xtdb/document-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
    }
  },
  "xtdb/tx-log": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
    }
  }
}
{:xtdb.rocksdb/block-cache {:xtdb/module 'xtdb.rocksdb/->lru-block-cache
			                      :cache-size (* 512 1024 1024)}
 :xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/document-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                                  :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/tx-log {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                          :block-cache :xtdb.rocksdb/block-cache}}}
{:xtdb.rocksdb/block-cache {:xtdb/module xtdb.rocksdb/->lru-block-cache
			                      :cache-size 536870912}
 :xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/document-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                                  :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/tx-log {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                          :block-cache :xtdb.rocksdb/block-cache}}}

Parameters

  • cache-size (int): Size of the cache in bytes - default size is 8Mb, although it is recommended this is set to a higher amount.