Google Cloud Storage

You can use Google’s Cloud Storage (GCS) as XTDB’s 'document store' or 'checkpoint store'.

Documents are serialized via Nippy.

Project Dependency

In order to use GCS within XTDB, you must first add this module as a project dependency:

  • deps.edn

  • pom.xml

com.xtdb/xtdb-google-cloud-storage {:mvn/version "1.21.0"}
<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-google-cloud-storage</artifactId>
    <version>1.21.0</version>
</dependency>

Using GCS

Replace the implementation of the document store with xtdb.google.cloud-storage/->document-store

  • JSON

  • Clojure

  • EDN

{
  "xtdb/document-store": {
    "xtdb/module": "xtdb.google.cloud-storage/->document-store",
    "root-path": "gs://bucket/prefix"
  },
}
{:xtdb/document-store {:xtdb/module 'xtdb.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}
{:xtdb/document-store {:xtdb/module xtdb.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}

Follow the GCS Authentication Guide to get set up.

Parameters

  • root-path (string/Path, required): path where documents will be stored, gs://bucket/prefix

  • cache-size (int): size of in-memory document cache (number of entries, not bytes)

  • pool-size (int, default 4): size of thread-pool for GCS operations

Checkpoint store

GCS can be used as a query index checkpoint store.

Checkpoints aren’t GC’d by XTDB - we recommend you set a lifecycle policy on GCS to remove older checkpoints.

;; under :xtdb/index-store -> :kv-store -> :checkpointer
;; see the Checkpointing guide for other parameters
{:checkpointer {...
                :store {:xtdb/module 'xtdb.google.cloud-storage/->checkpoint-store
                        :path "gs://bucket/prefix"}}

Parameters

  • path (string/URI, required): URI of the form "gs://bucket/prefix"