Google Cloud Storage

You can use Google’s Cloud Storage (GCS) as either:

  • XTDB’s 'document store'

  • An implementation of the XTDB query index checkpoint store.

Documents are serialized via Nippy.

In both cases, you will require a Google Cloud Project with a bucket to store the files in. See the "Create Buckets" documentation from the Google Cloud docs.

Authentication

Authentication for both the document store and checkpoint store components within the module is handled via Google’s "Application Default Credentials" - see the relevant documentation to get set up.

Whichever method of Authentication you use, ensure that they at least have the permissions to:

  • Read objects from a bucket

  • Write objects to a bucket

If using the checkpoint store, you will need the following permissions in addition to the above:

  • List objects from a bucket

  • Delete objects from a bucket

Project Dependency

In order to use GCS within XTDB, you must first add this module as a project dependency:

  • deps.edn

  • pom.xml

com.xtdb/xtdb-google-cloud-storage {:mvn/version "1.24.3"}
<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-google-cloud-storage</artifactId>
    <version>1.24.3</version>
</dependency>

Using GCS as a document store

Replace the implementation of the document store with xtdb.google.cloud-storage/->document-store

  • JSON

  • Clojure

  • EDN

{
  "xtdb/document-store": {
    "xtdb/module": "xtdb.google.cloud-storage/->document-store",
    "project-id": "your-project-id",
    "bucket": "your-storage-bucket",
    "prefix": "optional-containing-folder/"
  },
}
{:xtdb/document-store {:xtdb/module 'xtdb.google.cloud-storage/->document-store
                       :project-id "your-project-id",
                       :bucket "your-storage-bucket",
                       :prefix "optional-containing-folder/"}}
{:xtdb/document-store {:xtdb/module xtdb.google.cloud-storage/->document-store
                       :project-id "your-project-id",
                       :bucket "your-storage-bucket",
                       :prefix "optional-containing-folder/"}}

Parameters

  • project-id (string, required) - The name of the GCP project that the bucket is contained within

  • bucket (string, required) - The cloud storage bucket which the documents will be stored within

  • prefix (string, optional) - An optional prefix/folder name which will be used within document names (ie, could create a subfolder on a container called 'xtdb-document-store', and all of the document store files will be located under the folder)

Checkpoint store

GCS can be used as a query index checkpoint store. For more advice/configuration concerning the checkpointer itself, see Checkpointing.

;; under :xtdb/index-store -> :kv-store -> :checkpointer
;; see the Checkpointing guide for other parameters
{:checkpointer {...
                :store {:xtdb/module 'xtdb.google.cloud-storage/->checkpoint-store
                        :project-id "your-project-id",
                        :bucket "your-storage-bucket",
                        :prefix "optional-containing-folder/"}}

Parameters

  • project-id (string, required) - The name of the GCP project that the bucket is contained within

  • bucket (string, required) - The cloud storage bucket which the checkpoints will be stored within

  • prefix (string, optional) - An optional prefix/folder name which will be used within checkpoint names (ie, could create a subfolder on a container called 'xtdb-checkpoints', and all of the checkpoint files will be located under the folder)