DataStore

The DataStore is a file system-based storage system used to manage content during the fetch and transform process. It provides a simple API for storing and retrieving content, with support for namespaces and documents.

Core Concepts

Namespaces

Namespaces represent collections of documents from a single source. Each content source gets its own namespace, identified by the source's ID.

Documents

Documents are individual files containing content data. They can be either single JSON files or batched files (for paginated content).

API Reference

DataStore

createNamespace(namespaceId: string)

Creates a new namespace in the data store. Returns a Result containing the namespace.

namespace(namespaceId: string)

Gets an existing namespace. Returns a Result containing the namespace.

getDocument(namespaceId: string, documentId: string)

Gets a specific document from a namespace. Returns a Result containing the document.

filter(ids?: DataKeys)

Filters documents based on namespace and document IDs. Returns grouped results by namespace.

Namespace

insert<T>(id: string, data: Promise<T> | AsyncIterable<T>)

Inserts a new document into the namespace. Data can be a Promise for single documents or an AsyncIterable for batched documents.

document(id: string)

Gets a document by ID from the namespace.

documents()

Gets all documents in the namespace.

waitFor(id: string)

Returns a promise that resolves when the document with the passed ID has finished being written.

Document

update(cb: (data: T) => T | Promise<T>)

Updates document content using a callback function.

apply(pathExpression: string, fn: (x: unknown) => unknown)

Applies a transformation to specific paths in the document using JSONPath.

query(pathExpression: string)

Queries document content using JSONPath expressions.

Error Handling

The DataStore uses the neverthrow library for error handling. Most methods return a Result or ResultAsync type:

const namespaceResult = dataStore.namespace('my-source');
if (namespaceResult.isErr()) {
  console.error('Error:', namespaceResult.error);
} else {
  const namespace = namespaceResult.value;
  // Use namespace...
}

For async operations, you can use the andThen method:

await dataStore
  .createNamespace('my-source')
  .andThen(namespace => namespace.insert('doc1', data));