Datasets
Each started Job creates one folder-backed SQLite dataset artifact. The dataset is both DataCat’s durable local output and a file you can inspect with standard SQLite-compatible tools.
Artifact layout
The dataset folder contains the main .db database and can contain SQLite -wal and -shm companion files. These files are one working artifact, not unrelated exports.
Keep the folder intact while a Job is active. The main database, WAL, and shared-memory companion files participate in one working SQLite artifact. Do not make a file-level copy of only the .db file while the Job is active.
Tables
Every schema 8 dataset contains three base tables. Indicator-related tables appear only when the Job uses at least one Indicator Instance.
| Table | Purpose |
|---|---|
meta |
Dataset identity, lineage, versions, Job configuration, coverage, and quality summary |
bars |
Canonical OHLCV periods and provider-data status |
gaps |
Missing-period, finality, retry, and resolution history |
indicator_definitions |
Self-describing mapping from Indicator Instances and outputs to persisted columns |
indicators / indicators_1..N |
Wide numeric indicator outputs and their effective state metadata |
Raw values and calculation-only normalization
DataCat does not write synthetic OHLCV into bars to hide missing provider input. Indicator calculations can use an internal calculation-only normalization for certain missing or pending periods when a previous usable close exists, but raw provider cells remain NULL and output states disclose the dependency.
This distinction lets you choose whether provisional or derived outputs are acceptable for a particular analysis.
Identity and lineage
Job ID and Dataset ID are independent canonical UUIDs. ShortID and display names help people orient themselves but are not safe join keys.
A continued dataset records its immediate parent Job and Dataset. Following parent_dataset_id repeatedly reconstructs the continuation chain.
Schema compatibility
The current public compatibility baseline is dataset schema 8. Check meta.schema_version before consuming a dataset. Future schema versions can change tables or semantics and require an explicit compatibility decision.
Dataset size
Size limits apply per dataset artifact and include the main database plus WAL usage. When a dataset reaches its configured limit, DataCat stops before writing the next bar. Continue in New Job is the normal way to keep collecting into a new artifact.
Keep the working dataset read-only
You own the dataset file, but DataCat manages the schema and write lifecycle of the working artifact attached to a Job. Treat that working dataset as read-only from external tools. Do not insert, update, or delete rows; create or drop database objects; change persistence-related PRAGMAs; or run schema migrations. External writes can break schema and metadata invariants and prevent DataCat from continuing or reconciling the Job safely.
You can read an active Dataset and write derived tables, indexes, edits, or exports to another file or database without stopping the Job. Keep the DataCat-managed source connection read-only and direct every write to your separate destination.
If you need an independent snapshot, use a SQLite-aware online backup or export while the Job continues, or stop the Job and copy the complete dataset folder. A normal file copy of only the active .db file is not a safe snapshot because committed data can still reside in the WAL. See the SQLite Online Backup API for the live-backup model.
Reading from external tools
Use read-only access when inspecting a dataset that DataCat is actively writing.
SQLite does not guarantee row order without ORDER BY. Always specify an ordering for reproducible exports, especially for bars, gaps, and indicator_definitions.
Continue with Dataset Reference for practical queries.