Skip to main content

Streams Reference (Website Summary)

CDC streams are how MonoTS performs reliable data synchronization from local tables to Kafka, Delta Lake, or the filesystem. Full detail lives in docs/streams.md.

Management SQL​

CREATE STREAM [IF NOT EXISTS] <name> WITH (
  'key' = 'value'
);

SHOW STREAMS;
SHOW STREAM <name>;
SHOW STREAM STATUS FOR <name>;

DROP STREAM <name>;
DROP STREAM <name> WITH CHECKPOINT;

Global properties​

PropertyRequiredDescription
sink.typeYesfilesystem, delta, or kafka
source.tableYesOne table per stream
cdc.modeNobatch or hybrid
cdc.auto_endNoEnd after historical export (true / false)

Sinks​

Apache Kafka​

  • Default cdc.mode: hybrid
  • Format: JSON only
  • Keys: sink.kafka.brokers, sink.kafka.topic
  • Best for real-time row sync into message queues

Delta Lake​

  • Default cdc.mode: batch
  • Keys: sink.delta.path, optional sink.delta.endpoint (S3-compatible)
  • Credentials via AWS_* env vars for object storage
  • Best for analytical Data Lake landing zones

Filesystem​

  • Default cdc.mode: batch
  • Key: sink.filesystem.path
  • Best for local or NFS Parquet export

Guarantees​

Server-driven push, at-least-once, automatic checkpoints. Downstream should handle duplicates.