Skip to main content

MonoTS Edge-to-Cloud Streams Reference

In MonoTS, a Stream is the core mechanism for reliable edge-to-cloud data synchronization. Using standard SQL, you can push time-series CDC (Change Data Capture) from a local table to Kafka, Delta Lake, or the filesystem — without external sync middleware.

1. Synchronization guarantees​

Understand these guarantees before you configure a stream:

GuaranteeMeaning
Server-driven pushThe MonoTS process embeds the sync engine and actively pushes data to the sink
Automatic checkpointsSync offsets are recorded automatically; after outages or restarts, sync resumes from the last checkpoint
At-least-once deliveryData is not lost under network jitter, but rare duplicates may appear — design downstream consumers for idempotency

One stream captures exactly one source table.

2. Stream management SQL​

Managing streams is similar to managing tables.

CREATE STREAM​

Define and start a sync task from a local table to an external sink. Connection parameters go in the WITH clause.

CREATE STREAM [IF NOT EXISTS] <stream_name> WITH (
  'property_key' = 'property_value',
  ...
);

SHOW STREAMS / SHOW STREAM / SHOW STREAM STATUS​

-- List all streams
SHOW STREAMS;

-- Show configuration for one stream
SHOW STREAM sync_to_kafka_01;

-- Runtime status (offset, phase, errors, …)
SHOW STREAM STATUS FOR sync_to_kafka_01;

Useful status fields include:

  • phase — lifecycle state (inactive, syncingbatch, syncinglog, active, completed, failed, …)
  • batch_files_done / totals — historical backfill progress
  • acked_lsn — highest LSN acknowledged by the sink

DROP STREAM​

Dropping a stream does not delete local table data or downstream cloud data — it only stops the sync process and removes the stream definition.

DROP STREAM sync_to_kafka_01;

3. Global properties​

All sink types share these base properties:

KeyRequiredValues / notes
source.tableYesLocal table to sync (one table per stream)
sink.typeYeskafka, delta, or filesystem
cdc.modeNobatch — sealed / historical Parquet only (default for Delta & filesystem). hybrid — historical export, then live WAL tailing (default for Kafka)
cdc.auto_endNotrue / false (default). If true, the stream ends after the current historical export finishes (one-shot backup)
note

Legacy flat keys such as sink.path may still work for compatibility. Prefer prefixed keys like sink.delta.path.

4. Sink configuration and examples​

4.1 Apache Kafka​

Push edge rows as JSON into Kafka — good for low-latency streaming, pipelines, and real-time alerting.

SettingValue
Default cdc.modehybrid
FormatJSON only today
PropertyDescription
sink.kafka.brokersKafka brokers, e.g. 192.168.1.100:9092,192.168.1.101:9092
sink.kafka.topicDestination topic
CREATE STREAM kafka_metrics_sync WITH (
  'sink.type' = 'kafka',
  'source.table' = 'edge_metrics',
  'sink.kafka.brokers' = '192.168.1.100:9092,192.168.1.101:9092',
  'sink.kafka.topic' = 'edge-telemetry-live',
  'cdc.mode' = 'hybrid'
);

5. Operations notes​

FLUSH for batch modes​

In batch mode, only sealed Parquet SSTs are exported. Rows still in the memtable are not synced until flush (automatic size threshold or manual):

INSERT INTO edge_metrics (time, device_id, temperature)
VALUES (1718000000000, 'sensor-1', 21.5);

FLUSH TABLE edge_metrics;

Typical lifecycle​

SHOW STREAMS;
SHOW STREAM STATUS FOR kafka_metrics_sync;
DROP STREAM kafka_metrics_sync;