Cold Archive & Long Retention

NDJSON+gzip · Rotation · Restore

Overview

JetStream is a great hot buffer and transport, but long retention is a storage problem. The cold archive keeps retention cheap: write append-only NDJSON, rotate, gzip.

You can restore archived events back into the engine for re-processing. Details are in Docs.

Archive Format

Each line is a JSON object representing a canonical SecurityEvent. This means files are stream-friendly and easy to query with standard tooling.

Example: NDJSON
{"timestamp":"...","type":"auth_failure","details":{...}}
{"timestamp":"...","type":"http_request","details":{...}}

Rotation & Compression

Rotation is controlled by byte size and/or time interval, then optionally compressed. This avoids huge files and supports predictable storage usage.

Config snippet
archive:
  enabled: true
  dir: "./data/archive"
  rotate_bytes: 104857600
  rotate_interval: "1h"
  compress: true

Sampling

High-volume, low-severity events can be sampled while keeping all Medium+ or threat events. Sampling rules apply at archive time so the hot pipeline remains unaffected.

Restore

Restore replays archived events back into the engine, which means new detectors can be evaluated against historical data.

CLI restore
# Restore a date range
1sec archive restore --from 2026-02-01 --to 2026-02-15

# Restore by event types
1sec archive restore --from 2026-02-20 --types auth_failure,login_failure

Examples

Inspect archive status
GET /api/v1/archive/status
List files
1sec archive ls --dir ./data/archive