Correlation Workflow

Stormlog correlation is a derived investigation view over existing artifacts. It does not change TelemetryEvent v3; instead, the query layer projects telemetry, markers, rollups, OOM bundles, diagnose manifests, and external attachments into one evidence list around an investigation anchor.

Minimum Contract

A trustworthy correlation row needs enough identity, time, and provenance to let users understand why it appeared:

session_id: the strongest single-run grouping key
job_id: the distributed-run grouping key when ranks write separate sessions
rank and world_size: rank-local evidence placement
start_ns and end_ns: Unix epoch nanosecond bounds for point or interval evidence
source_path and source_kind: where the evidence came from
metadata: source-specific details such as event type, rollup counters, or attachment attributes
confidence and reasons: a plain explanation of the match quality

Correlation uses both identifiers and timestamps. Timestamp-only matches are kept as low-confidence fallbacks because they can be useful during partial recovery, but Stormlog labels them clearly instead of presenting them as proven links.

Single-Run And Distributed Correlation

For a single run, session_id is the primary pivot. Evidence with the same session and an overlapping or nearby timestamp window is high confidence. When a rank is supplied, same-rank evidence is preferred; cross-rank evidence in the same session is still shown with a lower confidence reason.

For distributed investigations, job_id links evidence across sessions and ranks. Stormlog keeps the same Unix epoch nanosecond clock domain used by telemetry records and reports cross-rank evidence when it overlaps the anchor window. If evidence lacks session_id and job_id, it can only match by time and is downgraded.

Stormlog does not currently rewrite cross-host clocks. Producers should keep host clocks synchronized; consumers should inspect observed_timestamp_ns, source paths, and confidence reasons when debugging rank-to-rank timing.

Correlation Versus Listing

Artifact listing answers “what exists?” Correlation answers “what evidence is near this suspicious point, and why is it related?” A correlated result is anchored by --at-ns or a projected telemetry record_id, filters evidence by identity and time, then sorts by confidence and distance from the anchor.

This makes correlation suitable for pivots such as:

memory spikes to nearby phase markers
degraded collector periods to alert rows
OOM events to dump bundles and diagnose manifests
rollup windows to nearby raw telemetry
telemetry windows to external profiler or experiment-tracking links

External Attachments

External attachments are discovered from stormlog_attachments.json sidecars. The sidecar is local JSON; Stormlog records URLs or local paths but does not fetch remote content during correlation.

Minimal shape:

{
  "schema_version": 1,
  "format": "stormlog.attachments",
  "attachments": [
    {
      "attachment_id": "wandb-run-123",
      "title": "W&B run",
      "kind": "experiment",
      "url": "https://wandb.ai/example/project/runs/run-123",
      "session_id": "session-123",
      "job_id": "job-42",
      "rank": 0,
      "start_ns": 1700000000000000000,
      "end_ns": 1700000060000000000,
      "updated_at_utc": "2026-06-14T21:00:00Z",
      "metadata": {"owner": "training"}
    }
  ]
}

Relative path values resolve against the sidecar directory. Each attachment must provide either url or path, and should provide identifiers and time bounds whenever possible. attachment_id is optional but recommended for sidecars that are updated over time because it gives correlation output a stable evidence identity.