An agent can read everything ffprobe reports: duration, codec, bitrate, keyframe positions. That is a set of coordinates on a map with no picture of the terrain. It does not know that 0–12s is a title card, that the cut it is about to make at 1:04 lands in the middle of a spoken sentence, or which scene the user actually cares about.

The obvious fix — render every frame and describe it — is correct in principle and unworkable in practice.

Built today: structural pass, scene detection, transcription + alignment, frame extraction, and tiling. With an injected labeler (Studio's claude -p), analyze also writes one audio-visual label per scene; with no labeler it stays inference-free. Proven on our test clip.

Tiered by design: the eager pass writes a one-line label per scene. Full per-scene descriptions are not eager — they're computed lazily and cached on drill-down (L3), so analyze stays fast and you pay for depth only where you ask.

Two ffprobe passes, no video decode. The first reads container and stream metadata; the second enumerates keyframe timestamps. The most agent-useful output is edit_constraints.safe_cut_points — the timestamps you can cut at with -c copy and no quality-degrading re-encode.

• codec, resolution, fps, duration from -show_format -show_streams
• keyframes via -skip_frame nokey -show_entries frame=pts_time
• nearest_safe_cut bridges visual understanding to a re-encode-free edit

// vision.json — structural section
{
  "structural": {
    "codec": "h264", "width": 1280, "height": 720,
    "fps": 30.0, "duration_seconds": 12.0,
    "nb_frames": 360
  },
  "edit_constraints": {
    "safe_cut_points": [0.0, 8.333]
  }
}

real output — test_out/vision.json

real artifact · a random ~2-min clip pulled from YouTube · 38 scenes → 3 sheets · this is sheet 1, scenes 01–16. scene-id top-left, timecode bottom-left, burned per cell.

AdaptiveDetector (threshold 15) finds scene boundaries; the midpoint frame of each scene is extracted, scaled with Lanczos, and tiled into one image. Each cell carries its scene ID so the VLM can reference cells by name, and its timecode so the agent can map a description straight to an edit point.

• scene detection + tiling — built in temporal.py
• burned overlays — id + timecode on every cell, so the VLM can cite cells by name
• eager labels — one audio-visual label per scene at analyze (injected labeler); deep descriptions stay lazy in L3

The grid is descended from IG-VLM, but scene-aware rather than uniform: cells land on real visual changes, not arbitrary time slices.

Sheets are sequence-named with their timecode range, so lexical sort matches temporal order and an agent can pick a sheet without opening a file.

live in Studio — the resolution slider + a min-scene-length control regenerate sheets on the fly

Understanding is tiered: every scene already carries a cheap one-line label from analyze (L2). Layer 3 is the lazy deep tier — and describe always runs on a zoom, not one frame, because a single midpoint frame breaks down two ways: cells get downscaled until fine detail is illegible, and one frame can't show what changes across a multi-second scene. It also gets the scene's transcript (§Acoustic), and the result is cached.

The temporal zoom is Layer 2's tiling applied recursively, on a known window, so it needs no scene detection. Take the scene, pad it by a neighbor each side, uniformly sample N frames across that window, tile them at higher resolution. Fewer cells → more pixels each; several frames → motion recovered; neighbor padding catches mislabeled boundaries.

# agent-callable — prints a path to read
lookingglass zoom --input vision.json \
  --scene scene-12 --context 1 --frames 9

• autonomous — the agent runs it itself when a scene is too ambiguous, reads the sharper sub-sheet, continues
• or manual — a human triggers the same drill-down
• cached by scene + window + frames + resolution — repeat zooms are free

real zoom · the test clip, scene-23 ±1 neighbor · 9 frames, 00:01:07→00:01:14 · uniform sampling, no detection. f5 is a hard cut to a road shot — exactly the within-scene change the single macro frame misses. Built end to end: vision.zoom + CLI + Studio.

Grounding DINO does open-vocabulary detection from a natural-language query; SAM 2 returns tracked masks. Together they turn "cut just after the user clicks Save" from a guess into a grounded edit.

The cardinal rule: annotations are metadata, not duplicated images. Detections are cached as a .objects.json sidecar beside the clean sheet; the overlay is rasterized on demand only when an annotated view is requested. Storage stays compact and the clean sheet stays canonical.

• partial-cache hits: a query that's a subset of prior detections reuses them
• masks memoized per canonical visual state — computed once, reused everywhere

sidecar holds geometry · overlay drawn on the clean sheet only when asked

The discipline, then: small by default, deep on demand, never compute the same thing twice.

LookingGlass behaves like a finder over a video — build a map once, then every later request is a search or a drill-down. The data layer is foundational: it is stood up before any layer feature code, because every layer writes through it.

A perceptual hash of each representative frame, clustered by Hamming distance, folds near-duplicate frames into one canonical visual state. A login screen revisited four times is one state, embedded once.

• SigLIP 2 (so400m / base), OpenCLIP fallback — Apache/MIT, commercial-safe
• not JinaCLIP v2 — CC-BY-NC forbids in-house production use
• the map is the bounded index; depth records live in the cache

Finder

A "what's here / find X" query embeds the text into the same SigLIP 2 space and brute-force cosine-ranks the state vectors in NumPy. Below ~10k vectors a vector database is unnecessary. The only model call is the text embed, and that is cached by normalized query string.

It returns matching scenes, timecodes, the nearest safe cut, a coarse label, and a confidence score.

Cache key

key = xxh3(source_video)
    : canonical_state_dhash
    : op          # describe | segment
    : model_version
    [ : params ]

Because dedup maps every near-identical frame to one state, a description or mask is computed once per state and reused by every scene that shares it. The source content-hash namespaces every key, so editing the source invalidates transparently — content-addressed, not mtime.

L1 and L2 are cheap and local — FFmpeg, PySceneDetect, Pillow, NumPy. That is the entire current requirements.txt, and it runs anywhere FFmpeg does.

The map's SigLIP 2 embeddings and especially L4 Grounded SAM 2 are different: PyTorch, optional CUDA, ~600 MB of weights, and 30–60s per clip on CPU. Requiring all of that just to tile a sheet would defeat the lightweight core.

So the heavy layers run through a swappable compute backend. The boundary is the detect / segment / embed contract, not the CLI. An agent that never asks for L4 never installs PyTorch; a host with no GPU still reaches L4 remotely.

compute.backend in config · infra detail in PRD §8.5

Everything an agent needs to understand a clip before issuing a command, in one file: the structural profile, safe_cut_points, the sheet manifest with time ranges, and the scene list with labels, timecodes, and the nearest safe cut for each.

• query reads it for agents — --scene-at, --safe-cuts, --sheet-at
• context --agent strips binary fields for clean VLM injection
• summary records calls made and tokens used — the cost is legible

Heading toward a tiered shape: a compact index plus addressable detail records, so loading the map never re-pays for the depth.

{
  "edit_constraints": { "safe_cut_points": […] },
  "context_sheets": [
    { "id":"sheet-001", "time_range":{…},
      "scenes_included":["scene-01",…] }
  ],
  "scenes": [
    { "id":"scene-04", "label":"New Test Form",
      "start_timecode":"00:00:14.000",
      "nearest_safe_cut":12.012 }
  ],
  "summary": { "analysis_cost":{ "estimated_tokens_used":14800 } }
}

# the agent reasons over vision.json
"Remove the intro (scene-01), cut straight to
 the walkthrough (scene-03). Nearest safe cut
 to scene-03 start is 12.012s — stream-copy,
 no re-encode."

# then emits the command
ffmpeg -ss 12.012 -i demo.mp4 -c copy out.mp4

scene IDs and safe cuts turn understanding into a re-encode-free edit

Every operation is a structured command an agent can emit:

• analyze — L1 + L2, write the map built
• zoom --scene scene-12 — temporal drill-down built
• query --scene-at 00:01:04 — which scene? partial
• annotate --queries "save button" — L4 designed

Built today: analyze, zoom, and query with --safe-cuts, --scenes, --scene-at. Studio drives all of it through claude -p.

For the common case — 1–3 sheets plus a few targeted descriptions — total inference is single-digit pennies per video. The budget is sized to the strictest image limit (Opus); the default model stays Sonnet for cost.

Originally visual-only — but on real footage the dialogue carries half the meaning. When a clip has an audio stream, LookingGlass transcribes it once with faster-whisper (an opt-in extra, lazy-loaded), then aligns each transcript segment to a scene by timecode overlap. Cheap, because the temporal map already exists.

It's not a deeper tier — it's a second input track that enriches the map. Each scene gains a transcript, which feeds the eager labels and the deep describe, so the VLM knows what a scene is about, not only what it looks like. Spoken content becomes searchable too.

• only when present — ffprobe gates it; silent clips skip it cleanly
• core stays light — Whisper is an opt-in [audio] extra, never imported otherwise
• alignment is free — a timecode overlap, no extra model