# Sparse traffic-camera interpolation: October 2026 research

## Review: model the scene, not the frames

Independent review, October 19, 2026. Five research passes covered this repository, background modeling, low-frame-rate tracking, generative interpolation and evaluation. [Full report with all sources](interpolation-review-2026-10.md).

The research below is careful and honest. Nothing indefensible has been published. It also correctly found that a plain crossfade matches or beats every neural interpolator tested. But most of the effort went into the wrong layer of the problem. About thirteen frame-synthesis systems were tried against what were essentially two hard intervals. Meanwhile, the inputs those systems depend on went unmeasured. The repository holds ten stills per camera, the clock it uses is the wrong one, and no code detects or associates a single person or car, or places one in ground coordinates.

The background-plate idea is correct and already proven in miniature: a four-frame plate kept every background pixel exactly still. It is under-built, not wrong. But layering simplifies a different problem from the one being attacked. Once the street is known, interpolation becomes detecting, matching and moving people and cars about 22–32 px tall. That is a tracking problem, not a synthesis problem. No 2026 model this review found is built for objects that small moving farther than their own height. A walker also completes about two strides between stills, so no method can observe the gait in between. The realistic target is a deterministic, labelled reconstruction, scored against hold and crossfade baselines before any generative model is reconsidered.

### The cameras already record capture time

The project measures intervals with receipt time, which its own documents call unverified. The cameras supply better. Five of the six are Axis units that write a JPEG comment record (`0x0A 0x01`) holding Unix seconds plus hundredths ([Axis VAPIX](https://developer.axis.com/vapix/network-video/trigger-data/)). The Times Sq 45 St unit carries EXIF `SubsecTimeOriginal` instead. All 60 sample JPEGs in [frames.json](../frames.json) were decoded, and every file's SHA-256 matched the manifest.

| Camera | Receipt intervals | Embedded capture intervals | What receipt time hides |
|---|---|---|---|
| Canal × Allen, Baxter, Chrystie | 2.87–3.01 s, nearly uniform | Alternating ~2.0 and ~4.0 s | The original 3-second poller skipped every other 2-second refresh. |
| Times Sq 46 St | 1.48–2.26 s | 1.93–2.07 s | The “1.48 s” benchmark interval is 2.07 s. |
| Times Sq 45 St | 1.49–2.26 s | 1.97–2.03 s | The 0.75-second poll grid, not camera timing. |
| Herald Sq 33 St | 1.50–2.26 s | 1.76–2.30 s | A less regular source cadence. |

Averaged over all intervals, receipt time misstates the capture interval by 26%. The documented crossing pedestrian moves about 36 px in 2.07 s, not 1.48 s. That is about 17 px/s rather than 24 px/s. For an assumed 1.7 m adult it is 1.0–1.3 m/s, an ordinary walking pace ([usual pace 1.31 m/s](https://pubmed.ncbi.nlm.nih.gov/33030707/)). Some heldout claims below invert as a result; corrections are marked in place. On Canal Street, any constant-velocity path drawn on receipt time would show apparent speed alternating by a factor of two, and some motion judged “unnatural” may come from this. Axis does not document whether the stamp is the exposure or the encode time. Either way its error is a few hundredths of a second, about 1 px here, against 0.2–1.0 s for receipt time. Re-encoding strips these segments, so the archive must keep the raw bytes.

### What is overcomplicated, and what is overlooked

**Overcomplicated.**

- **Breadth across models, not data.** Nearly every trial re-tested one Times Square crossing and a few Allen pairs. On heldout frames, crossfade tied ABC-Inter (MAE 11.29 vs 11.25–11.27), and SPEED was worse than both hold and crossfade on every case ([holdout benchmarks](holdout-benchmarks.json)).
- **Large models on CPU.** LDF-VFI spent 1,005 s and 7.5 GiB to produce a crosswalk-colored ghost. A resumable CPU attention adapter was then built after the method had already been rejected.
- **Upscaling before motion.** 2816 × 1920 Allen stills (about 5.5 MB each) were published, although the pipeline says to upscale only after motion is validated ([8× review](upscale8-review.json)).

**Overlooked.**

- **Capture timestamps**, as above. The capture worker currently writes `capture_time: None`.
- **Data volume.** Each camera's ten stills cover 18–27 s at a single lighting condition. The capture worker is designed to archive about 259,200 images per day, but no capture host is connected.
- **The plate is never used.** `estimate_background` in [worker/layered_motion.py](../worker/layered_motion.py) is called only by unit tests.
- **Correspondence.** There is no detector, association step, motion model or ground homography. Every result that kept a pedestrian moving forward used a hand-supplied trajectory.
- **Photometry and effects.** Static pixels shift by up to +16.9 levels between neighbouring stills, but that correction exists only inside one experiment. Shadows are not modeled anywhere. The stills and motion pipelines also disagree on whether road lettering is protected overlay or background. The motion pipeline's rule, background, is the right one.

### Where a clean plate is easy, and where it breaks

Background initialization is a mature problem. On SBMnet scenes, robust-median, patch-selection and low-rank methods reach an average gray-level error of 1.4–3.0 on basic scenes. Under illumination change, median-style methods degrade to 15–20, while low-rank MSCL holds at 2.8; camera jitter gives 6–12 ([comparison, arXiv 2003.12931](https://arxiv.org/abs/2003.12931)). A per-pixel median also has a 50% breakdown point. Where a person or car occupies a pixel in more than half the frames, no frame count fixes it. The closest precedent is WebCamT: 212 city webcams at 352 × 240 with 1–3 s between frames, using a light-traffic reference image brightness-adjusted for other periods ([arXiv 1703.05868](https://arxiv.org/abs/1703.05868)). Night is the hardest case for unseen-scene background subtraction, with best CDnet F-measures of 0.59–0.70 ([BSUV-Net 2.0](https://arxiv.org/abs/2101.09585)). Every Times and Herald sample here was captured at 01:49.

| Scene element | Recovered by a many-frame plate? | Handling |
|---|---|---|
| Road, quiet sidewalks, facades, lane lettering | Yes, as exact real pixels | Robust median or clustering per time-of-day bin |
| Pixels occupied more than half the time | No; the median breaks down | Low-occupancy hours, semantic masking, or a crowd layer |
| Billboards, LED screens, signals | No; not stationary | Dynamic-region mask filled from the nearest real frame |
| Exposure and white-balance jumps | Partly | Per-frame gain and bias against the plate |
| Daylight drift, building shadows, night | Only within a time bin | A plate bank keyed by time of day and illumination |
| Cast shadows, headlight beams, wet-road reflections | No; they travel with movers | Assign to the moving object's layer |
| Parked cars, standing people | Absorbed into the plate | Leave them; they need no interpolation |
| HUD timestamp | Must never be estimated | Copy from the source frame |

A measured plate makes most 2025–2026 object-removal research unnecessary for these cameras. Generative Omnimatte and ROSE exist mainly to hallucinate disoccluded background in dense 480p video, and the ECCV 2026 BeyondMasks benchmark still finds systematic failures in removing shadows and reflections ([arXiv 2608.20107](https://arxiv.org/abs/2608.20107)). The plate also fixes foreground isolation, the step where the project's masks failed. Text-prompted SAM 3.1 missed the moving person in all fourteen frames, and SAM tends to treat shadows as background ([ShadowSAM](https://arxiv.org/abs/2305.16698)). Differencing against a photometrically matched plate supplies a scene-specific prior that no generic segmenter has. A detector can then gate the blobs, and box-prompted SAM 2 can refine them on 2–4× crops. Every rejected whole-frame method instead re-synthesized the roughly 97% of the frame that is street. Much of the damage reviewers saw, such as morphing stripes and eroded lettering, landed on pixels the cameras had already observed many times.

### Layering turns interpolation into tracking

Published tracking evidence stops at 1 Hz. On MOT17 subsampled to every tenth frame, ByteTrack's HOTA falls from 67.5 to 57.6, while query-based ColTrack holds at 65.5→65.8 ([arXiv 2308.05911](https://arxiv.org/abs/2308.05911)). At 1 Hz on crowded MOT20, Kalman and IoU baselines reach HOTA 15.4–37.4, while Re-ID-driven StableTrack reaches 63.5 ([arXiv 2511.20418](https://arxiv.org/abs/2511.20418)). These cameras run at 0.5 fps, twice as sparse as any published protocol, so some Times Square identities will stay genuinely ambiguous.

Geometry makes association physical. Clicking four or more ground points per camera gives a homography. Walking speed can then be gated to 0–3 m/s ([MOTChallenge](https://arxiv.org/abs/1504.01942)), and lane polylines with a direction forbid reversed cars by construction. Vehicles should be parameterized along their lane, which cut trajectory error 11% in the 2026 LoRFT benchmark ([arXiv 2607.19911](https://arxiv.org/abs/2607.19911)). Each sequence is short and processed offline, so association can be global, using min-cost flow or K-best assignment, and must allow objects to arrive or leave between frames. Confident pedestrian tracks are smoothed in metres with a constant-velocity Kalman filter and RTS smoother, using the true time steps. An object is marked unknown, and its observed frames held, if it was seen once, violates a speed or lane gate, has an occluded foot point at an endpoint, or has hypotheses that disagree on direction.

The current no-tracking stance merges two different things. Matching anonymous blobs within one camera over a 20-second window, from position, size and color, and discarding the result after rendering, is not identity recognition. Cross-camera re-identification and stored appearance embeddings are what the privacy concern rightly excludes.

### Two strides per gap make gait a labelled guess

Usual-pace walkers take about 117 steps per minute, roughly one stride per second. Over the true 2.07-s interval a walker completes about two strides, so the endpoint poses look nearly identical. Interpolators then produce frozen or backward-shuffling legs, matching the “natural gait was not established” verdicts below. Generators cannot help at this scale. A Wan2.1 token covers 16 × 16 px and an LTX token 32 × 32 px, so a 25-px pedestrian is one or two tokens tall and must be invented from the model's prior. That explains the color drift. The 2026 KeyFrame-Compass benchmark found the best open keyframe model, LTX-2.3, still “melting, dissolving, or fluid-like” under fast motion ([arXiv 2607.14202](https://arxiv.org/abs/2607.14202)). DSG-GI lists multiple moving subjects among its own failure cases.

For a 20-px figure, the visible gait is a few-pixel scissor of the lower half and about a 1-px bob. A procedural walk cycle on the existing amodal sprite captures nearly all of it. Its cadence comes from the measured ground speed, it crossfades into the observed pose over 100–200 ms near each keyframe, and its colors come only from source pixels, so they cannot drift. That removes the three recurring rejection reasons: fading, color shift and frozen gait. Vehicles are rigid at this resolution and move as their own sprites along lane paths, scaled by the homography. Go-with-the-Track (SIGGRAPH 2026, Apache-2.0) is the one open-code model that conditions on both endpoint sprites along known tracks. It is the right candidate for an optional GPU refinement pass later. Whether a procedural gait reads as natural at 22–32 px is unknown, so visual review decides it.

### Score objects, and label every synthetic frame

People cover about 3% of the frame below the HUD in Times 46 St, so erasing a person and painting a clean street can improve whole-frame MAE, SSIM or LPIPS. Distribution metrics answer the wrong question: FVD barely registers temporal corruption ([Ge et al., CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/html/Ge_On_the_Content_Bias_in_Frechet_Video_Distance_CVPR_2024_paper.html)). The project needs its own object-level suite: foreground-masked error per object, detection recall and phantom objects, counts by class, centroid error, identity persistence, and near-zero change on static regions. Methods should climb a fixed ladder (hold, crossfade, plate plus linearly moved sprites, flow, learned VFI, generative), and a mode publishes only if it beats plate plus sprites without phantom people. MOT17 subsampled 1-in-60 and MOT20 or UA-DETRAC 1-in-50, downscaled to 352 × 240 with JPEG damage, test the deployed 2-s gap against true frames using their ground-truth tracks.

Labelling obligations have tightened; this is not legal advice. EU AI Act Article 50 transparency duties have applied since August 2, 2026, and its deepfake definition covers content resembling real places and events ([European Commission](https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content)). Platforms often strip metadata, so a burned-in per-frame “SIMULATED” label and distinct timeline marks beat a mode name alone. C2PA should be attached to exports, and AI Enhanced needs a label too. NYC DOT says its cameras “do not record any footage” ([NYC DOT](https://www.nyc.gov/html/dot/html/motorist/atis.shtml)), and in 2024 it sent a cease-and-desist to a traffic-camera art project. Short retention, no long per-person tracks, attribution and documented permission are prudent before scaling capture.

### Prioritized recommendations

| When | Action | Done when |
|---|---|---|
| 01 · Now | Parse capture time in `worker/batch.py` from the Axis comment record, falling back to EXIF `SubsecTimeOriginal`. Keep raw JPEG bytes, and keep receipt time only as provenance. | All six cameras populate `capture_time`, and frames.json uses it. |
| 02 · Now | Re-capture the Canal samples with the 1-second poller, then re-run heldout tests at capture-time fractions. | Regenerated heldout reports and corrected timing claims. |
| 03 · Now | Pause new generative and VFI trials and further 8× upscaling. | Steps 04–08 exist. |
| 04 · Next | Host the capture worker and keep several days per camera across day, dusk, night and rain, under a stated retention limit. | A published curve of plate coverage against frame count, per camera. |
| 05 · Next | Build a plate bank from `estimate_background`: HUD-excluded registration, time-of-day and illumination bins, per-frame gain and bias, coverage maps, dynamic-region masks, and one road-lettering rule. | Plate error is reported on heldout low-occupancy frames. |
| 06 · Next | Isolate foreground by plate differencing, gated by a detector and refined with box-prompted SAM 2. Compute alpha from the plate, and find shadows by intensity ratio to the plate. | Detection recall is reported per camera, day and night. |
| 07 · Next | Annotate a ground homography, directed lane polylines, sidewalk polygons and static occluders for each camera. | Six calibrated cameras, checked against standing-pedestrian heights. |
| 08 · Next | Build the evaluation harness: the baseline ladder scored with object-level metrics on subsampled MOT17, MOT20 and UA-DETRAC with ground-truth tracks, split by camera and time block. | A per-method results table is published on this page. |
| 09 · Then | Associate anonymous objects offline within each camera using speed, lane, size and color costs, then smooth with Kalman and RTS or along the lane. Mark ambiguous objects unknown. | IDF1, ID switches and the share marked unknown are reported. |
| 10 · Then | Build the deterministic compositor: rigid vehicle sprites with perspective scaling, and amodal pedestrian sprites with a procedural walk cycle, locked colors, short keyframe crossfades and fades for unmatched objects. | It beats crossfade and linear sprites on object metrics and passes visual review. |
| 11 · Before release | Burn a SIMULATED label into every synthetic frame, mark recorded and synthetic spans on the timeline, label AI Enhanced, attach C2PA to exports, attribute NYC DOT, and never upscale people. | Labels are visible without interaction. |
| 12 · Optional | Only if step 10 looks too synthetic, try Go-with-the-Track, Wan-Move or ATI on upscaled crops with hosted GPUs, with gates on color statistics and outside-mask pixels. | It beats step 10 on the same metrics. |

> The likely end state is a labelled, partly diagrammatic reconstruction rather than photoreal video. Every pixel comes from an observation, every assumption is visible, and ambiguity is shown as ambiguity. If the deterministic compositor cannot beat crossfade on object-level metrics, the current “Motion withheld” default remains the correct public answer.

## What the actual footage shows

We now have executable tests rather than model-catalog recommendations. ABC-Inter and SPEED run on CPU against the original camera images. Neither has passed the crossing-person visual review. Larger input resolution alone did not fix the failure.

A particularly difficult Times Square interval contains a person moving roughly 36 native pixels into the view of another person already standing there. The visible moving-person mask changes from 146 to 83 pixels. That is partly **occlusion**, not a reason to shrink the person's complete body. Ordinary interpolation, visible-mask morphing, and unconstrained flow blending all confuse these cases.

The source observations are only 352 × 240 pixels, with small, compressed people and irregular receipt intervals. Receipt times are not verified camera exposure times. A realistic reconstruction can be a plausible hypothesis; it cannot recover the unrecorded path as fact.

## Recent primary research and availability

| Work | Date | Useful idea | What can actually run |
| --- | --- | --- | --- |
| [DSG-GI](https://arxiv.org/abs/2610.13187) | October 9, 2026 | Progressive destination conditioning and semantic/geometric alignment to reduce drift | The paper says code will be released. No executable official release was verified. |
| [ABC-Inter](https://arxiv.org/abs/2609.23408) | September 20, 2026 | Explicit quadratic Bézier motion controls, including externally supplied controls | [Official code/weights](https://github.com/SHH-Han/ABC-Inter) tested at pinned commit `ccdf96b970d5d0782c42ec25d89aa74b8364496e`. CPU ports were used only in the research workspace. |
| [MGMVFI](https://arxiv.org/abs/2608.22861) | August 24, 2026 | Motion-guided feature serialization | [Official repository](https://github.com/Jaep0805/MGMVFI) contained only a README when checked. Accepted at ECCV 2026; still README-only on October 19. |
| [SNM-VFI](https://arxiv.org/abs/2608.13460) | August 13, 2026 | Four-frame nonlinear flow, occlusion confidence, flow-guided diffusion and selective fusion | No official executable implementation was verified. The architecture is a useful reference for our own pipeline. An ECCV 2026 workshop paper; training-free on RAFT and the SVD-based Generative Inbetweening weights. |
| [SPEED](https://arxiv.org/abs/2607.15585) | July 17, 2026 | One-step synthesis directly in pixel space | [Official code](https://github.com/bbldCVer/SPEED), pinned `40fadbe85c88cc6e4015062389da464fd7e85ab9`, and [weights](https://huggingface.co/bbldCVer-hf/SPEED) execute on CPU with the existing native attention fallback. The repository has no LICENSE file, so its code cannot safely be vendored. |
| [Amodal SAM](https://arxiv.org/abs/2604.20748) | April 22, 2026 | Complete object shape should be separated from the visible, occluded mask | Relevant architectural evidence; not a model tested on our footage. |
| [FC-VFI](https://arxiv.org/abs/2603.04899) | March 5, 2026 | Fidelity conditioning and semantic motion guidance | No executable official release was verified in this search. Published at ICASSP 2026. |
| [LDF-VFI](https://arxiv.org/abs/2601.14959) | January 2026 | Joint sequence conditioning using an autoregressive diffusion transformer | [Official code](https://github.com/xypeng9903/LDF-VFI), pinned `61b34d2379df8a313e8e4cb467cc2f74c52b45d7`, and [weights](https://huggingface.co/onecat-ai/LDF-VFI). The joint CPU benchmark completed: all 17 frames were reviewed and both intervals rejected. See [reproduction and limitations](ldf-cpu-benchmark.md). A v2 of the paper was posted on March 30, 2026. |

These methods are not equivalent. SPEED's public model predicts a midpoint; a diffusion timestep is not an arbitrary video timestamp. ABC supports arbitrary interpolation time, but nonlinear controls cannot correct an already wrong endpoint correspondence. A list-of-frames API is not evidence that a model jointly reasons over the complete observed sequence.

The October 19 review found these further 2025–2026 works relevant:

| Work | Date | Useful idea | Status and fit |
| --- | --- | --- | --- |
| [KeyFrame-Compass](https://arxiv.org/abs/2607.14202) | July 2026 | Benchmark of nine keyframe-conditioned video systems on 386 samples | LTX-2.3 adhered to keyframes best but showed “melting, dissolving, or fluid-like artifacts” under fast motion, like the smears seen here. |
| [TGI](https://arxiv.org/abs/2603.17651) | March 2026 | Training-free inbetweening on Wan2.1-FLF2V-14B (CVPR 2026) | Code released without a LICENSE file; GPU-only. |
| [MASS](https://arxiv.org/abs/2606.27718) | June 2026 | Large-motion VFI along flow-guided curved trajectories (ECCV 2026) | Improves matching but still warps and blends; not aimed at objects under 40 px. |
| [Multiple-hypothesis flow](https://arxiv.org/abs/2608.07120) | August 2026 | Keeps top-K correspondences to reduce ghosting from wrong matches | Useful only to refine a match that association has already made. |
| [Go-with-the-Track](https://arxiv.org/abs/2606.20891) | June 2026 | Multi-reference, track-anchored compositing and motion (SIGGRAPH 2026) | [Apache-2.0 code](https://github.com/eyeline-labs/Go-with-the-Track); needs a GPU. The best optional refiner, since it can take both endpoint sprites. |
| [FlexComposer](https://arxiv.org/abs/2607.29627) | July 2026 | Separates an object's own motion from its global displacement | Supports the sprite-over-plate design; no runnable code was verified. |
| [StableTrack](https://arxiv.org/abs/2511.20418) | November 2025 | Re-ID-driven association evaluated at 1 Hz | HOTA 63.5 on MOT20 at 1 Hz against 15.4–37.4 for Kalman/IoU baselines; no code link found. |
| [LoRFT](https://arxiv.org/abs/2607.19911) | July 2026 | Road-aligned vehicle trajectory reconstruction from fixed cameras | Code and annotations released; 11% lower displacement error than the strongest baseline. |
| [BeyondMasks](https://arxiv.org/abs/2608.20107) | August 2026 | Object-removal benchmark including shadows and reflections (ECCV 2026) | Finds systematic failures removing secondary effects; a measured plate avoids most of this problem. |

## New experiments

### ABC-Inter

Both base and largest checkpoints ran with learned and linear controls. Four heldout predictions used the actual receipt-time fraction. The largest model took roughly 1.4 seconds per native frame on this worker.

> **Capture-clock correction:** the receipt-time fraction used for the Times46 heldout case was t = 0.396; the cameras’ embedded capture times put it near 0.52.

The official checkpoint keys contain `contral_estimator.*`, while the code expects `control_estimator.*`. The benchmark explicitly corrected that prefix and required strict loading. Silently skipping those weights would leave part of the model uninitialized.

The crossing-person flow was near zero instead of the approximately 36-pixel observed displacement. An additional diagnostic supplied the correct local displacement using observed semantic masks. The person still faded: average splatting mixed foreground into stationary background, and the synthesis network eroded it further. Better path curvature alone is insufficient.

The code/checkpoint license is CC BY-NC-ND 4.0. Modified upstream code and weights are not vendored into this repository. The implementation here is original, separate rendering/control code.

### SPEED

The exact released checkpoint loaded strictly: 117,131,648 parameters and about 1.16 GiB peak RSS in the initial native benchmark. Native midpoint inference took roughly 0.5–0.6 seconds on CPU in those runs.

We tested adjacent nine-frame sequences, heldout observations, all nine adjacent-pair midpoints for Times46 and Allen, 2× conventional input resizing, and 4× motion crops. The crossing people still faded. The larger input experiments cost more and did not pass visual review.

Midpoint-only evaluation must respect time: Times46 observations 1 and 3 predict a time 7 ms away from observation 2; Allen's residual is 0.5 ms. The 0-to-2 Times46 midpoint is 388.5 ms away from observation 1 and must not be called a time-correct heldout prediction.

> **Capture-clock correction:** these figures use receipt times. On the embedded capture times, Times46 observation 2 is about 50 ms from the 1-to-3 midpoint. Allen’s observation 2 sits at 66% of its 1-to-3 span, about 1 s from the midpoint. The Times46 0-to-2 midpoint is within about 70 ms of observation 1, so that case is the more time-correct one.

### Layered and aligned synthesis

An original opaque-layer renderer keeps the complete person geometry separate from front occlusion. The amodal diagnostic maintains a 146-pixel body mask and moves forward without alpha decay. It still looks like a rigid cutout and snaps to the observed endpoint pose.

A second experiment aligns the endpoint sprites before neural RGB refinement, then restores fixed amodal alpha and explicit front occlusion. It preserves contrast much better than visible-mask conditioning. The missing endpoint pixels are partly copied from an observation, so this is an ablation rather than independent validation of recovered shape.

SAM 3.1 text segmentation detects many people, but missed the moving person in all fourteen generated Framer frames. The compositor checks that specific component instead of accidentally reporting the stationary person's trajectory. Focused prompts and mask initialization are separate diagnostics; failed masks cannot certify motion.

### LDF-VFI joint sequence benchmark

The 1.410-billion-parameter transformer ran on a 4-vCPU / 8-GiB CPU worker with
16 sampling steps. Three native observations at receipt times 0, 1480 and
3737 ms were conditioned jointly. The complete run took 1005.287 seconds and
peaked at 7490.699 MiB process RSS. The decoder returned 20 frames; 17 bounded
by the observed anchors were reviewed.

> **Capture-clock correction:** on the cameras’ embedded timestamps, these anchors are near 0, 2070 and 4000 ms.

Every raw frame was inspected. The crossing figure becomes a crosswalk-colored
ghost or textured smear around frames 3–7 and 9–14, then returns at the anchors.
Both intervals fail. Raw anchor errors were small, but that did not establish
credible intermediate motion. The deterministic wrapper preserves all three
anchors and all 17 HUD strips exactly.

The CPU attention adapter preserves the original mask predicates using chunked
SDPA. A portable optional runner verifies pinned source/weight hashes and saves
resumable denoising checkpoints. Its CLI, masks and small SDPA equivalence were
checked; full weighted inference used the earlier experiment runner, not a
second complete execution of the portable CLI.
See [protocol and license](ldf-cpu-benchmark.md) and
[measured report](ldf-vfi-benchmarks.json).

### Trajectory-guided masked LTX refinement

A single 33-frame LTX 2.3 quality inpainting job used a 384 × 256 foreground
crop, fixed rightward trajectory, opaque amodal support, an unchanged front
occluder, black endpoint masks and 15 sampling steps. The 1.48-second source
interval was represented at 21.622 fps. Original input/mask encoding and the
final deterministic composite passed lossless pixel checks.

> **Capture-clock correction:** the true capture interval is about 2.07 s, so the clip was sized for the wrong duration.

All 33 raw and all 33 composited frames were visually inspected. The body
remained solid and followed the guide, but dark/red source clothing became
olive/yellow immediately, then snapped back at the final source. The generated
limbs also clipped at mask boundaries, and natural gait was not established.
This trial is rejected. Exact background pixels and forward motion are useful
controls, but do not compensate for appearance and pose failures.

The listed provider rate gives a nominal $0.00781 calculation for the input
pixel volume; the actual invoice was not available and no billed-cost claim is
made. No further requests were submitted.
See [recipe, provenance, contact sheets and review](ltx-guided-benchmark.json).

## Original pipeline now implemented

- `worker/motion_controls.py`: timestamp-aware four-observation Bézier controls with confidence masks, independent forward/backward fallback, analytic reversal/overshoot checks and bounded speed. Incorrect flow can still pass a mathematical path check.
- `worker/layered_motion.py`: observed-background coverage, fixed amodal foreground masks, explicit depth, optional RGB-only refinement, exact source anchors and deterministic overlay restoration. Exposed unobserved background returns an unsupported result instead of a silent fabricated fill.
- `worker/benchmark_ldf.py` and `worker/ldf_cpu_attention.py` provide an optional CPU research backend with verified provenance, bounded attention memory and per-step checkpoints. It is separate from normal capture/render jobs and does not approve clips.
- `worker/bounded_download.py` enforces byte limits, socket/total timeouts, checksum verification and atomic publication for research data downloads.
- The heldout evaluation tooling separates model inputs from target observations, verifies provenance, and measures moving regions as well as the full image. Its reports require separate visual review.

Camera HUD and timestamps are held as observed bitmaps until the next anchor. Road lettering belongs to the static background and can be covered by moving foreground; painting it over a person would be incorrect. An upscaler must not invent new source timestamps.

## What to build or train next

Adapt a pretrained motion/synthesis system before attempting to train a complete video model from scratch. Keep the pipeline modular:

1. Retain original observations, receipt times, hashes, camera geometry and source quality.
2. Estimate a stationary background with a coverage map. Distinguish observed clear pixels from inferred fills.
3. Obtain scene-local visible masks and correspondences using several observations. Preserve complete body hypotheses separately from visible masks and reject ambiguous instance merges.
4. Estimate timestamp-aware motion. Gate wrong correspondences, overshoot and reversals before synthesis.
5. Render opaque foreground with depth-aware occlusion over the observed background.
6. Refine pose and texture in aligned foreground crops or with joint-sequence diffusion. Use uncertainty to limit generated detail, rather than allowing a model to repaint street geometry.
7. Restore exact observed anchors and deterministic HUD. Upscale after motion has been validated.
8. Publish only reviewed artifacts with their method, source hashes and timeline.

For adaptation, dense fixed-camera footage supplies the missing supervision. [UA-DETRAC](https://faculty.ucmerced.edu/mhyang/papers/cviu2020_detrac.pdf) is 25 fps. [MOT17](https://motchallenge.net/data/MOT17/) includes a stationary square and elevated night street at 30 fps; [MOT20](https://motchallenge.net/data/MOT20/) includes dense crowds at 25 fps. Downloaded preview videos must be decoded using their own timestamps; their encoding need not equal the original dataset frame rate.

Split by original camera/video before extracting windows. MOT17's DPM/FRCNN/SDP variants duplicate the video and must not cross training/evaluation splits. Generate sparse inputs at realistic intervals with JPEG damage, irregular sampling and duplicates, while keeping dense observations exclusively as targets.

Train a small confidence/occlusion compositor or a limited pretrained-model adapter first. Relevant losses include foreground reconstruction, silhouette support, temporal consistency, correspondence confidence and correct occlusion. Penalize body disappearance separately from valid occlusion. An opaque mask by itself does not prove a natural gait.

Evaluation must include heldout cameras, night/headlight scenes, overlap, entrances/exits and stationary people. Report foreground error, duplicate bodies, opacity, scale, direction and flicker. Whole-image similarity is dominated by the street and can reward a missing person. A final human review of every published frame remains necessary.

## Prepared supervision and current status

The dataset tool was executed on two pinned official MOT17 raw preview videos,
both decoded at their actual 30 fps / 960 × 540 geometry. Six sparse windows
produce 24 model inputs and 818 unique dense targets at 352 × 240. A separate
space-time export keeps those same inputs and uses 704 × 480 targets.
All content frame indices used as inputs are excluded globally from targets,
including across overlapping windows. Two stale inputs are exact byte copies.

The cameras are split before window extraction and duplicate detector variants
are excluded. This is a small development testbed, not sufficient independent
evaluation for a trained production model. Dataset attribution and restrictions
are retained; no dataset imagery or model weights are committed. See the
[data audit](dense-motion-data-audit.json) and the worker README for commands.

No new model has been trained, and none of this turn's generated clips is
approved for the main viewer. The remaining work is appearance-preserving pose
synthesis and reliable occlusion/correspondence, rather than higher output FPS.

> **Review note:** the October 19 review finds that pose between stills is unrecoverable at about two strides per gap. It puts capture timing, a plate bank, geometry and association first; see the review section above.