A reproducible, low-cost model for sparse-camera interpolation

01Goal

The input is two or more stills from a fixed camera, and the output is the frames between them. Moving people and vehicles should keep a solid shape and follow a physically possible path. All other pixels should come from real observations. Each observed frame, including the camera's timestamp overlay, must be reproduced exactly at its position in the sequence.

Two stills record an object's position at two times. They do not record its path, its pose in between, or when another object passed in front of it. The pipeline estimates these, and every generated frame is labelled as an estimate.

02Source data and why standard interpolation fails

Most frame interpolation research targets dense video, where objects are large and move a few pixels between frames. The traffic cameras used here provide one 352 × 240 JPEG about every two seconds, with compression artifacts and irregular timing. Pedestrians are about 20–30 px tall, and many move farther than their own height between stills.

Walkers take about one stride per second, so a two-second gap contains about two strides. The leg positions at the two endpoints are often nearly the same, and models that interpolate between them produce legs that freeze or move backward. Walking motion between stills can only be estimated.

Current video models also work at a coarser resolution than these objects. One latent token covers about 16 × 16 px in Wan2.1 and 32 × 32 px in LTX, so a pedestrian spans one or two tokens. The model has to generate limbs and clothing from its training data, which causes color drift. A 2026 keyframe benchmark reported “melting, dissolving, or fluid-like artifacts” from the best open model under fast motion (KeyFrame-Compass).

The tests here showed the same problems. About thirteen interpolation and video-generation systems were tested. Failures included pedestrians fading into the road, reversed cars, added vehicles, clothing color changes and frozen poses. On heldout frames, a crossfade had a whole-image error of 11.29, against 11.25–11.27 for the best interpolator.

03Pipeline

A fixed camera records the same background in thousands of frames, so the background can be measured instead of generated. The remaining task is to find moving objects, match each one between stills, and move it across the known background. In a busy frame, people cover about 3% of the pixels, so generation is limited to a small part of each frame.

1. Capture and timing. Store the raw JPEG bytes and read the capture time embedded in each file. Receipt time is kept only as a record of when the file arrived.
2. Background plates. Build plates from many observations, grouped by time of day and lighting, with a map of how often each pixel was seen clear. Mask regions that change on their own, such as screens and signals.
3. Foreground layers. Detect moving objects by comparing each frame with the plate, confirm them with a detector, and refine the masks with box-prompted segmentation. Bodies are rendered opaque, gaps between legs are kept, and the full outline of a partly hidden object is stored separately from its visible mask.
4. Geometry and matching. Map image points to the ground plane. Match objects within one camera using speed limits, lane direction, size and color. An ambiguous match is marked unknown.
5. Motion and rendering. Vehicles move as rigid layers along their lanes. Pedestrians use a simple walk cycle, labelled as an estimate. Generated video can supply candidate pixels for foreground objects only.
6. Detail refinement. Refine only the region that contains motion, with zero noise and a fixed seed, and place it back at its exact coordinates.
7. Anchor restoration and checks. Insert the exact observed frames and timestamp overlay at each anchor, and check them before encoding.
8. Evaluation and labelling. Score each object rather than the whole frame, compare against simple baselines, and label every generated frame.

A background plate is reliable where the scene is static. A per-pixel median fails where an object occupies a pixel in more than half the frames. Median methods also degrade under lighting change: average gray-level error rises from about 1.4–3.0 on basic scenes to 15–20, while low-rank methods stay near 2.8 (comparison).

Published trackers have been evaluated down to about one frame per second. At that rate, matching that relies only on motion degrades sharply, and matching based on re-identification performs much better in crowds (StableTrack). These cameras run at half that rate, so some objects in crowded scenes will not be matchable. A ground-plane homography from a few marked points adds two constraints: walking speed between 0 and 3 m/s, and lane directions that rule out reversed cars.

Objects are matched only within one camera over a few seconds, and the matches are discarded after rendering. The pipeline does not identify people, link them across cameras or store appearance embeddings.

04Results so far

Generation limited to foreground masks

Full-scene video proposals contain useful information about how people walk, but they also add vehicles that were not there. Using a proposal only as a source of foreground pixels, filtered by reviewed masks, excluded every added vehicle. Limiting generation to moving regions also removed the street warping and damaged road lettering seen with whole-frame methods.

Mask stability

Changes in mask size can look like a body shrinking or growing. Normalizing each object's masks to a shared template and combining masks across neighboring frames reduced frame-to-frame area changes from 35% to at most 2.54%. Pedestrian masks keep the gap between the legs; filling the outline closes that gap.

Background repair

When an object moves, the area it covered must be filled with background. Patches copied from a distant observation had different lighting, including another vehicle's headlights. Patches from the nearest clear observation matched the surrounding lighting.

Changes at observed frames

Where two generated intervals meet at an observed frame, a person's pose and clothing can change abruptly. At the two joins tested, the mean frame-to-frame change was 3.45 and 5.72 on a 0–255 scale, compared with about 0.42 between neighboring frames. The difference was already present in the raw generator output, before masking and refinement.

The next test is a bridge clip that includes motion before and after the join, with the observed frame fixed in the middle. This requires keyframe or inpainting control in the model. A short optical-flow correction around the observed frame is a cheaper first test, but it cannot reconcile two different poses.

Timing

Each frame contains the camera's capture time, to one hundredth of a second, in a JPEG comment or EXIF field. Receipt times differ from the true intervals by 26% on average. The original three-second poller skipped every other camera refresh, so intervals recorded as three seconds were alternating two- and four-second gaps. One benchmark interval recorded as 1.48 s lasted 2.07 s, which overstated a pedestrian's speed by about 40%.

Upscaling

Direct 8× upscaling of still frames added texture that was not in the source and merged small figures with street furniture. Upscaling an accepted 4× result by a further 2× with zero noise avoided these problems and reused the earlier result. Still-image quality and motion quality are evaluated separately.

05Cost

Generating one three-second interval currently costs about $1.10 in inference at published list prices. These figures are estimates, not invoices.

Most of this cost is video generation. In the planned pipeline, the background comes from measured plates, and vehicles and pedestrians are rendered deterministically on a CPU. The only paid step is then an optional GPU refinement of a small crop.

• Moving regions only. Refining only the moving region instead of the full frame reduced output pixels by 61–82% in the reviewed intervals.
• Reuse of accepted results. Upscaling from an accepted 4× result cost about half a cent per frame, because the first stage was not repeated.
• Self-hosting. A planned benchmark will compare self-hosted refinement with hosted inference on matched, pinned crops, with a five-dollar pilot limit and a quality check. No results are recorded yet.

06Reproducibility

Each published interval is defined by a recipe, so it can be regenerated and checked.

• Pinned inputs. Each source image has a SHA-256 hash. Upstream model code is pinned to specific commits, and the research runner checks weight hashes.
• Recipe validation. The renderer rejects invalid boxes, repeated mask IDs, misspelled fields and incomplete sequences before writing any image.
• Deterministic refinement. Detail refinement uses zero noise and a fixed seed.
• Replay. A verification replay reproduced all 73 frames of an interval exactly, and a mask-consensus helper reproduced all 73 of its masks.
• Anchor checks. Before encoding, the observed endpoint frames and the previous timestamp overlay are compared pixel by pixel.
• Delivery. Published files are copied from a manifest that pins each file's size and hash. A missing or corrupt file stops the build.

License note: ABC-Inter is licensed CC BY-NC-ND 4.0, and the SPEED repository has no license file. Modified upstream code and weights are not included in this repository.

07Evaluation and labelling

Whole-frame metrics do not rank these methods correctly. People cover a small part of the frame, so removing a person and filling in the street can improve whole-image error, SSIM and LPIPS. FVD changes very little under temporal corruption (Ge et al., CVPR 2024).

Evaluation is therefore per object: foreground-masked error, detection recall, objects with no source in the observed frames, counts by class, centroid error, consistency of object identity, and change in static regions. Each method is compared with simpler baselines in this order: holding the last frame, crossfade, the background plate with linearly moved layers, optical flow, learned interpolation and generation. A method is published only if it outperforms the plate-and-layers baseline without adding people.

Dense fixed-camera datasets such as MOT17, MOT20 and UA-DETRAC can be subsampled to two-second gaps and reduced to 352 × 240 with JPEG compression. This provides real intermediate frames and ground-truth tracks for scoring.

Generated frames need labels that stay attached when a clip is shared. EU AI Act transparency rules have applied since August 2, 2026, and cover realistic depictions of real places (European Commission). Platforms often remove metadata, so each generated frame carries a visible label, and recorded and generated spans are marked differently on the timeline. This is not legal advice.

08Related work

The methods below differ in what they support. A model that predicts only the midpoint cannot produce frames at other times, and a diffusion timestep does not correspond to a video timestamp. Nonlinear motion controls cannot correct a wrong match between endpoints. A model that accepts a list of frames does not necessarily use all of them together.

Other 2025–2026 work relevant to this pipeline:

09Roadmap

10Detailed records

This page is a summary. The detailed records are:

• Lab notes: each experiment, its parameters and its review.
• Independent review: the assessment behind sections 02–09, with sources.
• Localized generation, motion assessment and validation.
• Cost and quality benchmark plan.