# Model the scene, not the frames

This project's interpolation research is careful and honest. It has never published a clip it could not defend, and it correctly found that a plain crossfade matches or beats every neural interpolator it tested. But most of the effort has gone into the wrong layer of the problem. About thirteen frame-synthesis systems were tried against what were essentially two hard intervals. Meanwhile, the inputs those systems depend on went unmeasured. The repository holds only ten stills per camera. The clock it uses is the wrong one. No code detects, associates or places in ground coordinates a single person or car. The most consequential oversight is timing. All six cameras embed sub-second capture timestamps in their JPEGs. Receipt times misstate the true intervals by about a quarter on average, and the Canal Street samples silently skip every other two-second refresh. The project's benchmark pedestrian interval lasts **2.07 s, not 1.48 s**. The background-plate hypothesis is correct and already proven in miniature: a four-frame plate held every background pixel exactly still. The plate is under-built, not wrong. It is easy to build for quiet, static street surfaces at a given time of day, and hard where pixels are occupied most of the time, where billboards change, and where light shifts or headlights dominate. Layering also simplifies a different problem from the one the project has been attacking. Once the background is known, interpolation becomes detecting, associating and moving people and cars about 22–32 px tall. No 2026 model this review found is built for objects that small moving farther than their own height, whether it does frame interpolation, generative inbetweening or trajectory-controlled video. A walker completes about two strides between stills, so no method can observe the gait in between. The realistic goal is therefore a deterministic, labelled reconstruction: an exact plate, tracked rigid vehicles, pedestrian sprites with a generic walk cycle, and an explicit "unknown" wherever association is ambiguous. It should be scored with object-level metrics against hold and crossfade baselines before any generative model is reconsidered.

## The cameras stamp capture time to the hundredth of a second

The data behind every experiment is thinner than the experiment count suggests. The repository ships **10 distinct 352×240 stills per camera, 60 in total**. Each camera's set covers 18–27 s at a single moment and a single lighting condition, and every camera reports `quality_approved_pairs: 0` ([frames.json](/frames.json)). The capture worker is designed to archive about **259,200 deduplicated images per day** across six cameras ([worker README](/worker/README.md)). However, "no persistent capture host is connected" ([README](/README.md)), and rendering is capped at a 12-frame excerpt ([worker/batch.py](/worker/batch.py)). An earlier 85-second experiment received 43–44 unique images per Canal camera at about 2-s intervals. Those images survive only as statistics ([motion-quality.md](/docs/motion-quality.md)).

The project's own documents say receipt time "is not an independently verified exposure timestamp" ([camera-selection.md](/docs/camera-selection.md)), and the capture code writes `capture_time: None` ([worker/batch.py](/worker/batch.py)). Yet the cameras already supply that timestamp. Axis cameras write a JPEG comment segment whose `0x0A 0x01` record holds seconds since the epoch plus hundredths of a second ([Axis VAPIX trigger data](https://developer.axis.com/vapix/network-video/trigger-data/)). Five of the six cameras carry it. The Times Square 45 St unit, a different Axis model on newer firmware, carries EXIF `SubsecTimeOriginal` instead. This review decoded all 60 source JPEGs referenced in [frames.json](/frames.json), and every file's SHA-256 matched the manifest. Their capture intervals differ sharply from the receipt intervals the project uses everywhere:

| Camera | Receipt intervals (s) | Embedded capture intervals (s) | What receipt time hides |
|---|---|---|---|
| Canal × Allen / Baxter / Chrystie | 2.87–3.01, nearly uniform | **alternating ~2.0 and ~4.0** | Every other 2-s refresh was skipped by the original 3-s poller |
| Times Sq 46 St | 1.48–2.26 | 1.93–2.07 | The "1.48 s" benchmark interval is **2.07 s** |
| Times Sq 45 St | 1.49–2.26 | 1.97–2.03 | Poll-grid artifacts of the 0.75-s poller |
| Herald Sq 33 St | 1.50–2.26 | 1.76–2.30 | Less regular source cadence |

Averaged over all intervals, receipt time misstates the capture interval by **26%**, and the receipt always lags capture by 0.06–1.61 s ([Axis VAPIX](https://developer.axis.com/vapix/network-video/trigger-data/); [frames.json](/frames.json)). The error reaches into the published conclusions. The crossing pedestrian moves about **36 px** between Times 46 St observations 0 and 1 ([abc-inter-benchmarks.json](/docs/abc-inter-benchmarks.json)). On the receipt clock that is 24 px/s; on the capture clock it is about 17 px/s, so receipt-based paths overstate this walker's speed by roughly 40%. Taking the person's 22–30 px mask height ([framer-composition-benchmarks.json](/docs/framer-composition-benchmarks.json)) and assuming a 1.7 m adult, the corrected speed is about 1.0–1.3 m/s. That matches the 1.31 m/s usual outdoor walking pace ([Sports Medicine meta-analysis](https://pubmed.ncbi.nlm.nih.gov/33030707/)), whereas the receipt-clock figure of 1.4–1.9 m/s is closer to the 1.72 m/s fast pace.

The held-out "time-correct" claims in [interpolation-research-2026-10.md](/docs/interpolation-research-2026-10.md) are partly inverted. The document calls Allen's 1→3 midpoint test exact, with a 0.5 ms residual. On the capture clock, however, observation 2 sits at 66% of that span, about **1 s from the midpoint**. Conversely, the Times 46 St 0→2 midpoint, dismissed as 388.5 ms off, is actually within about 70 ms. ABC-Inter was asked for t = 0.396 when the true fraction was about 0.52. The LDF-VFI anchors at 0/1480/3737 ms are really near 0/2070/4000 ms ([ldf-cpu-benchmark.md](/docs/ldf-cpu-benchmark.md)), and the LTX and Wan VACE clips were sized for a 1.48-s interval ([ltx-guided-benchmark.json](/docs/ltx-guided-benchmark.json); [wan-vace-localized-benchmark.json](/docs/wan-vace-localized-benchmark.json)). On Canal Street, any constant-velocity path drawn on receipt time would show apparent speed alternating by a factor of two between consecutive intervals. Some of the motion reviewers saw as "unnatural" may come from this.

Two uncertainties remain. Axis does not document that the stamp equals sensor exposure time rather than snapshot or encode time, and only these 60 frames were checked. Even so, an error of a few hundredths of a second is about 1 px at these speeds, against 0.2–1.0 s from receipt time. The current worker's 1.0-s default poll ([worker/batch.py](/worker/batch.py)) would already capture every refresh, because the skipped Canal frames came from the original three-second demo ([worker README](/worker/README.md)). Re-encoding strips these metadata segments, so the archive must keep the raw bytes.

## Thirteen synthesis systems, two hard intervals, zero approvals

The experiment ledger is broad and consistent. RAFT fixed-flow failed the reliability gate on **27 of 27 Canal pairs and 27 of 27 new-camera pairs** ([direction-review.json](/docs/direction-review.json); [new-camera-motion-review.json](/docs/new-camera-motion-review.json)). Pika and Seedance produced direction reversals; RIFE, FILM and AMT ghosted or faded the crossing person "into road stripes" ([motion-benchmark.json](/docs/motion-benchmark.json); [direction-review.json](/docs/direction-review.json); [amt-film-benchmarks.json](/docs/amt-film-benchmarks.json)). ABC-Inter estimated near-zero flow for a person who moved 36 px. SPEED rendered the body transparent. LDF-VFI spent **1,005 s and 7.5 GiB of RAM on CPU** to produce a "crosswalk-colored ghost" ([ldf-cpu-benchmark.md](/docs/ldf-cpu-benchmark.md)). Masked LTX 2.3 turned dark clothing olive-yellow, and Wan VACE kept the colour but held a rigid pose, then jumped at the endpoint ([ltx-guided-benchmark.json](/docs/ltx-guided-benchmark.json); [wan-vace-localized-benchmark.json](/docs/wan-vace-localized-benchmark.json)). The held-out numbers point the same way. On Times 46 St 0→2, whole-image MAE was 12.28 for previous-hold, **11.29 for crossfade and 11.25–11.27 for ABC-Inter**, a statistical tie. SPEED was worse than both trivial baselines on every case: 12.67 against 8.27 and 8.98 on Times 46 St, and 10.65 against 6.06 and 5.19 on Allen ([holdout-benchmarks.json](/docs/holdout-benchmarks.json)).

Several choices overcomplicated the work without reducing risk. The LDF run was followed by a resumable CPU attention adapter, engineered after the method had already been rejected. The project published **2816×1920 Allen stills at about 5.5 MB each**, even though its own pipeline says to "upscale after motion has been validated". Direct 8× upscaling had already been rejected for "invented road/graffiti texture and a small figure merging" ([upscale8-review.json](/docs/upscale8-review.json)). Approval files bound to SHA hashes, and HMAC media signatures, are sound hygiene, but they guard a pipeline that has approved nothing. Above all, the breadth was across models, not across data: nearly every trial re-tested one Times Square crossing and a few Allen pairs. **Every result that preserved a pedestrian's direction used a manually supplied correspondence or trajectory** ([framer-benchmark.json](/docs/framer-benchmark.json); [abc-inter-benchmarks.json](/docs/abc-inter-benchmarks.json)).

The record also deserves credit. The viewer labels the mode "Motion withheld · Source frames" rather than shipping a weak clip. The HUD is restored bit-exactly. The layer renderer refuses to inpaint unobserved background silently ([worker/layered_motion.py](/worker/layered_motion.py)). The research document separates amodal body shape from the visible mask, and it already warns that whole-image similarity "can reward a missing person." Its literature table is accurate: all eight arXiv IDs resolve to the described papers with matching dates. Four notes are due. SPEED's repository has **no LICENSE file**, so vendoring its code is legally unsafe ([SPEED repo](https://github.com/bbldCVer/SPEED)). MGMVFI is still README-only ([MGMVFI repo](https://github.com/Jaep0805/MGMVFI)). LDF-VFI has a v2 dated 30 March 2026 ([LDF-VFI arXiv](https://arxiv.org/abs/2601.14959)). SNM-VFI is an ECCV workshop paper built on SVD-based GenIn weights ([SNM-VFI arXiv](https://arxiv.org/abs/2608.13460)). The eight-step plan in the research document already names a background plate, correspondences and depth-aware layers. The problem is sequencing. Those steps remain unbuilt while the document frames the remaining work as "appearance-preserving pose synthesis."

Newer models will not close this gap, because the failures belong to the regime, not to configuration. A Wan2.1 DiT token covers 8×8 VAE pixels × 2×2 patch = **16×16 px** ([Wan technical report](https://arxiv.org/html/2503.20314v2)). An LTX token covers **32×32 px** ([LTX-Video](https://arxiv.org/abs/2501.00103)). A 25-px pedestrian is therefore one or two tokens tall and less than one token wide, so a generator must invent limbs and clothing from its prior, which explains the colour drift. Flow pyramids shrink the same person to 1–4 px at coarse levels, where a 36-px displacement exceeds the object's width. The 2026 KeyFrame-Compass benchmark covers 386 samples across nine systems. It found that the best open keyframe adherer, LTX-2.3, still shows "melting, dissolving, or fluid-like artifacts" under fast subject motion, and that keyframe adherence trades off against natural motion ([KeyFrame-Compass](https://arxiv.org/html/2607.14202v1)). The newest inbetweener, DSG-GI, lists multiple moving subjects among its failure cases and needs about 180 s per 49 frames on a workstation GPU ([DSG-GI](https://arxiv.org/html/2610.13187v1)). BiM-VFI's authors state that VFI models assume uniform motion at inference, which they describe as a limitation shared by all VFI methods ([BiM-VFI](https://arxiv.org/pdf/2412.11365)). Large-motion flow work such as MASS, multiple-hypothesis flow and SGM-VFI improves matching ([MASS](https://arxiv.org/abs/2606.27718); [Multiple Hypothesis Flow](https://arxiv.org/abs/2608.07120); [SGM-VFI](https://arxiv.org/abs/2404.06913)). But these methods still warp and blend, and no 2026 paper found targets objects of 40 px or less moving more than their own size.

## A clean plate is easy on quiet asphalt and hard under changing light

The repository has already tested the hypothesis in miniature. `estimate_background` computes a masked temporal median with per-pixel counts of clear observations, but **only unit tests call it** ([worker/layered_motion.py](/worker/layered_motion.py)). The one real plate used four Times 46 St frames and SAM people masks, and left vehicles unmasked. Even so, only **2.75% of below-HUD pixels were never observed clear**. Composited over that plate, the background showed a **maximum pixel change of 0** across intermediate frames, which "removes global street geometry morph" ([framer-composition-benchmarks.json](/docs/framer-composition-benchmarks.json)). Whole-frame renders without a plate are what turned the crosswalk into "a pixel mosaic" and eroded road lettering ([new-camera-motion-review.json](/docs/new-camera-motion-review.json)). The intuition is correct, and the project stopped at four frames when the capture design could supply thousands.

The background-initialization literature puts numbers on where it is easy. On SBMnet sequences, robust-median, patch-selection and low-rank methods reach an average gray-level error (AGE) of **1.4–3.0 on basic scenes** and 1.6–3.8 with intermittent motion. Under **illumination change**, median-style methods degrade to AGE 15–20: LaBGen-OF scores 19.6 and CPB 15.3, while low-rank MSCL holds at 2.8. Under camera jitter most methods score 6–12 ([CPB comparison, arXiv 2003.12931](https://arxiv.org/abs/2003.12931)). A per-pixel median also has a 50% breakdown point. Where a pixel shows a person or car in more than half the frames, as on Times Square sidewalks or in queued lanes, no frame count fixes it. The closest precedent is WebCamT, which uses **212 city webcams at 352×240** with 1–3 s between frames. Its background-subtraction baseline used a vehicle-free reference image taken in light traffic and brightness-adjusted for other periods ([WebCamT, arXiv 1703.05868](https://arxiv.org/abs/1703.05868)). That is the plate-plus-photometric design this project needs.

These cameras hit each failure condition somewhere. Between neighbouring Times 46 St stills, static pixels shifted by a fitted bias of up to **+16.9 levels**. A gain/bias fit cut static MAE from 9.85 to 5.49, but that fit exists only inside one JSON experiment ([framer-composition-benchmarks.json](/docs/framer-composition-benchmarks.json)). Times 45 St's protected rectangles cover large billboard areas, and reviewers saw "actual billboard illumination and scene color change between observations" ([worker/cameras.json](/worker/cameras.json); [camera-processing-review.json](/docs/camera-processing-review.json)). Every Times and Herald sample was captured at 01:49 local time, and night is the hardest category for unseen-scene background subtraction: the best CDnet 2014 night F-measures are 0.59–0.70 ([BSUV-Net 2.0, arXiv 2101.09585](https://arxiv.org/abs/2101.09585)). Shadows are modelled nowhere in the codebase, and promptable segmenters will not catch them, because "SAM tends to consider shadows as background" ([ShadowSAM, arXiv 2305.16698](https://arxiv.org/abs/2305.16698)).

| Scene element | Does a many-frame plate recover it? | Handling |
|---|---|---|
| Road, low-occupancy sidewalks, facades, lane lettering | Yes, as exact real pixels | Robust median or clustering per time-of-day bin |
| Pixels occupied more than half the time | No; the median breaks down | Low-occupancy hours, semantic masking, or a "crowd layer" |
| Billboards, LED screens, signals | No; not stationary | Dynamic-region mask filled from the nearest real frame |
| Exposure and white-balance jumps | Partly | Per-frame gain/bias against the plate |
| Diurnal light, building shadows, night | Only within a time bin | Plate bank keyed by time of day and illumination |
| Cast shadows, headlight beams, wet-road reflections | No; they travel with movers | Assign to the moving object's layer |
| Parked cars, standing people | Absorbed into the plate | Leave them; they need no interpolation |
| HUD timestamp | Must never be estimated | Copy from the source frame |

This plan makes most 2025–2026 removal research unnecessary for these cameras. Generative Omnimatte's public release processes 81–85-frame windows at 384×672 or 480×832 ([gen-omnimatte-public](https://github.com/gen-omnimatte/gen-omnimatte-public)), and ROSE needs 16n+1-frame clips at 480×720 ([ROSE](https://github.com/Kunbyte-AI/ROSE)). Both exist mainly to hallucinate disoccluded background in dense video. A measured plate supplies that background as real pixels. Removal is also not solved in general: the ECCV 2026 BeyondMasks benchmark finds "systematic failures in removing secondary physical effects despite high masked region fidelity" ([BeyondMasks, arXiv 2608.20107](https://arxiv.org/abs/2608.20107)). The project has one internal inconsistency to fix first. The stills pipeline copies road lettering from each frame as a protected overlay ([existing-image-review.json](/docs/existing-image-review.json)). The motion pipeline says lettering belongs to the background and can be covered by a passing person ([interpolation-research-2026-10.md](/docs/interpolation-research-2026-10.md)). A layered renderer needs one rule, and the motion pipeline's rule is the correct one.

The plate also fixes foreground isolation, the step where the project's masks failed. SAM 3.1 text prompts **missed the moving person in all fourteen frames**, and SAM 2 point prompts clipped the head and legs ([framer-composition-benchmarks.json](/docs/framer-composition-benchmarks.json)). The SAM 2 authors list crowded scenes and thin, fast details as known weaknesses ([SAM 2, arXiv 2408.00714](https://arxiv.org/abs/2408.00714)). Differencing against a photometrically matched plate uses a scene-specific prior that no generic segmenter has. Blobs found that way can be gated by a detector and refined with box-prompted SAM 2 (Apache-2.0) on 2–4× crops. SAM 3 is under a custom license with gated weights, so it needs a license check first ([sam3 repo](https://github.com/facebookresearch/sam3)). Alpha mattes can then be computed from the known plate on the Background Matting principle ([Background Matting, arXiv 2004.00626](https://arxiv.org/abs/2004.00626)). With a plate, a shadow is simply a darker, chromaticity-preserving region attached to an object's foot point. That is the sense in which the hypothesis is right to stop "treating the background as noise." Every rejected whole-frame method treated the roughly 97% of the frame that is street as something to re-synthesize. Much of its visible damage (morphing stripes, eroded lettering, mosaic crosswalks) therefore landed on pixels the cameras had already observed many times.

## Layering turns interpolation into tracking, and tracking needs geometry

With the plate solved, the remaining problem is correspondence: which blob in one still is which blob in the next. The repository answers this only with dense optical flow or by hand. It has no detector, association step, motion model or ground-plane homography, and the project describes avoiding "detectors, embeddings, or tracking" as a privacy stance ([worker/motion_quality.py](/worker/motion_quality.py); [motion-quality.md](/docs/motion-quality.md)). That stance merges two different things. Matching anonymous blobs within one camera over a 20-second window is not identity recognition. It can be done from position, size and colour histograms alone, and discarded after rendering. Cross-camera re-identification and stored appearance embeddings are what the privacy concern rightly excludes.

The tracking literature shows what survives at low frame rates, and its evidence ends at 1 Hz. On MOT17, ByteTrack's HOTA falls from 67.5 to 57.6 when the stream is subsampled to every tenth frame. ColTrack, with query-based association, holds at 65.5→65.8 ([ColTrack, arXiv 2308.05911](https://arxiv.org/pdf/2308.05911)). At 1 Hz on crowded MOT20, Kalman/IoU-family baselines reach HOTA 15.4–37.4, with association accuracy as low as **4.9**. StableTrack, which lets Re-ID drive matching under loose box-distance gating, reaches 63.5 ([StableTrack, arXiv 2511.20418](https://arxiv.org/html/2511.20418)). Motion-based association "collapses in low-frame-rate regimes, where object displacement exceeds the search radius of standard filters" ([arXiv 2602.00450](https://arxiv.org/html/2602.00450)). These cameras run at 0.5 fps, which is every 60th frame of 30-fps MOT17 and twice as sparse as any published protocol. The Times Square crowds will therefore leave some identities genuinely ambiguous. The design lesson transfers even though the networks do not: associate on content, gate geometrically, and stay loose.

Geometry makes that gating physical. Fully automatic traffic-camera calibration reaches **1.10 km/h** speed error, but it needs video-rate vehicle tracks ([Sochor et al., arXiv 1702.06451](https://arxiv.org/abs/1702.06451)), which 0.5-fps stills cannot provide. For six fixed cameras, clicking four or more ground points per view (curbs, crosswalk stripes, lane lines) takes minutes. The MOTChallenge benchmark treats **0–3 m/s** as realistic walking speed, with a 1.4 m/s comfortable mean ([MOTChallenge 2015, arXiv 1504.01942](https://arxiv.org/pdf/1504.01942)). In ground coordinates that becomes a hard gate against impossible pedestrian matches. Lane polylines with a direction of travel forbid the reversals that sank Pika and Seedance, by construction. Parameterizing vehicles by distance along the lane follows the 2026 LoRFT result: reconstructing paths in a road-aligned state space cut average displacement error 11% against the strongest baseline ([LoRFT, arXiv 2607.19911](https://arxiv.org/abs/2607.19911)). At 2-s gaps, urban vehicles often travel farther than the visible road segment. The associator must allow objects to appear and leave between frames instead of forcing matches, which is exactly the "vehicle arrives only at the endpoint" case reviewers flagged ([new-camera-motion-review.json](/docs/new-camera-motion-review.json)).

Each camera's sequence is short and processed offline, so global association over all frames is better than an online tracker: min-cost flow or K-best multi-frame assignment can use the frame after an occlusion to resolve it. Each confident pedestrian track can then be smoothed in metres with a constant-velocity Kalman filter and Rauch–Tung–Striebel smoother. With true capture Δt, the smoother yields both a rendered path and an uncertainty tube. With only two endpoints, BiM-VFI's "time-to-location ambiguity" applies: the timing of a turn or speed change inside the gap is not identifiable ([BiM-VFI](https://www.alphaxiv.org/abs/2412.11365)). The honest rule follows. If an object has one observation, violates a speed or lane gate, has an occluded foot point at an endpoint, or has K-best hypotheses that disagree on direction, it is marked unknown and its observed frames are held. The site already holds frames when motion is withheld.

## Two strides per gap make gait a labelled guess

The project's stated remaining problem, appearance-preserving pose synthesis, cannot be solved from these data. Usual-pace walkers take **116.65 steps per minute** ([Sports Medicine meta-analysis](https://pubmed.ncbi.nlm.nih.gov/33030707/)), a stride frequency of about 0.97 Hz. Sampling even the fundamental of that motion needs about 2 Hz, four times the camera rate. Over the true 2.07-s benchmark interval, a walker completes **about two full strides**, so the endpoint poses look nearly the same. Any model that interpolates between them produces frozen or backward-shuffling legs, the wagon-wheel effect. That matches the "natural gait was not established" verdicts on LTX, Wan VACE and the opaque layers. The receipt clock's 1.48 s implied 1.4 strides and near-opposite leg phases, a different and equally misleading picture. Heatmap pose estimators also degrade significantly for low-resolution people ([Hardy et al., arXiv 2107.02108](https://arxiv.org/abs/2107.02108v2)). Pose-driven animators such as UniAnimate-DiT need about 23 GB of GPU memory for 832×480 output ([UniAnimate-DiT](https://github.com/ali-vilab/UniAnimate-DiT)). Fed a 20-px reference, they would invent a body only to have it downsampled away.

A plausible gait remains worth drawing, as long as it is labelled as a hypothesis. For a 20-px figure (about 8.5 cm per pixel), a step separates the feet by roughly 8 px at most. The visible gait signal is a few-pixel scissor of the lower half plus about a 1-px vertical bob. A procedural walk cycle on the existing amodal sprite captures nearly all of it. The sprite is split at the hip, its cadence is set from the ground speed measured through the homography, and it crossfades into the observed pose over 100–200 ms near each keyframe. Colour cannot drift because every pixel comes from a real observation. That removes the three rejection reasons the project keeps recording: fading, colour shift, and frozen gait. Real-time crowd rendering has used image-based walk-cycle impostors for decades ([Tecchia et al. 2002](https://www.cs.ucy.ac.cy/~yiorgos/publications/crowd_CG&A02_draft.pdf)). A richer middle option is to retrieve real walking silhouettes at similar scale from the MOT17 windows the project has already prepared. Vehicles are rigid at this resolution. They should move as their own sprites along lane paths, scaled by the homography's depth ratio, with a swap between the two endpoint sprites on turns. Headlight glow and shadow travel with them.

Trajectory-controlled generators are the natural objection. Their controls exist and several carry permissive licenses, but none keeps the background exact by itself, because each regenerates every pixel of its crop. Wan-Move and ATI propagate first-frame features along known tracks, under Apache-2.0 ([Wan-Move](https://github.com/ali-vilab/Wan-Move); [ATI](https://github.com/bytedance/ATI)). Wan-Move still needs a 40 GB GPU with offloading. Go-with-the-Track (SIGGRAPH 2026, Apache-2.0) is the one open-code option that conditions on multiple reference images with anchored point tracks ([Go-with-the-Track](https://github.com/eyeline-labs/Go-with-the-Track)), so it could take both endpoint sprites. It is the right candidate for an optional refinement pass, and it is untested here. FlexComposer's 2026 design separates an object's intrinsic motion from its global displacement ([FlexComposer, arXiv 2607.29627](https://arxiv.org/abs/2607.29627)). That is the same decomposition as sprite-over-plate, which supports the architecture while offering no runnable code. The strongest counter-argument is aesthetic: the project's own opaque layers looked like a "rigid cutout" ([interpolation-strategy.md](/docs/interpolation-strategy.md)). The walk cycle addresses exactly that. Whether it reads as natural at 22–32 px is unknown, because no perception study at that scale was found, so the project's visual review has to decide it.

## Score objects, not pixels, and label every synthetic frame

Whole-frame metrics cannot rank these methods. People cover **2.8–3.0% of the below-HUD pixels** in Times 46 St ([framer-composition-benchmarks.json](/docs/framer-composition-benchmarks.json)), so erasing a person and painting a clean street can improve MAE, SSIM or LPIPS. VFI-specific perceptual metrics are only moderately aligned with viewers even on ordinary content. PSNR_DIV improves on FloLPIPS by 0.09 PLCC ([Daly et al., arXiv 2510.01361](https://arxiv.org/abs/2510.01361)), and an earlier divergence metric from the same group reached PLCC 0.51 ([arXiv 2508.09078](https://arxiv.org/abs/2508.09078)). Distribution metrics answer the wrong question: FVD barely moves under large temporal corruption, and drops when generated sets are filtered toward motionless videos ([Ge et al., CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/html/Ge_On_the_Content_Bias_in_Frechet_Video_Distance_CVPR_2024_paper.html)). No standard object-level interpolation metric exists, so the project has to define its own. The suite should cover foreground-masked error per object, detection recall and phantom objects on interpolated versus held-out frames, count error by class, centroid position error, identity persistence across observed-synthetic-observed spans, and near-zero change on static regions. Detector recall on the real held-out frame sets the ceiling for all of these, so it must be reported per camera and per lighting condition.

The protocol should rest on a fixed baseline ladder: hold-last-frame, crossfade, plate plus linearly translated sprites, classical flow, learned VFI, then generative inbetweening. A synthesized mode should not publish unless it beats the plate-plus-sprite baseline on foreground error, recall and count at the deployed gap, without phantom people. On the project's own feed, held-out triplets must use capture-time fractions. Dropping a middle frame doubles the gap to about 4 s, so these tests are conservative proxies. Dense static-camera footage tests the deployed 2-s gap more directly. MOT17 subsampled 1-in-60 and UA-DETRAC or MOT20 subsampled 1-in-50, downscaled to 352×240 with JPEG damage, allow scoring at t = 0.25, 0.5 and 0.75 against true frames. UA-DETRAC is the closest vehicle analogue, with static elevated cameras at 25 fps ([UA-DETRAC, arXiv 1511.04136](https://arxiv.org/pdf/1511.04136.pdf)). Several MOT17 sequences have moving cameras and must be excluded ([MOT17](https://motchallenge.net/data/MOT17/)). The dataset tool deliberately never loads MOT17's ground-truth tracks ([worker/prepare_motion_dataset.py](/worker/prepare_motion_dataset.py)). Those tracks are exactly what is needed to measure association quality, and the current evaluation of 2–4 held-out cases on two cameras ([holdout-benchmarks.json](/docs/holdout-benchmarks.json)) cannot rank anything.

Presentation obligations have tightened, and this is not legal advice. EU AI Act Article 50 transparency duties have applied since **2 August 2026**. The Act defines a deepfake as content that "resembles existing persons, objects, places, entities or events" and would falsely appear authentic, and a voluntary Code of Practice was finalized on 10 June 2026 ([European Commission](https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content)). A synthesized clip of a real NYC intersection beside real camera stills plausibly fits that definition if it reaches EU viewers. Secondary summaries of the Code say labels should be perceivable at first exposure and embedded where possible ([Lewis Silkin](https://www.lewissilkin.com/insights/2026/07/24/the-eus-new-ai-labelling-rules-what-every-organisation-needs-to-know-102ne35)). C2PA 2.2 supports MP4 manifests with AI source types ([C2PA 2.2](https://spec.c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification.html)), but platforms commonly strip metadata. A burned-in per-frame label such as "SIMULATED between 01:49:16 and 01:49:18", plus distinct timeline marks for recorded and synthetic spans, therefore beats a mode name alone. Viewers judge speed changes in animated dots poorly ([Koval et al., IEEE TVCG 2024](https://hal-lara.archives-ouvertes.fr/HCI-SORBONNE/hal-04552326v1)), and single summary displays invite misreadings that ensemble displays avoid ([Ruginski et al.](https://www.sciencedaily.com/releases/2017/10/171005151140.htm)). A caption stating that path and speed between stills are unknown is warranted, and showing faint endpoint ghosts beats implying one confident path.

The privacy risk at 352×240 comes less from faces than from what the pipeline adds. Faces here are a few pixels across, far below the roughly 12-px inter-pupillary distance at which one patent reports face recognition collapsing ([US patent 10002286](https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10002286)). Upscaling and generation, however, can invent sharp but fictional detail, and the project's own 8× review saw a small figure merge ([upscale8-review.json](/docs/upscale8-review.json)). The "AI Enhanced" mode therefore needs the same persistent label, and person regions should not be upscaled. NYC DOT states that its cameras "only provide live feeds and do not record any footage" ([NYC DOT](https://www.nyc.gov/html/dot/html/motorist/atis.shtml)). A 259,200-image daily archive with reconstructed trajectories creates records the city says it does not keep. NYC.gov materials are "all rights reserved" ([NYC.gov Terms of Use](https://www.nyc.gov/home/terms-of-use.page)), and in 2024 DOT sent a cease-and-desist to the Traffic Cam Photobooth art project ([404 Media](https://404media.co/traffic-cam-photobooth-cease-and-desist)). Short retention, no long per-person tracks, attribution, and documented permission are prudent before scaling capture. Model licenses also constrain the stack. ABC-Inter is CC BY-NC-ND 4.0; SPEED and TGI ship no license; Wan2.1/2.2, VACE, Wan-Move, ATI, Go-with-the-Track, FILM and SAM 2 are permissive ([ABC-Inter](https://github.com/SHH-Han/ABC-Inter); [TGI](https://github.com/teunchoi/TGI); [Wan2.2](https://github.com/Wan-Video/Wan2.2)).

## Conclusion

The project framed its task as generating the missing frames. The data supports a different task: reconstruct a scene and state what is known about it. That shift moves the cheapest, highest-value work out of model research and into measurement: reading a timestamp the camera already writes, keeping the frames the worker already knows how to archive, and clicking four ground points per camera. The clean-plate hypothesis pays off indirectly. It does not make synthesis easier. It makes synthesis unnecessary for about 97% of a Times Square frame and turns the remaining 3% into a tracking problem, where physical priors such as walking speed, lane direction and ground-plane depth order do work that no 2026 generator can do for 25-pixel people.

The likely end state is a labelled, partly diagrammatic reconstruction rather than photoreal video. That is a feature for a public site showing real places and real people. Every pixel comes from an observation, every assumption is visible, and ambiguity is shown as ambiguity. If the deterministic compositor cannot beat crossfade on object-level metrics, the current "Motion withheld" default remains the correct public answer. The results would then become the evidence that 0.5-fps cameras support stills, not motion.

## Prioritized recommendations

| # | When | Action | Why | Done when |
|---|---|---|---|---|
| 1 | Now | Parse capture time in `worker/batch.py` from the Axis JPEG comment (`0x0A 0x01`, seconds + 1/100 s), falling back to EXIF `DateTimeOriginal` + `SubsecTimeOriginal`. Keep raw JPEG bytes; keep receipt time only as provenance | Receipt intervals are off by 26% on average; the benchmark interval is 2.07 s, not 1.48 s | `capture_time` is populated for all six cameras and `frames.json` uses capture time |
| 2 | Now | Re-capture the Canal samples with the 1.0-s poller. Re-run held-out tests at capture-time fractions. Correct the research page's 1.48-s, Allen "0.5 ms" and LDF-anchor statements | The Canal samples skip every other refresh; the Allen held-out target sits about 1 s off the midpoint | Regenerated holdout JSON and corrected docs |
| 3 | Now | Pause new generative and VFI trials and further 8× upscaling | Thirteen systems, zero approvals; crossfade ties or wins | Items 4–8 exist |
| 4 | Next | Host the capture worker. Retain several days per camera across day, dusk, night and rain, under a stated retention limit | Plates and evaluation need thousands of frames; the repo has 10 per camera | Published curve of plate coverage against frame count, per camera |
| 5 | Next | Build the plate bank from `estimate_background`: HUD-excluded registration, time-of-day and illumination bins, per-frame gain/bias, coverage maps, dynamic-region masks for billboards and signals. Adopt one road-lettering rule | A four-frame plate already gave zero background change; medians fail under light change | Plate error on held-out low-occupancy frames is reported |
| 6 | Next | Isolate foreground by plate differencing, gated by a detector and refined with box-prompted SAM 2 on 2–4× crops; plate-based alpha; shadows by intensity ratio to the plate | Text-prompted SAM 3.1 missed the moving person in 14 of 14 frames | Detection recall per camera, day and night |
| 7 | Next | For each camera, annotate a ground homography from four or more points, lane polylines with direction, sidewalk polygons and static occluders in `worker/cameras.json` | Enables 3 m/s gates, no-reversal paths and depth ordering | Six calibrated cameras, each checked against standing-pedestrian heights |
| 8 | Next | Build the evaluation harness: the baseline ladder (hold, crossfade, plate plus linear sprites, then flow, VFI, generative) scored on object-level metrics; MOT17, MOT20 and UA-DETRAC subsampled to about 2 s at 352×240 with JPEG damage, using their ground-truth tracks; splits by camera and time block | Whole-frame MAE rewards an erased person; 2–4 cases cannot rank methods | Per-method results table published on this page |
| 9 | Then | Associate anonymous objects offline within each camera (min-cost flow or K-best; speed, lane, size and colour costs; appearances and exits allowed), then smooth with constant-velocity Kalman plus RTS, or along the lane. Mark ambiguous objects unknown | Every successful trial so far relied on manual correspondences | IDF1 and ID switches on subsampled MOT; share marked unknown reported |
| 10 | Then | Build the deterministic compositor: rigid vehicle sprites with perspective scaling, pedestrian amodal sprites with a procedural walk cycle and locked colours, short keyframe crossfades, and fades for unmatched objects | Gait is aliased at about two strides per gap; generators melt small movers | Beats crossfade and the linear-sprite baseline on object metrics and passes visual review |
| 11 | Before any motion release | Burn a "SIMULATED" label into every synthetic frame, mark recorded and synthetic spans differently on the timeline, label AI Enhanced too, attach C2PA to exports, attribute NYC DOT, document permission, and never upscale people | EU Article 50 deepfake definition; NYC DOT's 2024 cease-and-desist precedent | Labels visible without interaction |
| 12 | Optional, later | Only if item 10 looks too synthetic, try Go-with-the-Track, Wan-Move or ATI on upscaled crops on hosted GPUs, with gates on colour histograms and outside-mask pixels. Update the literature table's license notes | Go-with-the-Track is the only open multi-reference option; these models need 40 GB or more of GPU memory | Must beat item 10 on the same metrics |