# Benchmark plan: lower-cost refinement for Canal × Allen

Status: planned. No alternative GPU inference has been submitted. The matched reference files have been downloaded and verified by SHA-256.

Playback fix: [f481f2a](https://github.com/edisonmoy/spatial-traffic/commit/f481f2a8f21869e12bacffd42b9920632742969b). Freeze this source revision for the experiment. Keep the production viewer and its accepted frames unchanged during testing.

## The decision we're trying to make

Can we run the refinement stage for less money, while keeping the quality people already accepted?

Start with SeedVR2. Keep Kling's proposed motion, the reviewed SAM masks, clear background repairs and compositor fixed. If several parts change together, we won't know which one caused a better picture or a new artifact.

The larger crop currently costs an estimated $0.07064064 to refine on fal. The estimate follows fal's published output-pixel rate; it isn't an invoice. The complete segment's conservative estimate is about $0.54, with Kling accounting for most of it. A 30% saving on that refinement would save roughly two cents on this segment, about 4% of the estimated whole pipeline. We should measure honestly before building another service.

## What changes in the comparison

The same submitted crop enters each path. Only the refinement implementation changes.

~~~mermaid
flowchart LR
  I["Pinned input crop"] --> F["Cached fal refinement"]
  I --> S["Self-hosted SeedVR2"]
  F --> A["Frozen compositor"]
  S --> B["Same frozen compositor"]
  A --> R["Anonymous quality review"]
  B --> R
  S --> C["Billed cost and runtime"]
~~~

## Fixed test cases

The machine-readable fixture manifest is [cost-quality-benchmark-cases.json](cost-quality-benchmark-cases.json). It pins the exact compressed input and reference output bytes, dimensions, frame count, URLs and settings.

| Case | Submitted input | Cached fal output | Reference refinement cost |
| --- | --- | --- | ---: |
| Small right-crosswalk pedestrians, span 00 | 352 × 376, 73 frames | 576 × 608, 73 frames | $0.025565184 |
| Opposing pedestrians and dog, span 02 | 1040 × 348, 73 frames | 1680 × 576, 73 frames | $0.07064064 |

Both clips run at 24 fps. Their last decoded frame is at 3.000 seconds; the 73-frame container lasts about 3.0417 seconds. Record that distinction when another provider rounds billing by duration.

These are the actual submitted proposal crops and raw refined outputs. Don't crop a finished composite and feed that back into the model as a replacement input.

The accepted taxi remains a protected-region control in span 00. That region must stay byte-identical before encoding. We don't have a matching taxi-only refinement pair in this fixture set, so this experiment won't claim to benchmark taxi refinement.

The crop output dimensions include the original provider's rounding. Use the same placement boxes when recomposing. Keep the source anchors, mask holes, occlusions and held timestamp bitmaps unchanged.

## Candidates and price assumptions

| Candidate | Role | Price checked October 20, 2026 |
| --- | --- | --- |
| Existing fal SeedVR2 results | Cached quality and cost reference; no new baseline request | $0.001 per output megapixel, including every frame |
| Modal L40S, 48 GB | First self-hosted candidate | $0.000542 per GPU-second, about $1.95/hour |
| Runpod RTX A6000, 48 GB | Compare the identical implementation on cheaper hourly hardware | $0.59/GPU-hour shown on the current pricing page |

Sources: [fal SeedVR2](https://fal.ai/models/fal-ai/seedvr/upscale/video), [Modal pricing](https://modal.com/pricing), [Runpod pricing](https://www.runpod.io/pricing).

Modal currently advertises $30/month of included compute. Report those credits separately. They can reduce our cash bill, but they don't make the underlying computation free. CPU, RAM, model storage and transfer charges still belong in the comparison. Recheck rates and availability immediately before execution.

A lower GPU-hour price doesn't guarantee a cheaper clip. The A6000 may take longer. Don't assume that either 48 GB card fits the model configuration until the smoke test succeeds.

## Preparation before renting a GPU

1. Verify all fixture hashes and decoded media properties. Archive the small matched inputs and references in project-controlled storage so a temporary CDN URL isn't the only copy.
2. Freeze the renderer, source observations, accepted taxi base, recipes and mask files. Hash them as well. Confirm that the existing fal refinement reproduces the accepted composites before swapping in any candidate output.
3. Build a versioned container and download weights using CPU resources where practical. Record this first-install cost and time separately from recurring worker startup.
4. Pin the model checkpoint SHA-256, code commit, container digest, dependency versions, precision, tiling, temporal window, overlap, offload and color-correction settings.
5. Configure a job deadline, a budget guard and automatic GPU shutdown. Never leave a Pod running between experiments.

Use the [official SeedVR repository](https://github.com/ByteDance-Seed/SeedVR) and a pinned [ComfyUI SeedVR2 implementation](https://github.com/comfyorg/comfyui_seedvr2) if memory optimizations are needed. Check the selected code and checkpoint terms before use.

First try the published SeedVR2-3B checkpoint in BF16, with input and latent noise disabled and seed 37. If it passes technical checks but visibly loses detail, a 7B checkpoint is a separate quality candidate within the same pilot budget. Quantization is also a separate candidate. Don't silently change precision or model size and call it the same configuration.

fal's internal checkpoint and precision may be undisclosed. Record those fields as unknown unless confirmed. Matching the exposed parameters gives us a comparable implementation, not proof of identical inference.

## Run sequence

### Feasibility and quality gate

Run one small-crop smoke test on Modal. Count its full cost, including a failed attempt.

Allow one documented memory-setting adjustment after an out-of-memory failure. Stop that configuration after a second failure. Don't automatically rent a larger GPU.

If the small crop is valid, run the harder crop and inspect both outputs before paying for repetitions. If the 3B candidate fails quality, test the 7B variant only while the remaining pilot budget supports it. Pick one configuration before comparing hardware.

### Timed comparison

For each hardware candidate that passes feasibility:

| Measurement | Runs | Setup |
| --- | ---: | --- |
| Cold small crop | 1 | Fresh worker, empty process/model memory, persistent weight cache allowed |
| Cold harder crop | 1 | A second fresh worker, same pinned configuration |
| Warm repetitions | 3 per case | Keep the second worker's model loaded; alternate case order |

That is eight timed outputs per hardware candidate, in addition to feasibility attempts. Disable completed-output caching during all timed runs. Each repetition must execute inference.

Keep the same model weights, precision and temporal settings on Modal and Runpod. If one GPU requires a different configuration, record a separate configuration rather than treating the change as hardware performance.

Distinguish model-warm from shape-warm execution. The first small crop in a worker that has only processed the large crop can trigger compilation or autotuning. Record those events and include their cost in batch economics. Do not move that work outside the bill just to improve a warm timing.

Record download, container startup, model loading, inference, encoding, upload, idle and shutdown times separately. Use provider billing records for the total; a stopwatch around the model call misses billable work.

Two cold runs and three warm repetitions per case are a pilot, not a reliable p95 latency estimate. Report each sample and the median/range. Expand the sample only if the initial comparison is close enough to justify it.

## Quality review

Compare both the raw refined crops and the recomposed full scene. The full-scene comparison must use the same reviewed masks, clear plates, placement boxes and source anchors.

Automated checks must pass before visual review:

- Exactly 73 decodable frames at 24 fps, with the expected dimensions and monotonic presentation timestamps.
- No corrupt, blank or near-white whole frames, and no unexpected dropped or duplicated frames. Expected source holds are classified separately.
- Before lossy encoding, both observed anchors equal the accepted enhanced source images.
- Protected pixels outside the allowed refinement regions remain exact. The span 00 taxi is unchanged.
- The on-frame timestamp and stationary street text use the existing deterministic compositor rules. They don't pass through a new text-generation step.

Review all frames, plus looping playback at normal size and enlarged crops. Use anonymous A/B labels and vary their left/right placement. Inspect feet, dog contours, poles and occlusion boundaries closely.

Reject a candidate that introduces material new trails, flicker, body scaling, fading, extra limbs, reversed motion or malformed objects relative to the accepted reference. A sharper still doesn't compensate for worse motion. Repeated runs must also avoid intermittent failures.

If possible, have a second person review the loops without provider labels. Record pass/fail, the reason, exact frame indices and review/repair minutes. With one reviewer, report that limitation.

Pixel differences, PSNR, SSIM or perceptual distances measure similarity to fal. They cannot prove the unseen motion is correct. Any flow-based flicker metric must account for occlusion rather than penalizing real movement. Use these numbers to locate problems, not as the release decision.

## Cost accounting and batch scenarios

Report two costs: actual cash charged after credits, and the undiscounted economic cost at the provider's rate. Include failed attempts, retries, startup, CPU/RAM, storage, transfer, idle time and shutdown.

Use the same accounting boundary on both sides. Include common compositing and delivery costs for both providers, or show why they cancel. Charge extra self-hosted encoding, orchestration or storage to the candidate when fal already includes that service.

For an executed batch:

    cost per accepted processing output =
      total attributable batch charges / outputs that pass quality review

Keep the total pilot spend visible as well. Repeated benchmark outputs don't create additional unique footage for the project.

For a recurring nightly forecast:

    batch cost =
      one worker startup and model load
      + sum of measured inference costs for the actual crop mix
      + CPU, storage, transfer and observed failure/retry allowance

Show a single-clip run, the two-case batch, and a six-request load test with three copies of each fixture. A six-request test has a fal refinement reference of $0.288617472. Label it as a representative load test, not the measured cost of the six unfinished Allen intervals; their eventual crop sizes may differ.

Validate the six-request projection against the measured warm batch. If the candidate looks competitive, run a fresh six-request batch to confirm end-to-end billing before migration. Include that confirmation in the same budget guard or stop when it won't fit.

GPU-only break-even for the larger crop is about 130 seconds on Modal L40S or 431 seconds on Runpod A6000. For the small crop it is about 47 or 157 seconds. CPU and other costs shorten those allowances. These are mathematical ceilings, not measured model runtimes.

## Proposed pilot budget and stop rules

Proposed total cap: $5 at undiscounted rates, including credits consumed. This is a planning ceiling, not an instruction to start paid jobs.

Allocate up to $0.25 to CPU preparation/storage, $2.75 to Modal and $2 to Runpod. All feasibility attempts, memory retries and confirmation runs consume those same allowances. Start with Modal; test Runpod only after a configuration passes quality.

Give retained model volumes a 24-hour pilot TTL or an explicit continuing storage allocation; include storage beyond the experiment in the report. Budget exhaustion means an incomplete benchmark, not a passing result.

Set a 15-minute maximum for any inference attempt and a separate overall allocation deadline. Stop sooner if the budget guard would be exceeded. Include provider billing granularity and shutdown delay in the guard.

Stop a configuration on repeated memory failure, invalid output, clear quality regression, or evidence that even GPU-only cost exceeds the fal reference without a credible remaining optimization. Record the failure; don't hide it by retrying indefinitely.

Don't submit new Kling proposals, regenerate SAM masks, process another camera or launch training for this benchmark.

## Decision and deliverables

Migrate refinement only when both cases pass visual review, all technical checks pass, and the measured expected batch cost is at least 30% below the corresponding fal refinement cost before promotional credits. A 50% saving is preferable if the new service needs meaningful maintenance.

Passing these two cases supports this Allen refinement configuration at these crop sizes. It does not establish quality for other cameras, lighting or vehicle refinement. Review each of the remaining Allen outputs before widening automatic routing.

Report savings per case as well as for the weighted mix. A cheaper large-crop batch can hide a more expensive small-crop path; routing can differ by case.

Report cold one-clip economics separately. If self-hosting wins only on batches, route only those batches to it. If fal wins at our current volume, keep fal.

Deliver a run ledger with configuration IDs and provider receipts, output hashes, synchronized A/B loops and contact sheets, a quality decision for every output, and a cost table with measured and projected values clearly marked. Keep candidate outputs out of the viewer until review is complete.

Only after this decision should we benchmark SAM on the same cached proposals. Replacing Kling is a later experiment: it changes the motion itself and needs its own endpoint and held-out-observation checks. Existing rejected Wan/LTX experiments remain useful negative examples; a cheaper request is not an accepted result.