PHOTO HOLDOUT

Frozen before generation · Every first attempt

New photos, separate results.

36 development outcomes and methods · Earlier pilots and failures

6 planned renders on 2 synthetic photos

The source photos were unused in the available prior project records. The catalog products were already used in development. This small study tests new photos of known products; it does not establish performance on shoppers or new products.

6 / 6first-attempt tasks succeeded; 0 failed, 0 uncertain, 0 unattempted
4 / 4plain-top alternatives within ΔE 10.0; 0 missing
2patterned reference tops excluded from single-colour scoring
2source photos, with three correlated renders each
0independent human rubric reviews; requested change assessed in 0 / 4 comparisons
12 unitsreconciled against task history; 958 remaining at the post-run check

Colour closeness is not overall fidelity. An unblinded AI observation flags pushed-up sleeves in one sage-top result. The catalog says long sleeves; this image does not verify the requested coverage. Independent product-detail and requested-change ratings remain unassessed.

Executed 2026-10-29T04:09:35.633530+00:00. No retries, corrections or selected rerenders. All six first outputs are below. No combined accuracy score is calculated.

All planned outcomes

CasePhoto / productStateColour ΔEElapsedUnits
heldout-v1-01model-d / Midnight Floral Blousesucceedednot scored20.6 s2
heldout-v1-02model-d / Sage Knit Topsucceeded0.920.1 s2
heldout-v1-03model-d / Ribbed Turtlenecksucceeded3.719.6 s2
heldout-v1-04model-e / Midnight Floral Blousesucceedednot scored20.3 s2
heldout-v1-05model-e / Sage Knit Topsucceeded5.519.3 s2
heldout-v1-06model-e / Ribbed Turtlenecksucceeded2.519.6 s2

Elapsed time includes upload and polling. Two units were charged per task. Development measurements retain their own denominators.

Reproduce the measurements

The design, measurement code and thresholds were committed before generation. Reporting glue was added afterwards. This command checks the frozen hashes and recomputes measurements from the exact public normalized inputs and stored output JPEGs, without an API key, database data or private images.

DATABASE_URL=sqlite:///local.sqlite3 .venv/bin/python manage.py evaluate_heldout --check

Download results and input hashes (JSON) · Registration commit

Protocol heldout-v1. Frozen manifest SHA-256: 9374a8edfd0040871446b5df48c19f0a4cb0b716f2a085ab180dab36a776dcc6.

The immutable registration still says “not run”: it records the pre-run plan, not current execution status. Results are recorded separately here. Private task identifiers are hashed in the public record.

heldout-v1-01 · reference · model-d

Midnight Floral Blouse

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-01

Synthetic source photo for heldout-v1-01
Exact normalized input photo
Catalog Midnight Floral Blouse
Catalog product
First YouCam output for heldout-v1-01
Uncorrected first output

Colour: pattern. Patterned top: the preview may simplify the pattern. Check the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

heldout-v1-02 · alternative · model-d

Sage Knit Top

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-01

Synthetic source photo for heldout-v1-02
Exact normalized input photo
Catalog Sage Knit Top
Catalog product
First YouCam output for heldout-v1-02
Uncorrected first output

Colour: close, ΔE 0.9. Colour check: close to the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

heldout-v1-03 · alternative · model-d

Ribbed Turtleneck

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-01

Synthetic source photo for heldout-v1-03
Exact normalized input photo
Catalog Ribbed Turtleneck
Catalog product
First YouCam output for heldout-v1-03
Uncorrected first output

Colour: close, ΔE 3.7. Colour check: close to the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

heldout-v1-04 · reference · model-e

Midnight Floral Blouse

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-04

Synthetic source photo for heldout-v1-04
Exact normalized input photo
Catalog Midnight Floral Blouse
Catalog product
First YouCam output for heldout-v1-04
Uncorrected first output

Colour: pattern. Patterned top: the preview may simplify the pattern. Check the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

heldout-v1-05 · alternative · model-e

Sage Knit Top

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-04

Synthetic source photo for heldout-v1-05
Exact normalized input photo
Catalog Sage Knit Top
Catalog product
First YouCam output for heldout-v1-05
Uncorrected first output

Colour: close, ΔE 5.5. Colour check: close to the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

AI observation, not blinded. The sleeves appear pushed up on this render, exposing forearms despite the long-sleeve product. A close colour measurement does not verify requested sleeve coverage. This is an unblinded AI observation, not a human rubric rating. Reviewer: Codex (AI assistant).

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

heldout-v1-06 · alternative · model-e

Ribbed Turtleneck

succeeded · 1 attempt · YouCam cloth-v4 · compared with heldout-v1-04

Synthetic source photo for heldout-v1-06
Exact normalized input photo
Catalog Ribbed Turtleneck
Catalog product
First YouCam output for heldout-v1-06
Uncorrected first output

Colour: close, ΔE 2.5. Colour check: close to the product photo.

Outside-region pixel change: 0.0%. Fixed approximate boxes and threshold 40; a zero does not establish identity, pose or background preservation.

Independent human review pending. Product detail, requested difference and misleading-result judgments are unassessed.

Limits