The Weights Are Not the Recipe: Reproducing an Open-Weights Generative Media Paper

Open weights are a gift. They are not a recipe. That distinction hit us last month when we tried to reproduce the headline results from a recent generative media paper that shipped a checkpoint, a model card, and a PDF—but no reference implementation. The weights downloaded fine. The numbers did not.

We are not naming the paper because the point is not to dunk on a team. The point is that this is now a common pattern in generative media: release the artifact, skip the pipeline, and let reproducibility become a community archaeology project. If open science means 'you can download it,' we are using the word too loosely.

What was actually released

The release included a 7B-ish diffusion transformer checkpoint, a model card with a few sample prompts, and a table claiming state-of-the-art scores on a text-to-image benchmark. Missing: training code, inference code, preprocessing scripts, scheduler config, evaluation prompt set, and the exact metric implementation. The model card said 'standard DDIM-like sampling' and 'standard preprocessing.' Those two phrases cost us three days.

Our reproduction setup

We built a minimal pipeline in PyTorch: load checkpoint, tokenize prompts with the stated text encoder, run a scheduler we guessed from the paper, decode latents with the released VAE, and save PNGs. Hardware was a single 80GB GPU, bf16, batch size 1. We started with 50 steps, guidance 7.5, a 512×512 center crop, and the prompt template from the model card. Then we compared 200 generated images against the paper's sample grid and ran the benchmark metric we reconstructed from the appendix.

Our first pass landed about 9% below the headline score. Not catastrophic, but enough to change the story. After matching a few hidden details, we closed most of the gap. Here is where the missing milliseconds—and missing points—went.

The details that moved the numbers

Preprocessing. The paper said 'resize and normalize.' We tried bilinear and bicubic resizing, with and without center crop. Bicubic plus a 512 crop improved our metric by roughly 4%. A different normalization range moved sample brightness enough that our human raters noticed. Preprocessing is not plumbing. It is part of the model.

Scheduler. 'DDIM-like' is not a config. We tested 25, 50, and 100 steps, two beta schedules, and eta values of 0 and 1. The 25-step run looked soft; the 100-step run was slower and barely better. The paper's sample grid matched our 50-step cosine schedule most closely, but we cannot prove that was the authors' setting.

Prompt formatting. The model card showed prompts with a style suffix. We initially omitted it. Adding the suffix changed composition and raised our metric by about 2%. Negative prompts were mentioned once in the appendix and never specified.

Evaluation prompts. This was the biggest hole. The paper reported a score on a benchmark, but the exact prompt set was not released. We recreated 200 prompts from the benchmark's public examples and wrote 50 of our own. That is not the same test. Our numbers are directionally useful, not directly comparable.

Precision and seeds. bf16 versus fp16 changed a handful of outputs. Seed policy was unspecified. We ended up reporting a mean over 10 seeds because single runs were noisy enough to mislead.

How close did we get?

After matching preprocessing, prompt suffix, and a 50-step cosine schedule, our reconstructed metric landed within about 2% of the headline. That is close enough to believe the weights are real and far enough to know the paper is not reproducible as shipped. We also found two failure modes the paper did not mention: hands at small scale and text rendering at long prompt lengths. Neither is a scandal. Both are the kind of thing a reference implementation would have surfaced in an afternoon.

A reproducibility checklist for generative media papers

If you release weights, release the recipe too. A practical checklist:

• One-command inference. Include a script that takes a prompt and writes an image or video with pinned dependencies.
• Exact preprocessing. Resize method, crop, color space, normalization range, and any augmentation at inference time.
• Scheduler config. Steps, beta schedule, eta, guidance scale, and any custom noise injection.
• Prompt templates. The exact formatting, system prompts, negative prompts, and truncation rules.
• Evaluation assets. Prompt set, seeds, metric code, and reference outputs or checksums.
• Hardware and precision. GPU type, dtype, batch size, and library versions.
• Smoke test. A tiny input with an expected output hash or perceptual distance.
• Known failures. A short list of what the model is bad at, written by the authors.

This is not bureaucracy. It is the difference between a checkpoint and a result. For developers, a missing pipeline means a week of guessing before you can even evaluate whether the model fits your product. For researchers, it means citations that cannot be checked. For everyone, it means 'open' becomes a vibe.

What we changed in our own work

We now ship a repro/ folder with every internal model release: a Dockerfile, a prompt file, a seed list, and a make check target that regenerates three sample outputs and compares them to stored hashes. It takes an afternoon. It has already saved us from two silent dependency upgrades.

Open weights are a starting point. Reproducibility is the product. If a paper cannot survive a determined developer with a free weekend and a single GPU, the headline number is not a result yet. It is a rumor with a download link.