Hedra AI All articles
Experiments

Seven Ways to Say No: What Seven Media APIs Actually Do When Your Prompt Trips a Filter

Hedra AI
Seven Ways to Say No: What Seven Media APIs Actually Do When Your Prompt Trips a Filter

Photo: Sucheta Ghoshal, CC BY-SA 4.0, via Wikimedia Commons

Everyone budgets for the happy path. Nobody budgets for the moment a filter decides your prompt is a problem and simply... doesn't tell you. Over two weeks in October and November, we ran an identical set of 240 prompts through seven generative media APIs — three image, two video, two voice — and logged every response verbatim. Not to name names (the vendors move too fast for that to age well), but to build a taxonomy of what a refusal actually looks like in practice. Because it turns out a refusal isn't one thing. It's at least five, and only one of them is honest.

The Setup

We built a deliberately boring prompt library: 120 innocuous requests (a bowl of soup, a kid's birthday cake, a dog catching a frisbee at a beach in Delaware), 60 edge cases (medical illustration, historical violence in a textbook context, a character described as "exhausted and gaunt"), and 60 prompts we fully expected to be blocked (we're not going to detail those, and you can guess the shape of them). Each prompt went out ten times to catch flakiness, since some filters are stochastic themselves. Total: 16,800 calls. We logged status code, latency delta, returned text, and — critically — whether the output was a generated asset, an error, or something in between.

Five Flavors of No

The hard block. A 4xx, a policy code, a human-readable reason. This is the version everyone imagines. It accounted for about 40% of our rejections. It's also the only mode you can build a real UI around, because it tells you what happened.

The silent rewrite. The API returns 200, bills you, and hands back an image that is not what you asked for. In one case a prompt mentioning a hospital corridor came back as a hotel lobby — same composition, same lighting, wrong room. No warning. If you weren't diffing prompts against outputs, you'd ship that to a client and find out on a Tuesday.

The soft degradation. No error, no rewrite, but the output is visibly worse — flatter lighting, mangled hands, a voice that drops its prosody on the second sentence. We saw this most on video: prompts near a policy boundary returned shorter clips with more artifacts, as though the model had been nudged toward safety at the cost of quality.

The latency throttle. Some stacks don't refuse, they just get slow. Median latency for our edge-case set was 2.3× the innocuous set across the seven APIs, and one video vendor consistently pushed boundary prompts past the 90-second mark before returning a result. That's a refusal wearing a stopwatch.

The post-hoc moderation flag. Asset generates fine, then a separate field in the response says it was flagged after the fact. If your client code only reads the asset URL, you never see it. Two of the seven APIs do this, and neither documents it prominently.

The Birthday Cake Problem

False positives deserve their own section because they cost the most in production. Our least controversial prompt — a six-year-old blowing out candles on a sheet cake — got blocked or rewritten by three of the seven APIs on at least one run out of ten. One returned a cake with no child. One returned an error referencing a policy category we still can't map to anything in its documentation. The third rewrote the child into an adult, which is a strange thing to do to a birthday party.

Other hits: a prompt about a surgical training mannequin (blocked by two image APIs), a voice line reading a prescription dosage as part of a test script (blocked by one voice API, allowed by the other), and a video prompt for a boxer training on a heavy bag (returned, but with the contact frame removed, which made the clip physically nonsensical).

None of these are scandals. They're the predictable cost of running classifiers on prompts instead of intent. But they're invisible until you go looking.

Where the Policies Genuinely Diverge

The gaps between vendors were widest on voice. Two voice APIs, same script: one refused a line describing a tense argument between two adults, the other synthesized it without comment. Multiply that by every script your product might generate and you've got a real portability problem — the same feature works on vendor A and quietly fails on vendor B.

Video was second. We saw one stack accept a prompt about a crowded street protest and return a clip of an empty plaza — the silent rewrite again, this time at 24 frames per second. Same prompt on a competitor returned a normal crowd. Neither vendor's docs mention crowds as a category.

Image models were the most consistent, which tracks — they've had the most years of policy pressure.

A Test Harness You Can Steal

Before you wire a media API into anything users touch, run something like this:

  1. Build a three-tier prompt set. Mundane, edge, and expected-fail. Sixty each is enough to see patterns.
  2. Repeat every prompt at least five times. Filters flake. A single pass tells you nothing.
  3. Log the raw response object, not just the asset. Grep for moderation fields you didn't know existed.
  4. Diff prompt against output description using a cheap vision model. Silent rewrites show up here first.
  5. Track p50 and p99 latency separately per tier. The throttle is real and it's tiered.
  6. Test the same prompt on two vendors in the same category before you commit to either.
  7. Write down what a refusal should look like in your UI — because 60% of the time, the API won't give you one.

What We're Changing Internally

The lesson wasn't that filters are too strict or too loose. It's that a refusal is a response type, and if you don't handle it as one, your pipeline will handle it for you — badly. We've started treating safety metadata as a first-class field in our asset records, alongside provenance and cost. When a prompt comes back rewritten, we want to know it happened, log it, and decide what to show the user.

That's the whole audit, really. The no is not an edge case. On a long enough timeline, it's a meaningful percentage of your traffic, and you should know what it looks like before your users do.

All Articles

Related Articles

Aspect Ratio Is a Contract: Auditing Ten Image APIs for True Vertical Output

Aspect Ratio Is a Contract: Auditing Ten Image APIs for True Vertical Output

Your Endpoint Is Not a Constant: Thirty Days of Watching Hosted Media APIs Shift Under Us

Your Endpoint Is Not a Constant: Thirty Days of Watching Hosted Media APIs Shift Under Us

Alt Text Is Not a Vibe Check: Auditing What Generative Models Say About Their Own Output

Alt Text Is Not a Vibe Check: Auditing What Generative Models Say About Their Own Output