Numbers, not adjectives

Rating prediction — measured

We harvested the official CARA rating rationale for 4,755 films from the MPA's own register (88% of our post-1990 corpus; the practice barely existed before 1990, so our scope is post-1990 wide releases, stated plainly). Of these, 4,544 parsed cleanly into intensity-category descriptors — the CARA descriptor corpus that is the basis for the per-descriptor rating marginals a report cites and for the model below. We report a marginal only at n ≥ 20; below that it is an anecdote, not a base rate, and the report falls back to the broader parent category. (The same official rationales, embedded, also power the nearest-comparables search on the rating panel and the What-If simulator — one corpus measures the decision boundary and locates your neighbourhood. A broader 6,302-film content-profile corpus remains behind the first-look pitch comparables, where topical similarity is the right instrument.) The model carries a distribution-free coverage guarantee (Mondrian split conformal):

held-out films: 937 (deterministic split, never trained on)
conformal coverage: 92.2% aggregate against a 90% target — guarantee per pooled group
top-1 accuracy: 83.9% · multiclass Brier: 0.239
prediction sets: mostly single ratings; boundary scripts get honest pairs like {PG-13, R}

The reliability diagram above is the unflattering version of that table, published on purpose. Dot size is bin population (71% of holdout films land in the 0.9–1.0 bin, where confidence and accuracy agree at 93%). In the thin middle bins the raw probability runs overconfident — a 0.74 answer is right about 54% of the time — which is exactly why a report never ships the raw probability as the claim: the conformal set is the guarantee, and it widens to an honest pair precisely where these dots sag below the diagonal.

Two findings from the same data: the R-line for language is the modifier, not the category — "pervasive language" lands R 99% of the time (n=187) while "strong language" lands PG-13 89% (n=384) — and the descriptor→rating mapping drifts measurably across decades: a model trained only on pre-2010 films loses 5.0 points of accuracy on post-2015 films (an earlier, narrower descriptor vocabulary measured 1.1 — the richer vocabulary sees drift the narrow one was blind to). The model you get is trained on the full corpus; recency weighting is a candidate improvement, and if we ship it, this number is the reason.

What the aggregate hides — the per-class table, published on purpose: G-rated wide releases are nearly extinct post-1990 (1 calibration film) and NC-17s barely exist (6), so a single-class 90% guarantee is mathematically unattainable for them — a fact we found stated nowhere until an external review computed it. The guarantee is therefore constructed and stated per pooled rating group — {G, PG}, {PG-13}, {R, NC-17} — each of which clears 90% on held-out data; thin-class rows below are best-effort and labeled, not guaranteed.

The population boundary: the guarantee is calibrated on post-1990 US wide-release features with published CARA rationales, and it is a guarantee about that population. A documentary short, an unrated festival cut, or a pre-1990 sensibility sits outside it — the prediction still renders, but the 90% number does not travel with it.

Cut guidance — grounded, validation growing

The cut list ("these changes buy the lower rating") is checked against a second regulator's paper trail: the BBFC publishes, per film, which cuts were made and what category they achieved. Our harvest holds 436 such records — including 57 that state the full trade ("uncut 15 was available; cuts achieved 12A") — the exact transaction our cut simulator predicts, documented by the regulator that watched it happen.

Descriptor extraction — pending, and it is the load-bearing wall

Everything above measures the decision layer: given descriptors, the boundary is measured and the coverage guarantee holds. But the descriptors come from a model reading your script, and that extraction step is not yet measured. If the desk reads your drug content as "some" where CARA would say "pervasive", the 92% coverage does not save the prediction — the guarantee is conditional on correct extraction, and we say that here in plain words rather than in a footnote. The planned measurement is validation against hand-labeled scripts; until it runs, "our model is calibrated" is the claim we make, and "our reports are accurate" is the claim we do not.

Run-to-run consistency — measured, and the news is mixed

Same script, same answer? We measure it by running the entire pipeline multiple times and matching findings across passes — and we publish what came out, not what we hoped. Judgment is consistent: when two passes examine the same item they agree flag-vs-clear 36 times in 37 on the feature-length measurement — raw agreement 97%, Gwet's AC1 = 0.97 (α = 0.93 on the short fixture; at feature scale Krippendorff's alpha degenerates because ~95% of co-examined items are cleared, leaving no variance for the chance correction — AC1 is the chance-corrected statistic built for exactly that regime, and we report all three numbers plus the prevalence so you can see why). Coverage is not yet consistent: across two complete feature-length passes, only 5 of the 16 distinct findings were surfaced by both — different passes chase different threads. And one pass in three had two desks return no dispositions at all, a failure the pipeline now refuses to grade or ship (it renders as an INCOMPLETE banner instead of silence). Coverage convergence is the current engineering target; this section will carry the numbers as they move.

Recall — pending, and we will not fake it

What fraction of findable issues do we find? Honest answer today: we do not publish a recall number yet. The estimator (capture–recapture across independent passes) is built, but it is only honest after validation against independently hand-labeled scripts — and its known biases point optimistic, so when it ships it ships as an upper bound ("no better than X%"), never a point estimate.

The score and the estimates — labeled for what they are

The Greenlight Score is a deterministic ordinal risk index — a severity summary of the cited findings, not a probability and not calibrated; compare drafts by findings, not by small score moves. Cost ranges are desk-estimated and summed as independent remedies (dependencies not modeled). Each flag's "desk self-assessment" percentage is the model's own uncalibrated confidence — displayed for transparency, used by nothing. The calibrated number on this page is the rating boundary; these are not, and we label the difference.

Verification — measured per run

Every finding faces a blinded verifier — claim and citations only, never the desk's reasoning — and rejections are shown on every report with the verifier's reason. Across recent runs the verifier rejects roughly 10–30% of draft findings. A finding rejected for citation quality (rather than substance) now gets exactly one re-sourcing round with fresh evidence and a second blinded verdict; each report carries its attempted/recovered counts. The aggregate recovery rate will be published here once enough production runs exist for it to be a statistic rather than an anecdote — until then it stays per-report.

What we do not claim

• This is a research pre-screen, not a legal opinion, and it does not satisfy E&O requirements — production counsel reviews before production relies.
• No accuracy number appears here before its measurement exists. When a label above says pending, that is the whole claim.