Golden-eyes review is structured subjective video quality evaluation by trained viewers. It's the human counterpart to VMAF — where VMAF gives you a number, golden-eyes gives you "the codec change introduced visible banding in dark scenes." For premium streaming, broadcast distribution, and content licensing where subjective quality matters most, golden-eyes review catches issues that automated metrics miss. This page is the engineering reference.
What golden-eyes review is
A golden-eyes panel is a group of trained viewers who:
- Have verified normal or corrected-to-normal vision and color vision.
- Are trained to identify specific video artifacts (blocking, ringing, banding, color shifts, etc.).
- Score consistently across studies (calibrated baseline).
- Operate in controlled viewing environments (calibrated displays, controlled lighting, specific viewing distance).
For comparison, "naive viewers" (general public, no training) score the "average viewer" perspective. Both have value; they answer different questions.
Golden-eyes panels do:
- Spot-check encoder upgrades — does v3.6 introduce visible artifacts vs v3.5?
- Validate codec migrations — does AV1 produce subjectively equivalent quality to HEVC?
- Catch corner cases — content types where automated metrics underestimate or overestimate quality.
- Final-deliverable QC — premium content review before release.
For most pipeline operations, automated metrics (VMAF, PSNR, SSIM) suffice. Golden-eyes is for cases where those don't.
When to use golden-eyes review
Golden-eyes review is worth the investment for:
Encoder version upgrades — before deploying a new encoder version to production, spot-check on representative content. Catch regressions VMAF might miss.
Codec migration evaluation — before adding AV1 to your ladder, golden-eyes validation that AV1 produces subjectively acceptable quality across content types.
Content licensing requirements — some licensing contracts mandate subjective quality verification.
Premium content release QC — for high-stakes releases (theatrical features, premium TV episodes), golden-eyes review before publication.
Customer escalations — when automated metrics show quality is fine but customers complain, golden-eyes review identifies the disconnect.
New encoder configurations — when introducing a new preset, parameter, or filter, golden-eyes confirms it doesn't introduce subjective issues.
For everyday pipeline operations (volume content, standard streaming), golden-eyes isn't needed. Save it for the high-stakes decisions.
Review environment
ITU-R BT.500 specifies the proper review environment:
Display:
- Calibrated reference monitor (e.g., Sony BVM, EIZO ColorEdge).
- Specific peak luminance: 100 cd/m² for SDR; 1000 cd/m² for HDR mid-grade; 4000 cd/m² for HDR premium grade.
- Proper color calibration (BT.709 for SDR; BT.2020 for HDR; D65 white point).
- Black level appropriate for the display type.
Ambient lighting:
- Subdued ambient (~10% of display brightness or less for SDR).
- Surrounding wall colors neutral gray.
- No direct light in viewer's field of view.
Viewing distance:
- 3-4× picture height for typical viewing.
- Closer (1-2× picture height) for detailed artifact analysis.
Audio:
- Calibrated playback (not from monitor speakers; reference studio monitors).
- Loudness calibrated to specification (typically -23 LUFS for broadcast review; user-controlled for general).
For pipelines doing this seriously, dedicated review rooms with proper equipment are the answer. For occasional review, a properly-configured workstation with a good calibrated display works.
Reviewer training
Trained reviewers (golden-eyes) require:
Initial training (~1-2 weeks):
- Theory of video artifacts: what causes blocking, ringing, banding, etc.
- Practical exercises: identifying artifacts in test content.
- Calibration to common scoring conventions.
- Vision testing for inclusion in panel.
Ongoing calibration:
- Periodic test content with known scoring profiles to verify reviewer consistency.
- Cross-reviewer correlation analysis.
- Recalibration as needed.
Specialized training:
- HDR-specific training for HDR content review.
- Audio-specific training for audio review.
- Codec-specific training for evaluating new codecs.
For full-time review staff at large streaming services, this is a real role. For startups and smaller services, contracted reviewers from specialized firms (Witbe, Fraunhofer, BBC R&D) provide ad-hoc capability.
Review methodology
Common methodologies:
DSCQS (Double-Stimulus Continuous Quality Scale) — show reference and test sequence; reviewer scores both on continuous scale; difference = quality cost.
Use case: comparing encoded version to source. The standard for codec quality evaluation.
SSCQE (Single-Stimulus Continuous Quality Evaluation) — show test only; reviewer scores in real-time on continuous scale.
Use case: evaluating standalone playback experience without reference.
Stimulus Comparison (PC) — show pairs of test sequences (e.g., two encoder configurations); reviewer picks the better one.
Use case: comparing two configurations head-to-head.
Hidden Reference Removal (HRR) — randomly insert reference (no encoding) into test sequences; verify reviewers consistently score reference highest.
Use case: validating panel calibration. If reviewers don't score reference highest, panel needs recalibration.
For most production review, DSCQS is the standard. SSCQE for cases where reference isn't available; PC for direct comparisons.
Sample sizes and statistics
For statistically meaningful results:
- Minimum panel size: 15 reviewers.
- Per-condition sample size: 4+ content samples per evaluation condition.
- Total trials: 50-100 per session.
- Session duration: <1 hour to avoid fatigue.
- Multiple sessions: scheduled across days to capture variability.
For a comprehensive evaluation:
- Multiple genres (sports, drama, animation, etc.).
- Multiple bitrate operating points.
- Multiple device targets (TV, mobile, etc.).
A full evaluation might span 2-4 weeks of panel time across 30-60 hours of content review.
Recording and analyzing results
For each review:
- Per-trial scores: each reviewer's score for each (content, condition) pair.
- Mean opinion score (MOS): average across reviewers per (content, condition).
- Confidence interval: 95% CI based on viewer variability.
- Outlier detection: identify reviewers whose scores deviate substantially from the panel; investigate or exclude.
For analysis:
- MOS comparison across conditions — which encoder configuration produces highest MOS?
- Per-content-type analysis — does the conclusion hold across genres or vary?
- Confidence in differences — do the differences exceed CI?
For pipeline decisions, look at both the MOS values and the confidence intervals. A difference of 0.2 MOS may or may not be statistically significant.
Outsourcing options
For pipelines without internal panels:
Specialized labs — Witbe, Fraunhofer IIS, BBC R&D, Telos. Premium pricing; expert results; lab-grade infrastructure.
Vendor-managed panels — some QA companies offer subjective video testing as a service. Mid-tier pricing.
Crowd-sourced — Subjectify, Crowd4Test. Lower cost; less controlled environment; appropriate for naive-viewer studies but not golden-eyes.
For one-off or annual evaluations, outsourcing is more cost-effective than maintaining internal panel infrastructure.
Cost-benefit analysis
Golden-eyes review costs:
- Specialized lab study — $5,000-50,000 per study depending on scope.
- In-house panel time — at $50-100/hour reviewer time × hours; for a meaningful study, $5,000-30,000.
- Equipment — calibrated displays, review rooms, calibration equipment. $20,000-100,000+ initial; ongoing maintenance.
Benefits:
- Catches regressions automated metrics miss.
- Validates premium content quality for licensing requirements.
- Reduces customer-impacting incidents from undetected quality issues.
For premium streaming services, the cost is justified by avoided incidents. For volume / mass-market streaming, automated metrics are sufficient.
Operational considerations
Things that matter for golden-eyes review in production:
- Scheduling — review studies take days to weeks. Plan in advance for major release cycles.
- Content rotation — refresh test content periodically so panels don't memorize.
- Documentation — record methodology, results, conclusions. Future engineers benefit.
- Cross-team alignment — review results inform engineer decisions; communicate findings broadly.
- Continuous calibration — periodic panel recalibration with hidden references.
- Cost tracking — review studies are expensive; track ROI vs alternative QA approaches.
Common golden-eyes review mistakes
Things that go wrong when running review studies:
Mistake 1: Insufficient panel size.
Running with 3-5 reviewers because that's what's available. Statistical significance requires 15+; results from smaller panels are noisy and not actionable.
Mistake 2: Unblinded studies.
Reviewers know which condition is which. Bias creeps in. Always blind: reviewers don't know if they're seeing config A or config B until after scoring.
Mistake 3: Fatigue.
Sessions over 1 hour produce fatigue-driven scoring drift. Schedule shorter sessions across multiple days.
Mistake 4: Memorized content.
Same content repeatedly produces familiarity that affects scoring. Rotate test content across studies.
Mistake 5: Self-selected panels.
Recruiting only enthusiastic volunteers biases toward viewers who care intensely about quality. Panel should mix viewer types.
Mistake 6: Skipping calibration.
Running studies without panel calibration. Without baseline calibration, you don't know if your panel is reliable or noisy.
Mistake 7: Over-interpreting marginal results.
A 0.1 MOS difference between conditions may not be statistically significant. Look at confidence intervals; don't conclude from small differences.
What MpegFlow does with golden-eyes review
Golden-eyes review is a human-gated workflow that MpegFlow's pipeline does not automate. The DAG runtime today doesn't include decision nodes that pause on human review and resume on operator approval — that's a roadmap item, not a current capability.
What MpegFlow's pipeline does support is the encoding-side work that feeds a review study: producing the test variants (parallel encode stages with different parameter sets), running the automated quality metrics (PSNR, SSIM, VMAF as discrete measurement stages), and emitting structured comparison output. The human review itself, and the integration of findings back into encoder configuration, is operator work outside the pipeline.
For customers requiring golden-eyes review for their content:
- We can recommend specialized firms with relevant expertise.
- We provide content packaging and metadata to support review studies.
- We integrate review findings into pipeline configuration when actionable — manually, via workflow YAML updates rather than automated decision-loop integration.
For internal MpegFlow encoder evaluation, we work with external specialized firms for occasional studies. Findings inform default encoder configurations and quality thresholds; the integration is editorial, not automated.
The strict-broker security model handles review materials with appropriate discipline — content for review may be premium licensed material; access is controlled; no reviewer-permanent storage of content beyond what's needed for the study.
For customers asking whether golden-eyes review is worth it, the standing recommendation: yes for premium streaming with high-stakes content; yes when codec or encoder migrations are being evaluated; no for everyday pipeline operations. Match the QA tool to the cost of getting it wrong.
The general guidance: golden-eyes review is the rigorous answer when automated metrics aren't enough. Use it judiciously — too much is wasteful; too little misses quality regressions that hurt your audience. For most production pipelines, automated metrics with periodic golden-eyes spot-checks is the right balance.