Human Evaluation - Program Manager

Netflix is expanding into new entertainment areas including AI, and is building frameworks, tools, and workflows to evaluate and train ML and Generative AI models used in product experiences.

The Human Evaluations Program Manager will design and operationalize rigorous human evaluation programs and own end-to-end execution to ensure AI quality, consistency, and alignment with Netflix values.

• Lead end-to-end execution of human evaluation and data operations initiatives from intake and scoping through delivery.
• Develop and operationalize frameworks for evaluating GenAI and ML outputs.
• Partner with research, product, UX, and engineering to embed evaluation into model development cycles.
• Build and maintain project timelines, proactively manage blockers, and ensure timely execution.
• Create scalable guidelines and scoring rubrics to drive consistent rater judgment.
• Oversee rater onboarding, calibration, and QA workflows.
• Define and monitor success metrics (e.g., speed to IRR, throughput, task effectiveness).
• Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality.
• Build foundational documentation and drive adoption of best practices across teams.
• Track evaluation health and communicate progress to stakeholders clearly and proactively.
• Anticipate and resolve bottlenecks and blockers; act as connective tissue across partners to ensure alignment and execution at scale.

• 4+ years of experience in human evaluations, data collection, labeling, or annotation operations in GenAI/ML environments.
• Experience implementing process improvements or quality control systems for data collection needs.
• Experience managing annotation vendors, raters, or data labeling teams.
• Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols.
• Proven end-to-end program management of complex, cross-functional programs with accountability for delivery.
• Experience with human labeling platforms.
• Excellent written and verbal communication skills.
• Ability to synthesize feedback into clear recommendations and process improvements.
• Familiarity with responsible AI principles and embedding them into evaluation design.
• Strong organizational and execution skills; able to manage details while maintaining big-picture alignment.

• Compensation range: $230,000 to $340,000 annual salary (varies by location); employee can choose allocation between salary and stock options; no bonuses.

• Benefits include health plans, mental health support, 401(k) with employer match, stock option program, disability programs, HSA/FSA, family-forming benefits, life and serious injury benefits, and paid leave programs.

• Time off: full-time hourly employees accrue 35 days annually PTO; full-time salaried employees receive flexible time off.

• Equal-opportunity employer; accommodations available during hiring.