The role, as published
About the company
Our client is a fast-growing technology company.
The role
Raydar is recruiting for this role on behalf of our client. Own the quality measurement framework that determines whether models and software features are ready to release. You will build test suites, reporting tools and human review rubrics, and partner with research, product and business colleagues so that quality is measurable and trusted.
What you'll do
- Build and maintain evaluation suites that act as a gate for each release, covering capability, behavior and regressions.
- Design human-rated review rubrics that catch issues automated checks miss.
- Develop dashboards and tooling that speed up experiment cycles and support clear decisions.
- Define and uphold the quality standard for what is ready to ship, including under deadline pressure.
- Work with researchers on measuring what matters, with product engineers on instrumenting real usage, and with external partners on explaining improvement in concrete terms.
- Audit existing evaluations early on, document what is useful, noisy or missing, and close the most significant gaps.
- Stand up new evaluation areas in the first months and help shape the research roadmap.
What we're looking for
- 3 to 6 years of experience building evaluation frameworks for AI systems whose outputs vary from run to run.
- Strong engineering skills in tooling, dashboards, instrumentation and fast experiment loops.
- Ability to design rubrics for human reviewers.
- Confidence to challenge a gamed metric while keeping the team aligned.
- High agency and a strong sense of urgency.
- Track record of owning a quality bar and protecting it under product pressure.
- Hands-on evaluation experience in a real-world setting, such as a data or evaluation company or an advanced AI research group.
Bonus points
- Experience evaluating agentic, on-device or tool-using systems.
- Evaluation work at an AI research lab or a data and evaluation company.
- Exposure to consumer AI or device-maker quality assurance.
- Early-hire or fast-growing company experience.
- Relocation support
Location and work model
- San Francisco, CA, United States
- On-site
- Full-time
Compensation, as the posting describes it
- Equity