Member of Technical Staff, Evals
Magic
- San Francisco, CA
- Remote
- Senior level
- ML Engineering
- Sponsors visas
- $200k–$550k
The role in 30 seconds
The Evals team at Magic measures the performance and capabilities of internal and external models to help teams make informed decisions. As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance, with a focus on correctness, reproducibility, and measurement quality.
Is this role a fit for you?
The backgrounds this role is most likely to hire from, as RoleStack reads the posting.
Strong match
- B2B infrastructure / API / developer tools
- Software, SaaS & AI Platform
- ML Platform
- Developers / technical users
- Internal users (employees, ops teams)
Workable stretch
- B2B enterprise platform / workflow software
- Internal tooling / back-office systems
- Enterprise users (complex workflows, admin panels, role-based access)
- Startup / greenfield
Harder transition
- B2C consumer-facing application
- Marketplace / multi-sided platform
- Consumer / general public
- SMB / small business
- Regulated environment (healthcare, finance, government)
Industry: Software, SaaS & AI Platform. See your personal match score
What they're really screening for
Must-haves
- CORE Experience developing model evaluations, benchmarks, or ML experiments
- CORE Strong programming skills and experience working with machine learning systems
- PLUS Track record of owning technical projects end-to-end
- CORE Skepticism toward results that cannot be reproduced, validated, or explained
- CORE Experience analyzing model performance and interpreting experimental results
- CORE Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
- CORE Commitment to raising the standard of rigor in AI evaluations
Nice-to-haves
- PLUS Experience with public benchmarks, research papers, and open-source implementations
- PLUS Experience building tools that make evaluation results accessible and useful
Résumé keywords for this role
Terms the posting stresses. Include the ones you can honestly back up.
Tools & technologies
- AGI
- RL
- Evals
- ML
- AI
Concepts & phrases
- Member of Technical Staff
- safe AGI
- automating research
- code generation
- model alignment
- frontier-scale pretraining
- domain-specific RL
- ultra-long context
- inference-time compute
- Evals team
- internal and external models
- model performance
- pretraining
- RL
- data
Words that count as the same thing: "AGI" ≈ Artificial General Intelligence · "RL" ≈ Reinforcement Learning · "Evals" ≈ Evaluations, Benchmarking, Model assessment
Check my résumé against these keywordsHiring at Magic
Open senior roles RoleStack tracks right now, posted in the last 30 days.
Full job description from Magic
From Magic's posting. View the original
Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pretraining, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal.
Show full description Show less
ABOUT THE ROLE
The Evals team measures the performance and capabilities of internal and external models, helping teams across Magic make informed decisions. The team supports pretraining, RL, data, inference, and product, and sits on the critical path of many of the company's most important decisions.
As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance.
Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality.
WHAT YOU'LL WORK ON
- Design and validate new evaluations and benchmarks across model training and deployment
- Investigate model behavior and identify gaps in existing evaluations
- Measure dataset quality and identify opportunities to improve training data
- Improve evaluation correctness, reproducibility, and reliability
- Audit and improve upon public benchmarks, research papers, and open-source implementations
- Define metrics that accurately reflect model quality
- Build tools that make evaluation results accessible and useful across Magic
WHAT WE'RE LOOKING FOR
- Experience developing model evaluations, benchmarks, or ML experiments
- Strong programming skills and experience working with machine learning systems
- Track record of owning technical projects end-to-end
- Skepticism toward results that cannot be reproduced, validated, or explained
- Experience analyzing model performance and interpreting experimental results
- Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
- Commitment to raising the standard of rigor in AI evaluations
OUR CULTURE
- Integrity. Words and actions should be aligned
- Hands-on. At Magic, everyone is building
- Teamwork. We move as one team, not N individuals
- Focus. Safely deploy AGI. Everything else is noise
- Quality. Magic should feel like magic
Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.
COMPENSATION, BENEFITS, AND PERKS (US)
- Annual salary range between $200K - $550K depending on experience
- Equity is a significant part of total compensation, in addition to salary
- 401(k) plan with 6% salary matching
- Generous health, dental, and vision insurance for you and your dependents
- Unlimited paid time off
- Visa sponsorship and relocation support for candidates moving to San Francisco
- A small, fast-moving, highly collaborative team working on frontier AI systems
FAQ
How much does a Member of Technical Staff, Evals at Magic make?
The posted pay for this role is $200k–$550k a year.
Does Magic sponsor visas for this role?
The posting says visa sponsorship is available.
Is this role remote?
It is a remote role.
What tools and technologies does the posting mention?
AGI, RL, Evals, ML, AI.