About the role
Help make AI performance measurable, reproducible, and useful in real operational workflows.
The opportunity
Turn ambiguous quality questions into evidence that guides product decisions. Partner with engineers and domain experts to evaluate model behavior, improve retrieval and prediction, and understand the cost of errors in sensitive or consequential work.
What you will do
- Design representative evaluation datasets, labeling guidance, and repeatable experiments for AI-assisted workflows.
- Measure factual quality, retrieval relevance, task completion, uncertainty, and performance across important user and data segments.
- Compare models and system configurations using quality, latency, inference cost, and human review effort.
- Investigate failure patterns and translate findings into product changes, better datasets, and release criteria.
- Document data provenance, evaluation limitations, and reproducibility while protecting sensitive information.
What you bring
- At least six years of relevant professional experience in applied data science, machine learning, or quantitative research.
- A bachelor’s degree or higher in Computer Science, Data Science, Statistics, Mathematics, Artificial Intelligence, Machine Learning, or a closely related quantitative field from an accredited or officially recognized institution.
- A record of independently leading significant AI or ML work, setting evaluation standards, and guiding technical decisions with clear evidence.
- Strong Python and SQL skills, practical statistics, experimental design, and experience working with imperfect real-world datasets.
- Experience evaluating machine learning systems and interpreting uncertainty, bias, and variation in results.
- Strong applied understanding of LLMs, embeddings, retrieval, and the limits of model-based evaluation.
- Ability to communicate findings and tradeoffs clearly to engineering, product, and domain stakeholders.
Helpful experience
- Experience with retrieval evaluation, human labeling programs, drift monitoring, or production model monitoring.
- Work with privacy-sensitive data or operational decisions that require defensible evidence.
- A master’s degree or PhD in a relevant technical or quantitative discipline.
Location & working arrangement
This position is based in our Burlingame, California office and requires working in the office five days a week.
This is a contract position. Scope, duration, and engagement terms will be discussed during the hiring process.
AI & ML knowledge
Robust theoretical and applied knowledge of AI and machine learning, including supervised and unsupervised learning, deep learning, generalization, calibration, sampling bias, and leakage-resistant validation.
Every team member should be able to discuss the capabilities and limitations of AI relevant to their work, verify AI-generated output, protect sensitive information, and keep their knowledge current as the technology evolves.
Before you apply
Direct applicants only
We work directly with individual applicants. We do not work with recruiting agencies, consulting firms, offshore or outsourcing companies, or freelancers offering external services for these openings. Please do not submit candidates or service proposals on behalf of a third party. Individuals may apply directly for our listed contract positions.
U.S. work authorization required
All applicants must be legally authorized to work in the United States without employer sponsorship. Metrosion does not provide employment visa sponsorship.
On-site in Burlingame, CA
All positions require working on-site in our Burlingame, California office five days a week. These roles are not remote or hybrid.
Different perspectives. Shared purpose.
We welcome applicants from all backgrounds and value an inclusive, respectful workplace.