Job Description

About the Role
We are seeking a Staff AI Scientist with deep expertise in machine learning fundamentals and a proven track record of building and evaluating advanced AI systems. This role is for someone who thrives at the intersection of data science, statistics, and ML engineering, with an emphasis on dataset rigor, metric design, and evaluation methodology.
At the staff level, you will set the direction for how we evaluate AI systems, work across teams to ensure scientific rigor is balanced with product and engineering needs, and deliver both datasets and pipelines that stand up to scrutiny.
What you'll do

• Dataset Preparation & Evaluation

• Design, prepare, and curate high-quality evaluation datasets with defensible methodology.
• Define criteria for dataset construction, ensuring statistical rigor, reproducibility, and fairness.
• Develop new metrics and evaluation frameworks to measure model performance in nuanced ways.

• Model Evaluation & ML Engineering

• Evaluate LLMs and other pre-trained models using carefully chosen datasets and metrics.
• Build scalable pipelines for training, fine-tuning, and benchmarking models.
• Contribute to projects involving fine-tuning, retrieval-augmented generation (RAG), and other adaptation methods.

• Cross-Functional Leadership

• Partner with product and engineering to align scientific rigor with business outcomes.
• Define evaluation standards and ML lifecycle practices that raise the bar across the company.
• Mentor scientists and engineers, guiding best practices in experimentation, statistics, and ML development.

Who you are

• Education: Master's degree (PhD preferred) in Computer Science, Statistics, Machine Learning, or a related quantitative field.
• Strong background in mathematical and statistical foundations of machine learning (probability, linear algebra, optimization, experimental design).
• Demonstrated experience in end-to-end ML lifecycle: dataset preparation, model training, evaluation, deployment, and monitoring.
• Proven expertise in evaluation dataset design and metric creation, not just using existing benchmarks but knowing when and how to improve them.
• Experience with LLM evaluation, fine-tuning, and RAG, with the engineering skills to build production-ready pipelines.
• Track record of strategic impact at a staff or principal level setting evaluation and research standards across teams.

Even better if you have

• Publications, open-source contributions, or widely adopted benchmarks in ML/AI.
• Prior experience shaping AI evaluation frameworks at an organizational or industry level.
• Comfort working at the interface of science, engineering, and product.

You will thrive in role if

• You define gold-standard evaluation datasets and metrics that internal teams adopt as a standard.
• You ship datasets and pipelines that are statistically rigorous, reproducible, and scalable.
• You elevate the organization's ML practice by embedding sound statistical and scientific methodology into product and engineering work.
• You are recognized as a thought leader who sets strategy while still contributing technically at depth.

Skills Required