Job Description

Checkmate is building advanced Voice AI systems for some of the largest restaurant and retail brands in the U.S., including several in the top 10. Our AI solutions are live in production with real customers, achieving over 80% accuracy. This represents a $1 billion market opportunity. Join us at this pivotal moment of growth to shape AI products used daily by thousands of staff and customers, combining cutting-edge LLM innovation with real business impact.
Essential Job Functions

• Design, test, and optimize LLM prompts for conversational AI, text classification, and structured data extraction tasks.

• Build evaluation pipelines to analyze prompt performance using quantitative metrics, human-in-the-loop feedback, and business KPIs.

• Conduct prompt experiments and regression testing to ensure stability, accuracy, and safety as models evolve.

• Collaborate with Machine Learning, Product, and Operations teams to translate business objectives into scalable, data-driven prompt-engineering strategies that enhance model accuracy, efficiency, and real-world usability.

• Use Python/SQL to analyze model outputs, identify anomalies, and automate quality checks.

• Document best practices and contribute to internal frameworks for prompt evaluation and continuous improvement.

• Communicate findings effectively to technical and non-technical stakeholders, driving measurable business impact through insight-driven decisions.

Requirements

• B.S. or higher in a quantitative discipline (Data Science, Computer Science, Engineering, or related field) or in a field relevant to language models (Linguistics, Philosophy, Cognitive Science, etc.).
• 5+ years of relevant experience with a B.S. degree, or 3+ years of experience with a Master's degree.
• Demonstrated proficiency in Python for automation, evaluation, and experimentation with LLM workflows.
• Proven experience in prompt engineering and working with LLMs (GPT-4, Claude, Gemini, and LLaMA) for text generation, reasoning, and structured data extraction.
• Proficiency in Python and SQL for data analysis, evaluation scripting, and workflow automation.
• Strong background in A/B testing, statistical analysis, and performance metric evaluation, with the ability to design experiments and interpret data-driven insights for continuous model optimization.
• Familiarity with prompt-evaluation tools such as LangFuse or Galileo, and Weights and Biases for experiment management and regression testing.
• Deep understanding of advanced prompting techniques, including few-shot prompting, reasoning-based prompting, multi-turn dialogue design, agentic orchestration, and DSPy/AdaFlow-style programmatic prompting frameworks.
• Experience applying CO-STAR and TIDD-EC! prompting frameworks for structured reasoning, instruction design, and context control in production-grade LLM systems.
• Excellent requirement-elicitation and communication skills, with the ability to translate business objectives into prompt-engineering solutions.
• Analytical mindset with a process-driven approach to optimizing model behavior, data quality, and operational workflows.
• Academic or applied research experience related to language models, prompt engineering, or LLM-based systems is a strong plus.
• Familiarity with LLM architectures, embeddings, and fine-tuning techniques preferred.
• Experience with LLM red-teaming, adversarial evaluation, or model safety testing is a plus.
• Candidates must be flexible and work during US hours at least till 5 pm EST, which is essential for this role & must also have their own system/work setup for remote work.

Skills Required