Job Description

Experience

: 5+ Years Experience

Education

: Bachelor's/Master's degree

Work Location

: Chennai, India (Chennai/Remote/Hybrid)

Key Responsibilities :

Design, develop, and maintain

automated testing frameworks

for Gen AI applications and agentic systems. Implement

evaluation pipelines

to measure and improve LLM reliability and response quality. Develop and operationalize

hallucination and bias detection systems

. Collaborate with LLM Engineers to integrate evaluation checkpoints into model deployment workflows. Define and manage

evaluation datasets

and metrics for automated quality reporting. Partner with Product and Governance teams to ensure

responsible AI testing and validation practices

. Continuously monitor and improve

AI performance, reliability, and trust metrics

.

Required Technical Skills :

Generative AI Quality Engineering

Design automated

testing frameworks

for generative AI and multi-agent systems. Develop

evaluation pipelines

to assess LLM responses for accuracy, coherence, and factual grounding. Implement

hallucination detection mechanisms

using reference comparison, confidence scoring, or retrieval checks. Create reproducible

test datasets

and

evaluation benchmarks

for prompt-response validation.

Automation Frameworks & Tools

Develop and maintain

Python-based QA frameworks

integrated with CI/CD systems. Experience with testing tools such as

pytest, LangSmith, DeepEval,

or

TruLens

. Automate testing of

API-driven AI systems, RAG pipelines,

and

multi-agent conversations

. Implement regression, performance, and stress testing for large-scale AI applications.

Evaluation Metrics & Analysis

Define and compute

LLM evaluation metrics

(precision, recall, coherence, diversity, factual accuracy, toxicity). Build

evaluation dashboards

and

reporting tools

for continuous model monitoring. Analyze model drift, prompt sensitivity, and performance degradation over time.

Model Safety & Reliability

Develop and integrate

toxicity, bias, and compliance detection

checks into QA pipelines. Collaborate with security and governance teams to ensure

AI safety and alignment standards

. Implement

guardrail testing

and policy compliance frameworks for generative systems.

Infrastructure & Integration

Integrate QA pipelines into

CI/CD workflows

for LLM deployment pipelines. Experience with

cloud-native tools

(AWS, Azure, GCP, or OCI) for automated test execution. Implement test orchestration using

Docker, Kubernetes

, or

serverless workflows

. Familiarity with

API mocking, synthetic test data generation,

and

version-controlled evaluation artifacts

.

Required Skills & Experience :

5+ years of experience in

QA Automation, AI Testing,

or

ML Evaluation Engineering

. Strong programming background in

Python

with experience in

PyTest, FastAPI

, or

Playwright

. Understanding of

LLM architectures, prompt-response mechanics,

and

RAG systems

. Experience with AI evaluation frameworks such as

DeepEval, TruLens,

or

LangSmith

. Familiarity with

data validation, model observability

, and

pipeline testing

. Proven ability to design

automated evaluation frameworks

for AI and ML models.
APPLY

Close

Drivestream's Employee Benefits

.

----------------------------------------

Remuneration

----------------



Drivestream offers competitive pay and attracts a diverse community of skilled individuals. We recognize the value of investing in our talent.

Medical, Disability and Life Insurance

------------------------------------------



We provide an array of coverage options including full medical, full dental and vision plans, employee life insurance, LTD and STD coverage, flexible spending account and employee accidental death and dismemberment

Leave Benefits

------------------



Drivestream's generous paid leave programs feature vacation/paid time off (PTO), holiday leave and bereavement leave

Professional Development

----------------------------



Our training and development programs include traditional classroom training, online courses, including leadership, communication and project planning development, strategic planning and management programs, and professional society membership incentive

Work/Life Programs

----------------------



Drivestream offers Work-Life Integration options that help individuals manage their personal and professional responsibilities. Options Include work from home and telecommuting, day care, flexible spending accounts, internal job transfer, and career mobility, and health and wellness programs

Community Involvement

-------------------------



Drivestream believes in supporting community and philanthropic activities that allow our employees to engage in outreach and educational programs.

Awards Programs

-------------------



Drivestream recognizes and rewards our staff through various annual awards programs.

Retirement Benefits

-----------------------



Drivestream offers complete 401(k) plans and annual profit-sharing contribution.