Job Description

GenAI & Databricks- R01556764

===================================

Data Scientist

Primary Skills

Data Science skill -

Machine Learning, NLP, and LLMs

Databricks

Experience- 5 to 8yrs

Specialization

Data Science Advanced:

Generative AI & Databricks

Job requirements

Job Title: Senior AI Engineer - Generative AI & Databricks

Experience:

5-8+ Years

About the Role

We are seeking a

Senior AI Engineer

with strong expertise in

Generative AI (GenAI)

,

Databricks

, and

end-to-end ML/LLM systems

. You will be responsible for designing, building, and deploying

intelligent, scalable GenAI solutions

integrated into enterprise-grade data and analytics platforms. The ideal candidate combines strong

software engineering

,

MLOps

, and

LLM engineering

experience -- with the ability to lead

AI agentic workflows

,

data pipeline optimization

, and

model-driven automation

using

Databricks

,

MLflow

, and

Azure/Snowflake ecosystems

.

Key Responsibilities

1. Solution Architecture & Implementation

Design and implement

end-to-end Generative AI solutions

on

Databricks

, leveraging

Unity Catalog

,

MLflow

,

Delta Lake

, and

Vector Search

. Architect

LLM-based multi-agent frameworks

for intelligent automation, chatbot systems, and document reasoning tasks. Integrate

Cortex AI

,

OpenAI

, or

Anthropic APIs

for retrieval-augmented generation (RAG), conversational reasoning, and workflow orchestration.

2. Model Development & Optimization

Fine-tune and evaluate

LLMs

and

domain-specific NLP models

(NER, Risk Assessment, Question Answering). Develop pipelines for

prompt engineering

,

context management

,

model evaluation

, and

hallucination detection

. Optimize inference performance, latency, and cost across multi-cloud and Databricks environments.

3. Data Engineering & Governance

Collaborate with data engineering teams to ensure

clean, well-governed, and vectorized data

pipelines. Build and maintain

feature stores

and

embeddings stores

using Databricks or Snowflake. Implement data validation, lineage, and monitoring using

Delta Live Tables

and

Unity Catalog

.

4. MLOps & Automation

Build reusable

ML pipelines

using

Databricks Repos, MLflow, and Feature Store

. Automate deployment, monitoring, and retraining workflows for continuous model improvement.

5. Collaboration & Leadership

Partner with product managers, data scientists, and business stakeholders to translate ideas into production-ready AI systems. Review code, mentor junior engineers, and enforce best practices in scalable AI/ML development. Contribute to internal knowledge bases, documentation, and reusable component libraries.

Required Skills & Expertise

Core AI/ML

Strong background in

Machine Learning, NLP, and LLMs

(Transformers, RAG, embedding models). Proven experience fine-tuning or implementing models using

Hugging Face

,

LangChain

,

LlamaIndex

, or

OpenAI API

. Knowledge of

retrieval-augmented generation

,

multi-agent orchestration

, and

context management

.

Databricks & Cloud Ecosystem

Expertise in

Databricks (Delta Lake, MLflow, Unity Catalog, Feature Store, Vector Search)

. Familiarity with

Azure Databricks

,

Azure OpenAI

, or

Snowflake Cortex AI

. Experience integrating

external APIs

and

cloud-native microservices

(FastAPI, REST, or gRPC).

Programming & Engineering

Strong proficiency in

Python

,

SQL

,

PySpark

, and

Databricks Notebooks

. Experience building modular codebases, deploying APIs, and working with CI/CD pipelines (GitHub Actions, Azure DevOps). Hands-on experience with

Streamlit

,

Gradio

, or other UI frameworks for AI app development.

MLOps & Validation

Hands-on with

MLflow tracking

,

model registry

, and

experiment management

. Experience in

AI validation

,

faithfulness scoring

,

drift detection

, and

integrity match metrics

. Working knowledge of

Docker

,

Kubernetes

, and

inference scaling

techniques.

Soft Skills

Strong communication, stakeholder management, and ability to translate business problems into AI solutions. Comfort working in agile, multi-disciplinary environments. Passion for innovation, experimentation, and applied AI problem-solving.

Mandates* o Need GenAI Data Scientist - Databricks certified ML Engineer and work closely with customers. o Use case will involve data extract from pdf-based documents. o Leverage Databricks native solutions.