Job Description

Key Responsibilities

------------------------

1. Machine Learning Development

Design, develop, and optimize

ML models

for predictive analytics, classification, regression, NLP, or other use cases. Perform

exploratory data analysis (EDA)

, feature engineering, data preprocessing, model selection, tuning, and evaluation. Implement responsible AI practices including model performance monitoring, drift detection, and interpretability.

2. Databricks Platform Expertise

Develop and maintain

Databricks notebooks

, jobs, Delta Lake pipelines, and MLflow tracking workflows. Optimize large-scale data workloads using

Spark

, Delta Live Tables, and Databricks clusters. Manage data access, lineage, and governance through

Databricks Unity Catalog

.

3. MLOps & Productionization

Build and maintain

end-to-end ML pipelines

using Databricks, MLflow, and CI/CD tools (Azure DevOps / GitHub Actions / Jenkins). Deploy models to production using

MLflow Models

, Databricks Model Serving, or containerized microservices. Implement automated monitoring for

model drift

, data quality, and inference performance. Support continuous model retraining strategies and versioning of datasets, features, and models.

4. Data Engineering Collaboration

Work closely with Data Engineering to design

scalable ETL/ELT pipelines

on Delta Lake. Ensure high availability of feature pipelines and support/maintenance via the

feature store (Databricks Feature Store)

.

5. API Integrations

Develop RESTful APIs for real-time model inference and analytics workflows. Integrate with internal and external systems using API gateways, event-driven architectures, or message queues. Ensure security, observability, and performance of deployed endpoints.

6. Governance, Security & Compliance

Apply data governance best practices across

Unity Catalog

, including permissions, lineage tracking, and data auditing. Comply with enterprise security controls, secrets management, and model governance frameworks.

Required Skills & Experience

---------------------------------


3-8+ years of experience in Data Science / ML Engineering (adjust as needed). Strong hands-on experience with

Databricks

,

Spark

,

Delta Lake

, and

MLflow

. Proficiency in Python, SQL, and common ML libraries (scikit-learn, PySpark MLlib, TensorFlow/PyTorch optional). Solid understanding of

MLOps concepts

: CI/CD, feature stores, monitoring, model deployment, pipelines. Experience integrating ML systems via

REST APIs

or event-driven services. Deep understanding of

ML lifecycle

: data ingestion training evaluation deployment monitoring. Familiarity with cloud platforms (

Azure

,

AWS

, or

GCP

, preferably Azure Databricks). * Experience with

Unity Catalog

data governance and access control.