Job Description

Designation: Assistant Manager - Data Engineering

Level: L3

Experience: 5 to 8 years

Location: Chennai / Bengaluru

:

We are seeking a highly skilled and motivated Assistant Manager with strong experience in the Databricks Domain to join our growing team.

Responsibilities:

Data Engineering & Architecture

Design and implement scalable and optimized data pipelines on Databricks using Delta Lake, PySpark, and SQL. Develop ETL/ELT frameworks for batch and streaming data processing. Ensure data quality, governance, and observability using Unity Catalog, Great Expectations, or custom validations. Optimize Spark jobs for performance, cost, and scalability.

Cloud & Infrastructure (Azure/AWS/GCP)

Deploy and manage Databricks clusters, workspaces, and Jobs. Work with Terraform or ARM templates for infrastructure automation. Integrate cloud-native services like Azure Data Factory, AWS Glue, or GCP Cloud Composer.

MLOps & CI/CD Automation

Implement CI/CD pipelines for Databricks notebooks, workflows, and ML models. Work with MLflow for model tracking and lifecycle management. Automate data pipelines using Azure DevOps, GitHub Actions, or Jenkins.

Leadership & Collaboration

Lead a team of data engineers, ensuring best practices and code quality. Collaborate with data scientists, analysts, and business teams to understand requirements. Conduct performance reviews, technical mentoring, and upskilling sessions.

Skills:

Strong hands-on experience in Databricks, Apache Spark (PySpark/Scala), and Delta Lake. Expertise in SQL, ETL/ELT pipelines, and data modeling. Experience with Azure, AWS, or GCP cloud platforms. Knowledge of MLOps, MLflow, and CI/CD best practices. Experience in workflow orchestration using Databricks Workflows, Airflow, or Prefect. Understanding of cost optimization, cluster tuning, and performance monitoring in Databricks. Communication and leadership experience, with experience initiating and driving projects. Strong stakeholder management and mentoring skills. Experience with data sets, Hadoop, and data modernisation tools. Development experience in at least one object-oriented language (Python, Java, etc.). BA/BS in Computer Science, Math, Physics, or other technical fields.

Good to Have:

Experience with data lakehouse architectures and Unity Catalog. Hands-on with Terraform, Infrastructure-as-Code (IaC), or Kubernetes. Familiarity with data governance, security, and privacy frameworks.

Job Snapshot

Updated Date
22-09-2025
Job ID
J_4146
Location
Chennai, Tamil Nadu, India ..+ 1
Experience
5 - 8 Years
Employee Type
Permanent