Job Description

Senior Data Engineer - Databricks

Location: Hyderabad/Chennai

Work Schedule:

Timings 2:30 PM to 11:30 PM

Role Overview:

We are seeking a highly skilled Senior Data Engineer with deep expertise in Databricks to design, build, and optimize scalable data pipelines and architectures. You will collaborate with cross-functional teams including data scientists, analysts, and platform engineers to deliver high-quality, reliable, and secure data solutions.

Key Responsibilities:

Design, develop, and maintain robust batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, and PySpark. Build and optimize ETL/ELT workflows to transform and integrate data from diverse sources. Implement Delta Live Tables (DLT) and Databricks Workflows for pipeline orchestration and automation. Ensure data quality, observability, and monitoring across all pipelines. Collaborate with cloud platform teams (AWS/Azure) to ensure proper setup of storage, networking, and security. Work within Agile teams, participating in sprint planning, code reviews, and deployments. Document pipeline processes, technical specifications, and architectural decisions. Stay current with emerging technologies and best practices in data engineering and cloud-native solutions.

Required Qualifications:

Bachelor's or Master's degree in Computer Science, Engineering, or related field. Around 5 years of experience in data engineering, with 3+ years hands-on in Databricks. Strong proficiency in PySpark, SQL, and Apache Spark. Experience with cloud platforms (Azure or AWS) and big data technologies (e.g., Kafka, Hadoop). Solid understanding of data warehousing, data modeling, and ETL processes. Familiarity with data governance, security, and compliance frameworks.

Preferred Skills:

Experience with Delta Live Tables (DLT) and Unity Catalog. Knowledge of CI/CD pipelines, Git, and DevOps practices. Certifications in Databricks, AWS, or Azure are an added advantage.

Soft Skills:

Excellent communication and collaboration skills. Strong problem-solving and analytical mindset. Ability to work independently and lead initiatives.
Job Types: Full-time, Permanent

Work Location: In person