Job Description

Roles & Responsibilities

Data Engineering Excellence: Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion. Cloud Data Migration Leadership: Lead cloud migration projects, developing scalable Spark pipelines. Medallion Architecture: Implement Bronze, Silver, and gold tables for scalable data systems. Spark Code Optimization: Optimize Spark code to ensure efficient cloud migration. Data Modeling: Develop and maintain data models with strong governance practices. Data Cataloging & Quality: Implement cataloging strategies with Unity Catalog to maintain high-quality data. Delta Live Table Leadership: Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management. Customer Collaboration: Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance.

Educational Qualifications

Experience: Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects. Education: Bachelor's or higher degree in Computer Science, Data Engineering, or a related field.

Skills

Must-have: Proficiency in Spark, SQL, Python, and other relevant data processing technologies. Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations. Expertise in on-premises to cloud Spark code optimization and Medallion Architecture.

Good to Have

Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus).

Soft Skills

Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams. Certifications AWS/GCP/Azure Data Engineer Certification.
Job Types: Full-time, Permanent

Work Location: In person