Job Description

We are seeking a highly skilled and motivated Data Engineer to join our dynamic technology team. The ideal candidate will have deep expertise in data engineering tools and platforms, particularly Apache Airflow, PySpark, and Python, with hands-on experience in Cloudera Data Platform (CDP). A strong understanding of DevOps practices and exposure to AI/ML and Generative AI use cases is highly desirable.

Key Responsibilities:

1. Design, build, and maintain scalable data pipelines using Python, PySpark and Airflow.



2. Develop and optimize ETL workflows on Cloudera Data Platform (CDP).



3. Implement data quality checks, monitoring, and alerting mechanisms.



4. Ensure data security, governance, and compliance across all pipelines.



5.Work closely with cross-functional teams to understand data requirements and deliver solutions.



6. Troubleshoot and resolve issues in production data pipelines.



7. Contribute to the architecture and design of the data platform.



8. Collaborate with engineering teams and analysts to work on AI/ML and Gen AI use cases.



9. Automate deployment and monitoring of data workflows using DevOps tools and practices.



10. Stay updated with the latest trends in data engineering, AI/ML, and Gen AI technologies.