Duration: 6+ months (extendable based on performance)
Experience Required: 8+ years
Work Timing: Flexible (supporting US & India)
Job Overview
We are seeking an experienced Senior Data Pipeline Engineer with strong expertise in modern data platforms. The ideal candidate will have hands-on experience in building, optimising, and scaling large-scale batch and real-time data pipelines using cutting-edge technologies. This role involves working with cross-functional teams to deliver reliable, high-quality data solutions for enterprise-grade applications.
Key Responsibilities
- Design, develop, and maintain scalable and reliable data pipelines for both batch and real-time workloads.
- Implement real-time event streaming and data ingestion using Apache Kafka.
- Build and optimise ETL/ELT workflows on Databricks and integrate with Snowflake for analytics and reporting.
- Write efficient, production-ready code in Python, Node.js, and Java.
- Collaborate with data scientists, analysts, and business stakeholders to deliver consistent data solutions.
- Ensure data quality, governance, and security across platforms.
- Monitor, debug, and optimise distributed data systems for high performance.
- Lead and mentor junior engineers, promoting best practices in data engineering.
Required Qualifications
- 8+ years of professional experience in data engineering or software engineering.
- Strong expertise in PySpark for distributed data processing.
- Hands-on experience with Apache Kafka for data streaming.
- Proficiency with Databricks for big data and ML-driven workloads.
- Experience with Snowflake for cloud data warehousing.
- Advanced programming skills in Python, with proficiency in Node.js and Java.
- Solid knowledge of ETL/ELT frameworks, data modelling, and pipelines.
- Strong understanding of distributed systems and cloud-based architectures.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Preferred Skills
- Experience with cloud platforms (AWS, Azure, GCP).
- Knowledge of CI/CD pipelines and containerization (Docker, Kubernetes).
- Familiarity with MLOps and data science workflows.
- Strong system design and debugging skills in large-scale environments.
- Excellent communication and problem-solving abilities.
What We Offer
- Opportunity to design and scale enterprise-grade data infrastructure.
- Competitive compensation with career growth opportunities.