Job Description

Location:

Pune / Remote

Experience Level:

3-8 years

Role Overview

We are looking for a Python-centric Data Engineer who can design and maintain scalable, high-performance data pipelines for large and complex datasets. The ideal candidate brings strong object-oriented Python programming skills, experience with distributed data systems (like Hadoop or Spark) and a mindset for building modular, reusable, and testable data solutions. While exposure to cloud technologies (AWS preferred) is valuable, this role emphasizes deep Python-based data engineering over platform administration.

Roles & Responsibilities

Design, develop, and optimize data ingestion and transformation pipelines using Python.
Apply OOP principles to build modular, maintainable, and reusable data components.
Work with distributed systems (e.g., Hadoop/Spark) for processing large-scale datasets.
Develop and enforce data quality, testing, and validation frameworks.
Collaborate with analysts, data scientists, and product teams to ensure data availability and reliability.
Participate in code reviews, CI/CD workflows, and infrastructure automation to maintain high engineering standards.
Contribute to ongoing evolution of data architecture and help integrate with cloud-based data ecosystems.

Technical Skills

Core Expertise

Python Programming:

Strong grasp of object-oriented design, modular code practices, and design patterns.
Experience with libraries/frameworks such as pandas, PySpark, dask, or custom ETL frameworks.
Proficiency in writing unit-tested, well-documented, and scalable code (pytest, unittest).

Data Engineering & Ecosystem:

Experience in building and maintaining ETL/ELT pipelines using Python.
Exposure to Hadoop or Spark ecosystems for distributed data processing.
Solid understanding of SQL and working with relational or analytical databases (e.g., PostgreSQL, Redshift).
Familiarity with CI/CD pipelines (Jenkins, GitHub Actions) and Infrastructure as Code (Terraform).
Basic exposure to AWS data services like S3, Lambda, Step Functions, Glue, or DynamoDB.

Qualifications

Bachelor's or master's degree in engineering or technology or related field.
3-6 years of hands-on experience in data engineering or backend development.
Proven track record of Python-based pipeline development and distributed data processing.
Strong foundation in data modeling, data quality, and pipeline orchestration concepts.
Excellent problem-solving and communication skills, with an ownership-driven mindset.

Desired Qualifications

Experience with Apache Airflow, Luigi, or Prefect for workflow orchestration.
Exposure to real-time data streaming (Kafka, Kinesis).
Familiarity with containerization tools (Docker, Kubernetes).
Knowledge of data governance, cataloguing and metadata management.