Job Description

ROLES & RESPONSIBILITIES

ROLE OVERVIEW

We are seeking a

Data Engineer Lead

who can

deliver end-to-end data projects

, lead a team of engineers, and collaborate with stakeholders to design scalable solutions. The ideal candidate will have strong technical expertise in

PySpark, AWS Glue, Lambda, Postgres, Jenkins, Cloud DevOps, SQS

, and experience in

data migrations

and

reporting (PDF/CSV)

. This role requires leadership, solution design, and the ability to work in

overlapping PST/IST time zones

.

KEY RESPONSIBILITIES

Lead and manage a team of up to 7 developers

, ensuring timely and high-quality deliverables.

Engage with stakeholders

to gather requirements, define scope, and design solutions aligned with business objectives. Deliver

end-to-end data engineering projects

, including architecture, development, testing, and deployment. Build and optimize

ETL pipelines

using

AWS Glue

and

PySpark

for large-scale data processing. Develop

serverless workflows

leveraging

AWS Lambda

and

SQS

for event-driven architectures. Oversee

data migrations

from on-premise systems to cloud environments. Implement and maintain

PostgreSQL databases

for analytics and transactional workloads. Set up and manage

CI/CD pipelines

using

Jenkins

and enforce

Cloud DevOps

best practices. Generate

reports in PDF and CSV formats

for operational and executive dashboards. Ensure

data security, compliance, and performance optimization

across all solutions. Work in

overlapping PST/IST time zones

to collaborate with global teams. Mentor team members and promote best practices in coding, testing, and deployment.

REQUIRED SKILLS & EXPERIENCE

8+ years

of experience in

data engineering

, with at least

3 years in a leadership role

. Proven ability to

manage teams of 10+ engineers

and deliver complex projects. Strong proficiency in

Python

,

PySpark

, and distributed data processing frameworks. Hands-on experience with

AWS Glue

,

Lambda

, and

SQS

. Solid understanding of

PostgreSQL

and relational database design. Expertise in

CI/CD automation

using

Jenkins

and

Cloud DevOps

tools. Experience in

report generation (PDF/CSV)

and data visualization. Excellent communication skills for stakeholder engagement and requirement gathering.

PREFERRED QUALIFICATIONS

AWS certifications (e.g.,

AWS Certified Data Analytics

,

Solutions Architect

). Experience with

data governance

,

cataloging

, and

metadata management

. Familiarity with

containerization (Docker/Kubernetes)

and

Infrastructure as Code (Terraform/CloudFormation)

.
EXPERIENCE


8-11 Years
SKILLS


Primary Skill: Data Engineering
Sub Skill(s): Data Engineering
Additional Skill(s): Python, SQL, Workflows