Job Description

Role Overview:
We are seeking a highly skilled and experienced Senior Data Engineer to join our team. The ideal candidate will have strong expertise in Python, PySpark, Apache Iceberg, Flink, and Kafka, with additional exposure to API development and integration. This role involves designing, building, and optimizing scalable data pipelines and systems to support our data-driven initiatives.
Key Responsibilities:
Data Pipeline Development:

• Design, develop, and maintain scalable and efficient data pipelines using Python and PySpark.
• Implement real-time and batch data processing workflows leveraging Apache Flink and Kafka.

Data Lakehouse Management:

• Work with Apache Iceberg to manage large-scale datasets, including schema evolution, partitioning, and time-travel queries.
• Optimize data storage and retrieval for performance and scalability.

API Development and Integration:

• Develop and integrate APIs to enable seamless data access and interaction with downstream systems.
• Ensure APIs are secure, scalable, and optimized for performance.

Data Streaming and Messaging:

• Build and manage real-time data streaming solutions using Kafka.
• Implement event-driven architectures to support business use cases.

Data Governance and Quality:

• Ensure data quality, consistency, and security across all pipelines and systems.
• Implement monitoring and alerting mechanisms to track data pipeline performance and reliability.

Collaboration and Mentorship:

• Collaborate with cross-functional teams, including data scientists, analysts, and DevOps engineers.
• Mentor junior engineers and contribute to best practices in data engineering.

Required Skills:
Programming and Frameworks:

• Proficiency in Python and PySpark for data processing and pipeline development.
• Strong understanding of distributed data processing frameworks.

Big Data Technologies:

• Hands-on experience with Apache Iceberg for managing large-scale datasets.
• Expertise in Apache Flink for real-time data processing.
• Proficiency in Kafka for data streaming and messaging.

API Development:

• Experience in developing and integrating RESTful APIs.
• Knowledge of API security, scalability, and performance optimization.

Data Engineering:

• Strong understanding of ETL/ELT processes and data transformation techniques.
• Familiarity with data lakehouse architectures and best practices.

DevOps and CI/CD:

• Experience with CI/CD pipelines for deploying and managing data pipelines.
• Familiarity with containerization tools like Docker and orchestration platforms like Kubernetes.

Preferred Skills:

• Experience with cloud platforms (AWS, Azure, or GCP) for data engineering solutions.
• Knowledge of data governance frameworks and compliance requirements.
• Familiarity with monitoring tools like Prometheus, Grafana, or similar.

Education:

• Bachelor's degree/University degree or equivalent experience

This job description provides a high-level review of the types of work performed. Other job-related duties may be assigned as required.
Job Family Group: Technology
Job Family: Applications Development
Time Type: Full time
Most Relevant Skills Please see the requirements listed above.
Other Relevant Skills For complementary skills, please see above and/or contact the recruiter.
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review .
View Citi's and the poster.