Job Description

Project Role :

Data Engineer

Project Role Description :

Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.

Must have skills :

PySpark

Good to have skills :

NA

Minimum

5

year(s) of experience is required

Educational Qualification :

15 years full time education



Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to effectively migrate and deploy data across various systems, contributing to the overall efficiency and reliability of data operations. Roles & Responsibilities: - Develop and maintain ETL pipelines using PySpark for data ingestion, transformation, and aggregation. -Write optimized and reusable PySpark code to process large-scale datasets. -Work with structured and unstructured data sources (CSV, JSON, Parquet, Avro, relational DBs, etc.). -Optimize Spark jobs for scalability, reliability, and performance. -Collaborate with data engineers, analysts, and business stakeholders to deliver clean and reliable datasets. -Implement data quality checks, monitoring, and logging for pipelines. -Work with cloud storage and big data frameworks (HDFS, Hive, Delta Lake, S3, ADLS, BigQuery). -Apply best practices for code versioning (Git), testing, and CI/CD deployment. -Document design, workflows, and technical solutions. Professional & Technical Skills: - Must To Have Skills: Proficiency in PySpark, Git, Testing, CI/CD deployment. - Work with cloud storage and big data frameworks (HDFS, Hive, Delta Lake, S3, ADLS, BigQuery). - Strong understanding of data modeling and database design principles. - Experience with data warehousing solutions and architecture. - Familiarity with cloud platforms such as AWS or Azure. - Knowledge of data governance and data quality frameworks. Additional Information: - The candidate should have minimum 7 years of experience in PySpark. - This position is based at our Bengaluru office. - A 15 years full time education is required.




15 years full time education