Job Description

Title: Data Engineer - AWS


Purpose of the Position:


You will be a critical member of the InfoCepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, Spark-Scala, PySpark, S3, AWS Lambda etc. Strong foundation on database concepts and SQL is also required.

Key Result Areas and Activities:

Study existing technology landscape and understand current data integration framework and do impact assessment for the requirements. Develop spark jobs using Scala for new project requirements. Enhance existing spark jobs for any ongoing product enhancement. Performance tuning of spark jobs, stress testing etc. Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache Airflow Responsible for database design process, logical design, physical design, star schema, snowflake schema etc. Analyze data processing, integration, modelling and reporting requirements & Define data loading strategies considering volume, data types, frequency and analytics specifications. Ensure optimal balance between cost and performance. Project documentation, Adheres to Quality guidelines & Schedules. Works hand in hand with PM for successful delivery of project and provide Estimation, scoping, scheduling assistance. Manage build phase and quality assure code to ensure fulfilling requirements and adhering to Cloud architecture. Resolve difficult design and develop issues.

Work and Technical Experience:

5 years of IT experience, with at least 3 years on cloud-related projects. Strong SQL skills and database fundamentals Hands-on experience with AWS services: S3, EC2, EMR, Athena, Glue, Lambda Big Data technologies: Hadoop, Hive, Spark, YARN Programming languages: Scala, Python, Shell Scripts Operating systems: Linux (any flavor), Windows Experience in developing ETL workflows with complex transformations Understanding of AWS networking: VPC, Subnets, Security Groups, Route Tables Quick learner with adaptability to new AWS and Big Data technologies

Qualifications:

Bachelor's degree in computer science, Engineering, or related field Technical certifications or evidence of continued learning 5 years of IT experience, with at least 3 years on cloud-related projects

Qualities:

Hold strong technical knowledge and experience Should have the capability to deep dive and research in various technical related fields Self-motivated and focused on delivering outcomes for a fast-growing team and firm Able to communicate persuasively through speaking, writing, and client presentations Able to consult, write, and present persuasively Able to work in a self-organized and cross-functional team Able to iterate based on new information, peer reviews, and feedback Prior experience of working in a large media company would be added advantage

Years Of Exp

5 to 7 years

Location

India