Job Description

Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties - Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue, Snowflake. Analyze, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS or 3rd party services. Design and build production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala . Design and implement data engineering, ingestion and curation functions on AWS cloud using AWS native or custom programming. Perform detail assessments of current state data platforms and create an appropriate transition path to AWS cloud. Skill: A WS (Pyspark, Lambda Glue, Kinesis). Experience: 4 yrs to 10 yrs. Well versed with SQL and RDBMS concepts. Good exposure in Python and Pyspark programming. Familiarity with AWS services such as SQS, EventBridge, Kinesis, Lambda and Glue. Ability to tr oubleshoot code independently. Mandatory Skills: AWS, SQL Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources like Hadoop, Spark, AWS Lambda, etc. Experience with AWS Cloud on Data Integration with Apache Spark, EMR, Glue, Kafka, Kinesis and Lambda in S3, Redshift, RDS, and MongoDB DynamoDB ecosystems. Strong real-life experience in Python development, especially in PySpark in AWS Cloud environment. Design, develop, test, deploy, maintain and improve data integration pipeline. Develop pipeline objects using Apache Spark Pyspark Python or Scala. Design and develop data pipeline architectures using Hadoop, Spark and related AWS Services. Load and performance test da ta pipelines built using the above-mentioned technologies. .

Job Requirements: PySpark, AWS Glue, Data Visualization, Data Analysis, Data Processing Design

Job Type

SUBCONTRACT

Location

CHENNAI

Mandatory Skills

• PySpark