:
Job Profile
Does working with data on a day-to-day basis excite you
Are you interested in building robust data architecture to identify data patterns and optimize data consumption for our customers, who will forecast and predict what actions to undertake based on data
If this is what excites you, then you'll love working in our Data Analytics team.
We are looking for a savvy Data Engineer to join our growing team of AI , BI and machine learning experts. You will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up.
The Data Engineer will support our software engineers, data analysts, dashboard developers and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.
ResponsibilitiesCreate and maintain optimal data pipeline architecture; assemble data sets that meet functional / non-functional requirements.
Design the right schema to support the functional requirement and consumption pattern.
Design and build production data pipelines from ingestion to consumption.
Build the necessary datamarts, data warehouse required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
Create necessary preprocessing and postprocessing for various forms of data for training/ retraining and inference ingestions as required
Create data visualization and business intelligence tools for stakeholders and data scientists for necessary business/ solution insights
Identify, design, and implement internal process improvements: automating manual data processes, optimizing data delivery, etc.
Requirements and SkillsYou should have a bachelor's or master's degree in computer science, Information Technology or other quantitative fields
You should have at least 3-4 years working as a data engineer in supporting large data transformation initiatives related to machine learning, with experience in building and optimizing pipelines and data sets
Strong analytic skills related to working with unstructured datasets.
Must-have Programming Skills:Programming experience python programming, spark is must.
Hands -on experience in SQL, writing analytical queries and windows functions.
Hands - on experience in creating external tables, partitioning, parquet files.
2-3 years of solid experience in Big Data technologies a must.
Data Engineering experience using AWS core services (Lambda, Glue, EMR and RedShift)
Knowledge of Python and Pyspark is an absolute must.
Qualifications:
Requirements/Skill sets:Experience with AWS cloud services: EC2, EMR, RDS, Redshift, S3, Athena and familiarity with various log formats from AWS.
Experience in AWS Glue ETL, AWS Crawler, AWS Lambda, Glue Data Catalog, AWS Glue Studio.
Hands on experience with python programming, spark, shell scripting
Knowledge of Database Concepts - Indexing, Partitioning is must.
Knowledge of Data warehousing - Normalization, Denormalization, Star/Snow-flake schemas.
Good Hands -on the table's creations, DDL, DML and TCL
Knowledge of Database Concepts - Indexing, Partitioning
Knowledge of Data warehousing - Normalization, Denormalization, Star/Snow-flake schemas
About Our Company:
MNCJobsIndia.com will not be responsible for any payment made to a third-party. All Terms of Use are applicable.