Job Description

Company Overview

Company Overview

Incedo is a US-based consulting, data science and technology services firm with over 2,500 people helping clients from our six offices across US and India. We help our clients achieve competitive advantage through end-to-end digital transformation. Our uniqueness lies in bringing together strong engineering, data science, and design capabilities coupled with deep domain understanding. We combine services and products to maximize business impact for our clients in telecom, financial services, product engineering and life science & healthcare industries.
br
Working at Incedo will provide you an opportunity to work with industry leading client organizations, deep technology and domain experts, and global teams. Incedo University, our learning platform, provides ample learning opportunities starting with a structured onboarding program and carrying throughout various stages of your career. A variety of fun activities are also an integral part of our friendly work environment. Our flexible career paths allow you to grow into a program manager, a technical architect or a domain expert based on your skills and interests.

Role Description

? Able to perform Data Engineering tasks using Spark and Databricks.
? Develop Data Processing Jobs (that focus on Data Engineering tasks like Data Massaging, Data Cleansing, Data Transformation - both ETL and ELT along with load into Data Base)
? Understand Business Requirements and help in developing High level and Low-level Data Engineering and Data Processing Documentation
? Help in Designing Solution architecture, work on Data Ingestion, Data Preparation, Data Transformation, Data Load
? Knowledge of developing efficient frameworks for development and testing using Hadoop (Sqoop/Nifi/Kafka/Spark/Streaming/ WebHDFS/Python) to enable seamless data ingestion processes on to the Hadoop platform.
? Enabling Data Governance and Data Discovery on Big data Platform (Spark and Databricks)
? Building data processing framework using Talend, SparkSQL, HQL
? Exposure of Job Monitoring framework along validations automation
? Exposure of handling structured, Un Structured and Streaming data.

Must have - Technical & Soft Skills

? Strong hands-on experience of programming language like Python, Scala with Spark and Databricks.
? Solid hands-on and Solution Architecting experience in Big-Data Cloud Technologies - Aws, Azure and GCP (AWS preferred)
? Good command and working experience on SparkSQL, Hadoop/Map Reduce, HDFS, Hive, HBase, and No-SQL Databases
? Knowledge of data engineering/analytics platform (Hortonworks/Cloudera/ MapR/Spark Databricks), AWS Spark Databricks preferred
? Hands-on experience on Data Ingestion using Apache Nifi, Apache Airflow, Sqoop, and Ozzie
? Hands on working experience of data processing at scale with event driven systems, message queues (Kafka/ Flink/Spark Streaming)
? Hands on working Experience with AWS Services like EMR, Kinesis, S3, Cloud Formation, Glue, API Gateway, Lake Foundation, AWS Spark Databricks
? Hands on working Experience with AWS Athena, Spark Databricks
? Feature Engineering/Data Processing to be used for Model development
? Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

Certifications (Preferred)

? Spark Databricks Certification
? AWS Certified Data Engineer
? AWS Certified Solution Architect - Associate

Nice to have

? Bash and Scripting: Unix or Shell scripting
? Exposure to various ETL and Business Intelligence tools
? Experience in data warehouse design and best practices
? A strong background in troubleshooting and technology support will be beneficial.

Educational Qualification

? A bachelor's degree in computer science, computer engineering, or a related discipline is required to work as a technical lead
? Certification in Spark Databricks and AWS would be a big plus

Company Value

Company Value

We are an Equal Opportunity Employer. We value diversity at Incedo. We do not discriminate based on race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.