Job Description

We are looking for a Data Engineer to build and implement Data Ingestion, Data transformation. You will work with other Data Engineers and Tech leads throughout the data pipeline development life cycle. You should be a team player with a keen eye for detail and problem-solving skills in this role. If you also have experience in Agile frameworks and popular ETL/ELT and SQL tools (e.g. ADF, Azure SQL), wed like to meet you. Your goal will be to build efficient, optimized, and dynamic data transformation notebooks in Azure Databricks that serve customers requirements. Responsibilities Convert existing SCALA HD Insights to Pypark Notebook in Azure Data Bricks Produce clean, efficient data transformation process based on specifications Integrate On-Prem data, API, Social Media, and various data sources data into Azure Validate, and deploy ETL pipelines and systems Troubleshoot, debug and upgrade Data Pipelines and transformations Gather and evaluate user feedback Recommend and execute improvements Create technical documentation for reference and reporting Requirements Hands-on 3 years experience in PySpark with Azure Databricks Proven experience as a Data Engineer, ETL Developer, or similar role Familiarity with Agile development methodologies Experience with Azure Databricks, and HD Insights Knowledge of coding languages (e.g. PySpark) and frameworks/systems (e.g. Git, Azure DevOps) Experience with databases i.e. Azure SQL Excellent communication skills Resourcefulness and troubleshooting aptitude Attention to detail Graduation

foundit