Job Description

"Responsibilities

Architect and build:

Design and implement scalable data architectures and pipelines on Azure, utilizing services like Azure Data Factory and Azure Synapse Analytics.

Develop and optimize:

Create and maintain data pipelines, including ETL/ELT processes, with a focus on performance and scalability, often using Python/PySpark and Databricks.

Ensure data quality:

Implement processes and checks to maintain data quality, integrity, and security across all data solutions.

Collaborate:

Work with data scientists, analysts, and other stakeholders to understand data needs and deliver effective solutions.

Troubleshoot and monitor:

Monitor data systems, troubleshoot performance issues, and ensure the smooth operation of data pipelines.

Mentor and lead:

Provide technical leadership and mentorship to junior data engineers.

Document:

Create and maintain comprehensive documentation for data workflows, processes, and best practices.

Qualifications

Education:

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Experience:

5+ years of experience in data engineering or a similar role.

Azure expertise:

Proven experience with Azure data services including Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake Storage.

Databricks expertise:

Advanced proficiency with Databricks, including cluster management, notebook development, and Delta Lake features.

Technical skills:

Strong knowledge of SQL, NoSQL databases, and big data technologies like Apache Spark (PySpark, Scala).

Programming:

Proficiency in programming languages like Python.

Other skills:

Experience with data warehousing, data modeling, and performance tuning. Excellent problem-solving and communication skills.

Sr Azure Data Engineer Template - Expertia AI

Responsibilities. Design and develop scalable data architectures on Azure. Implement and optimize data pipelines using Azure Data ...

"