Job Description

Job Summary:
We are seeking a highly skilled Data Engineer with expertise in leveraging Data Lake architecture and the Azure cloud platform to develop, deploy, and optimize data-driven solutions. You will play a pivotal role in transforming raw data into actionable insights, supporting strategic decision-making across the organization.

Key Responsibilities

Develop and optimize scalable data pipelines using

Python

and

PySpark

Build and orchestrate data workflows with

Azure Data Factory

Design and implement solutions using

Azure Databricks

and

Synapse Analytics

Manage and maintain data storage solutions in

Azure Data Lake Storage Gen2

, leveraging cost-efficient architecture Implement and manage

CI/CD pipelines

using

Azure Logic Apps

, with integration of tools such as

Azure DevOps

,

GitHub Actions

, or

Jenkins

Model and maintain

Medallion Architecture

across Bronze, Silver, and gold layers based on business requirements Collaborate with data scientists, analysts, and business stakeholders to ensure reliable data availability Monitor and optimize performance and cost efficiency of data solutions across the ecosystem

Required Skills & Experience

Strong proficiency in

Python

and

PySpark

for data manipulation and transformation Hands-on experience with

Azure Data Factory

,

Azure Databricks

, and

Synapse Analytics

Familiarity with

Azure Logic Apps

for CI/CD pipeline management Knowledge of CI/CD tools such as:

Azure DevOps Pipelines

GitHub Actions

Jenkins

Expertise in managing

Azure Data Lake Storage Gen2

environments with emphasis on security and cost optimization Deep understanding of

Medallion Architecture

principles:

Bronze Layer

: Raw data ingestion

Silver Layer

: Cleansed, enriched, and business-ready data

Gold Layer

: Aggregated and analytics-ready data models Strong problem-solving and communication skills * Experience on Cost optimization