Databricks, Azure Data Services ecosystem (ADF, ADLS, Microsoft Fabric, Synapse, etc.), Python, PySpark, and databases
. The ideal candidate will be responsible for designing, building, and optimizing scalable data pipelines and solutions to enable advanced analytics, business intelligence, and AI-driven initiatives.
Key Responsibilities
Design, develop, and maintain
data pipelines
using
Azure Data Factory (ADF)
and
Databricks
.
Work with
Azure Data Lake Storage (ADLS)
,
Microsoft Fabric
, and
Azure Synapse Analytics
to build robust data platforms.
Develop scalable
ETL/ELT solutions
leveraging
Python, PySpark, and SQL
.
Optimize data workflows for
performance, reliability, and cost efficiency
.
Collaborate with data scientists, analysts, and business teams to ensure
data availability and quality
.
Implement best practices for
data governance, security, and compliance
in the Azure ecosystem.
Monitor, troubleshoot, and enhance data processes and pipelines in
production environments
.
Stay current with evolving
Azure cloud technologies, big data frameworks, and modern data architectures
.
Required Skills & Qualifications
Strong experience with
Azure Data Services
:
Azure Data Factory (ADF)
Azure Data Lake Storage (ADLS)
Microsoft Fabric (nice-to-have but preferred)
Azure Synapse Analytics
Expertise in
Databricks
for data engineering and analytics.
Proficiency in
Python
and
PySpark
for large-scale data processing.
Strong knowledge of
SQL
and relational/non-relational
databases
.
Experience with
data modeling, ETL/ELT pipelines, and data warehousing
concepts.
Understanding of
CI/CD, version control (Git), and DevOps practices
.
Excellent problem-solving, communication, and collaboration skills.
Preferred Qualifications
Experience with
data governance
frameworks and
security best practices
in Azure.
Knowledge of
machine learning integration
within Databricks.
Familiarity with
Power BI
or other visualization tools.
Prior experience in
agile environments
.
Education & Experience
Bachelor's or Master's degree in
Computer Science, Information Systems, Data Engineering, or related field