to join our growing data team. This role is ideal for professionals who are passionate about building scalable data pipelines, ensuring data quality, and enabling advanced analytics and AI workloads. You will work closely with cross-functional teams to design, implement, and optimize data solutions that drive business insights and innovation.
Key Responsibilities:
Design, develop, and optimize end-to-end data pipelines using
PySpark notebooks
within the
Fabric Lakehouse
environment.
Architect and maintain
Delta Lake
structures with efficient partitioning and versioning in
OneLake
.
Implement robust data validation, testing, and monitoring frameworks to ensure data quality and integrity.
Collaborate with Data Architects, Analysts, and AI Engineers to deliver high-quality datasets for
Power BI
and other reporting tools.
Build and manage
CI/CD pipelines
for Fabric notebooks, dataflows, and other assets using
Azure DevOps
or
GitHub
.
Optimize distributed
Spark workloads
for performance and cost-efficiency.
Ensure compliance with data governance and security standards using tools like
Microsoft Purview
.
Provide technical leadership, conduct code reviews, and mentor junior engineers.
Contribute to architecture discussions, solution design, and documentation of data workflows.
Required Qualifications:
Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.
5+ years of experience in big data engineering, with at least 2 years of hands-on experience with
PySpark
.
Proven experience with
Microsoft Fabric
,
Lakehouse architecture
,
Delta Lake
, and
OneLake
.
Strong SQL and Spark optimization skills.
Experience with
Power BI
,
DirectLake
, and semantic model design.
Familiarity with
medallion architecture
(bronze, silver, gold layers) and data warehousing principles.
Experience implementing
CI/CD pipelines
and version control for data workflows.
Strong understanding of data testing frameworks and monitoring solutions.
Excellent communication and problem-solving skills.
Preferred Qualifications:
Certifications in
Microsoft Fabric
,
Azure Data Engineering
, or
Databricks
.
Experience with
Microsoft Purview
for data governance and lineage.
Familiarity with other modern data platforms like
Azure Synapse
,
Databricks
, or
Snowflake
.
Exposure to
AI/ML workloads
integrated into data pipelines.
Why Join Us?
At Prodat IT Solutions, we are committed to building a data-driven culture. You'll work with cutting-edge technologies, collaborate with talented professionals, and contribute to impactful projects that shape the future of data and analytics.
How to Apply:
Send your updated resume to
Rakes.k@prodatit.com / 9606018324 .
Job Type: Full-time
Pay: Up to ?2,000,000.00 per year
Benefits:
Health insurance
Provident Fund
Work from home
Application Question(s):
What is your Notice Period ?
Work Location: In person
Beware of fraud agents! do not pay money to get a job
MNCJobsIndia.com will not be responsible for any payment made to a third-party. All Terms of Use are applicable.