Job Description

Position: Senior Data Engineer

We are seeking an experienced

Senior Data Engineer

to design and implement high-performance data pipelines, ensuring the smooth flow of large datasets across systems. In this role, you will leverage your expertise in

Databricks

,

Pyspark

, and

ETL design

to build efficient data solutions. You will also contribute to optimizing database performance and ensure that data systems meet both scalability and security standards.

Key Responsibilities:

Develop, optimize, and maintain scalable

ETL pipelines

for processing large datasets using

Databricks

and

Pyspark

. Apply advanced techniques such as

chunking

and

partitioning

to handle large file volumes efficiently. Tuning and optimizing databases for better performance and storage efficiency. Collaborate with cross-functional teams to ensure the architecture meets business requirements and data quality standards. Design and implement solutions with an eye toward scalability and long-term performance. Work with

Azure

services, with exposure to

Azure Function Apps

,

Azure Data Factory

, and

Cosmos DB

(preferred). Communicate effectively with stakeholders to align on project goals and deliverables.

Key Skills and Experience:

Databricks

(advanced-level skills in data engineering workflows)

Pyspark

(intermediate-level skills for processing big data) Strong

ETL design skills

, particularly in partitioning, chunking, and database tuning for large datasets

Azure

experience (a plus, including

Function Apps

,

Cosmos DB

, and

Azure Data Factory

) * Excellent communication skills for effective collaboration with teams and stakeholders