Job Description

Position Overview:We are seeking a highly skilled Senior Data Engineer - Python, PySpark & Azure Databricks to join our dynamic data engineering team. This role focuses on building scalable, high-performance data pipelines using Python and PySpark within the Azure Databricks environment. While familiarity with broader Azure services is valuable, the emphasis is on distributed data processing and automation using modern big data frameworks. Prior experience in the Property & Casualty (P&C) insurance industry is a strong plus.Key Responsibilities:Data Pipeline Development & Optimization:Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.Leverage Azure Databricks to process large volumes of structured and semi-structured data efficiently.Implement data quality checks, error handling, and performance tuning across all stages of data processing.Data Architecture & Modeling:Contribute to the design of cloud-based data architectures that support analytics and reporting use cases.Develop and maintain data models that adhere to industry best practices and support business requirements.Work with Delta Lake, Bronze/Silver/Gold data architecture patterns, and metadata management strategies.Cloud Integration (Azure):Integrate and orchestrate data workflows using Azure Data Factory, Azure Blob Storage, and Event Hub where applicable.Optimize cloud compute resources and manage cost-effective data processing at scale.Collaboration & Stakeholder Engagement:Partner with data analysts, data scientists, and business users to understand evolving data needs.Collaborate with DevOps and platform teams to ensure reliable, secure, and automated data operations.Participate in Agile and contribute to sprint planning, demos, and retrospectives.Documentation & Best Practices:Maintain clear and comprehensive documentation of code, pipelines, and architectural decisions.Contribute to internal data engineering standards and promote best practices for code quality, testing, and CI/CD.

Qualifications

------------------

Position Overview:We are seeking a highly skilled Senior Data Engineer - Python, PySpark & Azure Databricks to join our dynamic data engineering team. This role focuses on building scalable, high-performance data pipelines using Python and PySpark within the Azure Databricks environment. While familiarity with broader Azure services is valuable, the emphasis is on distributed data processing and automation using modern big data frameworks. Prior experience in the Property & Casualty (P&C) insurance industry is a strong plus.Key Responsibilities:Data Pipeline Development & Optimization:Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.Leverage Azure Databricks to process large volumes of structured and semi-structured data efficiently.Implement data quality checks, error handling, and performance tuning across all stages of data processing.Data Architecture & Modeling:Contribute to the design of cloud-based data architectures that support analytics and reporting use cases.Develop and maintain data models that adhere to industry best practices and support business requirements.Work with Delta Lake, Bronze/Silver/Gold data architecture patterns, and metadata management strategies.Cloud Integration (Azure):Integrate and orchestrate data workflows using Azure Data Factory, Azure Blob Storage, and Event Hub where applicable.Optimize cloud compute resources and manage cost-effective data processing at scale.Collaboration & Stakeholder Engagement:Partner with data analysts, data scientists, and business users to understand evolving data needs.Collaborate with DevOps and platform teams to ensure reliable, secure, and automated data operations.Participate in Agile and contribute to sprint planning, demos, and retrospectives.Documentation & Best Practices:Maintain clear and comprehensive documentation of code, pipelines, and architectural decisions.Contribute to internal data engineering standards and promote best practices for code quality, testing, and CI/CD.

Job Location

----------------