Job Description

Requisition ID
41772BR
Business Unit
COR

We are seeking a highly skilled and experienced Senior Data Engineer with deep expertise in Databricks to join our Digital Capital team. The ideal candidate will have over 6 years of experience working in Databricks to create operational excellence across the platform, develop, optimize, and maintain data pipelines, and has a solid foundation in traditional enterprise data warehousing. You will play a critical role in building and maintaining our next-generation data platform, ensuring data quality, reliability, and accessibility for various analytical and operational needs across CDM Smith.
Key Responsibilities:

• Databricks Platform: Act as a subject matter expert for the Databricks platform within the Digital Capital team, provide technical guidance, best practices, and innovative solutions.
• Databricks Workflows and Orchestration: Design and implement complex data pipelines using Azure Data Factory or Qlik replicate.
• End-to-End Data Pipeline Development: Design, develop, and implement highly scalable and efficient ETL/ELT processes using Databricks notebooks (Python/Spark or SQL) and other Databricks-native tools.
• Delta Lake Expertise: Utilize Delta Lake for building reliable data lake architecture, implementing ACID transactions, schema enforcement, time travel, and optimizing data storage for performance.
• Spark Optimization: Optimize Spark jobs and queries for performance and cost efficiency within the Databricks environment. Demonstrate a deep understanding of Spark architecture, partitioning, caching, and shuffle operations.
• Data Governance and Security: Implement and enforce data governance policies, access controls, and security measures within the Databricks environment using Unity Catalog and other Databricks security features.
• Collaborative Development: Work closely with data scientists, data analysts, and business stakeholders to understand data requirements and translate them into Databricks based data solutions.
• Monitoring and Troubleshooting: Establish and maintain monitoring, alerting, and logging for Databricks jobs and clusters, proactively identifying and resolving data pipeline issues.
• Code Quality and Best Practices: Champion best practices for Databricks development, including version control (Git), code reviews, testing frameworks, and documentation.
• Performance Tuning: Continuously identify and implement performance improvements for existing Databricks data pipelines and data models.
• Cloud Integration: Experience integrating Databricks with other cloud services (e.g., Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Key Vault) for a seamless data ecosystem.
• Traditional Data Warehousing & SQL: Design, develop, and maintain schemas and ETL processes for traditional enterprise data warehouses. Demonstrate expert-level proficiency in SQL for complex data manipulation, querying, and optimization within relational database systems.

Group
COR
Employment Type
Regular
Minimum Qualifications

• Minimum of 6+ years of relevant experience in data engineering, with a significant portion dedicated to building and managing data solutions.
• Demonstrable expert-level proficiency with Databricks, including:

o Extensive experience with Spark (PySpark, Spark SQL) for large-scale data processing.
o Deep understanding and practical application of Delta Lake.
o Hands-on experience with Databricks Notebooks, Jobs, and Workflows.
o Experience with Unity Catalog for data governance and security.
o Proficiency in optimizing Databricks cluster configurations and Spark job performance.

• Strong programming skills in Python.
• Expert-level SQL proficiency with a strong understanding of relational databases, data warehousing concepts, and data modeling techniques (e.g., Kimball, Inmon).
• Solid understanding of relational and NoSQL databases.
• Experience with cloud platforms (preferably Azure, but AWS or GCP with Databricks experience is also valuable).
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and collaboratively in a fast-paced environment.

Preferred Qualifications

• Databricks Certifications (e.g., Databricks Certified Data Engineer Associate/Professional).
• Experience with CI/CD pipelines for data engineering projects.

Why CDM Smith?
Join Us! CDM Smith - where amazing career journeys unfold.
Imagine a place committed to offering an unmatched employee experience. Where you work on projects that are meaningful to you. Where you play an active part in shaping your career journey. Where your co-workers are invested in you and your success. Where you are encouraged and supported to do your very best and given the tools and resources to do so. Where it's a priority that the company takes good care of you and your family.
Our employees are the heart of our company. As an employer of choice, our goal is to provide a challenging, progressive and inclusive work environment which fosters personal leadership, career growth and development for every employee. We value passionate individuals who challenge the norm, deliver world-class solutions and bring diverse perspectives. Join our team, and together we will make a difference and change the world.
Agency Disclaimer
All vendors must have a signed CDM Smith Placement Agreement from the CDM Smith Recruitment Center Manager to receive payment for your placement. Verbal or written commitments from any other member of the CDM Smith staff will not be considered binding terms. All unsolicited resumes sent to CDM Smith and any resume submitted to any employee outside of CDM Smith Recruiting Center Team (RCT) will be considered property of CDM Smith. CDM Smith will not be held liable to pay a placement fee.
Amount of Travel Required
0%
Assignment Category
Fulltime-Regular
Background Check and Drug Testing Information
CDM Smith Inc. and its divisions and subsidiaries (hereafter collectively referred to as "CDM Smith") reserves the right to require background checks including criminal, employment, education, licensure, etc. as well as credit and motor vehicle when applicable for certain positions. In addition, CDM Smith may conduct drug testing for designated positions. Background checks are conducted after an offer of employment has been made in the United States. The timing of when background checks will be conducted on candidates for positions outside the United States will vary based on country statutory law but in no case, will the background check precede an interview. CDM Smith will conduct interviews of qualified individuals prior to requesting a criminal background check, and no job application submitted prior to such interview shall inquire into an applicant's criminal history. If this position is subject to a background check for any convictions related to its responsibilities and requirements, employment will be contingent upon successful completion of a background investigation including criminal history. Criminal history will not automatically disqualify a candidate. In addition, during employment individuals may be required by CDM Smith or a CDM Smith client to successfully complete additional background checks, including motor vehicle record as well as drug testing.