Job Description

We are seeking an experienced Databricks Developer with 5 to 7 years of expertise in big data engineering and analytics platforms. The ideal candidate will have deep experience in Apache Spark Databricks PySpark and cloud platforms such as Azure or AWS and a strong background in data pipeline development performance optimization and data governance.




Key Responsibilities


Design develop and maintain scalable data pipelines using Apache Spark on Databricks


Write efficient and production ready PySpark or Scala code for data transformation and ETL processes


Integrate data from various structured and unstructured sources into a unified platform


Implement Delta Lake and manage data versioning updates and schema evolution


Optimize data processing workflows for performance scalability and cost efficiency


Collaborate with data scientists analysts and business stakeholders to deliver high quality datasets


Implement data quality checks validation routines and logging mechanisms


Monitor and debug production jobs using Databricks jobs notebooks and clusters


Ensure security privacy and compliance standards are met throughout the data lifecycle


Provide guidance and mentorship to junior team members




Required Skills and Qualifications


5 to 7 years of experience in Big Data development


Hands on experience with Databricks including Workflows Notebooks Delta Live Tables Unity Catalog


Strong programming skills in PySpark and or Scala


Solid understanding of Delta Lake architecture


Proficient in SQL for data analysis and transformation


Experience with cloud platforms such as Azure Azure Data Lake Data Factory Synapse or AWS S3 Glue Redshift


Familiarity with CICD for Databricks deployments using GitHub Actions Azure DevOps


Knowledge of data governance cataloguing and security best practices


Experience working in an Agile Scrum environment




Preferred Skills


Experience with Databricks Unity Catalog and Delta Live Tables


Exposure to machine learning workflows in Databricks


Experience with Apache Airflow Kafka or other orchestration messaging tools


Certifications Databricks Certified Data Engineer Associate Professional Azure AWS certification

About Virtusa

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth -- one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.


Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.


Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.