Job Description

Role description

5 days working from Client office inManyata Tech Park Bengaluru



About the Role



We are seeking hands on Data Engineers to design build and operate secure data pipelines and analytical platforms on Azure with Databricks Youll work inside a cleanroom controlled environment to ingest transform and publish high quality datasets for analytics ML and reportingensuring governance privacy and performance at scale



Key Responsibilities



1Data Pipelines ETLELT



oDesign and implement batchstreaming pipelines using Azure Databricks PySpark and Delta Lake



oBuild ingestion frameworks for diverse sources files APIs databases applying robust data quality checks and observability logging metrics s



2Lakehouse Architecture



oModel curated layers BronzeSilverGold with Delta tables optimize for ACID Z Ordering partitioning and costperformance



oImplement schema evolution CDC and data versioning



3Azure Data Services



oIntegrate Azure Data Lake Storage ADLS Gen2 Azure Data FactorySynapse pipelines Azure Key Vault Event HubService Bus as applicable



oManage secrets identities Managed Identity and role based access Azure RBAC



4Operational Excellence



oOwn CICD for Databricks ReposJobs infra automation TerraformBicep and environment promotion



oEnsure SLAsSLOs implement cost optimization auto scaling and cluster policies



5Data Quality Governance



oDefine DQ rules eg Great Expectations Deequ and lineage adhere to privacy and compliance standards in cleanroom operations



oWork closely with InfoSec to maintain air gappedcontrolled workflows and review access requests



6Collaboration Documentation



oPartner with AnalyticsML teams to productionize feature stores write clear runbooks data dictionaries and operating procedures



Must Have Qualifications



2 to 6 years in Data Engineering with strong PySparkSpark experience



Hands on with Azure Databricks clusters jobs notebooks repos and Delta Lake



Practical experience with ADLS Gen2 Azure Data FactorySynapse Azure DevOpsGitHub Actions



Solid SQL performance tuning window functions and Python dataETL utilities



Experience in production data pipelines scheduling monitoring ing



Understanding of data modeling dimensional wide tables partitioning and performance optimization



Comfort working onsite five days in a cleanroom with strict governance



Good to Have Preferred



Streaming Structured Streaming KafkaEvent Hub



Orchestration AirflowAzure Data Factory advanced patterns



Testing Quality Great ExpectationsDeequ unitintegration tests for data pipelines



Security Key Vault Private Endpoints VNET network rules familiarity with PII handling



Infra as Code TerraformBicep for Azure resources Databricks workspacecluster policies



Cost Observability Photon auto optimize Delta cache Lakehouse performance tuning



Exposure to ML feature pipelinesMLflow and feature stores nice to have



Evaluation Rubric high level



DatabricksSpark 30 Coding quality performance Delta features



Azure Data Stack 25 ADLS ADFSynapse security identities



Data Modeling Quality 20 BronzeSilverGold DQ rules lineage



OpsCICDCost 15 Automation monitoring cost controls



Cleanroom Readiness 10 Compliance discipline documentation

Skills

Mandatory Skills :

Azure Data Factory, Databricks

About LTIMindtree

LTIMindtree is a global technology consulting and digital solutions company that enables enterprises across industries to reimagine business models, accelerate innovation, and maximize growth by harnessing digital technologies. As a digital transformation partner to more than 700 clients, LTIMindtree brings extensive domain and technology expertise to help drive superior competitive differentiation, customer experiences, and business outcomes in a converging world. Powered by 87,000+ talented and entrepreneurial professionals across more than 40 countries, LTIMindtree -- a Larsen & Toubro Group company -- solves the most complex business challenges and delivers transformation at scale. For more information, please visit https://www.ltimindtree.com/. Please also note that neither LTIMindtree nor any of its authorized recruitment agencies/partners charge any candidate registration fee or any other fees from talent (candidates) towards appearing for an interview or securing employment/internship. Candidates shall be solely responsible for verifying the credentials of any agency/consultant that claims to be working with LTIMindtree for recruitment. Please note that anyone who relies on the representations made by fraudulent employment agencies does so at their own risk, and LTIMindtree disclaims any liability in case of loss or damage suffered as a consequence of the same. Recruitment Fraud Alert - https://www.ltimindtree.com/recruitment-fraud-alert/