Job Description

Job Category: KMBL
Degree Level: Bachelor's Degree
:
Job Title : Data Engineer

As a Data Engineer, you will be part of a dynamic team responsible for developing and maintaining data pipelines, databases, and analytical systems @Kotak811

• You will work closely with other data engineers, data scientists, and business stakeholders to ensure data integrity, availability, and reliability.
• You should have expertise in designing, implementing, and operating stable, scalable, solutions to flow data from production systems into analytical data platform (big data tech stack) and into end-user facing applications for both real-time and batch use cases

• Data Ingestion and Integration
• Collaborate with cross-functional teams to gather data requirements and identify data sources.
• Design, develop, and maintain data ingestion pipelines from various structured and unstructured sources.
• Implement data integration processes to ensure data quality and consistency.
• Monitor and troubleshoot data ingestion issues and optimise performance.
• Data Transformation and ETL:
• Translate business requirements into technical specifications (data models)
• Extract, transform, and load data from diverse sources into appropriate data models and structures.
• Develop and maintain ETL workflows and processes using tools like Apache Spark, Apache Kafka, or other data integration frameworks.
• Cleanse, validate, and transform data to meet business and analytical requirements.
• Strong engineering mindset - build automated monitoring, alerting, self healing (restartability/graceful failures) features while building the consumption pipelines
• Database Management:
• Create and maintain scalable and efficient data storage systems, including relational databases, data warehouses, or NoSQL databases.
• Optimise database performance, including indexing, partitioning, and query optimization.
• Implement data security and access controls to ensure data privacy and compliance with industry regulations.
• Monitor database health, performance, and capacity, and take proactive measures to ensure optimal system operation.
• Data Governance and Documentation:
• Ensure adherence to data management policies and standards.
• Document data pipelines, ETL processes, and data flows to facilitate knowledge sharing and maintain data lineage.
• Contribute to the development and maintenance of data dictionaries, data catalogues, and metadata repositories.
• Participate in data quality initiatives and data governance reviews to maintain high data integrity.
• Collaboration and Communication:
• Collaborate effectively with cross-functional teams, including data scientists, business analysts, and software engineers.
• Communicate technical concepts and solutions to both technical and non-technical stakeholders.
• Participate in team meetings, code reviews, and knowledge-sharing sessions to foster a collaborative work environment.
• Stay up-to-date with emerging technologies, industry trends, and best practices in data engineering.

• 2-3 years' experience with a Bachelor's Degree in Computer Science, Engineering, Technology or related field required.
• Proficiency in one or more programming languages such as Python, Java, or Scala.
• Good knowledge of Agile, SDLC/CICD practices and tools
• Experience with data integration tools and frameworks such as Apache Spark, Apache Kafka, or similar.
• Experience is leveraging Cloud / OSS Data platform to build scalable data models and data pipelines for analytical, data science, systemic consumption use cases
• Must have good knowledge of performance tuning/optimising data processing jobs, debugging time consuming jobs.
• Good understanding of distributed systems
• Experience working extensively in multi-petabyte DW environment
• Strong problem-solving skills and attention to detail.