Job Description

As a Data Engineer, you apply broad knowledge and technical know-how in the Data Engineering domain to deliver the following responsibilities: Validate design, build, test and deploy software solutions into production Support your team to develop innovative software that maintains data/network security management, leveraging set processes to reduce the time between commission and production Collaborate effectively with your team to translate customer needs into leading-edge software application products, services or systems that meet company s time, cost and quality standards Seek diverse viewpoints from colleagues in your team, and apply analytical thinking to resolve complex problems that don t have an immediate or tried-and-tested solution Contribute to continuous improvement initiatives of our systems and processes, apply best practice corporate testing standards and our quality assurance approach to drive improved outcomes (eg productivity, customer experience and/or profitability) Apply a continuous learning and development mindset to your work, using tried and tested methodologies but also finding opportunities to experiment to incrementally improve or resolve challenges Take ownership of your own actions and seek guidance from more experienced engineers in order to support your development and contribute to the delivery of team/mission outcomes Essential Hands-on experience in the following: Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing (AWS Redshift) Skilled in the following: Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc) Azure data factory/Airflow/control-M/Luigi PL/SQL Exposure to NOSQL(Hbase / Cassandra / GraphDB(Neo4J) / MongoDB) File formats (Parquet/ORC/AVRO/Delta/Hudi etc.) Kafka/Kinesis/Eventhub Highly Desirable Spark Streaming Cloud exposure (Azure/AWS/GCP) Azure data offerings ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc. Presto/Athena Azure DevOps Jenkins/ Bamboo/Any similar build tools Power BI Prior experience in building or working in team building reusable frameworks, Data modelling. Data Architecture and design principles. (Delta/Kappa/Lambda architecture) Exposure to CI/CD Code Quality Static and Dynamic code scans Agile SDLC

foundit