Job Description

Job Summary: Under general supervision, supports the design, build, test, and maintenance of data pipelines at big data scale. Assists with updating data from multiple data sources. Other functionalities under general supervision include working on batch processing of collected data and matching its format to the stored data, making sure that the data is ready to be processed and analyzed. Assists with keeping the ecosystem and the pipeline optimized and efficient, troubleshooting standard performance, data related problems and providing L3 support. Provides recommendations to moderately complex issues. : Experience with distributed and cloud environments and an adequate understanding of how to balance computational considerations with theoretical properties Detailed knowledge of the Microsoft Azure tooling for large-scale data engineering efforts and deployments is highly preferred. Experience with designing and deploying large scale technical solutions, which deliver tangible, ongoing value Direct experience having built and deployed robust, complex production systems that implement modern, data scientific methods at scale Ability to context-switch, to provide support to dispersed teams which may need an \'expert hacker\' to unblock an especially challenging technical obstacle, and to work through problems as they are still being defined Demonstrated ability to deliver technical projects with a team, often working under tight time constraints to deliver value An \'engineering\' mindset, willing to make rapid, pragmatic decisions to improve performance, accelerate progress or magnify impact Comfort with working with distributed teams on code-based deliverables, using version control systems and code reviews Ability to conduct data analysis, investigation, and lineage studies to document and enhance data quality and access Use of agile and devops practices for project and software management including continuous integration and continuous delivery Demonstrated expertise working with some of the following common languages and tools: Spark (Scala and PySpark), HDFS, Kafka and other high volume data tools SQL and NoSQL storage tools, such as MySQL, Postgres, Cassandra, MongoDB and Elasticsearch Pandas, Scikit-Learn, Matplotlib, TensorFlow, Jupyter and other Python data tools Must have: Knowledge of data integration tools such as Ab Initio Knowledge of database structures/ data warehousing principles, and experience in SQL across different platforms including structured and unstructured data from various enterprise sources. Ability to manipulate and analyze data with a scientific approach in order to determine the best and most optimal solution. Knowledge of message queuing, stream processing, and highly scalable \'big data\' data stores and experience with building and optimizing \'big data\' data pipelines, architectures and data sets. Experience in databases like SQL, Oracle, cloud databases Good to have: Experience with implementing large-scale data engineering efforts. Build processes supporting data transformation, data structures, metadata, dependency and workload management. Concepts and work experience with SAFe/Agile methodology Experience in Informatica, Azure data Factory. Minimum Qualifications: Bachelor\'s degree in computer science, Information Systems, a related quantitative field such as Engineering or Mathematics or equivalent formal training or work experience. 3-6 years equivalent work experience in measurement and analysis, quantitative business problem solving, simulation development and/or predictive analytics and 2 years of relevant work experience as a data engineer. Strong knowledge in data engineering and machine learning frameworks including design, development and implementation of highly complex systems and data pipelines. Strong knowledge in Information Systems including design, development and implementation of large batch or online transaction-based systems. Strong understanding of the transportation industry, competitors, and evolving technologies. Experience as a member of multi-functional project teams. Strong oral and written communication skills. A related advanced degree may offset the related experience requirements.

foundit