Job Description

About the Role

Uber's data infrastructure is composed of a wide variety of compute engines, scheduling/execution solutions, and storage solutions. Compute engines such as Apache Spark(TM), Presto, Apache Hive(TM), Neutrino, Apache Flink, etc., allow Uber to run petabyte-scale operations on a daily basis. Further, scheduling and execution engines such as Piper (Uber's fork of Apache Airflow(TM)), Query Builder (user platform for executing compute SQLs), Query Runner (proxy layer for execution of workloads), and exist to allow scheduling and execution of compute workloads. Finally, a significant portion of storage is supported by HDFS, Google Cloud Storage (GCS),Apache Pinot(TM), ElasticSearch, etc. Each engine supports thousands of executions, which are owned by multiple owners and sub-teams.



With such a complex and diverse big data landscape operating at petabyte-scale and around a million applications/queries running each day, it's imperative to provide the stakeholders a holistic view of the right performance and resource consumption insights.




DataCentral, is a comprehensive platform that provides users with essential insights into big data applications and queries. It empowers data platform users by offering detailed information on workflows and apps, improving productivity by reducing debugging time and improving the cost efficiency by providing detailed resource efficiency insights



As an engineer in the Data Central Team, you will be solving some of the most complex problems in Observability and efficiency of Distributed Data Systems at Uber scale.

What You'll Do

Work with Uber data science and engineering teams to improve Observability of Batch Data use-cases at Uber. Leverage knowledge of spark internals to dramatically help improve customer's Spark job performance. Design and implement AI based solutions to improve the application debuggability. Design and implement algorithms to optimize Resource consumption without impacting reliability Design and develop prediction and forecasting models to proactively predict system degradations and failures Work with multiple partner teams within and Uber and build cross-functional solutions in a collaborative work environment. Work with the community to upstream Uber's contributions to open source and also keep our internal fork up to date

What You'll Need

Bachelor's degree in Computer Science or related field. 5+ years of experience building large scale distributed software systems. Solid understanding of Java for backend / systems software development. MS / PhD in Computer Science or related field. Experience managing production systems with a strong availability SLA. Experience working with Apache Spark or similar analytics technologies. Experience working with large scale distributed systems, HDFS / Yarn. Experience working with SQL Compiler, SQL Plan / Runtime Optimization. * Experience working with Kubernetes