Job Description

We are seeking a highly skilled and experienced

Senior Advisor - Data Engineering

with 10 plus years of experience to lead the design, development, and implementation of end-to-end software and data engineering solutions. In this role, you will be at the forefront of building scalable, high-performance systems for data processing, storage, and analytics while maintaining best practices in software engineering.



As a Senior Advisor, you will guide the technical direction of both software and data projects, mentor junior engineers, and ensure that data engineering solutions are efficient, reliable, and align with business goals. You will work across cloud technologies (primarily

Google Cloud Platform

) and be responsible for building data pipelines, orchestrating workflows, and leveraging cloud-native tools for modern data solutions.

Key Responsibilities:

Solution Design & Architecture:

Lead the design and architecture of complex software and data engineering solutions, ensuring they are scalable, reliable, and secure. Build and maintain

end-to-end data pipelines

for efficient data ingestion, processing, and transformation in cloud environments (primarily

Google Cloud

). Design and implement cloud-based data solutions using

Cloud Run

,

Cloud Functions

,

Big Query, Datastore

,

GCS

,

Dataplex

,

Pub/Sub, Cloud Scheduler

and

Composer

. Ensure data quality, consistency, and integrity across data sources and throughout the ETL pipeline. Implement

data governance

best practices, ensuring compliance, security, and data privacy in the processing of sensitive information.

Software Engineering & Development:

Contribute to the development of scalable, maintainable, and efficient software applications in

Python, Scala, Spark

. Write code to integrate various data systems and ensure seamless data flow between storage, processing, and analysis layers. Develop and optimize

SQL queries, Stored Procedure

for

Big Query

to support efficient data retrieval and analytics.

Mentorship & Team Guidance:

Mentor junior software and data engineers, providing guidance on technical aspects (e.g., data pipeline design, cloud technologies, OOP principles). Provide technical leadership in both

software engineering

and

data engineering

, helping to define best practices and standards for the team.

Collaboration & Communication:

Work closely with cross-functional teams to define requirements and deliver high-quality solutions. Communicate technical solutions clearly and concisely

Code Quality & Best Practices:

Promote a culture of clean, maintainable code and ensure best practices for software and data engineering are followed. Implement design patterns and architectural principles in both software and data engineering projects to ensure long-term maintainability.

Continuous Improvement:

Stay up to date with emerging trends in both

software/data engineering to

bring new tools, technologies, and approaches to the team. Advocate for continuous improvement in the development and deployment processes, particularly around data management, cloud services, and automation.

Required Skills & Qualifications:

Proven Expertise in:

Object-Oriented Programming (OOP)

and software design principles.

Python

for building cloud-native applications, data processing, and automation. Extensive experience with

Google Cloud Platform (GCP)

, particularly with

BigQuery

,

Cloud Functions

,

Cloud Run

,

Datastore

,

GCS, Pub/Sub, Cloud Scheduler

,

Dataplex

, and

Composer

. Strong expertise in

data engineering

, including building

ETL pipelines

,

data warehousing

, and

data storage solutions

. Proficiency in

PostgreSQL

,

MS-SQL

, and designing relational database solutions. Familiarity with

Apache Airflow

for workflow orchestration and managing data pipelines. Ability to design scalable

data architectures

and work with both structured and unstructured data.

Architectural Skills:

Deep understanding of

design patterns

and their application to both software engineering and data engineering solutions. Experience designing and implementing large-scale

data systems

and distributed architectures.

Mentorship & Leadership:

Proven experience mentoring and guiding junior engineers

Nice to Have:

Experience with

Apache Spark and Scala

for large-scale data processing. Exposure with

Azure

cloud services.

Education:

* Bachelor's in computers science or related.