Job Description

Introduction
At IBM Global Sales, we bring together innovation, collaboration, and expertise to help clients solve their most complex business challenges. Working across industries and geographies, you'll partner with colleagues, clients, and partners to co-create solutions that drive digital transformation and lasting impact.Success in Global Sales is built on curiosity, empathy, and collaboration. You'll connect technical understanding with strong people skills, building trusted relationships and shaping solutions that improve business and society. With world-class onboarding, continuous learning, and a supportive culture, IBM offers the tools and opportunities to grow your career. Join us and be part of a global team that's passionate about driving innovation and making a difference.
Your role and responsibilities
We are seeking a highly skilled and experienced Spark Cluster Admin to join our dynamic team. This role is critical for building, optimizing, and supporting our cutting-edge data platform, leveraging Apache Spark, Apache Iceberg, and a robust DevOps approach within an OpenShift environment. The ideal candidate will be adept at both developing high-performance data solutions and ensuring their stability and reliability in a production setting.
Key Responsibilities

• Design, develop, and optimize scalable and resilient data processing applications using Apache Spark (batch, streaming, and real-time).
• Implement and manage data pipelines, ensuring data quality, consistency, and performance.
• Perform Spark job performance tuning and optimization to handle large-scale datasets efficiently.
• Manage and automate the deployment of Spark applications within OpenShift clusters, utilizing Docker and Kubernetes.
• Establish and maintain CI/CD pipelines for automated testing, deployment, and release management of Spark workloads.
• Provide comprehensive production support for critical Spark jobs, including proactive monitoring, troubleshooting, debugging, and participation in on-call rotations.
• Work extensively with Apache Iceberg table format, leveraging its capabilities for schema evolution, time travel, hidden partitioning, and ACID transactions.
• Collaborate closely with data scientists, other data engineers, and operations teams to deliver robust and integrated solutions.
• Develop and maintain documentation for data pipelines, job configurations, and operational procedures.

Required education
Bachelor's Degree
Preferred education
Master's Degree
Required technical and professional expertise

• 9+ years of hands-on experience in Spark cluster administration roles.
• Expert-level proficiency in Apache Spark (Spark Core, Spark SQL, Spark Streaming).
• Strong programming skills in Scala, Python (PySpark), or Java.
• Significant experience with OpenShift, including deploying, managing, and automating containerized applications within the platform.
• Solid understanding of Docker and Kubernetes for containerization and orchestration.
• Proven experience implementing and maintaining CI/CD pipelines using tools like Jenkins, GitLab CI, or similar.
• Demonstrable experience with Apache Iceberg, including practical application of its features like schema evolution, time travel queries, and ACID compliance.
• Strong background in production support for data applications, including monitoring, troubleshooting, and incident resolution.
• Understanding and practical application of DevOps principles (Infrastructure as Code, automation, continuous monitoring).
• Strong SQL skills and experience working with various data sources.
• Excellent analytical and problem-solving abilities for diagnosing and resolving complex issues in distributed environments.
• Familiarity with distributed systems concepts and architectures.

Preferred technical and professional experience

• Experience with other data lake table formats (e.g., Delta Lake, Apache Hudi).
• Familiarity with cloud platforms (AWS, Azure, GCP) beyond OpenShift.
• Experience with messaging queues or streaming platforms like Apache Kafka.
• Contributions to open-source data projects.

ABOUT BUSINESS UNIT
IBM has a global presence, operating in more than 175 countries with a broad-based geographic distribution of revenue. The company's Global Markets organization is a strategic sales business unit that manages IBM's global footprint, working closely with dedicated country-based operating units to serve clients locally. These country teams have client relationship managers who lead integrated teams of consultants, solution specialists and delivery professionals to enable clients' growth and innovation. By complementing local expertise with global experience and digital capabilities, IBM builds deep and broad-based client relationships. This local management focus fosters speed in supporting clients, addressing new markets and making investments in emerging opportunities. Additionally, the Global Markets organization serves clients with expertise in their industry as well as through the products and services that IBM and partners supply. IBM is also expanding its reach to new and existing clients through digital marketplaces.
YOUR LIFE @ IBM
In a world where technology never stands still, we understand that, dedication to our clients success, innovation that matters, and trust and personal responsibility in all our relationships, lives in what we do as IBMers as we strive to be the catalyst that makes the world work better.
Being an IBMer means you'll be able to learn and develop yourself and your career, you'll be encouraged to be courageous and experiment everyday, all whilst having continuous trust and support in an environment where everyone can thrive whatever their personal or professional background.
Our IBMers are growth minded, always staying curious, open to feedback and learning new information and skills to constantly transform themselves and our company. They are trusted to provide on-going feedback to help other IBMers grow, as well as collaborate with colleagues keeping in mind a team focused approach to include different perspectives to drive exceptional outcomes for our customers. The courage our IBMers have to make critical decisions everyday is essential to IBM becoming the catalyst for progress, always embracing challenges with resources they have to hand, a can-do attitude and always striving for an outcome focused approach within everything that they do.
Are you ready to be an IBMer?
ABOUT IBM
IBM's greatest invention is the IBMer. We believe that through the application of intelligence, reason and science, we can improve business, society and the human condition, bringing the power of an open hybrid cloud and AI strategy to life for our clients and partners around the world.
Restlessly reinventing since 1911, we are not only one of the largest corporate organizations in the world, we're also one of the biggest technology and consulting employers, with many of the Fortune 500 companies relying on the IBM Cloud to run their business.
At IBM, we pride ourselves on being an early adopter of artificial intelligence, quantum computing and blockchain. Now it's time for you to join us on our journey to being a responsible technology innovator and a force for good in the world.
IBM is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, neurodivergence, age, or other characteristics protected by the applicable law. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
OTHER RELEVANT JOB DETAILS
When applying to jobs of your interest, we recommend that you do so for those that match your experience and expertise. Our recruiters advise that you apply to not more than 3 roles in a year for the best candidate experience. For additional information about location requirements, please discuss with the recruiter following submission of your application.
Job Title
Spark Cluster Admin
Job ID
81348
City / Township / Village
BANGALORE
State / Province
Karnataka
Country
India
Work arrangement
Hybrid
Area of work
Consulting
Employment type
Regular
Position type
Professional
Travel required
No Travel
Company
(0063) IBM India Private Limited
Shift
General (daytime)
Is this role a commissionable/sales incentive based position?
No

Skills Required