Job Description

Hi ,

Hope you are doing well.

I am hiring for position with client. Please go through the JD below and let me know your interest on swapnil.t@ipeopleinfosystems.com

Role:

Data Engineer

Location: Gr. Noida , Hyderabad , Pune (Hybrid)

Type: Full time

Experience -: 3.5-7 Years

Mandatory Skills : Kafka + Databricks + Spark

Interview Process-

First round- Technical (Virtual) Second round- Technical (Face to Face Mandatory) Third round- Virtual (Technical+Managerial)

Job description

We are seeking a skilled and detail-oriented

Data Engineer (Kafka/Apache Flume)

to join our team. The ideal candidate will have 4+ years of hands-on experience in development, with a strong grasp of Apache Flume, data modeling, and migration.

You will work closely with cross-functional teams to deliver data-driven solutions that empower strategic decision-making across the organization.

To lead the migration of existing logs from SQL Server databases to an Apache Flume application and establish a robust pipeline for daily log ingestion.

The ideal candidate will have hands-on experience in data engineering, log management, and automation of ETL processes.

Key Responsibilities:

Log Migration:

Design and implement a solution to extract and transfer historical logs from SQL Server to Apache Flume.

Ensure data integrity, completeness, and consistency during migration.

Daily Log Ingestion:

Develop and automate a pipeline to move newly generated logs from SQL Server to Flume on a daily basis.

Monitor and maintain the pipeline for performance and reliability.

System Integration & Optimization:

Collaborate with infrastructure and application teams to ensure seamless integration.

Optimize data flow and troubleshoot issues in real-time.

Documentation & Reporting:

Maintain detailed documentation of architecture, processes, and configurations.

Provide regular updates and reports to stakeholders.

Required Skills & Qualifications:

3+ years of experience in data engineering or ETL development.

Strong expertise in SQL Server and Apache Flume.

Proficiency in scripting languages (e.g., Python, Shell) for automation.

Experience with log management, data pipelines, and system integration.

Familiarity with big data ecosystems (e.g., Hadoop, Kafka) is a plus.

Excellent problem-solving and communication skills.

Preferred Qualifications:

Experience with cloud platforms (Azure, AWS, GCP).

Knowledge of data security and compliance standards.

Certification in data engineering or related technologies.

Best regards,

Swapnil Thakur

Recruitment & Delivery Lead

iPeople Infosystems LLC

Contact No: +91 8956457311

Email ID: swapnil.t@ipeopleinfosystems.com

Visit us at www.ipeopleinfosystems.com

Job Type: Full-time

Pay: ?1,000,000.00 - ?2,200,000.00 per year

Work Location: In person