Lead the design, architecture, and implementation of scalable and maintainable web scraping solutions using the Scrapy framework, integrated with tools such as Kafka, Zookeeper, and Redis
Develop and maintain web crawlers to automate data extraction from various sources, ensuring alignment with user and application requirements
Research, design, and implement automation strategies across multiple platforms, tools, and technologies to optimize business processes
Monitor, troubleshoot, and resolve issues affecting the performance, reliability, and stability of scraping systems and automation tools
Serve as a Subject Matter Expert (SME) for automation systems, providing guidance and support to internal teams
Analyze and validate extracted data to ensure accuracy, integrity, and compliance with client's data standards
Define, implement, and enforce data requirements, standards, and best practices to ensure consistent and efficient operations
Collaborate with stakeholders and end users to define technical requirements, business goals, and alternative solutions for data collection and reporting
Create, manage, and document reports, processes, policies, and project plans, including risk assessments and goal tracking
Conduct code reviews, enforce coding standards, and provide technical leadership and mentorship to development team members
Proactively identify and mitigate technical risks, recommending improvements in technologies, tools, and processes
Drive the adoption of modern development tools, frameworks, and best practices
Contribute to strategic planning related to automation initiatives and product development
Ensure clear, thorough communication and documentation across teams to support knowledge sharing and training
*Minimum Qualifications:
Bachelor's degree in Computer Science, Software Engineering, or related field.
5+ years of professional software development experience.
Strong proficiency in HTML, XML, XPath, XSLT, and Regular Expressions for data extraction and transformation
Hands-on experience with Visual Studio
Strong proficiency in Python
Some experience with C# .NET
Solid experience with MS SQL Server, with strong skills in SQL querying and data analysis
Experience with web scraping, particularly using the Scrapy framework integrated with Kafka, Zookeeper, and Redis
Experience with .NET automation tools such as Selenium
Understanding of CAPTCHA-solving services and working with proxy services
Experience working in a Linux environment is a plus
Highly self-motivated and detail-oriented, with a proactive, goal-driven mindset
Strong team player with dependable work habits and well-developed interpersonal skills
Excellent verbal and written communication skills
Demonstrates willingness and flexibility to adapt schedule when necessary to meet client needs.
Core Technical Skills- Python (strong proficiency)
1. HTML, XML, XPath, XSLT, Regex (for data extraction)
2. Visual Studio
3. C# .NET (some experience)
4. MS SQL Server (SQL querying & analysis)
5. Web Scraping Framework: Scrapy (integrated with Kafka, Zookeeper, Redis)