Senior Lab Engineer \xe2\x80\x93 Lab Operations Bangalore, India
Wind River In a world increasingly driven by software innovation, Wind River is pioneering the technologies to accelerate the digital transformation of our customers with a new generation of Mission Critical AI Systems in an AI-first world with the most exacting standards for safety, security, performance, and reliability. Success will be determined by our ability to innovate with velocity and sell at the solutions level.
ABOUT THE OPPORTUNITY The Lab Operations team supports the engineering teams inside Wind River (WR). We provide reliable computer labs, networks, target devices and tools for engineers to develop, test and validate their software and solutions. We strive to always improve availability, reliability and ease of use for all our devices. Our primary customers are the cloud platform, Linux, VxWorks and Wind River Studio teams. We assist the Professional Services, documentation, training and sales teams as well. The devices we manage range from small, embedded computers, to large-scale distributed cloud solutions. Our team supports 6 global engineering development labs globally and we provide remote support whenever possible. The lab infrastructure and target functionality must be available 24/7, whereas the test software on target systems is dependent on the engineer. You will be helping to build and maintain a new engineering lab at our site in Bangalore. You will add racks, PDU, terminal servers and networking as needed to provide the infrastructure to support embedded and cloud labs. You will work with the existing team to support you and implement our best practices on installation, layout, inventory and tooling. You will be required to work on site as the work dictates. Some remote work is possible so long as our lab customers are being supported. You will be asked to provide some weekend and after hours support to manage high priority alerts and outages. Cloud platform labs require a hands-on approach to working with our engineering teams, to install, inventory, build, configure, manage, upgrade, improve and modify cloud infrastructure labs. These clouds may be running Kubernetes or OpenStack software. Lab Operations\xe2\x80\x99 manages the hardware, firmware, networking and network services to prepare the lab environment for our customers to test their cloud software. Familiarity with L2 and L3 protocols such as OSPF, VXLAN, BGP, NAT, NAT64, DNS64 as well as strong skills in Linux networking are a must. Our embedded hardware labs support the VxWorks, Wind River Linux and Wind River Studio development teams for both manual and automated testing. You will help manage our lab infrastructure equipment worldwide. A strong networking background is needed. Remote debugging skills are needed. We use several tools to manage inventory and are migrating to NetBox. You will also be monitoring, automating data collection and validating data. Familiarity with using CURL, REST APIs and SQL can be a strong benefit to you. You must be organized and like helping solve our customers\xe2\x80\x99 issues. You must love learning and working with technology. You must be willing to document, update and improve procedures that can help you, and the team in the future. We are always working on a wide variety of technologies, so willingness to learn and share new skills is a must. Teamwork and collaboration is a must and is highly encouraged.
Responsibilities

• Support the various Engineering teams in WR including Cloud Platform, VxWorks, Wind River Linux, Professional Services, Sales Engineering, Training and Documentation teams.
• Provide remote access to tools and devices for users globally. Help engineers and developers use the labs effectively and point them to documentation that can help.
• Help configure servers and networking to create \xe2\x80\x9ccloud\xe2\x80\x9d environments. Document this in NetBox. Allocate VLANs, IPv4 and IPv6 addresses, tenants, multicast addresses. Configure VXLAN and routing using VXLAN. Enable static and OSPF routing between clouds. Draw lab diagrams in Visio and document in detail.
• Upgrade and configure BIOS, BMC, iDRAC and firmware of NICs, disks, SSDs, and other controllers on a wide variety of hardware.
• You must have electronics experience and be comfortable working on embedded and server hardware.
• Manage serial communication via RS232 and USB serial, as well as terminal servers.
• Work with various network interfaces and cabling including CAT5/6/7, SFP, SFP+, SFP28, QSFP, QSFP28, DAC, MTP, LC, SC and breakouts.
• Be familiar with NICs, memory, disk and systems hardware.
• Install new target boards and/or servers as teams require.
• Configure networking via CLI, including Cisco, Quanta, Dell (OS9/OS10) and HP to provide the correct environment for our cloud systems. Support tools such as NAT, NAT64, DNS64(BIND), and Squid proxy.
• Help diagnose and resolve customer lab issues. Modify labs as needed to support the engineering teams\xe2\x80\x99 feature requests.
• Configure monitoring of systems. Configure CheckMK to potentially monitor thousands of systems. Watch for system performance, service outages and infrastructure issues. Help migrate, use and validate data from our various tools and databases. Be able to chain tools together using APIs, scripts, CURL, Linux CLI to migrate data into new systems.
• Support our SIMICS servers (These are Linux based systems that emulate various target boards)
• Use Bitbucket/Git to capture configuration changes.
• Enable remote access tooling where possible to allow teams to use the labs effectively from remote locations. (Our goal is to keep users out of the labs and keep them working efficiently remotely.)
• Enable network booting of embedded devices and servers PXE/gPXE/iPXE, PXE +EFI, DHCP, HTTP
• Support our lab services including NTP, PTP, SSH, telnet, NFS, DHCP, SyncE, GPS signals distribution,
• Be able to enable and work remotely on systems using a combination of serial console, Serial-Over-LAN, IPMI, remote console, KVM, SSH or other methods. Document these to help the engineering community.
• Help migrate data off older tools into WR Studio\xe2\x80\x99s DevOps platform and NetBox environments.
• Use scripting and monitoring tools to identify lab issues, collect data and proactively fix issues. Automate tasks. Use REST API and tools like, Python, Bash, expect to link tools and improve data and quality. Create reports highlighting any errors.
• Manage and improve inventory accuracy and details of devices.
• Be neat and organized.
• Share interesting findings and technology with the team.
• Plan and develop larger projects and schedule work to completion.
• Work with the team to solve lab issues.

ABOUT YOU
Core Competencies & Demonstrated Success

• Managing Intel based servers and systems (Intel, Dell, HPE and others).
• Understand IPMI and Redfish on BMC, iDRAC, iLO, ipmitool, serial console redirection, Serial-over-LAN (SOL), IPMI over SSH and remote console access
• Understands server hardware, CPU, RAM types and speeds, SATA, SAS, NVMe. M.2, BIOS settings, memory channels, PCI bus and configuring for system performance
• Be able to install server hardware and have mechanical aptitude for a variety of hardware and rack mounting options.
• Understand cloud architectures for Kubernetes and OpenStack
• Understand containerization using Docker or LXD
• Understand hypervisors like ESXi
• Understand cloud storage like Ceph, Minio is an asset
• Have a strong understanding of network configuration in Linux and network routers via CLI e.g.: VLANs, VLAN tagging, L3 routing, OSPF, LACP, LLDP, storm control, multicast, VXLAN, BGP, in both IPv4 and IPv6 stacks.
• Understand IP networking concepts like TCP, UDP, ICMP, unicast, multicast, broadcast etc.
• Able to update and configure various firmware, BIOS, uboot and flash firmware (eg cards, disks, NIC, BIOS, uboot, VxWorks boot loaders, networking gear and other infrastructure equipment. Use new and old tools as needed to get the job done.
• Work with a wide variety of embedded systems. (We support ARM, Intel, PPC and other architectures from vendors such as NXP, Intel, Raspberry Pi, Beagle Bone, TI, Xilinx, among others.)
• Work carefully with expensive, static sensitive boards and devices.
• Be able to configure uboot to initiate a network boot.
• Experience with VxWorks and VxWorks bootloaders is an asset.
• Manage serial ports, RS232, terminal servers and KVM connectivity.
• Be familiar with USB serial ports and using udevadm.
• Be able to solder and build RF cabling (for GPS signals)
• Expertise using the Linux shell and tools like bash, grep, cat, awk, lspci, man, ipmitool, ip, iptables, tcpdump. Be able to resolve network issues, collect system data, debug issues, look at devices and configure services.
• Update and validate inventory using various tools like NetBox. Update provisioning info as you work with equipment. Be able to import data efficiently in to NetBox.
• Scripting, cURL, SQL and automation skills preferred.
• Skilled with vi/vim, emacs or another Linux editor.
  • You must understand vi basics for machines with no other editor option.
• Understand SSH and key based authentication.
• Understand SNMP and MIBs
• Must be a self-starter and have research skills to find solutions to obscure problems. You are a \xe2\x80\x9cGoogle pro\xe2\x80\x9d! You \xe2\x80\x9creach out\xe2\x80\x9d to the team as needed for help or ideas.
• You love to learn about new technologies and look for a better way to do things.
• Be accurate, organized, and clear in your communications.
• Helpful to have: experience with Confluence, Jira (Kanban and sprint), Agile processes, SharePoint
• Use Visio to make professional network graphs and diagrams.
• Use of CheckMK, Nagios, Grafana or other monitoring tools

Qualifications

• Computer Science, Computer Networking, Electrical Engineering or IT degree/diploma
• Minimum of 4 years of relevant experience
• Self-starter and commitment
• Passion and drive for technology, learning and researching solutions to technical problems
• Mechanical aptitude is needed and great hands-on approach to be able to migrate, install and reconfigure boards, drives, networking gear and servers. Our work includes hardware, firmware and software.
• Electronics background and the ability to be able to work with delicate hardware and provide some basic debugging.
• Positive attitude, be eager to share findings and work well with others.
• Provide customer support to our internal engineering teams

BENEFITS & PERKS
Wind River is an Equal Opportunity Employer with a commitment to diversity. We prohibit discrimination based on race, color, religion, gender, national origin, age, disability, veteran status, marital status, pregnancy, gender expression or identity, sexual orientation or any other legally protected status.