Senior Operations Engineer at CETI AI

CETI AI is a leader in the decentralized AI (DeAI) and decentralized physical infrastructure (DePin) movements. We are seeking an experienced Senior Operations Engineer to drive all aspects of maintaining and growing a healthy fleet of HGX machines. The ideal candidate will have a strong background in networking, particularly with Infiniband and Cisco routers, physical infrastructure, and systems administration.

The engineer will join an aggressive, accomplished team in a rapidly expanding industry, and will have direct access to some of the best hardware in the world and unparalleled opportunity to sharpen their skills at management of high performance AI clusters.

Key Responsibilities

• Set up and configure the Infiniband network, including subnet management and LID assignment.
• Configure networking hardware for the out-band network and optimize network speeds.
• Implement network boot and ansible playbooks to reliably configure and scale fleets of machines.
• Manage scalable network storage systems.
• Perform routine maintenance tasks, such as power cycling machines as needed.
• Installing remote switches and similar devices to automate routine tasks.
• Troubleshoot live issues on production systems.
• Create bill of materials and purchase orders to expand existing infrastructure.

Qualifications

• Strong knowledge of networking fundamentals, including network topologies, Infiniband, and subnet management.
• Experience with router, switch, and gateway configuration.
• Familiarity with Linux administration.
• Proven experience solving complex network issues.
• Physical presence may be required at our Vancouver facility from time to time to perform maintenance tasks or resolve hardware issues.

What We Offer

• 150-200k/year total compensation including equity.
• Convenient remote work arrangement (although occasional on-site visits may be required)
• Opportunities for professional growth in a rapidly expanding AI crypto company

Conclusion

The Senior Operations Engineer plays a vital role in ensuring the smooth operation and security of an organization’s network infrastructure. This role requires a combination of technical expertise, problem-solving skills, and the ability to work effectively in a team environment. Your goal will be to enable the rest of the team to do their work seamlessly and without interruption, making sure the machines are in top working condition.