The posting
Firmus Technologies
Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific.
Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability.
At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally.
Firmus AI Cloud
Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers.
It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale.
AI FactoryOS Operations
AI FactoryOS is Firmus' proprietary operating system for the AI Factory. It governs GPU telemetry, cooling, power and grid interaction as one integrated layer, so that every Firmus site can be optimised and monitored as a single system.
AI FactoryOS Operations runs that platform in production and owns the 24/7 reliability of AI FactoryOS, Firmus AI Cloud and the platforms built on them, together with the service levels the estate is measured against.
The remit is an engineering one. The function builds the guarded automation, remediation and operational tooling that turn manual response into a software-defined capability, and builds and operates the shared services the estate's own operation depends on. The function works closely with the engineering teams that build the platform, supplying the production evidence that shapes what they fix and what they build next.
Role Summary
Firmus runs large-scale, state-of-the-art AI infrastructure built on the latest generation of GPU rack-scale systems and operated as one estate to power the next generation of AI innovation. The Senior Platform Reliability Engineer, Fabric and Interconnect, owns the reliability of the fabrics this estate runs on: the GPU-to-GPU interconnect domains, the high-performance network fabrics carrying training and inference traffic, and the DPU-based host networking that binds compute to the rest of the platform.
This is a hands-on senior role with deep technical expertise. Automation is a first-class part of the role: the team builds and maintains the guarded automation and remediation tooling that turn manual fabric response into a self-healing capability, and the role engages fabric vendors at engineering level, reproducing faults to their standard and holding them to their answers.
Key Responsibilities
- Triage, diagnose and resolve incidents across the stack, including Kubernetes clusters and workloads, storage, virtualisation, network connectivity and the underlying Linux hosts, owning each incident from escalation through to resolution.
- Operate the multi-tenant control and management plane that Firmus' AI and infrastructure services depend on, including the tenancy, quota and access controls that support separation between tenant workloads and data.
- Operate the virtualisation platform that hosts the management plane, and the shared services that run on it, to their declared service levels.
- Operate Firmus' exabyte-scale distributed and high-performance filesystems and S3-compatible object storage to their declared service levels, handling capacity, performance and availability incidents and escalating to specialist engineers and vendors (for example VAST or WEKA) with clear evidence when further investigation is needed.
- Operate the GPU compute fleet in production: node health and readiness, GPU and node fault detection and handling, firmware and driver currency to the supported baselines, remediation and return to service, and support for cluster provisioning and validation activities.
- Reduce toil by converting repeat fixes into runbooks and automated remediation delivered as controlled code, contributing to the team's automation and remediation tooling.
- Diagnose performance problems across the data path between compute and storage, gathering the evidence needed to resolve them and working with network and storage specialists where required.
- Deliver change as code through the agreed change process, with peer review, testing, staged rollout where applicable, and a rollback plan.
- Use the observability stack (for example Grafana, Prometheus, Loki and Tempo) to track service level indicators against their objectives, query metrics, logs and traces, correlate signals across services and diagnose faults, and improve dashboards and alerting so that each alert is actionable and tied to a runbook.
- Execute the patching cycle and urgent vulnerability remediation across the services the team operates.
- Manage vendor support cases across the hardware and software portfolio, raising and tracking cases under the relevant support agreements, supplying logs and diagnostic evidence, and following cases through to agreed response and resolution times.
- Take part in technical recovery during major incidents and in blameless post-incident reviews, feed recurring issues back to the engineering teams as clear requirements, share the follow-the-sun on-call roster, and keep incident tickets, operational procedures and runbooks accurate and current so that others can execute them successfully.
Skills & Experience
Required Skills
- Solid experience operating production infrastructure in a 24/7 environment using site reliability engineering practices, including service level objectives, toil reduction, owning incidents from escalation to resolution, on call participation, post-incident review and writing runbooks that others can execute successfully.
- Strong working knowledge of Linux systems administration and troubleshooting, including storage, host networking, processes, memory and I/O, and a systematic approach to isolating faults.
- Working experience with scale-out, parallel or enterprise storage (for example VAST, WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance and capacity problems using evidence rather than assumption.
- Familiarity with a virtualisation platform (for example Proxmox, VMware or KVM).
- Working experience using observability tooling to diagnose faults (for example Grafana, Prometheus, Loki, Tempo or Elasticsearch), including writing queries across metrics and logs (such as PromQL and LogQL) and correlating signals across services.
- Working experience with infrastructure-as-code and GitOps practices (for example OpenTofu or Terraform, Ansible, Argo CD), and writing scripts or small tools for operational automation in a language such as Python or Go.
- Familiarity with server hardware operation and support, including working with hardware and software vendors to raise support cases and supply diagnostic evidence.
- Familiarity with cyber security principles and how they apply to operating production infrastructure.
- Clear communication skills, with the ability to write incident updates, ticket notes and escalations that others can act on, and a working style suited to a team that shares accountability for a broad portfolio across regions.
Preferred Experience
- Experience in infrastructure operations for GPU or HPC environments and the demands of large-scale distributed training and inference, including GPU health and fault signals (for example DCGM health checks, NCCL tests and XID errors).
- Experience in multi-tenant service provider, cloud or colocation environments with contractual response commitments and customer-facing reporting.
- Working knowledge of networking fundamentals, including TCP/IP, DNS, routing, VLANs, load balancing and firewalls, and the ability to trace a connectivity fault across hosts, switches and services.
- Knowledge of high-performance fabrics and data-path technologies, such as InfiniBand, RoCE, RDMA and GPUDirect Storage, and DPU or SmartNIC-based networking.
- Familiarity with bare-metal lifecycle and provisioning automation (for example NVIDIA Infra Controller, MAAS, Ironic or Metal3), and standard operating environments with known-good-state enforcement.
- Hands-on experience operating and troubleshooting Kubernetes, including workloads, networking, storage and cluster health, and multi-tenant and GPU-enabled Kubernetes patterns.
- Experience with ITSM and incident management tooling (for example Jira Service Management, ServiceNow or PagerDuty).
- Experience working under ISO 27001 or SOC 2 controls.
- Relevant certification, such as Certified Kubernetes Administrator (CKA), a Linux certification (for example RHCSA or LFCS), or a vendor certification.
- A Bachelor's degree in computer science, engineering or a related discipline, or an equivalent combination of relevant experience and training.
Location & Reporting
Location: Based in Australia or Singapore, with travel to Australian AI Factory sites as required.
On-call: The function runs 24/7. First line monitoring and first response sit with the operations centre. This role shares the after-hours escalation roster for its domain with the other senior engineers in the function.
Reporting to: Reports to the Head of AI FactoryOS Operations while the function is being established, working under broad direction with a high degree of autonomy and direct access to the decision makers. As the function reaches its planned structure, the role will report to the Infrastructure Operations Manager, with the Head of AI FactoryOS Operations remaining accountable for the function. The scope, level and remit of the role do not change under either arrangement.
Employment Basis
Permanent full-time
Diversity
At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.
Join us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.
