The posting
Company Description
Join us and make YOUR mark on the World!
Lawrence Livermore National Laboratory (LLNL) has turned bold ideas into world-changing impact advancing science and technology to strengthen U.S. security and promote global stability.
Our mission spans four critical national security areas nuclear deterrence, threat preparedness, energy security, and multi-domain defense empowering teams to take on the toughest challenges of today and tomorrow. With a culture built on innovation and operational excellence, LLNL is a place where your expertise can make a real impact.
Job Description
We have an opening for an HPC Operations Engineer to develop, deploy, and maintain the tools, automation, and processes that keep Livermore Computing high-performance computing systems available and performing at high levels of reliability and performance. You will work in a challenging, team-oriented environment, applying technical knowledge of HPC systems and software development to engineer monitoring, alerting, and diagnostic solutions for Livermore Computing’s (LC) high-performance computing clusters, storage systems, networks, and data center facilities. You will contribute to technical projects as a member of a technical team, and collaborate with system administrators, software developers, and facilities engineers to improve the reliability and efficiency of LC operations. This position is in the Livermore Computing Division within the Computing Directorate.
This position will be filled at either level based on knowledge and related experience as assessed by the hiring team. Additional job responsibilities (outlined below) will be assigned if hired at the higher level.
This position requires full-time on-site presence due to the nature of the work.
You will
- Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for LC HPC clusters, storage systems, high-speed networks, and data center facility infrastructure.
- Create and maintain software tools and processes that streamline alarm response, diagnostics, and routine operational tasks using established methods and procedures.
- Configure and integrate data from building management and facility systems, such as power and cooling, with HPC system monitoring to provide a unified operational view under established technical direction.
- Share alarm response and monitoring duties with operations staff, and analyze, diagnose, and resolve hardware, software, network, and facility problems of limited complexity, determine root cause, and implement corrective actions in collaboration with system administrators, vendors, and facilities staff.
- Participate in the deployment, testing, and validation of new HPC systems and components, and the decommissioning of retired systems.
- Develop and maintain technical documentation, procedures, and runbooks for operational tools and processes.
- Respond to customer problems and questions, working with Hotline, system administration, and other LC groups to improve the quality of service for end users.
- Perform other duties as assigned.
Additional job responsibilities, at the SES.2 level
- Design and develop moderately complex software tools and automations that improve the operation, reliability, and efficiency of LC computing and facility systems.
- Analyze system, alarm, and incident data to identify trends, and implement moderately complex strategies to improve monitoring coverage, alarm accuracy, and operational processes.
- Lead discrete technical projects or work packages, such as the deployment, validation, and decommissioning of HPC systems, coordinating tasks with system administration, facilities, and other LC groups.
- Diagnose and resolve moderately complex to complex system and facility problems and serve as a technical escalation resource.
- Develop and deliver training and provide guidance and informal mentoring to less experienced staff.
- Independently prioritize assigned work across multiple concurrent operational and project tasks.
Qualifications
- Ability to secure and maintain a U.S. DOE Q-level security clearance, which requires U.S. citizenship.
- Bachelor’s degree in computer science, computer engineering, or related field or the equivalent combination of education and related experience.
- Fundamental knowledge of and experience with Linux/UNIX systems and enterprise data storage, including installation, configuration, networking, RAID concepts, system logs, and troubleshooting.
- Understanding of programming and scripting languages, such as C/C++, Rust, Go, Python, or bash/csh/ksh.
- Experience with version control and configuration management systems, such as Git, Ansible, or CFEngine.
- Fundamental knowledge of system monitoring, alerting, and incident management practices and tools.
- Sufficient verbal and written communication and interpersonal skills necessary to collaborate effectively with system administrators, software developers, facilities staff, LC staff, and end users, and to present and explain technical information.
- Ability to work all shifts, including Owl shift (12am – 8am), weekends, and holidays.
Additional qualifications at the SES.2 level
- Comprehensive knowledge of HPC environments and technologies, such as Linux clusters, InfiniBand, Slurm, Lustre, and archival storage systems.
- Proficient experience developing software tools and automation in C/C++, Python, or similar languages, and broad application of technical principles to resolve moderately complex system, network, and facility issues.
- Experience leading technical work packages or discrete projects and mentoring less-experienced staff, and ability to work with limited direction in a dynamic environment with competing priorities.
- Relevant previous experience and demonstrated ability to perform independent technical work.
Qualifications We Desire
- Experience with monitoring and observability tools, such as Grafana, Prometheus, Splunk, or Nagios.
- Experience with data center utility infrastructure and building management systems, including power distribution, water distribution, and cooling distribution units supporting HPC environments.
- Experience with virtual machines, container technologies, and orchestration, such as Docker, Podman, or Kubernetes.
- Experience working in a Closed Area or other secure computing environment.
Pay Range
$121,830 - $185,544 annually
$121,830 - $154,500 annually for SES.1 level
$146,340 - $185,544 annually for SES.2 level
This is the lowest to highest salary we in good faith believe we would pay for this role at the time of this posting; pay will not be below any applicable local minimum wage. An employee’s position within the salary range will be based on several factors including, but not limited to, specific competencies, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, and business or organizational needs.
Additional Information
#LI-Onsite
Position Information
This is a Career Indefinite position, open to Lab employees and external candidates.
Why Lawrence Livermore National Laboratory?
- Included in 2026 Best Places to Work by Glassdoor!
- Flexible Benefits Package
- 401(k)
- Relocation Assistance
- Education Reimbursement Program
- Flexible schedules (*depending on project needs)
- Our values - visit https://www.llnl.gov/inclusion/our-values
Security Clearance
This position requires a Department of Energy (DOE) Q-level clearance. If you are selected, we will initiate a Federal background investigation to determine if you meet eligibility requirements for access to classified information or matter. Also, all L or Q cleared employees are subject to random drug testing. Q-level clearance requires U.S. citizenship.
Pre-Employment Drug Test
External applicant(s) selected for this position must pass a post-offer, pre-employment drug test. This includes testing for use of marijuana as Federal Law applies to us as a Federal Contractor.
Wireless and Medical Devices
Per the Department of Energy (DOE), Lawrence Livermore National Laboratory must meet certain restrictions with the use and/or possession of mobile devices in Limited Areas. Depending on your job duties, you may be required to work in a Limited Area where you are not permitted to have a personal and/or laboratory mobile device in your possession. This includes, but not limited to cell phones, tablets, fitness devices, wireless headphones, and other Bluetooth/wireless enabled devices.
If you use a medical device, which pairs with a mobile device, you must still follow the rules concerning the mobile device in individual sections within Limited Areas. Sensitive Compartmented Information Facilities require separate approval. Hearing aids without wireless capabilities or wireless that has been disabled are allowed in Limited Areas, Secure Space and Transit/Buffer Space within buildings.
How to identify fake job advertisements
Please be aware of recruitment scams where people or entities are misusing the name of Lawrence Livermore National Laboratory (LLNL) to post fake job advertisements. LLNL never extends an offer without a personal interview and will never charge a fee for joining our company. All current job openings are displayed on the Career Page under “Find Your Job” of our website. If you have encountered a job posting or have been approached with a job offer that you suspect may be fraudulent, we strongly recommend you do not respond.
To learn more about recruitment scams: https://www.llnl.gov/sites/www/files/2023-05/LLNL-Job-Fraud-Statement-Updated-4.26.23.pdf
Equal Employment Opportunity
We are an equal opportunity employer that is committed to providing all with a work environment free of discrimination and harassment. All qualified applicants will receive consideration for employment without regard to race, color, religion, marital status, national origin, ancestry, sex, sexual orientation, gender identity, disability, medical condition, pregnancy, protected veteran status, age, citizenship, or any other characteristic protected by applicable laws.
Reasonable Accommodation
Our goal is to create an accessible and inclusive experience for all candidates applying and interviewing at the Laboratory. If you need a reasonable accommodation during the application or the recruiting process, please use our online form to submit a request.
California Privacy Notice
The California Consumer Privacy Act (CCPA) grants privacy rights to all California residents. The law also entitles job applicants, employees, and non-employee workers to be notified of what personal information LLNL collects and for what purpose. The Employee Privacy Notice can be accessed here.
