The posting
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Staff/Principal Software Engineer (L6/L7): AI Platform Model Serving based in the United States, with remote work flexibility.
In this highly strategic engineering role, you will shape the next generation of AI and machine learning model-serving infrastructure at global scale. You will lead the architecture and development of a unified platform supporting predictive AI, generative AI, foundation models, and agentic workloads across real-time and asynchronous environments. Working at the intersection of distributed systems, GPU optimization, and advanced inference technologies, you will tackle complex challenges in performance, reliability, scalability, and cost efficiency. You will help establish technical standards, influence long-term engineering strategy, and guide multiple teams through architectural decisions. The role offers substantial research and development opportunities in emerging AI infrastructure, including intelligent model routing, speculative decoding, and multi-GPU inference. You will collaborate with senior technical leaders and cross-functional partners to build durable, high-performance systems that enable innovative AI-powered experiences.
Accountabilities
- Lead platform architecture and technical strategy: Define and drive the architecture of a unified AI model-serving platform that integrates internal inference infrastructure, third-party model hosting, and external large language model providers through consistent serving, routing, and experimentation capabilities.
- Build high-performance inference infrastructure: Design and optimize distributed systems capable of supporting large-scale production workloads, from low-latency predictive models to foundation models and complex generative AI applications.
- Optimize GPU utilization and inference economics: Develop and implement advanced techniques such as disaggregated serving, speculative decoding, prompt caching, intelligent routing, multi-GPU inference, and efficient capacity allocation to reduce latency and cost per inference.
- Develop intelligent model-serving capabilities: Enable teams to deploy, route, evaluate, and A/B test models consistently across different runtimes, infrastructure environments, and providers.
- Advance asynchronous and agentic workloads: Design reliable, resumable execution systems for long-running AI agents and asynchronous inference tasks, ensuring fault tolerance, operational consistency, and scalability.
- Drive technical innovation and research: Evaluate emerging AI serving technologies, identify infrastructure bottlenecks, and translate research and experimentation into production-ready solutions.
- Establish long-term technical direction: Own architectural decisions and multi-year roadmaps, ensuring that platform investments support evolving business needs, maintainability, security, and sustainable growth.
- Influence engineering teams across the organization: Build alignment across multiple teams through technical leadership, clear documentation, architecture reviews, and evidence-based decision-making, while mentoring engineers and raising engineering standards.
- Partner with cross-functional stakeholders: Collaborate with product, infrastructure, operations, and executive teams to balance technical trade-offs, business priorities, reliability requirements, and infrastructure costs.
- Education and experience: Bachelor's, master's, or doctoral degree in Computer Science, Applied Mathematics, Engineering, or a related field, with at least 12 years of relevant experience in AI/ML infrastructure, distributed systems, or related engineering domains.
- Distributed systems expertise: Extensive experience designing and operating high-traffic distributed services, high-performance RPC systems, and production infrastructure with demanding availability, security, reliability, and latency requirements.
- AI/ML model-serving experience: Demonstrated success building and scaling inference platforms for machine learning and foundation models that support predictive AI, generative AI, and other production applications.
- Inference optimization: Strong understanding of latency and throughput optimization, resource allocation, GPU/CPU utilization, and techniques for improving inference efficiency and reducing infrastructure costs.
- Modern inference technologies: Practical experience with model-serving platforms and runtimes such as NVIDIA Triton Inference Server, vLLM, SGLang, TensorRT-LLM, Text Generation Inference (TGI), DeepSpeed-Inference, or NVIDIA Dynamo.
- Cloud-native infrastructure: Experience deploying and scaling AI workloads using Kubernetes and public cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
- Distributed serving and control planes: Familiarity with model gateways, intelligent routing, control-plane architecture, experimentation infrastructure, and A/B testing for production AI systems.
- Technical leadership: Ability to set multi-year technical roadmaps, lead complex architectural initiatives, and influence decisions across teams without relying on formal authority.
- Communication and collaboration: Strong written and verbal communication skills, including the ability to explain complex technical concepts to non-technical stakeholders and produce clear design proposals, architecture decision records, and technical documentation.
- Business and product alignment: Proven ability to work with product managers, operations teams, and senior leadership to translate business objectives into scalable, maintainable technical solutions.
- Contributions to open-source projects related to AI/ML, distributed systems, model serving, or infrastructure.
- Hands-on experience developing AI models or implementing optimization techniques such as quantization, model compression, and model pruning.
- Proficiency in using AI-assisted coding tools to improve software development productivity.
- Experience with intelligent model gateways that select models or providers based on request context, performance requirements, or cost considerations.
- Exposure to multi-adapter model serving, disaggregated prefill and decode, and large-scale key-value (KV) cache management.
- Experience with evaluation frameworks and benchmarking tools for large language models and AI agents.
- Knowledge of GPU topology, NVIDIA CUDA programming, NCCL, machine learning compilers, and performance profiling tools such as NVIDIA Nsight.
- Familiarity with deep learning frameworks and ecosystems, including PyTorch, TensorFlow, JAX, and Hugging Face Transformers.
- Competitive compensation: Annual salary range of $890,000–$1,690,000, depending on location, experience, skills, and other relevant market factors.
- Flexible compensation structure: Compensation is primarily salary-based, without annual bonuses. Eligible employees can choose how to allocate their compensation between salary and stock options each year.
- Comprehensive healthcare: Access to health plans, mental health support, health savings accounts, and flexible spending accounts.
- Retirement planning: 401(k) retirement plan with employer matching.
- Equity opportunities: Stock option program offering eligible employees an opportunity to participate in long-term company value creation.
- Family and personal protection: Family-forming benefits, disability programs, life insurance, and serious injury benefits.
- Flexible paid time off: Full-time salaried employees have access to flexible time off, while full-time hourly employees accrue up to 35 days annually for vacation, holidays, and sick leave.
- Paid leave programs: Access to paid leaves of absence, subject to applicable policies.
- Remote work flexibility: Opportunity to work remotely from the United States.
- High-impact technical challenges: The opportunity to influence the direction of a large-scale AI platform, solve complex infrastructure problems, and contribute to emerging advances in foundation model serving.
- Senior-level ownership: Significant autonomy in technical decision-making, long-term architecture, and cross-team engineering strategy.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
