The posting
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Engineering Manager, Serving/API based in United States.
This is a technical leadership role overseeing the software layer that connects customer API requests to production AI inference. You will lead and grow an engineering team responsible for API serving, tokenization, structured generation, reasoning, speculative decoding, and emerging model capabilities. The role combines deep systems thinking with people leadership, product accountability, and cross-functional execution. You will shape the technical roadmap while ensuring API compatibility, correctness, latency, reliability, and scalable model integration. You will collaborate closely with compiler, accelerator, production platform, hardware, and customer-facing teams in a rapidly evolving environment. As the model catalog and customer base expand, you will help establish durable ownership structures and engineering practices that can scale with the organization. This is an opportunity to directly influence both the technology and team responsible for delivering high-performance AI inference services to customers.
Accountabilities
- Lead, coach, hire, and develop engineers working across API serving, tokenization, structured generation, reasoning, and model integration.
- Establish and maintain a clear technical and organizational roadmap covering API compatibility, chat templates, tool calling, reasoning, request budgets, speculative decoding, and new-model support.
- Own API fidelity, including OpenAI-compatible behavior, HTTP endpoints, request validation, streaming, usage accounting, and robust error handling.
- Lead support for reasoning models, including reasoning formats, reasoning-effort controls, token limits, and per-request accounting.
- Partner with compiler and execution teams on speculative decoding, including draft-model integration, request-side infrastructure, acceptance metrics, and policies for evaluating when speculation improves performance.
- Define and lead the roadmap for vision-language model support and interoperability with modern serving frameworks, making informed decisions about what to adopt, integrate, or build internally.
- Ensure serving-layer overhead remains off the critical path for time-to-first-token and streaming latency.
- Design the API layer to remain independent of underlying hardware topology as inference platforms evolve and expand across multiple hosts.
- Establish clear ownership boundaries for shared host-level load balancing, request policies, and system protections in partnership with production platform teams.
- Build robust release gates, including API conformance tests, tool-calling and reasoning evaluations, and regression suites, so new models and capabilities can ship with confidence.
- Collaborate with production platform and orchestration teams to turn new serving capabilities into reliable, supportable production endpoints.
- Translate customer and business priorities into sequenced engineering initiatives while protecting the team from reactive or unstructured demands.
- Establish measurable outcomes around API fidelity, tool-call accuracy, serving latency, throughput, reliability, and customer-visible quality.
- Develop clear ownership boundaries and grow emerging technical leaders as the organization and product surface expand.
- Support longer-term evolution of the function as it scales into areas such as API and protocol compatibility, structured generation, tool calling, speculative decoding, and multimodal serving.
- Demonstrated success managing and growing engineering teams responsible for LLM inference, model serving, ML systems, or closely related technical domains.
- Strong systems engineering background with expertise in C++ and working proficiency in Python, including the ability to review performance- and correctness-critical code.
- Proven ability to turn ambiguous product and customer requirements into coherent roadmaps, clear ownership models, and measurable engineering outcomes.
- Experience recruiting, coaching, developing, and retaining engineers across different levels of experience while maintaining high technical standards.
- Strong understanding of LLM serving concepts, including tokenization, chat templates, sampling, streaming APIs, and OpenAI-compatible APIs.
- Hands-on understanding of tool calling, function-calling formats, structured output, and the practical failure modes associated with these capabilities.
- Experience with serving frameworks or open-source technologies such as vLLM, SGLang, TensorRT-LLM, llguidance, XGrammar, or Hugging Face tokenizers is highly valuable.
- Familiarity with multimodal serving, including vision-language models, image preprocessing, vision encoders, and multimodal token handling.
- Experience supporting reasoning models and associated output formats, such as OpenAI Harmony, is advantageous.
- Experience working with GPU, FPGA, ASIC, or other AI accelerators, particularly for memory-bandwidth-bound inference, is valuable.
- Experience developing customer-facing API products where engineering teams are responsible for compatibility, escalations, and release readiness.
- Ability to make sound technical and organizational trade-offs and communicate decisions clearly to engineers, cross-functional partners, and executive stakeholders.
- Hands-on leadership style with enough technical depth to engage meaningfully in architecture and code discussions without becoming a delivery bottleneck.
- Strong written and verbal communication, prioritization, collaboration, and organizational skills.
- Ability to operate effectively in a fast-moving environment where models, modalities, serving technologies, and customer requirements evolve rapidly.
- Candidates must currently be authorized to work in the United States. New visa sponsorship is not available, though eligible H-1B transfers may be supported.
- Base salary range of $225,000–$350,000, with final compensation determined by experience, qualifications, internal equity, and expected impact.
- Competitive equity as part of the total compensation package.
- Fully company-paid medical, dental, and vision insurance for employees and dependents.
- Company-paid life and disability insurance, with voluntary supplemental coverage options.
- Supplemental hospital, critical illness, and accident insurance options.
- Unlimited paid time off, with an emphasis on taking meaningful time to rest and recharge.
- 13 paid company holidays.
- Remote-first work environment.
- Company-provided computer and home office setup.
- 401(k) retirement plan with company matching, eligible from day one.
- Opportunity to shape both advanced AI inference technology and the engineering organization supporting it.
- Opportunity to work on rapidly evolving accelerator, model-serving, and production AI technologies.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
