Lead AI Software Engineer (Remote)

The posting

The Lead AI Software Engineer will report to the Sr. Director, AI Center of Excellence, in the Technology Department. This role is a hands-on builder across the enterprise AI engineering foundation, including the semantic router, model and harness hosting, the enterprise AI gateway, and the AI operations layer used across the business. The role builds to the architecture and standards set by CoE technical leadership and contributes design input to help shape them. Primary duties include building and operating the shared AI platform and semantic routing layer; partnering with AI-Ops on monitoring, evaluation, and governance guardrails; building secure, compliant AI integration patterns for a regulated financial services environment; building and scaling agentic pipelines and multi-agent orchestration; delivering production AI services for CoE-enabled business functions; building reusable components to the CoE's reference architectures; evaluating AI workload unit economics; prototyping emerging AI capabilities; mentoring engineers across the CoE and partner teams; and partnering with security, risk, compliance, and data governance. This role does not have direct reports.

1. Build and operate the shared AI platform for the enterprise, including core semantic data, model, harness, and hosting capabilities used across developer and in-product personalization use cases.
2. Build intelligent request routing using semantic analysis and deterministic logic to direct requests to the best-fit LLM, internal tool, or agentic pool, using cost, latency, and privacy as routing criteria.
3. Partner with AI-Ops to enable metric-based reporting, quality, performance, and drift monitoring across live systems, build model evaluation harnesses, and support governance guardrails for safe AI use at scale.
4. Build secure, compliant AI integration patterns appropriate for a regulated financial services environment.
5. Build and help scale agentic pipelines and multi-agent orchestration across the software development lifecycle, from pilot to measured engineering capacity across product teams.
6. Deliver production AI services supporting document intelligence and claims adjudication, conversational and voice self-service, call intelligence, fraud and anomaly detection, and regulatory monitoring.
7. Build reusable components and standard patterns aligned to the CoE's reference architectures so other teams and embedded consultants can build AI solutions without rebuilding the foundation.
8. Evaluate unit economics for AI workloads, including cost per transaction, token and inference spend, latency budgets, and analysis supporting build-versus-buy and self-host-versus-vendor decisions.
9. Identify and prototype emerging AI capabilities relevant to the business and technology roadmap, with a clear building-block approach for scaling.
10. Build fast, focused feasibility spikes and use them to build foundational components that scale with teams through clean handoff.
11. Lead applied continuous learning sessions and pair directly with engineering teams on new development methods and prototype or pattern adoption.
12. Coach embedded AI consultants and the coaching network on CoE patterns and standards to scale AI delivery capacity beyond the CoE.
13. Mentor Software Engineer I, II, and Senior-level engineers across the CoE and partner teams, participate in AI design reviews, and help resolve complex agentic and integration problems alongside Principal-level engineers.
14. Partner with security, risk, compliance, and data governance teams to support safe and compliant AI delivery.

Education & Experience:

Minimum required:

• Bachelor of Science in Computer Science
• 7+ years building and operating production software, including 2+ years delivering production AI, LLM, or ML systems, with a track record of building and shipping components of an enterprise-scale platform.

Skills & Abilities:

• Building and operating production agentic workflows as state machines, including branching logic, retry and recovery paths, and precise control over multi-step process flow.
• Multi-agent system design, including role assignment and coordination, agent-to-agent conversation patterns, and code execution loops for iterative problem solving.
• Deep fluency with AI-assisted development environments such as GitHub Copilot, Claude Code, or ONA, with experience helping teams adopt them and measure their impact.
• Semantic routing and model orchestration, including embedding-based intent analysis, deterministic fallback logic, and policy-based model selection across multiple providers on cost, latency, and privacy criteria.
• API and integration experience across REST, event-driven, and MCP patterns, including integrating AI into systems of record and workflow platforms.
• Observability and AI Ops for non-deterministic systems, including distributed tracing, behavioral analysis, usage and quality metrics, drift detection, and cost and latency telemetry for executive reporting.
• Security-aware engineering with demonstrated experience handling PII and sensitive data, including encryption, secrets management, least-privilege access, and audit trails in a regulated environment.
• Model hosting and serving experience, including deploying self-hosted or open-weight models, inference optimization, and gateway or proxy patterns in front of multiple models.
• Evaluation and reliability engineering for probabilistic systems, including eval harnesses, golden datasets, offline and online scoring, guardrails, red-teaming, hallucination mitigation, and human-in-the-loop review design.
• Data engineering foundations for AI, including document ingestion and extraction pipelines, data source integration and mapping, embeddings, and data quality controls.
• Technical leadership without formal authority, including contributing to architecture decision records and design reviews, mentoring early-career and senior engineers, and communicating trade-offs clearly.
• Ability to deliver roadmap capabilities, moving work from discovery through prototype, production validation, and scale against defined business metrics.
• Ability to move fluidly between abstract system thinking and tactical, incremental delivery while finding a critical path and managing stakeholders.
• Financial services, healthcare benefits, or other regulated industry experience preferred, with HSA/FSA, claims, custody, or payments domain exposure especially valuable.
• Knowledge of AI governance and compliance frameworks preferred, such as NIST AI RMF, ISO/IEC 42001, model risk management, SOC 2, or HIPAA control environments.
• Experience with rapid prototyping and feasibility assessment in an R&D or innovation capacity preferred.
• Experience facilitating technical learning sessions or pairing with engineers at varying skill levels preferred.
• Document intelligence and intelligent document processing experience preferred, including OCR, extraction, and classification pipelines with measured accuracy targets.
• Experience with decisioning and adjudication systems where machine decisions are reviewed, audited, and held to an accuracy and turnaround standard is a plus.
• Conversational and voice AI experience preferred, including chat deflection, speech-to-text, call transcription and summarization, and authenticated, account-specific responses is a plus.
• Experience with fraud, anomaly, and identity signal detection preferred, including behavioral baselines, relationship or graph analytics, and detection of doctored or AI-generated media is a plus.
• Experience with workflow and low-code automation platforms preferred.