Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 143 days ago

Senior Backend Engineer, AI Evaluations

ellipsis-health18 open roles

Pay
$175,000 – $250,000 a year
Where
San Francisco - Hybrid
Work mode
Hybrid
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowSenior Backend Engineer, AI Evaluationsellipsis-health · San Francisco - Hybrid

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on ellipsis-health's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

7.8% of postings close within 7 days. Measured by our own scanner across the market.

Share of postings closed within
  1. 1.7%1 day
  2. 3.5%3 days
  3. 7.8%7 days
  4. 14.6%14 days
  5. 34.1%30 days
This job: posted 143 days ago

The posting

ABOUT THE ROLE

We’re seeking a strong Senior Backend Engineer to join our AI Evaluation team, focused on building the infrastructure and internal tooling that enable reliable, repeatable evaluation of AI systems in production. In this role, you’ll develop the scaffolding around our evaluations platform - API/data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. You’ll also build and maintain evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, productionalizing these frameworks to support regular analysis, regression detection, and continuous monitoring.

In addition, you’ll create debugging tools that make it easy to inspect end-to-end calls, trace failures, and surface all relevant signals in one place - empowering internal teams to diagnose issues quickly and confidently. The ideal candidate is a pragmatic, infrastructure-minded engineer who enjoys turning ad hoc analysis into durable systems, cares deeply about developer experience, holds a high bar for software-quality and takes pride in building tooling that makes complex AI systems observable, testable, and easier to operate.

RESPONSIBILITIES:

- Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools.

- Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails.

- Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling.

- Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use.

- Ensure evaluation tooling meets production standards for reliability, performance, and maintainability.

QUALIFICATIONS:

- 5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling.

- Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines.

- Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar).

- Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems.

- Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows.

- Strong debugging skills and a methodical approach to diagnosing production and evaluation issues.

- Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems.

- Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability.

PREFERRED:

- Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)

- Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling.

- Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery.

- Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks).

- Background in digital health, care coordination, or patient-facing systems.

SALARY AND BENEFITS

We offer competitive salary and benefits, including 401(k) matching, health, vision, and dental insurance, and very flexible paid time off.

The typical salary range for this role is $160,000 to $210,000 USD, depending on skills, qualifications, and relevant experience.

BACKGROUND CHECKS

As a health technology company, we reserve the right to run background checks on candidates to whom we extend offers, in compliance with applicable laws. We evaluate candidates holistically and comply with all “ban the box” regulations.

ASSISTANCE

If you have a disability or require accommodations during the application or recruitment process, please contact [email protected].

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against ellipsis-health's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on ellipsis-health's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    ellipsis-health's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.