Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 59 days ago

AI Engineer, Evals & Agent Quality

Town16 open roles

Pay
$250,000 – $300,000 a year
Where
San Francisco
Work mode
On site
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowAI Engineer, Evals & Agent QualityTown · San Francisco

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Town's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

12.3% of postings close within 7 days. Measured by our own scanner across the market. Town postings stay open a median of 61 days.

Share of postings closed within
  1. 2.1%1 day
  2. 4.5%3 days
  3. 12.3%7 days
  4. 19.7%14 days
  5. 35.6%30 days
  6. 58.0%60 days
This job: posted 59 days ago

Town median: 61 days open

The posting

ABOUT TOWN

Town is building the AI coworker for every team. Everyone gets their own Townie, an AI assistant that learns how you work (your voice, judgment, relationships, and priorities) and does real work on your behalf across email, calendar, docs, Slack, and CRM. Townies don't wait to be prompted. They observe, learn, and act. And because every teammate has one, Townies can ask each other questions and hand off work, so the whole team moves faster together.

Four months after our public launch, we became a $1B company. We recently raised a $108M Series B, co-led by Forerunner Ventures and Menlo Ventures, with continued support from Andreessen Horowitz, First Round Capital and angels such as Zach Perret, Soleio Cuervo, Scott Belsky, and more. Our founders, Jean-Denis and Tony, previously led EPD at Plaid and applied AI product at Google.

ABOUT THE ROLE

Town is building the most personalized, most capable AI assistant for everyone — one that knows you deeply, works across every tool you use, and gets sharper over time. Building the best assistant means proving it's the best: every model, prompt, and system change has to be measurably better, on every surface it touches.

That's what you'll own. You'll build the evals and quality systems that turn assistant performance into numbers the whole team can trust, measuring and improving the full multi-step trajectory the assistant takes to do real work. You'll build the model routing that puts the right model in the right place balancing cost, quality, and speed.

This is a foundational, 0→1 build with ownership to match: the eval framework, the golden datasets and labeling loop, model routing, and online measurement, and you set the bar for what "best" means at Town.

WHAT YOU'LL DO

- Build a generalized eval system that measures assistant quality across every surface it touches — and, crucially, across multi-step agent trajectories.

- Stand up golden datasets and the labeling loop that keeps them up to date and constantly checking to validate improvements and avoid regressions.

- Build model routing and online evaluation tooling to help us learn and route to the best models.

- Make every prompt and system change measurable, so the team can move fast without breaking what works.

- Partner with engineers across the product to instrument quality and close the loop from signal to fix.

YOU MIGHT THRIVE HERE IF YOU...

- Have built or owned LLM eval systems, or offline/online quality measurement at scale.

- Think rigorously about measurement. Maybe that came from an MLE or applied-ML background, maybe not, the instinct for how to measure "better" matters more than the exact pedigree.

- Know the eval landscape hands-on, off-the-shelf tooling and eval frameworks, and have opinions on what to reach for when.

- Are comfortable reasoning about model routing and the tradeoffs between models.

- Ship the fixes, not just the dashboards and metrics.

- Are a senior or staff engineer comfortable in greenfield, where the system doesn't exist yet.

LOCATION

San Francisco, CA. Five days a week in person at our Financial District office.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Town's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on Town's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    Town's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.