Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 76 days ago

Staff AI Evaluation Lead

Fetch81 open roles

Pay
$129,000 – $152,000 a year
Where
US - Remote
Work mode
Remote
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowStaff AI Evaluation LeadFetch · US - Remote

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Fetch's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

11.9% of postings close within 7 days. Measured by our own scanner across the market. Fetch postings stay open a median of 1 days.

Share of postings closed within
  1. 2.1%1 day
  2. 4.4%3 days
  3. 11.9%7 days
  4. 18.6%14 days
  5. 35.6%30 days
  6. 45.1%60 days
This job: posted 76 days ago

Fetch median: 1 days open

The posting

About the Role: At Fetch, we’re building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models, automations, and LLM systems perform with quality, reliability, and measurable impact.

As a Staff AI Evaluation Lead, you’ll own automation and evaluation programs across AI Operations. You’ll translate business and functional goals into scalable systems, define how we measure quality, and ensure automation and evaluation become durable, high-impact capabilities across the organization. This role is ideal for someone who combines deep technical problem-solving with systems-level thinking and strong cross-functional leadership.

This is a full-time role that can be held from one of our US offices or remotely in the United States.

Role Responsibilities: Own programs: Lead complex, high-impact automation and evaluation initiatives across workflows or teams Design scalable solutions: Architect end-to-end workflows integrating datasets, evaluations, automations, and HITL processes; build the harness and reusable components other analysts build against; build evaluation pipelines that run against production without hands-on operation Establish standards: Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations; verify that projects you don't run are meeting them; own the process by which those standards get made and used Own metric definitions: Create and maintain the metric definitions the org evaluates against, validate them against real production behavior, and revise them as models, tooling, and system architectures change Set the bar for production: Define the quality bar a system clears before it reaches production and where human review stays in the loop, and pull a system back when it stops meeting the bar Allocate evaluation depth by risk: Decide which systems get what depth of evaluation given finite capacity, name the risk accepted on the rest, and make that tradeoff visible to project stakeholders Keep the evaluation stack current: Define when a change to a model or platform requires re-baselining across the org and own the process for doing it; evaluate new models and AI capabilities as they ship and decide what the org adopts Improve systems at scale: Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations, not only within your own programs Drive cross-functional alignment: Influence priorities and partner with Engineering, Product, and AI teams to deliver solutions Measure and communicate impact: Define success metrics and communicate performance and recommendations to leadership Elevate the team: Raise the bar through your work and actively mentor others by sharing approaches, guiding problem-solving, and enabling the team to build stronger automation and evaluation capabilities Drive innovation: Stay current on emerging tools and approaches; pilot and translate them into actionable improvements for the team

Minimum Requirements: 8+ years of professional experience AI, machine learning, operational automations, or a related field. Proven ability to lead complex automation or evaluation initiatives across systems or teams Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products Fluency in SQL, JSON, APIs, and scripting with AI assistance, and ownership of the technical direction of the evaluation stack Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems you have built Experience with data pipelines, APIs, and production systems Experience in influencing cross-functional stakeholders and aligning priorities Demonstrated ability to define metrics and drive measurable business impact

Preferred Requirements: Experience mentoring or leading technical contributors Experience evaluating agentic systems in production at scale Experience setting technical standards adopted across an organization

Compensation: At Fetch, we offer competitive compensation packages including base, equity, and benefits to the exceptional folks we hire. The base salary range for this position is $129,000 - $152,000. Discover our benefits and how our employees live rewarded at https://fetch.com/careers.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Fetch's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on Fetch's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    Fetch's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.