Skip to content

Open nowPosted 6 days ago

Senior AI Quality & Evaluation Engineer

Solvd, Inc.24 open roles

Where
USA
Work mode
Remote
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowSenior AI Quality & Evaluation EngineerSolvd, Inc. · USA
  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Solvd, Inc.'s own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

14.3% of postings close within 7 days. Measured by our own scanner across the market. Solvd, Inc. postings stay open a median of 59 days.

Share of postings closed within
  1. 2.2%1 day
  2. 4.7%3 days
  3. 14.3%7 days
  4. 22.9%14 days
  5. 39.1%30 days
  6. 59.3%60 days
This job: posted 6 days ago

Solvd, Inc. median: 59 days open

The posting

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

We are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer.

WHAT YOU’LL DO

- Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.

- Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.

- Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.

- Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to the quality dashboard with trend lines and regression alarms.

- Define the QA gate content analysts apply to remediated content.

- Run the suite before every pilot change in weeks 7–9 and report misses with root cause — content, chunking, ranking, or permissions — to the right owner.

- Write the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.

REQUIRED SKILLS & EXPERIENCE

- 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.

- Hands-on evaluation of a search or RAG system in production — writing ground truth sets, scoring relevance and grounding, and tracking drift over time.

- Experience designing security-focused negative tests — authorization, entitlement, data exposure — and reporting them at the severity a client security team expects.

- Ability to work in AWS — invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.

- Clear written reporting; the suite’s output is read by client stewards, not only engineers.

PREFERRED QUALIFICATIONS

- AWS Certified AI Practitioner or Machine Learning Engineer Associate.

- Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.

- Contact center or knowledge management domain exposure.

- Familiarity with SOC 2 or ISO 27001 evidence practices.

When you join Solvd, you'll…

- Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

- Be part of a global team with equal opportunities for collaboration across continents and cultures.

- Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Ready to make an impact?

If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.

Solvd is an equal opportunity employer.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Solvd, Inc.'s own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.
  3. We apply on Solvd, Inc.'s form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.
  4. Every reply, sorted

    Solvd, Inc.'s answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.
  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

By hand5–10
With CV Rocket100
applications a day

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.