Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 32 days agoWe saw it 12 min after it went up

Research Engineer, Evals & RL Environments

Nolla Health6 open roles

Pay
$180,000 – $240,000 a year
Where
New York City
Work mode
On site
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowResearch Engineer, Evals & RL EnvironmentsNolla Health · New York City

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Nolla Health's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

12.4% of postings close within 7 days. Measured by our own scanner across the market. Nolla Health postings stay open a median of 14 days.

Share of postings closed within
  1. 2.1%1 day
  2. 4.5%3 days
  3. 12.4%7 days
  4. 20.0%14 days
  5. 35.5%30 days
  6. 58.6%60 days
This job: posted 32 days ago

Nolla Health median: 14 days open

The posting

New York City · Full-time

BUILD THE ENVIRONMENTS THAT TEACH US WHAT HEALTHCARE AI CAN DO—AND WHERE IT NEEDS TO IMPROVE.

At Nolla Health, we’re building AI-powered healthcare. To improve the systems involved in delivering care, we need realistic ways to test them: cases that unfold over time, tools an agent can use, and clear criteria for judging what happens.

We’re looking for a research engineer to build evaluation harnesses, simulated environments, and data pipelines that turn clinical knowledge and appropriately governed data into useful evaluation and training assets.

You’ll work with engineers and clinicians to define tasks, build the systems that run them, and make the results trustworthy. The work will support our own agents and models, with the opportunity to develop benchmarks and environments for external AI teams.

WHAT YOU'LL DO

- Build harnesses that run agents through reproducible tasks, capture their actions, and make failures easy to investigate.

- Develop multi-step and multi-visit environments with patient state, clinical tools, and realistic consequences for an agent’s decisions.

- Work with clinicians to turn clinical workflows into well-defined tasks, rubrics, and success criteria.

- Build data pipelines for preparing, validating, versioning, and tracing evaluation and training datasets.

- Develop scoring and reward mechanisms, combining programmatic checks, model-based grading, and clinician review where appropriate.

- Test graders and environments for ambiguity, shortcuts, data leakage, and reward hacking.

- Run experiments, analyze trajectories, and identify whether changes improve performance across meaningful slices of cases.

- Work with our applied AI engineer to turn evaluation findings into product improvements.

- Build with privacy, access controls, and permitted data use in mind.

WHAT YOU BRING

- Strong programming skills and experience building reliable software for experiments, data processing, or agent systems.

- Hands-on experience building an agent evaluation harness, an RL environment, or closely related infrastructure.

- The ability to translate an open-ended problem into a runnable task with measurable outcomes.

- Careful judgment about data quality, reproducibility, grading reliability, and what a benchmark result actually supports.

- Comfort debugging across datasets, environment state, agent behavior, and scoring code.

- The ability to work with clinicians and turn domain expertise into precise specifications and tests.

- Evidence of independent execution through shipped systems, research implementations, open-source work, or projects you’ve owned.

We expect many candidates to have roughly two to six years of relevant experience, but we’re flexible for someone with exceptional work. Research experience is valuable; you should also enjoy building and maintaining the systems behind the experiments.

BONUS POINTS

- RL environments, simulated tool use, or long-horizon agent benchmarks.

- Synthetic data generation, human review workflows, or dataset quality systems.

- Reward design, model-based graders, and validation against expert judgments.

- Building evaluation or training data products for AI labs.

- Healthcare data, clinical reasoning evaluations, or longitudinal patient workflows.

Healthcare experience is welcome, but it isn’t required. Curiosity about the domain and a willingness to work closely with clinicians matter.

You’ll help define how we measure progress toward AI that can support care over time. Your work will make it possible to investigate failures, compare approaches, and build training signals grounded in realistic tasks.

You’ll own both the experimental design and the engineering needed to make it useful, with room to shape a new area of work at Nolla.

We’d ideally like you to be based in New York City.

ROLE LOGISTICS, COMPENSATION & BENEFITS

- Role Type: Engineering

- Salary: $180,000–$240,000, based on experience

- Job Type: Full-time

- Work Setup: New York City preferred

- Equity: Meaningful equity, commensurate with experience

- Health Insurance: Medical, dental, and vision

- HSA/FSA: Eligible

- Time Off: Flexible, unlimited vacation

- Additional Perks: Meal stipends, team retreats

- Work Hours: Flexible but demanding. We're building something that matters

- Growth: Founding team members step into expanded roles as we scale

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Nolla Health's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on Nolla Health's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    Nolla Health's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.