Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 15 days ago

Evaluation Engineer

freshfields25 open roles

Pay
$100,000 – $200,000 a year
Where
New York
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowEvaluation Engineerfreshfields · New York

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on freshfields's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

12.8% of postings close within 7 days. Measured by our own scanner across the market. freshfields postings stay open a median of 11 days.

Share of postings closed within
  1. 2.2%1 day
  2. 4.5%3 days
  3. 12.8%7 days
  4. 20.6%14 days
  5. 35.6%30 days
  6. 58.4%60 days
This job: posted 15 days ago

freshfields median: 11 days open

The posting

Role purpose

Build and run tests that assess the quality and reliability of the firm’s AI applications. Help maintain evaluation datasets, investigate failures and provide clear evidence of how AI systems perform on legal and business tasks.

Key responsibilities

  • Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.
  • Prepare and maintain test datasets, source documents and reference examples with legal specialists. Follow requirements for confidentiality, permitted data use and version control.
  • Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation. Apply agreed scoring criteria and flag cases requiring expert judgement.
  • Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling, in coordination with application QA.
  • Review outputs and execution traces to identify failures. Reproduce issues, document supporting evidence and help distinguish data, retrieval, model and application defects.
  • Compare results across model, prompt and application versions. Report changes accurately, including uncertainty and limitations, rather than relying solely on headline scores.
  • Support the calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
  • Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases.
  • undefined

Required skills and experience

  • Minimum five years’ experience developing software demonstrated through employment, placements or substantial projects using programming languages such as Python, Java or C#.
  • Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes, and an ability to apply that understanding when developing evaluation criteria, test cases and acceptance thresholds with guidance from senior engineers.
  • Ability to use coding agents effectively to support development, testing and analysis, with awareness of best practices for prompting, context management, validation, security, responsible use of generated code and when to seek human review.
  • Working knowledge of APIs, Git, automated testing and software development lifecycle practices and CI/CD tools such as Jira and Jenkins.
  • Demonstrated curiosity and ability to learn in a rapidly evolving AI landscape, assess new models, tools and evaluation techniques, and apply relevant learning pragmatically with appropriate guidance.
  • Foundational understanding of statistics and measurement, including sampling, precision and recall, together with strong analytical skills, attention to detail and the ability to document issues clearly and seek technical or legal guidance when needed.
  • Practical experience evaluating AI applications is a plus but is not required, given the newness of the field; relevant academic, internship or project experience will also be considered.

For individuals assigned and/or hired to work in New York and California, Freshfields is required by law to include a reasonable estimate of the compensation range for this role. This compensation range is specific to the States of New York and California and takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled, and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is $100,000 to $200,000.

EEO Statement Freshfields US LLP is proud to be an equal employment employer. Our policies and practices will be free from unlawful discrimination based upon race, color, ethnicity, religion, creed, sex (including pregnancy, childbirth or related medical conditions), national origin, citizenship, immigration status, ancestry, age, marital status, protected veteran status, military service, disability, medical condition, genetic information, sexual orientation, gender identity, or any basis prohibited under federal, state or local law. We strive to promote an atmosphere that encourages equal opportunities and prohibits discriminatory practices, including sexual harassment.

Disability Accommodation for Applicants to Freshfields US LLP Freshfields US LLP is an Equal Employment Opportunity employer and provides reasonable accommodation for qualified individuals with disabilities and disabled veterans in job application procedures. If you have any difficulty using our online system and you need an accommodation due to a disability, you may use the alternative email address below to contact us about your interest in employment at [email protected] , or you can send your resume to [email protected] , or you can call us at +1-212-277-4000.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against freshfields's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on freshfields's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    freshfields's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.