Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 8 hours ago

Senior Data Scientist AI Evaluation

Alpaca75 open roles

Where
Remote - Americas
Work mode
Remote
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowSenior Data Scientist AI EvaluationAlpaca · Remote - Americas

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Alpaca's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

8.1% of postings close within 7 days. Measured by our own scanner across the market. Alpaca postings stay open a median of 32 days.

Share of postings closed within
  1. 1.7%1 day
  2. 3.6%3 days
  3. 8.1%7 days
  4. 15.0%14 days
  5. 33.9%30 days
This job: posted 8 hours ago

Alpaca median: 32 days open

The posting

Who We Are:

Alpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more.

Amongst our subsidiaries, Alpaca is a licensed financial services company, serving hundreds of financial institutions across 40 countries with our institutional-grade APIs. This includes broker-dealers, investment advisors, wealth managers, hedge funds, and crypto exchanges, totalling over 10 million brokerage accounts.

Our global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet. We're deeply committed to open-source contributions and fostering a vibrant community, continuously enhancing our award-winning, developer-friendly API and the robust infrastructure behind it.

Alpaca is proudly backed by $400 million in funding from top-tier global investors including Portage Ventures, Spark Capital, Tribe Capital, Social Leverage, Horizons Ventures, Opera Tech Ventures, SBI Group, Derayah Financial, Unbound, Peak XV, Elefund, and Y Combinator.

Our Team Members:

We're a dynamic team of 400+ globally distributed members who thrive working from our favorite places around the world, with teammates spanning the USA, Canada, Japan, Hungary, Nigeria, Brazil, the UK, and beyond!

We're searching for passionate individuals eager to contribute to Alpaca's rapid growth. If you align with our core values—Stay Curious, Have Empathy, and Be Accountable—and are ready to make a significant impact, we encourage you to apply.

Your Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems.

This role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role.

What You'll Do

  • Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
  • Build repeatable eval loops: Track quality over time and catch regressions before release.
  • Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
  • Drive iteration: Translate eval results into actionable recommendations for system improvements.
  • Establish quality standards: Set evaluation guidelines, documentation, and review practices.
  • Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.

What We're Looking For

  • Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).
  • Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.
  • Proficiency in Python and SQL, with experience evaluating models in production environments.
  • Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
  • Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
  • Strong problem-solving ability in fast-paced, greenfield environments.
  • 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome.

Nice to Have

  • Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.
  • Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.
  • Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.
  • Fluency with AI tools in research and engineering workflows.

How We Take Care of You:

  • Competitive Salary & Stock Options
  • Health Benefits
  • New Hire Home-Office Setup: One-time USD $500
  • Monthly Stipend: USD $150 per month via a Brex Card

Alpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.

Recruitment Privacy Policy

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Alpaca's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on Alpaca's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    Alpaca's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.