Skip to content

Open nowPosted 98 days ago

Data Scientist

Sunset13 open roles

Where
New York
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowData ScientistSunset · New York
  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Sunset's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

14.3% of postings close within 7 days. Measured by our own scanner across the market. Sunset postings stay open a median of 39 days.

Share of postings closed within
  1. 2.2%1 day
  2. 4.7%3 days
  3. 14.3%7 days
  4. 23.1%14 days
  5. 39.5%30 days
  6. 59.2%60 days
This job: posted 98 days ago

Sunset median: 39 days open

The posting

ABOUT REPLAY

We are building the first-ever bank for enterprise data. There are over 35 million businesses in the US alone, many of which are sitting on an asset they don’t know they have. At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training. Replay sits at the intersection of advancing the frontier of AI, while helping companies generate millions in growth capital without having to give up equity, take on debt, or lay off employees.

WHY JOIN REPLAY NOW

- We have scaled from $0 to a multi-eight-figure run rate in a matter of months

- We have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund

- We are small enough that you will carry outsized responsibility and grow as quickly as the company does

- You will partner with and build for some of the fastest and most important companies in the world

- You will help build a massive, category-defining business from the ground floor

THE ROLE

Replay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. That creates a difficult measurement problem. A system can improve aggregate F1 while missing a high-risk slice, remove more sensitive information while also destroying useful context, or pass one stage while defects escape somewhere else in the pipeline.

As Replay's first Data Scientist focused on evaluation, you will establish how we know whether that data is actually getting better. You will build the datasets, experiments, quality measures, and feedback loops that expose hidden failures, accelerate model and pipeline improvement, and give the team confidence in what it delivers.

This is a hands-on, zero-to-one role at the intersection of data science, AI, and a real production system. You will write Python and SQL, construct evaluation corpora, study failure patterns, design comparisons, calibrate human and model-based judgments, and turn the result into a clear decision. The questions are scientifically difficult, but the output must be practical enough to change what the team builds and ships.

You will work closely with Machine Learning, Product Engineering, Data Engineering, Security, Quality, domain experts, and the team making delivery decisions. Machine Learning Engineers own changing model behavior. You own the credibility of the evidence used to decide whether a model, pipeline, or delivery change actually made the data safer or more useful.

QUESTIONS YOU MIGHT ANSWER

- Did a higher NER or entity-resolution score actually reduce sensitive misses across the messages, documents, tables, and providers that matter?

- Is a new model finding more sensitive information, or simply removing more of the useful structure our customers need?

- Can we trust a golden dataset, a human review process, or an LLM judge enough to use it for a release decision?

- Which customer, modality, entity, language, or format slices are hidden by a strong aggregate result?

- Where did a quality loss enter between source data, processing, de-identification, review, and delivery?

- What is the smallest credible experiment that would tell us whether to ship, revise, or stop a change?

WHAT YOU'LL DO

- Define what high-quality and safe-to-deliver data mean across de-identification, structure preservation, semantic coherence, and customer utility

- Design representative samples and build golden, adversarial, replay, and production-like corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning

- Turn ambiguous concepts such as “useful,” “clean,” or “safe” into measurable claims with known uncertainty and clear decision consequences

- Evaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes using comparisons that can support a real decision

- Break aggregate results into the modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers that reveal consequential failures

- Connect local measures to escaped sensitive information, avoidable over-redaction, preserved data utility, review burden, rework, and delivery acceptance

- Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers

- Establish holdout and evaluation practices that keep the evidence trustworthy while model and product teams iterate quickly

- Use modern AI tools deeply for analysis, corpus development, coding, review, and hypothesis generation while independently verifying their output

WHAT SUCCESS LOOKS LIKE

- The team has a decision-grade baseline for a priority Clean Data quality claim and trusts it enough to use in model, pipeline, release, and delivery decisions

- Improvements are judged by the slices and failure costs that matter, not only by an aggregate benchmark

- The company can distinguish a true gain from label noise, sample bias, leakage, evaluator error, or a shifted workload

- Changes that improve one stage cannot hide escaped defects, over-redaction, utility loss, or review burden somewhere else

- At least one consequential decision changes because the evidence reveals a risk, tradeoff, or opportunity that was previously unclear

- Evaluation becomes faster and more repeatable without sacrificing independence or rigor

- Quality claims communicate uncertainty honestly and remain understandable to engineers, customers, and risk owners

YOU MIGHT THRIVE HERE IF

- You have at least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role

- You have designed evaluations or experiments that changed a product, model, release, or operational decision

- You understand sampling, uncertainty, precision, recall, F1, calibration, agreement, class imbalance, distribution shift, and imperfect labels

- You can investigate messy, multi-stage data systems and determine where an apparent gain or loss actually came from

- You are comfortable writing Python and SQL and building reproducible technical artifacts rather than handing requirements to an engineering team

- You can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates

- You have startup experience and enjoy broad ownership, changing context, and building the measurement foundation while decisions are already moving quickly

- You use AI tools fluently but do not confuse an articulate model output with valid evidence

- You communicate uncertainty and difficult findings directly, without hiding behind false precision

THIS ROLE MAY NOT BE FOR YOU IF

- You want to optimize models as your primary job rather than determine whether changes actually improve delivered data

- You prefer descriptive dashboards that stop short of changing a decision

- You treat labels, benchmarks, or model-based judges as ground truth without investigating how they fail

- You need a perfectly defined dataset and research plan before you can make progress

- You are uncomfortable disagreeing with a technically strong team when the evidence does not support its conclusion

- You do not want AI tools to be part of your daily scientific and technical workflow

BONUS

- Experience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems

- Experience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems

- Experience designing human-review, adjudication, weak-supervision, or active-learning systems

- Experience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations

- Experience connecting offline measures to escaped defects, customer outcomes, review effort, or preserved data utility

- Experience measuring quality across multi-stage batch or data pipelines

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Sunset's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.
  3. We apply on Sunset's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.
  4. Every reply, sorted

    Sunset's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.
  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.