Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 11 days ago

AI Evaluation Engineer

siteground5 open roles

Where
Sofia
Work mode
Hybrid
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowAI Evaluation Engineersiteground · Sofia

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on siteground's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

7.8% of postings close within 7 days. Measured by our own scanner across the market.

Share of postings closed within
  1. 1.6%1 day
  2. 3.4%3 days
  3. 7.8%7 days
  4. 14.3%14 days
  5. 33.7%30 days
This job: posted 11 days ago

The posting

YOUR ROLE:

YOUR RESPONSIBILITIES:

  • Build high-quality evaluation datasets with ground truth and define task-specific metrics such as correctness, task completion, efficiency, cost, token usage, and safety;
  • Run model bake-offs and before/after prompt experiments — every meaningful prompt change should be a measured experiment, not a guess;
  • Design evaluations for harder cases where there is no clean ground truth, including agentic workflows, tool-calling correctness, and structured-output validity;
  • Own production observability through trace analysis, per-tool dashboards, error classification, and session-level review;
  • Root-cause failures, identify recurring patterns, understand their impact on quality and cost, and turn findings into prompt improvements and engineering tickets;
  • Help close the loop between users and product teams by translating real customer behaviour and product questions into clear, measurable insights about how our AI features are used and where they can improve.
  • Design, version, test, and continuously improve system prompts for domain-specific agents;
  • Architect agent behaviour, including tool orchestration, multi-turn flows, and safeguards such as plan-before-execute, read-before-write, and always-confirm for destructive actions;
  • Build reusable, modular prompt components for areas such as safety, tone, context, and tool documentation;
  • Select the right model for each task, balancing quality, latency, and cost.
  • Work across the API layer our AI agents act through, integrating the REST endpoints that expose application data and operations to the LLM;
  • Help design and implement guardrails that prevent agents from corrupting user data, including permission enforcement and safe defaults;
  • Debug the full request path end to end — from an agent’s decision and tool call to the live result — correlating application logs with agent traces.

OUR EXPECTATIONS:

  • Strong prompt engineering and LLM application experience — you’ve shipped agents or LLM-powered features to production, not just prototyped them;
  • Hands-on evaluation experience and a data-driven mindset — you’ve built evaluation datasets and metrics, and you measure results before claiming something works;
  • Python experience and comfort consuming and integrating REST APIs;
  • Familiarity with modern LLM tooling and patterns, including function/tool calling, RAG, agentic workflows, observability platforms, and current model families;
  • Healthy scepticism toward AI output and a strong instinct for safety, failure modes, and guardrail design.

GREAT ADVANTAGE WILL BE:

  • Experiment design and statistics literacy, including A/B testing, significance, and sampling, so your evaluation conclusions hold up under scrutiny;
  • Experience designing LLM-as-a-judge evaluations and calibrating automated judges against human labels;
  • Red-teaming and adversarial testing experience, including prompt injection, jailbreaks, and structured-output abuse;
  • Comfort working with data tools for exploring and slicing evaluation results, such as SQL, pandas, DuckDB, notebooks, or similar;
  • Experience with platforms/solutions such as Langfuse, Langchain, n8n, Google ADK.

WHAT WE OFFER:

  • Competitive remuneration and a performance-based bonus system;
  • Shorter workday on Fridays – we finish at 3:00 PM;
  • 5 extra days off at 5 years of service, plus 1 more each year after – up to 10;
  • 2 extra paid days off for volunteering to support the causes you care about;
  • Premium additional health insurance coverage and annual medical check-ups;
  • Modern and cozy offices across Bulgaria, plus hybrid work options;
  • Free parking and a metro shuttle;
  • Chef-prepared breakfast and lunch provided daily at our HQ restaurant – all on us;
  • Fully equipped in-office gym with expert fitness guidance;
  • On-site sports at our HQ: yoga, spinning, Brazilian Jiu-Jitsu, dancing, table tennis, and more;
  • Multisport or Coolfit cards fully covered by the company;
  • Free massages;
  • Memorable team events and internal gatherings;
  • Knowledge-sharing culture through meetups, events and conferences;
  • Free employee access to the SiteGround products;
  • Anniversary and company gifts along the way.

Come and make a difference at SiteGround with people who inspire you to grow!

Please note: Only shortlisted candidates will be contacted for further steps.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against siteground's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on siteground's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    siteground's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.