Skip to content
CV Rocket
Get started
How it worksThe enginePricingBlogContactSign in

Open nowPosted 56 days ago

Senior AI Evaluation & Reliability Engineer

Aubergine Solutions Pvt Ltd4 open roles

Where
Ahmedabad
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowSenior AI Evaluation & Reliability EngineerAubergine Solutions Pvt Ltd · Ahmedabad

  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Aubergine Solutions Pvt Ltd's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

11.5% of postings close within 7 days. Measured by our own scanner across the market.

Share of postings closed within
  1. 2.1%1 day
  2. 4.4%3 days
  3. 11.5%7 days
  4. 18.1%14 days
  5. 35.6%30 days
  6. 44.6%60 days
This job: posted 56 days ago

The posting

Why Aubergine

Aubergine is a global transformation and innovation partner, shaping next-gen digital products through consulting-informed execution that integrates strategy, design, and development.

Since 2013, we’ve built 400+ B2B and B2C products worldwide, turning powerful ideas into impact-driven experiences. We are one of the top global B2B companies on Clutch, rated highest among more than 80,000 technology service providers.

With more than 150 digital thinkers, we are home to some of the brightest, most passionate people around the world who are committed to delivering excellence.

We’re not just another workplace. We’re officially Great Place To Work® certified, with an exceptional trust index rating, making Aubergine a community where you can thrive and grow.

Role Overview: Build AI Systems We Can Trust

We are looking for a Senior AI Evaluation & Reliability Engineer who is passionate about solving one of the most important challenges in AI:

How do we know an AI system is actually working, improving, and delivering business value?

You will design and build production-grade evaluation systems for LLMs, RAG applications, and multi-agent systems, while helping enterprise clients and our engineering teams adopt AI with confidence.

This role goes beyond building evals. You will be a trusted AI consultant to clients, a technical mentor to engineers, and a key contributor to our journey towards becoming an AI superagency.

What You Will Own

  1. Make AI Performance & ROI MeasurableBuild evaluation strategies that connect AI performance to business outcomes and ROI. Define quality benchmarks, SLOs, risk thresholds, and success criteria. Track hallucinations, reliability issues, and production risks. Build executive-friendly AI quality and ROI scorecards. Help clients determine where AI should be autonomous, supervised, or avoided. Don't just measure model accuracy. Measure business impact.
  2. Build Production-Grade Evaluation PipelinesArchitect automated evaluation pipelines for LLMs, RAG, and agentic systems. Integrate evaluations into CI/CD using GitHub Actions, GitLab CI, or equivalent. Build regression suites for prompts, models, tools, and workflows. Measure faithfulness, context precision, answer relevance, semantic drift, and task completion. Establish statistically meaningful benchmarks and continuously monitor AI quality. Evaluation should become part of engineering, not a final QA step.
  3. Engineer LLM-as-a-Judge SystemsDesign reference-based and reference-free LLM evaluation frameworks. Create structured rubrics and scoring systems. Build calibration loops using human-labelled datasets. Identify and mitigate judge biases such as position, verbosity, and self-preference bias. Measure judge reliability and optimize evaluation quality, latency, and cost.
  4. Own Evaluation EconomicsLLM evaluations can become expensive quickly. You will: Design tiered evaluation strategies. Use deterministic and heuristic graders for simple checks. Reserve powerful LLM judges for complex evaluations. Optimise batching, concurrency, and parallel execution. Track evaluation cost and balance quality, latency, and inference spend.
  5. Benchmark RAG & Agentic SystemsBuild systematic benchmarks across: Retrieval quality, faithfulness, and answer relevance Embedding, chunking and re-ranking strategies Vector databases and retrieval architectures Agent task completion and tool-calling accuracy State retention, multi-step workflows and failure recovery Latency, reliability and cost Build synthetic datasets, golden test suites, and adversarial scenarios to continuously expand evaluation coverage.

Be a Trusted AI Consultant

  • You will work directly with North American enterprise clients as a technical AI advisor.
  • You will:
  • Lead AI architecture and evaluation discussions.
  • Translate complex AI metrics into clear business recommendations.
  • Define AI quality, reliability, and risk frameworks.
  • Present evaluation telemetry and ROI scorecards to technical and business stakeholders.
  • Challenge assumptions and recommend the right AI solution, even when that means saying "don't use AI here."
  • You should be equally comfortable discussing LLM evaluation with engineers and ROI with a CTO/COO/CEO/CFO.

Drive AI Consulting & Presales

Your consulting mindset will extend beyond delivery. You will actively participate in presales and new business opportunities, helping us win strategic AI engagements with enterprise clients.

You will:

  • Participate in discovery calls, solution workshops, and technical discussions with prospective clients.
  • Understand client challenges and identify where AI, agents, RAG, or automation can create meaningful business value.
  • Shape AI solution approaches, evaluation strategies, and technical proposals.
  • Contribute to statements of work, estimates, solution architectures, and project proposals.
  • Build compelling technical narratives and demonstrations that showcase our AI capabilities.
  • Present AI solutions and recommendations to technical and executive stakeholders.
  • Help identify opportunities to expand existing engagements through new AI capabilities.
  • Partner closely with Sales, Delivery, Product, and Engineering teams to convert opportunities into successful engagements.
  • Bring a consultative, outcome-focused approach rather than simply responding to client requirements.

You won't just help us deliver AI solutions. You'll help us win the right AI problems to solve.

Coach Engineers. Raise the Bar.

As our organisation evolves towards an AI superagency, you will help shape how we build AI.

You will:

  • Mentor engineers working on AI and LLM systems.
  • Establish AI engineering and evaluation best practices.
  • Conduct technical workshops and knowledge-sharing sessions.
  • Review AI architectures and evaluation strategies.
  • Help engineers move from prompt experimentation to disciplined AI engineering.
  • Build reusable frameworks, playbooks, and internal accelerators.
  • Stay ahead of emerging AI technologies and bring valuable ideas into the organisation.

Your impact shouldn't be limited to the systems you build. It should be reflected in the engineers you make better.

What We Are Looking For

AI Evaluation & Reliability

  • Proven experience building production-grade AI evaluation systems.
  • Strong practical experience with LLM-as-a-Judge.
  • Experience with calibration, human ground truth, and evaluation bias.
  • Strong understanding of RAG and agent evaluation.
  • Knowledge of Faithfulness, Context Precision, Answer Relevance, Task Completion, and semantic similarity.
  • Experience with DeepEval, Ragas, TruLens, LangSmith, DSPy, Promptfoo, or equivalent.

Engineering

  • Advanced Python and/or TypeScript.
  • Strong backend, API, and asynchronous systems experience.
  • Experience with CI/CD and automated testing.
  • Experience with vector databases such as Pinecone, Qdrant, Milvus, or Chroma.
  • Understanding of LLM infrastructure, observability, and inference economics.

Consulting & Leadership

  • Experience working directly with enterprise clients.
  • Excellent communication and presentation skills.
  • Strong consulting and problem-solving mindset.
  • Ability to translate technical complexity into business outcomes.
  • Strong mentoring and coaching abilities.
  • Curiosity, ownership, and enthusiasm for emerging AI technologies.

Why This Role Matters

The next generation of AI companies won't simply be the ones with access to the best models.

They will be the ones that can measure AI, trust AI, improve AI, and turn AI into business value.

If you want to build that future with us, we'd love to hear from you.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Aubergine Solutions Pvt Ltd's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.

  3. We apply on Aubergine Solutions Pvt Ltd's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.

  4. Every reply, sorted

    Aubergine Solutions Pvt Ltd's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.

  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

Get the CV for this job
By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.