Skip to content

Open nowPosted 171 days ago

Lead AI QA Engineer

Kobie31 open roles

Where
Bengaluru, Karnataka
Work mode
Hybrid
Get the CV for this job

From $25 per CV, paid once. No subscription.

Your applicationOpen nowLead AI QA EngineerKobie · Bengaluru, Karnataka
  1. YouYes, apply to this one.

  2. CV RocketCV written for this posting.

  3. 25 readersRecruiter, hiring manager, skeptic. Round after round.

  4. CV RocketApplied on Kobie's own form.

The reply lands in your private mailbox

3×more interviews than doing it yourself with ChatGPT.

The clock on this job

Early applications get read.

14.3% of postings close within 7 days. Measured by our own scanner across the market. Kobie postings stay open a median of 28 days.

Share of postings closed within
  1. 2.2%1 day
  2. 4.7%3 days
  3. 14.3%7 days
  4. 23.0%14 days
  5. 39.3%30 days
  6. 59.3%60 days
This job: posted 171 days ago

Kobie median: 28 days open

The posting

Join our India Tech Hub – Be among the first hires!

Kobie, a 35-year veteran of the loyalty industry, a multi-year Forrester Leader, and USA Top Workplace is expanding its global footprint by establishing a Tech Hub in India. Kobie partners with global brands to build deep connections with their customers through personalized, data-driven loyalty experiences and has a mission of growing enterprise value through loyalty. The Tech Hub will serve as a Global Capabilities Center for a broad range of technology roles, and this is your chance to play a pivotal role in shaping our presence in India. Join us as we continue to lead in loyalty, delivering innovative customer experiences for some of the world’s most recognized brands while working alongside some of the best and brightest in loyalty.

About the team and what we will build together

We’re looking for an Lead AI QA Engineer with 6+ years of experience who thrives on designing test strategies and evaluation harnesses for production-grade, agentic AI systems in addition to experience in ETL. You have strong Python skills, hands-on experience testing LLM-powered features (prompt regression, tool/function-call validation, RAG correctness, and structured-output schema checks), and working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith or Langfuse. You are comfortable writing solid SQL, automating tests with PyTest, exercising APIs through Postman or REST clients, and shipping test pipelines using Git, Docker and CI tooling like Jenkins or GitHub Actions.

Kobie runs some of the largest loyalty programs in the world. We are building an internal agent platform on Dataiku that automates analyst workflows, surfaces insights from program data in Snowflake, and gives our teams an LLM-native way to work with complex loyalty logic. As an Lead AI QA Engineer on the India Tech Hub team, you will play a key role in protecting that platform — designing golden datasets, running LLM-as-judge and regression suites, and owning the quality bar for what goes to production. This is not a manual-only role: you will automate, build qa & automation strategies, roadmaps, instrument, monitor and partner closely with our U.S. AI & Innovation team and cross-functional partners across Engineering, Data, AI and Product.

How you will make an impact

  • Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions. In addition, develop framework to verify generated AI output.
  • Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows
  • Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation
  • Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships
  • Build observability into testing by instrumenting traces with LangSmith, Langfuse, MLflow or OpenTelemetry and triaging production drift back into the eval harness
  • Own quality end-to-end — define release criteria, run pre-prod and shadow tests, and partner with engineering to root-cause and fix regressions quickly
  • Partner with data engineers on Snowflake-backed retrieval testing patterns (Cortex Analyst and Cortex Search Services) and with platform teams on observability, security and cost
  • Help shape internal QA standards for AI & Data engineering as the stack evolves, contributing to design reviews and sharing knowledge across the India and U.S. teams
  • Participate in a collaborative DevOps environment, working closely with developers, AI engineers, Data Engineers, DBAs and product partners across environments

What you need to be successful

  • 3+ years of professional QA / SDET experience, with production experience automating tests for backend services or data pipelines
  • 1+ years of hands-on experience testing LLM or AI features in production: prompt regression, tool / function-call validation, structured outputs and RAG correctness
  • Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse or comparable LLM-as-judge tooling
  • Strong Python and PyTest skills; solid SQL skills and comfort with at least one cloud platform (AWS, Azure or GCP)
  • Fluency with Git, Docker, REST APIs and at least one CI tool (GitHub Actions, Jenkins, GitLab CI or CircleCI)
  • Solid understanding of data security and responsible AI practices, particularly in PCI-compliant or regulated environments
  • Proven ability to work independently and within a team, managing priorities across concurrent projects and time zones
  • Strong written and verbal communication skills; able to work effectively with both technical and non-technical stakeholders
  • A bachelor’s degree is not required — equivalent practical experience (including bootcamps, self-taught work, career changes or non-CS technical degrees) counts
  • Bonus Skills:
  • Hands-on experience with Dataiku DSS (Python / SQL recipes, scenarios, code environments, the dataiku and dataikuapi clients) or Dataiku Evaluations
  • Experience with Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents
  • Experience with Snowflake, Snowpark, or Snowflake Cortex (Search, Analyst, Agents)
  • Experience with red-teaming, prompt-injection testing or adversarial test generation for LLMs
  • Familiarity with multi-agent patterns: supervisor / router, subagent / handoff, reflection, human-in-the-loop
  • Experience with performance and load testing tools such as Locust, JMeter or k6
  • ISTQB, AI Testing or comparable QA certification
  • Experience in loyalty, martech, adtech or a comparable data-rich B2B domain

About Kobie

Named a Top Workplace in the USA and Top Remote Workplace, Kobie is where the best minds in loyalty come together, driven by passion and innovation. We’re always looking for talented individuals ready to join a collaborative, growth-focused culture.

As a trusted partner to some of the world’s most recognized brands, we are loyalty leaders, helping brands build lasting emotional connections with their consumers. We do this with a strategy-led technology approach that uncovers the truth behind what drives consumers on an emotional level.As we launch our India Tech Hub, we are excited to bring our award-winning culture to a new region - creating an environment where collaboration, flexibility, and career growth come together to build something truly special.

We are proud to be the only loyalty provider to be externally recognized for their culture. We believe people thrive when they feel valued, supported, and empowered to be their authentic selves. Our commitment to diversity, equity, and inclusion ensures every teammate has a voice and the opportunity to be heard.

Giving back is in our DNA at Kobie,through an annual fundraiser, charitable partnerships, and volunteer opportunities, we encourage our teammates to make a difference in their communities.

To support our teammates beyond just their careers, we offer highly competitive benefits, comprehensive health coverage, and well-being perks that support our teammates and their dependents. We understand the importance of time for life outside of work - recognizing public holidays,offering flexible time off, and prioritizing work-life balance.

As we expand into India, our new teammates will be fully integrated with our U.S. teams, working on global projects and gaining exposure to top industry leaders. With continued growth, we will establish a physical office in Bengaluru, India, giving teammates aspace for collaboration and fostering connection.

Now is the perfect time to join Kobie. Be part of something big and help shape the future of our global capabilities center, the Kobie India Tech Hub.

From $25, paid onceGet the CV for this job

What happens when you press

One press. We do the rest.

  1. A CV for this posting

    Written against Kobie's own wording, from every piece of relevant proof in your profile.

  2. 25 readers review it

    Recruiter, hiring manager, skeptic and more read every draft, round after round. You get the best round.

    The review screen in CV Rocket: how each CV was read, round by round.
  3. We apply on Kobie's form

    Our application engine gets through the hardest forms there are. Where a question needs you, AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

    An application in CV Rocket: every answer filled in on the employer's form.
  4. Every reply, sorted

    Kobie's answer lands in your private mailbox, and we classify it on arrival: interview, question, rejection.

    The CV Rocket inbox: each employer reply classified as an interview, an action or a rejection.
  5. Reply with AI

    AI helps you write the email, checks it and sends it. We show you whether the recruiter read it.

  6. The interview in your calendar

    Full integration with your calendar. The invitation goes straight in.

    An interview invitation in the CV Rocket inbox, added to the candidate's calendar.
Get the CV for this job

From $25 per CV, paid once. No subscription.

Why it works

3×

more interviews than doing it yourself with ChatGPT.

ChatGPT writes a CV and never learns what happened to it. We see every reply. For each CV we know:

  • How it was written, and how the review scored it
  • When we applied, and how long after the posting went up
  • Which posting, which company, which city
  • Who got the interview, and who heard nothing

That is how we know which CVs get called.

Get the CV for this job

From $25 per CV, paid once. No subscription.

The numbers game

More applications. More interviews.

Every application goes out with its own CV, written for that posting and paid once. Send enough of them and the law of large numbers finds you the job.

By hand5–10
With CV Rocket100
applications a day

Nearby

Live postings like this one

Same employer first, then the same role elsewhere.

Before you press

Straight answers

Get the CV for this job

From $25 per CV, paid once. No subscription.

What if my background isn't good enough?

We make the most of the background you have. The CV uses every piece of relevant proof your profile holds, and one of the 25 readers reads your whole profile and flags what the CV left out.

Do you really apply for me?

Yes, on the employer's own form, the hardest ones included. Where a question needs you, you answer it right there and AI suggests the best answer. Don't want us applying from our IP addresses? Use our Chrome extension: we apply straight from your own browser.

Is it a subscription?

No. You pay once per CV, from $25. Every application goes out with its own CV, written for that posting.

One job. One CV.
Paid once.

Pick the posting you want. We write for it, apply for you and catch the reply.

Get the CV for this job

From $25 per CV, paid once. No subscription.