The posting
Working at CreateFuture
CreateFuture is an AI-native consulting partner where people do work that matters and are supported to do it well. We work alongside organisations such as PayPal, adidas, NatWest, FanDuel and Money Saving Expert, building digital products and services that make a difference while always putting people first.
We’re a team of creators. We write code, shape delivery, build go-to-market strategies, develop AI solutions and create the practices that support our people. We work side by side with our clients, challenging what’s not working and helping them to build the future. Our commitment to craft, quality, and culture has helped us scale to over 600 people in just a few years.
Our UK Benefits
35 days leave (including bank holidays).
- Private medical insurance.
- Enhanced parental and adoption leave.
- Financial coaching + 5% pension match.
- 40 hours of paid learning and development.
View our full list of UK benefits.
CreateFuture is a Great Place to Work-Certified™ company and has won Best Workplaces UK multiple years in a row.
Join us on our journey. Let’s create tomorrow, together, today.
About the engagement
Most organisations shipping AI features have no reliable way of knowing whether those features are any good. They have vibes, a demo that went well, and a support queue. We're engaging two Senior AI Engineers to fix that properly on a major AI platform programme with a client in a high-traffic, heavily regulated consumer sector.
You'll own the evaluation and observability layer of the platform. That covers the golden datasets, the scoring pipelines, the eval gates that sit in the promotion path, and the production monitoring that tells the client when quality has quietly drifted. It's the part of AI engineering that decides whether the rest of it can be trusted, and in a regulated business it's also the part that has to stand up to scrutiny.
This is a hands-on role embedded with the client's engineers and product people, and a large part of the job is persuasion. An eval result that nobody acts on has cost the programme money and taught it nothing. You need to be the person who can say "this score is real, here's what we should do" and be believed by a team that didn't ask for your help.
Key Responsibilities
Technical Delivery & Implementation
- Eval platform: Design and build the evaluation pipelines on a platform such as Braintrust, LangSmith, Arize or Weights & Biases, and own the decision about which one fits the problem.
- Datasets and scoring: Curate golden datasets that represent real user behaviour. Design LLM-as-judge and human-in-the-loop scoring pipelines with a clear view of where each one is trustworthy and where it isn't.
- Production observability: Build token-level tracing, quality signal monitoring, hallucination and refusal detection, and per-use-case cost attribution, with alerting that catches degradation before users report it.
- Quality gates: Wire eval gates into CI/CD promotion paths so a regression blocks a release rather than being discovered in it.
- Regression and drift: Establish the baselines, regression suites and drift detection that let the client's teams change prompts, models and retrieval without holding their breath.
- Statistical honesty: Know the difference between a signal and noise, size your eval sets accordingly, and be straight with people when the data can't answer the question they're asking.
Client Delivery & Stakeholder Management
- Turning data into decisions: Present eval and observability findings to engineering and product audiences in a way that leads to a decision, and follow through until something changes.
- Setting the standard: Help the client's teams adopt eval-first habits, and challenge the "ship it and see" instinct where you find it, with evidence rather than dogma.
- Delivery ownership: Plan and prioritise your own stream, estimate accurately, manage shifting requirements without dropping quality, and flag risk to timelines early.
- Cost awareness: Understand the cost of what you're measuring. Eval runs and judge models aren't free, so design for a sensible ratio of cost to confidence and be able to justify that to a budget holder.
- Fitting in fast: Work within the client's existing tooling and release process where it's sound, and make the case for change where it isn't.
Documentation & Handover
- Documentation as you go: Leave the dataset provenance, scoring rationale and runbooks that let the client's engineers maintain and extend the eval suite without you.
- Knowledge transfer: Bring the client's engineers along in evaluation practice, prompt and context engineering, and critical review of model output, so the discipline outlasts the engagement.
- Exit readiness: Treat a clean handover as part of the definition of done from the first sprint, not something arranged in the final fortnight.
Skills & Experience
Core Technical Capabilities
We're looking for a mix of AI engineering (50%), software engineering (30%), data engineering (20%).
- Python: Strong and production-grade. You write code others maintain.
- Eval and observability platforms: Hands-on production experience with Braintrust, LangSmith, Arize, Weights & Biases or an equivalent, including the parts that didn't work well.
- Scoring design: Demonstrable experience designing LLM-as-judge and human-in-the-loop pipelines, and calibrating them against human judgement.
- Production AI systems: Experience operating LLM-backed features in production, covering tracing, latency, token cost, failure modes and retrieval quality.
- CI/CD: Enough fluency with GitHub Actions, GitLab CI or similar to own an eval gate in a promotion pipeline yourself.
- Data handling: Comfortable building the data pipelines and stores behind datasets, traces and results at volume.
- Communication: The ability to make a technical result land with a non-technical audience, tailoring the framing to the room.
Domain & Sector Experience
- Regulated industries: Experience where model behaviour has compliance or consumer-protection consequences, such as iGaming, financial services or health, is a strong advantage. Safer gambling and responsible-messaging contexts are directly relevant here.
- Contract and consulting delivery: A track record of arriving on an unfamiliar programme and being productive quickly, including building credibility with engineers who didn't ask for your help.
Useful credentials
Track record matters more than certification here. If you have an eval framework, a write-up or an open-source contribution you're proud of, lead with that. The following are useful supporting evidence:
- AWS Certified Machine Learning Engineer (Associate) or AWS Certified AI Practitioner.
- An associate-level AWS or GCP certification for the platform fundamentals around your work.
Contract details
These are contract engagements delivering a defined scope of work on a client programme. Duration, rate and IR35 status will be confirmed with our Talent Acquisition team at first contact. The work is largely remote, with travel to the client's site or to CreateFuture offices as the programme requires.
Our hiring process
- A call with our Talent Acquisition team, covering rate, availability and notice
- A capability and technical interview focused on the client and project work.
We move quickly for contract roles and will be straight with you about timelines.
Inclusion at CreateFuture
We want CreateFuture to be a place where people from every background can do their best work. If you need any adjustments to the process, or there's something that would help you show us what you can do, tell us and we'll sort it.
What we’ll offer you:
We trust people to do their best work. That means flexibility over rigid rules, impact over activity, and real investment in your growth both professionally and personally. You’ll be part of a supportive, and friendly culture, surrounded by smart, curious people who care deeply about what they do.
We offer flexible working, including hybrid and remote options. Our office hubs are located in Edinburgh, Leeds, Manchester, London and Bulgaria, with occasional travel to client sites or CreateFuture offices when needed.
We trust you to manage your time balancing collaboration with client time and focused work. What matters is the impact you have, not how busy you look.
Our hiring process
We try to keep our hiring process clear, fair and respectful of your time. We aim to get back to everyone who applies and we will be upfront about where you are in the process.
It usually looks like this:
- Call with our Talent Acquisition Team
- Role specific capability interview
Depending on the role, we might also ask you to do a short presentation, a practical or technical task or have a values focused conversation. We will explain what is involved before anything happens.
Inclusion at CreateFuture
We believe diverse teams build better workplaces and better products. We want CreateFuture to be a place where people feel able to be themselves and do their best work.
If you need any adjustments or support during the application process, just. We will do what we can to help.
We look forward to your application!
