The posting
About SOFICO Intelligence
SOFICO Intelligence applies advanced AI and intelligence techniques to transform and extract value from the deep business controls embedded in back-office systems. We build agentic AI that reasons over the complex financial, leasing, fleet, and mobility-management data that runs the enterprise, surfacing fraud, billing errors, hidden risk, and value that traditional tools miss. We are backed by SOFICO, a global leader in ERP software for automotive finance, leasing, fleet, and mobility management, which gives us deep domain and back-office expertise to build on. Remote work is part of how we operate, and our team is made up of curious, self-directed people who like to build and ship.
Our flagship AI platform, Velona, is a multi-agent system that turns years of business and operational data into financial intelligence. It finds fraud, ghost assets, billing errors, and risk on its own, and reports each finding as a specific asset, a dollar amount, and a recommended action. Rather than acting as a chatbot or a dashboard, it performs real analytical reasoning, runs investigations overnight without supervision, and answers in plain language.
What the job is about
This role is responsible for making our autonomous AI trustworthy. As an SDET on the AI team, you own quality for non-deterministic, multi-agent systems, which raises questions most testing roles never face. How do you prove an AI reached the right answer for the right reason? How do you catch a hallucination before a customer sees it? How do you guarantee one tenant can never see another's data, that an agent never exceeds its guard rails, and that quality does not quietly drift when a model or prompt changes?
You will help define how SOFICO Intelligence evaluates agentic AI, designing and building the evaluation harness that underpins our AI governance and our confidence to ship. This is real engineering against a production AI system at scale, not script-running against a static application. You write code to test code, and code to assess the quality of the AI itself. It is one of the more interesting testing problems in software today, and you would own it.
What you'll do (about 85% of the role):
- Design and build SOFICO Intelligence's evaluation harness for agentic AI: golden datasets, automated scoring, faithfulness and grounding checks, and regression detection across prompt, model, and agent changes.
- Verify grounding and faithfulness, confirming every insight and every number traces back to real data, and build the automated checks that stop hallucinations from reaching production.
- Test multi-agent orchestration end to end: agentic tool-calling loops, supervisor-specialist pipelines, autonomous background jobs, and the synthesis of findings into trustworthy reports.
- Red-team the system for prompt injection, jailbreaks, and tool misuse, and rigorously prove multi-tenant data isolation so one organization can never reach another's data.
- Validate that AI guard rails hold under pressure: iteration limits, agent-recursion depth, and token/cost ceilings behave correctly at the edges.
- Pressure-test performance and scale: real-time streaming, high-concurrency conversations, latency, and cost under realistic and adversarial load.
- Build automated tests for the Go microservices, data/SQL tools, and APIs that power the platform, and wire them into CI/CD for continuous evaluation.
- Partner with AI and data engineers to define quality bars and acceptance criteria, and help evolve the supervisor review-and-retry loop that keeps the system honest.
The remaining ~15%, breadth across the platform:
- Strengthen automated testing across the broader SOFICO Intelligence platform and, where it intersects, the wider SOFICO engineering environment, bringing modern test practices to established systems and tooling.
- Help raise quality engineering standards consistently across the combined group.
Why this role matters
- Hard, important work. Evaluating autonomous, reasoning AI is one of the harder open problems in software today, and you would help define how we approach it.
- Real ownership. You won't inherit a mature test suite to maintain. You'll build the evaluation foundation from the ground up and see it become core to how we ship.
- Real impact. Your work decides whether customers can trust an AI with their money. The stakes and the visibility are both high.
- Strong technology. Modern Go services, LLM orchestration, a multi-year proprietary data asset, and a team that has shipped at scale for years.
Your skills and interests
You are comfortable with the ambiguity of testing systems that don't give the same answer twice. You take pride in shipping software people can trust, and you want to see the products you work on succeed in the real world. You do well in collaborative problem-solving environments, prefer coding and collaborating over sitting in meetings, and can prioritize your own work while working closely with a team. You give and take constructive criticism well, and you bring a structured, analytical approach to building scalable evaluation systems.
What else is expected
Mandatory skills:
- Proven experience spanning both software development and quality assurance / quality engineering.
- Strong coding skills, able to program confidently in at least one language (we work primarily in Go and Python).
- Hands-on experience with open-source automated testing tools and frameworks.
- A solid understanding of RESTful APIs and the ability to perform HTTP requests from the command line or a tool like Postman.
- Comfortable working from the command line and reasoning about distributed, concurrent systems.
- Hands-on experience testing, evaluating, or building with LLM-based or agentic systems, not just interest. You have worked directly with model APIs, tool-calling, or multi-agent workflows and understand how they fail.
- Demonstrated ability to evaluate non-deterministic AI output against a measurable quality bar. For example, building or running eval harnesses, scoring grounding and faithfulness, detecting quality regressions, or red-teaming for prompt injection and jailbreaks.
- Working knowledge of LLM concepts that drive quality: prompting, context windows and token limits, hallucination and grounding, and the sources of non-determinism in model output.
- A sense of pride and a strong desire to ensure trustworthy, bug-free software ships every day.
- Fluent in English, as the role involves interaction with international teams.
Bonus points:
- Depth in AI evaluation methodology, such as designing golden datasets, LLM-as-judge scoring, supervisor/critic patterns, or automated regression detection for agentic pipelines.
- Experience with Go, data warehouses / SQL analytics (e.g. Databricks), event-driven systems, or real-time streaming.
- Familiarity with enterprise back-office or ERP environments, including Java back-ends and relational databases (Oracle, MSSQL, Postgres), is a plus for the cross-platform work.
- Exposure to the automotive finance, leasing, fleet, or mobility domain.
What you'll work with
- AI & data platform: a production multi-agent AI system built on modern Go microservices, LLM orchestration, and a multi-year proprietary fleet-data asset.
- Evaluation & testing: AI evaluation harnesses, golden datasets, Playwright, REST Assured, PyTest, JMeter, and similar open-source frameworks.
- Languages: Go and Python primarily, with JavaScript across the stack.
- Data & infrastructure: PostgreSQL, Redis, RabbitMQ, Databricks, Docker, and Kubernetes.
- APIs: RESTful and real-time WebSocket / streaming interfaces, exercised via Postman, REST Assured, and the command line.
- Organizational tools: Jira, TestRail, and Git, with tests integrated into CI/CD pipelines.
Benefits & how we work
We have operated as a remote-first team for years and have a long track record of running productive, professional remote teams. We offer competitive benefits, including health care (medical, dental, and vision) for US employees and a flexible PTO policy, and we support your growth through conferences, advanced training, and certifications.
