The posting
Help product teams turn customer feedback into better product decisions.
We are building AI capabilities for the Feedback app within Product Collection, helping customers find relevant evidence, discover meaningful themes, and receive useful suggestions and shape their roadmap in Jira Product Discovery.
As the first ML engineer on the team, you will establish the ML foundations of a new data product within the fast-growing Product Collection. You will have clear product direction and autonomy to experiment, make technical decisions, and ship improvements, working with product engineers, product management, and design.
You will build on Atlassian’s shared infrastructure and expertise, with the opportunity to help shape and grow the ML team as the product develops.
Our customers bring varied datasets, terminology, and ways of working. The challenge is to deliver trustworthy results with minimal setup, while keeping response times and processing costs practical.
Finding the right customer evidence
A useful answer starts with relevant evidence. You will help us improve how we retrieve feedback for a product question, balancing fast interactive search with deeper analysis in background workflows. This includes evaluating how we prepare and chunk data, choose embeddings, retrieve candidates and rerank results.
Discovering meaningful themes at scale
Customers need to understand recurring and emerging needs across many feedback items. You will help us combine clustering, retrieval and language-model synthesis to identify well-supported themes, preserve differences between customer segments and avoid duplicate or weak conclusions. You will also investigate how to refresh this analysis efficiently as new feedback arrives.
Making suggestions more useful over time
AI can suggest tags, populate fields and connect feedback, insights and product ideas. You will help us use customer acceptance, rejection and edits to evaluate and improve these suggestions. The challenge is to learn from imperfect signals without reinforcing bias or repeatedly presenting suggestions customers have already dismissed.
What you will do
- Turn product problems into testable ML hypotheses, then design experiments that compare approaches against clear baselines.
- Build representative evaluation datasets and workflows, combining measurable checks with human judgement where quality is subjective.
- Develop and improve retrieval, clustering and language-model pipelines, working with product engineers to integrate them into customer workflows.
- Make evidence-based trade-offs between quality, latency, cost and operational complexity. Use simpler approaches when they meet the need, and evaluate fine-tuning where it offers a measurable benefit.
- Evaluate performance across customer segments and dataset sizes, including sparse and imbalanced data, rather than relying only on aggregate metrics.
- Help take capabilities from experimentation into production, with regression checks, monitoring, controlled rollouts and clear ways to diagnose failures.
- Work with product, design and engineering colleagues to interpret customer feedback and translate findings into priorities.
- Reuse shared AI infrastructure and collaborate with platform teams, while developing evaluation methods and tooling that can support other Product Collection use cases.
What success would look like
Your initial focus would be agreed with the team. A proposed starting point is finding relevant customer evidence, because it supports the quality of downstream answers and insights.
- Establish a shared baseline: We understand the most important failure patterns and can measure quality, response time and cost on representative examples.
- Improve a priority workflow: We have evidence that a selected approach improves the customer experience, with its limitations and operational trade-offs understood.
- Make improvement repeatable: The team can detect regressions, observe production behaviour and use the results to prioritise further work.
The role offers the opportunity to shape both customer-facing AI capabilities and the evaluation practices behind them, starting with Feedback and identifying useful opportunities for reuse across Product Collection.
We are looking for someone who combines applied ML judgement with strong software engineering. You should be comfortable working through ambiguous product problems and explaining what the evidence supports, what remains uncertain and what to try next.
- Experience delivering applied ML systems: You have taken an ML capability beyond a prototype and can explain how you evaluated it, integrated it into a product or service, and monitored or improved it after release.
- Strong programming and engineering fundamentals: Expertise in Python with and the ability to write performant production-quality code, familiarity with SQL, knowledge of Spark and cloud data environments (e.g. AWS, Databricks)
- Relevant technical depth: You have practical experience in one or more of information retrieval, semantic search, natural language processing, clustering or LLM-based applications, and can apply that experience to unfamiliar problems.
- Evaluation and experimentation skills: You can construct useful datasets, choose appropriate metrics, compare against baselines and investigate why offline improvements may not translate into better customer outcomes.
- Sound data judgement: You understand issues such as data leakage, selection bias, imbalanced data, sparse labels and overfitting, and can explain how they affect design and evaluation choices.
- Production awareness: You consider latency, inference cost, observability and changing data as part of the solution, and can work with engineers to diagnose and improve production behaviour.
- Clear communication and collaboration: You can explain technical trade-offs to colleagues across disciplines and work with them to agree useful outcomes and practical next steps.
Experience that would be useful
You do not need experience in every area below. These are opportunities to bring additional depth to the team.
- Building retrieval-augmented generation systems, including chunking, embeddings, hybrid search and reranking.
- Clustering or discovering themes in large collections of unstructured text, with traceable supporting evidence.
- Designing human evaluation, annotation or relevance-judgement processes for subjective outputs.
- Using implicit or explicit user feedback to improve ranking, recommendations or other ML behaviour.
- Working with cold-start scenarios, limited labelled data or heterogeneous customer datasets.
- Running online experiments or controlled releases and investigating differences between offline and production results.
- Model fine-tuning, model selection or inference optimisation, with evidence of when the additional complexity is worthwhile.
- Building reusable evaluation tooling, regression checks or monitoring for multi-stage ML and LLM systems.
Working at Atlassian
Atlassians can choose where they work – whether in an office, from home, or a combination of the two. That way, Atlassians have more control over supporting their family, personal goals, and other priorities. We can hire people in any country where we have a legal entity.
