The posting
Care Lumen | Remote (US Only)
Employment Type: Contractor (3-6 months, with possibility of extension)
Salary Range: $129 to $150 an hour
Scope
The Senior Generative AI Engineer is an individual contributor responsible for testing, evaluating, optimizing, and monitoring enterprise Generative AI systems. The primary focus is improving the quality, reliability, and performance of existing GenAI solutions through systematic testing, evaluation of new model versions, prompt optimization, and application architecture improvements. The role will also help drive evaluation methodology, monitoring capabilities, and AI governance requirements for selected initiatives. Approximately 25% of the role is expected to support AI governance activities, including model documentation, versioning, registry management, experiment tracking, auditability, and related controls.
The role collaborates with engineering, data science, architecture, security, compliance, governance, and business stakeholders. It does not have direct reports or a formal leadership mandate.
What You’ll Do
- Test and evaluate GenAI applications, AI agents, and LLM-powered workflows for quality, reliability, accuracy, safety, and task effectiveness.
- Develop and execute repeatable GenAI evaluation methodologies, test datasets, benchmark suites, regression tests, and automated evaluation pipelines.
- Use DeepEval and related evaluation approaches to measure GenAI output quality and identify regressions across prompts, workflows, application changes, and model versions.
- Optimize GenAI solution architecture by decomposing complex prompts and workflows into appropriate subtasks, tools, and specialized agents.
- Build and improve multi-agent workflows using orchestration frameworks such as LangGraph and LangChain.
- Optimize prompts, system instructions, context management, agent interactions, and model selection / configurations using evaluation results (including latency and model performance) and production feedback (including user feedback).
- Evaluate new foundation models and model versions to determine quality, behavior, risks, performance, (groundedness, completeness, bias, etc) and suitability for specific use cases.
- Design and implement monitoring capabilities for GenAI systems, including quality metrics, operational telemetry, audit trails, alerts, and performance reporting.
- Help drive AI governance activities for selected initiatives, including model and application documentation, risk assessments, evaluation evidence, model versioning, registry management, experiment tracking, and change traceability.
- Collaborate across technical, governance, security, compliance, and business teams to improve GenAI solutions and support responsible production use.
Competencies
- GenAI testing and quality evaluation
- GenAI evaluation methodology and automated evaluation
- Prompt, context, and application architecture optimization
- Multi-agent architecture and orchestration
- Foundation model and model-version testing
- AI governance and model lifecycle management
- GenAI monitoring, observability, and traceability
- Traditional machine learning testing and validation
- Analytical problem-solving and cross-functional collaboration
What’s Required
- 10+ years of overall professional experience in data science, machine learning, AI engineering, or a related technical field.
- 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems.
- Demonstrated experience testing GenAI applications and establishing systematic methods for assessing LLM output quality.
- Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries.
- Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows.
- Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking.
- Practical experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability.
- Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with a strong understanding of model testing and validation methods.
- Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders.
- Ability to maintain at least four hours of overlap with U.S. Eastern Time (EST/EDT) each workday.
- Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline.
What’s Preferred
- Experience with Databricks Mosaic AI.
- Experience working with PHI / PII data inside highly regulated domain like healthcare.
- Experience testing or optimizing multi-agent GenAI systems in production or production-like environments.
- Experience working with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures.
- Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment.
Compensation & Benefits
The expected total cash compensation for this role is market-driven, with an expected compensation rate of $129 to $150 per hour. Hourly rate is ultimately determined based on a candidate's skills, expertise, and experience.
Offers are contingent upon successful completion of a background screening. Background screenings will be conducted in compliance with applicable laws and will not be initiated without your consent.
Care Lumen is an equal opportunity employer committed to building a diverse and inclusive workplace. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other legally protected status. We are committed to providing reasonable accommodations to qualified individuals with disabilities in our job application process. If you need assistance or an accommodation due to a disability, please contact us at [email protected].
About Care Lumen Care Lumen is transforming healthcare by connecting people, care, and data—so that every decision leads to better outcomes. We empower care teams with a unified platform that delivers intelligent, real-time insights, streamlined workflows, and stronger collaboration across clinical, operational, and administrative domains.
Our vision is bold: to reimagine how care is delivered by unifying clinical operations and unlocking insights that improve lives. We believe in the power of great people and purposeful technology—and with all of us aligned around a common goal, our potential grows even stronger. We're building for impact today and evolving through innovation tomorrow. We’re a remote-first company that values flexibility, autonomy, and results over hours logged. We're guided by values that push us to excel, stay curious, and take smart risks. The best solutions come from teams that listen, learn, and build together. And we're focused on outcomes that matter—not just for our clients, but for the people they serve.
Care Lumen’s LinkedIn Care Lumen’s website
Ready to Apply?
