The posting
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Content Specialist III | AI Evaluation & Prompting based in United States.
This remote contract role offers the opportunity to directly influence the quality and behavior of advanced AI systems. You’ll evaluate model responses across diverse topics, conversation types, and real-world use cases. Your work will help identify strengths, uncover failure patterns, and improve how AI systems communicate with users. You’ll combine expert writing, analytical judgment, prompt engineering, and structured quality assessment to shape model performance. The role involves close, hands-on experimentation with evolving AI products in a fast-moving environment. You’ll translate detailed observations into actionable feedback, stronger prompts, and more effective evaluation frameworks. Success requires balancing quality, speed, consistency, and independent judgment while adapting to changing product priorities.
Accountabilities
- Test new AI model versions across a broad range of topics, use cases, and conversation scenarios.
- Evaluate AI-generated responses against established rubrics, guidelines, and quality standards.
- Identify, document, and communicate examples of both successful and unsuccessful model behavior.
- Write, test, and refine system prompts designed to influence model personality, tone, responses, and behavior.
- Assess whether evaluation rubrics and quality frameworks accurately measure model performance and identify opportunities for improvement.
- Conduct quality reviews of human- and agent-based evaluations to ensure accuracy, consistency, and compliance.
- Perform hands-on experiments with AI models and products to investigate capabilities, limitations, and unexpected behaviors.
- Investigate model failures, identify potential causes or patterns, and document findings clearly.
- Apply detailed instructions and evaluation criteria consistently across a high volume of work.
- Perform fact checking and verify model claims against reliable original sources when required.
- Translate observations and research findings into clear, actionable feedback for product and AI teams.
- Adapt evaluation approaches as model behavior, product priorities, and quality requirements evolve.
- 5+ years of professional experience in writing, editing, journalism, production, linguistics, STEM, coding, policy, or another relevant subject-matter discipline.
- Bachelor’s degree or equivalent professional experience.
- At least 1 year of hands-on AI experience is preferred, including prompting, annotation, model evaluation, red teaming, or related work.
- Strong experience evaluating AI-generated content, large language model outputs, or other complex digital content is highly desirable.
- Deep expertise in at least one subject area, with the ability to evaluate content accurately as a subject-matter expert.
- Strong writing, editing, fact-checking, and research capabilities.
- Experience writing and refining prompts and assessing how prompt changes influence AI behavior.
- Strong analytical judgment and exceptional attention to detail.
- Ability to consistently apply detailed rubrics, guidelines, and evaluation criteria across high-volume work.
- Ability to identify subtle differences in AI responses and determine why a response succeeds or fails.
- Strong documentation and communication skills, with the ability to clearly explain findings, questions, blockers, and recommendations.
- Ability to verify claims against original or authoritative sources.
- Experience conducting detailed failure investigations is preferred.
- Comfortable working independently while escalating important questions, risks, and blockers early.
- Highly adaptable and comfortable shifting priorities as AI products and business needs evolve.
- Professional fluency in English is required.
- Must be authorized to work in the United States without requiring current or future sponsorship.
- 6-month remote contract based in the United States.
- Full-time schedule of approximately 40 hours per week.
- Standard schedule of Monday through Friday with 8-hour shifts.
- Health insurance.
- Dental insurance.
- Vision insurance.
- Health savings account (HSA).
- Life insurance.
- Paid time off.
- Retirement plan.
- Opportunity to work directly on the evaluation and improvement of advanced AI systems.
- Hands-on exposure to AI model evaluation, prompt engineering, quality frameworks, and emerging AI products.
- Opportunity to apply specialized subject-matter expertise to real-world AI development and product improvement.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
