Talent.com
This job offer is not available in your country.
AI Agent Quality Specialist - AI Trainer

AI Agent Quality Specialist - AI Trainer

MindriftWorkFromHome, Auckland, New Zealand
6 days ago
Job description

Overview

AI Agent Evaluation Analyst - AI Trainer at Mindrift. This is a flexible, project-based opportunity suitable for remote, async work. The Mindrift platform connects domain experts with AI projects powered by Toloka. Our mission is to unlock the potential of GenAI by leveraging real-world expertise from across the globe.

We are looking for curious, intellectually proactive contributors who double-check assumptions and think critically. If you enjoy ambiguity and complexity and want to learn how modern AI systems are tested and evaluated, this role may be a fit.

About the project

We are seeking QA-like support for autonomous AI agents on a project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. You will balance quality assurance, research, and logical problem-solving to help ensure robust testing of systems.

You do not need a coding background, but you must be curious, rigorous, and able to evaluate the soundness and consistency of complex setups. If you have experience in consulting, case solving, systems thinking, or related activities, you might be a great fit.

Responsibilities

  • Review evaluation tasks and scenarios for logic, completeness, and realism
  • Identify inconsistencies, missing assumptions, or unclear decision points
  • Help define clear expected behaviours (gold standards) for AI agents
  • Annotate cause–effect relationships, reasoning paths, and plausible alternatives
  • Think through complex systems and policies to ensure agents are tested properly
  • Collaborate with QA, writers, or developers to suggest refinements or edge-case coverage

What you’ll be doing

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism
  • Identifying inconsistencies, missing assumptions, or unclear decision points
  • Helping define clear expected behaviours (gold standards) for AI agents
  • Annotating cause–effect relationships, reasoning paths, and plausible alternatives
  • Thinking through complex systems and policies as a human to ensure proper testing of agents
  • Working with QA, writers, or developers to suggest refinements or edge-case coverage
  • Requirements

  • Excellent analytical thinking : ability to reason about complex systems, scenarios, and logical implications
  • Strong attention to detail : identify contradictions, ambiguities, and vague requirements
  • Familiarity with structured data formats; ability to read JSON / YAML is helpful
  • Ability to assess scenarios holistically : identify what’s missing or unrealistic and what could break
  • Good communication and clear writing in English to document findings
  • What we value

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design
  • Background in consulting, academia, Olympiads, or research
  • Exposure to LLMs, prompt engineering, or AI-generated content
  • Familiarity with QA or test-case thinking (edge cases, failure modes)
  • Understanding of how scoring or evaluation works in agent testing (precision, coverage)
  • Benefits

  • Compensation up to $38 / hour depending on skills and project needs
  • Flexible, remote, freelance project that fits around commitments
  • Opportunity to gain experience on an advanced AI project
  • Contribute to shaping how AI models understand and communicate in your field
  • How to get started

    Apply to this post, qualify, and contribute to a project aligned with your skills on your own schedule. Shape the future of AI while building tools that benefit everyone.

    #J-18808-Ljbffr

    Create a job alert for this search

    Trainer • WorkFromHome, Auckland, New Zealand

    Related jobs
    • Promoted
    AI Quality Assurance Engineer (AI QA Solutions & Automation)

    AI Quality Assurance Engineer (AI QA Solutions & Automation)

    DatacomAuckland, Auckland, New Zealand
    AI Quality Assurance Engineer (AI QA Solutions & Automation).Join to apply for the AI Quality Assurance Engineer (AI QA Solutions & Automation) role at Datacom. With over 6,200 people, and centres o...Show moreLast updated: 30+ days ago
    • Promoted
    Ai Quality Assurance Engineer (Ai Qa Solutions & Automation)

    Ai Quality Assurance Engineer (Ai Qa Solutions & Automation)

    DatacomAuckland, New Zealand
    AI Quality Assurance Engineer (AI QA Solutions & Automation).Join to apply for the AI Quality Assurance Engineer (AI QA Solutions & Automation) role at Datacom. With over 6,200 people, and centres o...Show moreLast updated: 30+ days ago
    • Promoted
    Ai Architect

    Ai Architect

    Microsoft CorporationAuckland, New Zealand
    Microsoft Industry Solutions Delivery is a global organization hosting over 6,000 strategic sellers, industry and security experts, elite engineers, world-class data scientists and architects, cons...Show moreLast updated: 30+ days ago
    • Promoted
    • New!
    Data and AI ML Solution Architect

    Data and AI ML Solution Architect

    Accenture New ZealandAuckland, Auckland, New Zealand
    Seeking a Data & AI Solution Architect with a strong focus on Artificial Intelligence (AI), Machine Learning (ML), and Generative AI. You will be responsible for designing and implementing advanced ...Show moreLast updated: 20 hours ago
    • Promoted
    Ai And Analytics Engineer

    Ai And Analytics Engineer

    Electric KiwiAuckland, New Zealand
    Be among the first 25 applicants.The Energy Collective (TEC) exists to #makeitbetter by creating and scaling digital businesses which disrupt utility retailing markets. We're at an exciting time whe...Show moreLast updated: 5 days ago
    • Promoted
    Ai Architect

    Ai Architect

    MicrosoftAuckland, New Zealand
    Responsibilities (Text Only)Architecture Design & Deployment - Lead enterprise-grade Azure cloud and AI platform design and implementation, focusing on scalability, security, performance, and cost ...Show moreLast updated: 30+ days ago
    Search Specialist

    Search Specialist

    YoungShandAuckland, Auckland, NZ
    Quick Apply
    We’re on the hunt for a success-driven Search Specialist to lead our search strategy and execution across a portfolio of key clients. You’ll be a proven search expert – someone with deep hands-on ex...Show moreLast updated: 30+ days ago
    SEO Analyst / Data Specialist

    SEO Analyst / Data Specialist

    RM Staffing B.V.Auckland, AUK, NZ
    SEO measurement, reporting, and performance insights.This role is critical in turning raw data into actionable recommendations that directly influence strategy, experimentation, and growth.You will...Show moreLast updated: 30+ days ago
    • Promoted
    AI LLM Technology Arch Sr. Analyst

    AI LLM Technology Arch Sr. Analyst

    Accenture Australia & New ZealandAuckland City, Auckland, New Zealand
    Ignite your passion for innovation! You will have the opportunity to perform independently and become a subject matter expert while actively participating in discussions that contribute to providin...Show moreLast updated: 3 days ago
    Senior AI Research Engineer, Model Inference (100% Remote)

    Senior AI Research Engineer, Model Inference (100% Remote)

    Tether Operations LimitedAuckland, AUK, NZ
    Join Tether and Shape the Future of Digital Finance.At Tether, we’re not just building products, we’re pioneering a global financial revolution. Our cutting-edge solutions empower businesses—from ex...Show moreLast updated: 8 days ago
    • Promoted
    Ai And Innovation Manager

    Ai And Innovation Manager

    Keller Executive SearchAuckland, New Zealand
    OverviewThis is a position within Keller Executive Search and not with one of its clients.As the AI and Innovation Manager in Auckland, New Zealand, this senior role is accountable for shaping ai a...Show moreLast updated: 19 days ago
    AI and Innovation Manager

    AI and Innovation Manager

    Keller Executive SearchAuckland, Auckland, NZ
    Quick Apply
    This is a position within Keller Executive Search and not with one of its clients.As the AI and Innovation Manager in Auckland, New Zealand, this senior role is accountable for shaping ai and innov...Show moreLast updated: 21 days ago
    • Promoted
    Ai Engineer

    Ai Engineer

    Kpmg New ZealandAuckland, New Zealand
    At KPMG, we believe in fuelling prosperity for all New Zealanders.We bring this purpose to life through the work we do and the way we do it. Our people live by our shared values : Integrity, Courage,...Show moreLast updated: 7 days ago
    • Promoted
    • New!
    Freelance Mechanical Engineering Expert - AI Trainer

    Freelance Mechanical Engineering Expert - AI Trainer

    MindriftWorkFromHome, Auckland, New Zealand
    The Mindrift platform connects specialists with AI projects from major tech innovators.Our mission is to unlock the potential of Generative AI by tapping into real-world expertise from across the g...Show moreLast updated: 20 hours ago
    Senior AI Engineer

    Senior AI Engineer

    DatacomAuckland, Auckland, NZ
    Quick Apply
    Datacom works with organisations and communities across Australia and New Zealand to make a difference in people’s lives and help organisations use the power of tech to innovate and grow.About the ...Show moreLast updated: 18 days ago
    • Promoted
    • New!
    Data And Ai Ml Solution Architect

    Data And Ai Ml Solution Architect

    Accenture New ZealandAuckland, New Zealand
    OverviewSeeking a Data & AI Solution Architect with a strong focus on Artificial Intelligence (AI), Machine Learning (ML), and Generative AI. You will be responsible for designing and implementing a...Show moreLast updated: 5 hours ago
    • Promoted
    • New!
    Freelance Earth Science Expert - AI Trainer

    Freelance Earth Science Expert - AI Trainer

    MindriftWorkFromHome, Auckland, New Zealand
    Freelance Earth Science Expert - AI Trainer.Be among the first 25 applicants.This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility...Show moreLast updated: 20 hours ago
    AI and Automation Specialist

    AI and Automation Specialist

    Auckland AirportAuckland, Auckland, NZ
    Quick Apply
    AKL is searching for a top-tier communicator, bot commander, and AI wrangler to join our Digital Team.If you can speak fluent automation, keep the machines working for us (not against us), and make...Show moreLast updated: 15 days ago
    AI Quality Assurance Lead

    AI Quality Assurance Lead

    DatacomAuckland, Auckland, NZ
    Quick Apply
    With over 6,200 people, and centre’s of technical excellence spread across Australasia, Datacom is committed to developing and growing its capability across Asia. Founded in 1965 in New Zealand, and...Show moreLast updated: 30+ days ago
    AI Quality Assurance Automation Engineer

    AI Quality Assurance Automation Engineer

    DatacomAuckland, Auckland, NZ
    Quick Apply
    The way we work and our customer needs are changing! We are going through a huge AI uplift that will be a real game changer in the NZ Market. Our customers are loving our story and excited about our...Show moreLast updated: 30+ days ago