Talent.com
This job offer is not available in your country.
AI Agent Evaluation Analyst

AI Agent Evaluation Analyst

MindriftWorkFromHome, Northland, New Zealand
3 days ago
Job description

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What We Do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.

Who We're Looking For

We're looking for curious and intellectually proactive contributors, the kind of person who double‑checks assumptions and plays devil’s advocate. Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

We are ideal for analysts, researchers, or consultants with strong critical thinking; students; or people open to part‑time non‑permanent opportunity.

About the Project

We're on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you'll have to balance quality assurance, research, and logical problem‑solving.

What You'll Be Doing

  • Review evaluation tasks and scenarios for logic, completeness, and realism.
  • Identify inconsistencies, missing assumptions, or unclear decision points.
  • Help define clear expected behaviors (gold standards) for AI agents.
  • Annotate cause‑effect relationships, reasoning paths, and plausible alternatives.
  • Think through complex systems and policies as a human would to ensure agents are tested properly.
  • Work closely with QA, writers, or developers to suggest refinements or edge‑case coverage.

How to Get Started

Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills on your own schedule. Shape the future of AI while building tools that benefit everyone.

Requirements

  • Excellent analytical thinking – can reason about complex systems, scenarios, and logical implications.
  • Strong attention to detail – can spot contradictions, ambiguities, and vague requirements.
  • Familiarity with structured data formats – can read, not necessarily write JSON / YAML.
  • Ability to assess scenarios holistically : Identify missing, unrealistic, or break‑potential aspects.
  • Good communication and clear writing in English to document findings.
  • Additional Strengths

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
  • Background in consulting, academia, olympiads (logic / math / informatics), or research.
  • Exposure to LLMs, prompt engineering, or AI‑generated content.
  • Familiarity with QA or test‑case thinking (edge cases, failure modes, what could go wrong).
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.).
  • Benefits

  • Get paid for your expertise, with rates that can go up to $47 / hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.
  • #J-18808-Ljbffr

    Create a job alert for this search

    Analyst • WorkFromHome, Northland, New Zealand

    Related jobs
    • Promoted
    Real Estate Agent (Whangarei)

    Real Estate Agent (Whangarei)

    REAP RecruitmentWhangarei, Northland, New Zealand
    Opportunities for Existing & New Real Estate Salespeople | Multiple Offices & Brands | Apply Today to See Them AllREAP Recruitment specialise in (and are the largest advertiser of) recruitment oppo...Show moreLast updated: 30+ days ago
    • Promoted
    Engineering Director - AI Platform

    Engineering Director - AI Platform

    CanvaWorkFromHome, Northland, New Zealand
    Be among the first 25 applicants.Get AI-powered advice on this job and more exclusive features.Join the team redefining how the world experiences design. Hey, g'day, mabuhay, kia ora, 你好, hallo, vít...Show moreLast updated: 3 days ago
    • Promoted
    Mathematics Expert (Remote)

    Mathematics Expert (Remote)

    TaskifyWorkFromHome, Northland, New Zealand
    Join to apply for the Mathematics Expert (Remote) role at Taskify.Get AI-powered advice on this job and more exclusive features. As a Mathematics Domain Expert, you will leverage your advanced subje...Show moreLast updated: 3 days ago
    • Promoted
    Consultant Geriatrician

    Consultant Geriatrician

    WaikatodhbnewsroomWhangarei, Northland, New Zealand
    Health New Zealand | Te Whatu Ora is firmly grounded in the principles of Te Tiriti o Waitangi and is committed to building a health system that serves all New Zealanders.Whangarei Hospital is a 25...Show moreLast updated: 18 days ago
    • Promoted
    Engineering Director - AI Platform (ANZ Remote)

    Engineering Director - AI Platform (ANZ Remote)

    CanvaWorkFromHome, Northland, New Zealand
    Engineering Director - AI Platform (ANZ Remote).Join the team redefining how the world experiences design.Hey, g'day, mabuhay, kia ora, 你好, hallo, vítejte! Thanks for stopping by.We know job huntin...Show moreLast updated: 3 days ago
    • Promoted
    Real Estate Agent

    Real Estate Agent

    REAP RecruitmentWhangarei, Northland, New Zealand
    EVES Realty are interested in hearing from New & Existing Salespeople for their Whangarei Office.EVES Realty are interested in hearing from New & Existing Salespeople for their Whangarei Office.The...Show moreLast updated: 30+ days ago
    • Promoted
    Remarkable AI Expert

    Remarkable AI Expert

    Remarkable AIWorkFromHome, Northland, New Zealand
    Become a Customer Support Expert for brands you love.Chatdesk (DBA Remarkable AI) Experts are freelance customer support agents who help companies provide the best support for their customers throu...Show moreLast updated: 18 days ago
    • Promoted
    Principal Machine Learning Engineer

    Principal Machine Learning Engineer

    Vital.ioWorkFromHome, Northland, New Zealand
    We are looking for a Principal Machine Learning Engineer to join our remote-first team, in a New Zealand & Australia friendly timezone. You will be leading from the front on our ML efforts, ensuring...Show moreLast updated: 30+ days ago
    • Promoted
    Restaurant Manager

    Restaurant Manager

    Kawakawa Foods LimitedFar North, Northland, New Zealand
    We are currently seeking an experienced Restaurant Manager to join our team on a permanent basis.The role guarantees a minimum of 30 hours per week, with the potential for additional hours based on...Show moreLast updated: 16 days ago
    • Promoted
    • New!
    Electrical Technician

    Electrical Technician

    Top EnergyFar North, Northland, New Zealand
    The Top Energy Group operates throughout the Far North delivering geothermal electricity generation and electricity network operations. Our purpose is providing affordable and sustainable energy for...Show moreLast updated: 18 hours ago
    • Promoted
    Search Analyst New Zealand (Work From Home)

    Search Analyst New Zealand (Work From Home)

    TELUS Digital AI Data SolutionsWorkFromHome, Northland, New Zealand
    Search Analyst New Zealand (Work From Home).Search Analyst New Zealand (Work From Home).TELUS Digital AI Data Solutions.What You’ll Do As A Media Search Analyst : . Assess diverse media categories : mu...Show moreLast updated: 28 days ago
    • Promoted
    Data Scientist / Algorithm Engineer (LLM) – AI Safety

    Data Scientist / Algorithm Engineer (LLM) – AI Safety

    BinanceWorkFromHome, Northland, New Zealand
    Binance is a leading global blockchain ecosystem behind the world’s largest cryptocurrency exchange by trading volume and registered users. We are trusted by over 280 million people in 100+ countrie...Show moreLast updated: 2 days ago
    • Promoted
    Division CFO, Trilogy (Remote) - $400,000 / year USD

    Division CFO, Trilogy (Remote) - $400,000 / year USD

    TrilogyWorkFromHome, Northland, New Zealand
    Division CFO, Trilogy (Remote) - $400,000 / year USD.Division CFO, Trilogy (Remote) - $400,000 / year USD.This range is provided by Trilogy. Your actual pay will be based on your skills and experience —...Show moreLast updated: 7 days ago
    • Promoted
    Product Marketing Director - AI

    Product Marketing Director - AI

    EnvatoWorkFromHome, Northland, New Zealand
    Product Marketing Director - AI at Envato.Product Marketing Director - AI.At Envato, everything we do is about empowering creatives to thrive. We help creatives bring their ideas to life, supercharg...Show moreLast updated: 12 days ago
    • Promoted
    Lead Research Scientist - Generative AI

    Lead Research Scientist - Generative AI

    CanvaWorkFromHome, Northland, New Zealand
    Lead Research Scientist - Generative AI at Canva, Wellington, New Zealand.This role is remote-friendly within Australia or New Zealand. Join Canva to coach a team of world-class scientists, advance ...Show moreLast updated: 30+ days ago
    • Promoted
    Real Estate Agent (Northland)

    Real Estate Agent (Northland)

    REAP RecruitmentFar North, Northland, New Zealand
    Opportunities for Existing & New Real Estate Salespeople | Multiple Offices & Brands | Apply Today to See Them AllREAP Recruitment specialise in (and are the largest advertiser of) recruitment oppo...Show moreLast updated: 30+ days ago
    • Promoted
    Payroll Data Entry Administrator

    Payroll Data Entry Administrator

    Alpha Personnel Recruitment LtdWhangarei, Northland, New Zealand
    An exciting opportunity for an experienced data-entry administrator with payroll exposure has arisen to join a high-performing team in Whangārei to support the . This full-time contract ensures you'...Show moreLast updated: 30+ days ago
    • Promoted
    Senior AI Engineer

    Senior AI Engineer

    DatacomWorkFromHome, Northland, New Zealand
    Datacom works with organisations and communities across Australia and New Zealand to make a difference in people’s lives and help organisations use the power of tech to innovate and grow.As an AI E...Show moreLast updated: 24 days ago