AI Agent Evaluation Analyst - AI Trainer

Overview

AI Agent Evaluation Analyst - AI Trainer at Mindrift. This is a flexible, project-based opportunity suitable for remote, async work. The Mindrift platform connects domain experts with AI projects powered by Toloka. Our mission is to unlock the potential of GenAI by leveraging real-world expertise from across the globe.

We are looking for curious, intellectually proactive contributors who double-check assumptions and think critically. If you enjoy ambiguity and complexity and want to learn how modern AI systems are tested and evaluated, this role may be a fit.

About the project

We are seeking QA-like support for autonomous AI agents on a project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. You will balance quality assurance, research, and logical problem-solving to help ensure robust testing of systems.

You do not need a coding background, but you must be curious, rigorous, and able to evaluate the soundness and consistency of complex setups. If you have experience in consulting, case solving, systems thinking, or related activities, you might be a great fit.

Responsibilities

• Review evaluation tasks and scenarios for logic, completeness, and realism
• Identify inconsistencies, missing assumptions, or unclear decision points
• Help define clear expected behaviours (gold standards) for AI agents
• Annotate cause–effect relationships, reasoning paths, and plausible alternatives
• Think through complex systems and policies to ensure agents are tested properly
• Collaborate with QA, writers, or developers to suggest refinements or edge-case coverage

What you’ll be doing

Requirements

What we value

Benefits

How to get started

Apply to this post, qualify, and contribute to a project aligned with your skills on your own schedule. Shape the future of AI while building tools that benefit everyone.

#J-18808-Ljbffr