Senior Software Engineer | AI Evaluation

Who we’re looking for

• Strong software engineering experience: 8+ years in any modern programming language, with a willingness and ability to work primarily in Python in our projects;
• Hands-on experience using coding agents such as Claude Code and Codex as part of practical engineering workflows, while maintaining ownership of architecture, correctness, code quality, and delivery;
• Problem-solving and clarity of thinking;
• Ability to communicate technical decisions and reasoning;
• Practical understanding of Docker and development workflows;
• Overall readiness to contribute in a technical, execution-focused role.

What you’ll do

• You create tasks and the criteria to evaluate them. Each task consists of: An isolated environment - an emulation of a developer's workstation: a Linux machine with dev tools (terminal, CLI), MCP servers (repository, task tracker, messenger, documentation, etc.), and a real web application codebase; A task - a prompt, just like in real work (a ticket link, a messenger conversation, anything). The agent figures it out on its own - reads the ticket, checks discussions, searches docs, explores the code. All of that context is also part of the task and needs to be created and thought through; Tests and evaluation criteria - how to verify the agent solved the task correctly.
• Building the virtual company - following a high-level plan, you create a company: codebase, infrastructure, context (conversations, documentation, tickets). The result is a realistic environment with development history.
• Assembling and calibrating tasks - from intermediate states of the company, task drafts are assembled (based on a taxonomy). You turn a draft into a full task: write the prompt, define evaluation criteria, ensure the task is solvable and the evaluation is fair.
• Then an AI agent receives the task and tries to solve it the way a developer would - and you evaluate the result.
• Iterating with an AI agent on tests - they should catch real problems, not miss bad solutions, not break on good ones;
• Reviewing code written by agents;
• Analyzing why an agent failed or succeeded;
• Designing edge cases and adversarial scenarios.

About the position

At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages . We are a well-funded startup with an enviable portfolio of clients including Anthropic , Amazon , Microsoft , Poolside , Recraft , and Shopify . Recently, we secured strategic investment led by Bezos Expeditions and Nebius Group with participation from Mikhail Parakhin , CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA , UK , the Netherlands , Serbia , and more.

Benefits

• Freelance collaboration, with a path into a project team if things go well;
• Flexible, fully remote schedule;
• Work at the leading edge of AI development, alongside a dedicated and dynamic team of experts;
• Projects with customers that are AI industry leaders and well-known household names;
• Friendly community.