O pozici
About Toloka
At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers—and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages . We are a well-funded startup with an enviable portfolio of clients including Anthropic , Amazon , Microsoft , Poolside , Recraft , and Shopify .
Recently, we secured strategic investment led by Bezos Expeditions and Nebius Group with participation from Mikhail Parakhin , CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA , UK , the Netherlands , Czech Republic , Serbia , and more.
About the Position
Led applied ML initiatives within the Delivery division, integrating LLM and AI-agent technologies into active client engagements with a focus on Anti-Fraud.
Optimize output quality and automate manual workflows to scale operational capacity and improve business margins.
Co budeš dělat
- Solution architecture: Design and deploy agentic workflows (tool use, planning, retrieval, critique loops) for data generation, evaluation, anti-fraud, and safety use cases.
- Automated evaluation: Build judge models, auto-grading systems, and self-verification pipelines to codify acceptance criteria into repeatable checks.
- Productization: Create reusable reference architectures, templates, and components to scale delivery across multiple client accounts.
- AI-driven scaling: Integrate LLMs into contributor workflows (pre-label → verify → escalate) to eliminate operational toil and maximize expert leverage.
- Observability: Implement production measurement frameworks (task KPIs, drift), online A/B tests, and cost/latency dashboards.
- Customer collaboration: Partner with Delivery Leads, PMs, and client teams to translate business requirements into pragmatic ML designs.
- Technical stewardship: Set engineering standards for prompts, agents, data pipelines, and CI/CD via active code and design reviews.
- Team building: Hire, coach, and mentor applied ML and agent engineers to maintain a high-bar, delivery-focused culture.
Koho hledáme
- Production experience: Over 5–8+ years of expertise in applied ML and LLM development, with a documented history of launching agentic workflows into production.
- Domain expertise: Strong background in building anti-fraud platforms, trust & safety systems, or real-time anomaly detection frameworks.
- Delivery focus: A pragmatic approach, capable of transforming vague project needs into executable solutions defined by strict timelines and measurable KPIs.
- AI quality assurance: Extensive experience in building automated evaluation pipelines and grading systems to maintain consistency at scale.
- Automation mindset: A natural drive for automation, with a track record of identifying and optimizing manual operations to enhance throughput and margins.
- System architecture: Skill in pragmatic engineering, prioritizing simple, observable, and efficient designs that align with latency and cost goals.
- Hands-on engineering: A technical profile featuring advanced Python proficiency and the ability to tune agents and prompts within live systems personally.
Benefity
- You will be part of an international, dynamic environment that drives innovation and sets new standards in the AI and technology sector.
- Competitive compensation package including base salary, bonus, and ESOP
- Paid PTO and benefits will vary depending on location
- We offer a full remote or hybrid model (if you are based in NL or Serbia)
- IT setup and home office allowances.
