Directeur.rice, Évaluations

About the role

LoiZéro, a nonprofit dedicated to designing safe‑by‑design AI systems, is seeking a Director of Evaluations to build and lead its Evaluation team. This founding position will define world‑class evaluation practices for the organization’s flagship Scientist AI and ensure independent verification of model capabilities and safety.

Key responsibilities

• Define the evaluation strategy and roadmap, prioritising metrics in close collaboration with research and product teams.
• Recruit and grow an Evaluation team of 8‑10 specialists in research, engineering, dataset design, benchmarking and red‑teaming within the first 3‑6 months.
• Maintain independence from core research and product streams to avoid conflicts of interest.
• Design novel benchmarks that enable fair, apples‑to‑apples comparison between Scientist AI and frontier LLMs.
• Oversee creation of new datasets, tasks and virtual environments to assess performance, safety, causality, and resistance to adversarial attacks.
• Lead evaluation of Scientist AI when deployed as a guardrail, ensuring compliance with harm‑prevention specifications, explainability, and robustness against jailbreaks, prompt injection and data poisoning.
• Establish and manage automated and manual red‑team programmes, both internal and with external partners.
• Drive development of internal tools and infrastructure for large‑scale evaluation.

Required profile

• Proven experience leading evaluation or safety teams in AI/ML contexts.
• Strong track record of designing benchmarks, datasets and red‑team exercises.
• Ability to operate independently from product and research agendas.
• Excellent strategic thinking and team‑building skills.

Required skills

What we offer

• Founding leadership role in a mission‑driven nonprofit.
• Opportunity to shape industry‑leading AI safety standards.
• Collaborative environment with top AI researchers.