Director, Evaluations

About the role

LawZero is a non‑profit building safe‑by‑design AI systems. The Director of Evaluations will define world‑class evaluation practices, build and lead the Evaluations Team, and ensure independent assessment of the Scientist AI and frontier LLMs.

Key responsibilities

• Define the evaluations strategy and roadmap in coordination with research and product teams.
• Build and scale the Evaluations Team to 8–10 members across research, engineering, dataset and benchmark design, and red‑teaming.
• Design novel benchmarks and datasets to measure capabilities, safety, explainability, and adversarial robustness.
• Lead automated and manual red‑teaming programmes, both in‑house and with external partners.
• Develop internal tooling and infrastructure to run evaluations at scale.
• Own public communication of evaluation results through model cards, technical reports, publications and blog posts.

Required profile

• Advanced degree (MSc or higher) in machine learning, computer science or a related field.
• 10+ years of ML experience with at least 5 years in a leadership role building technical teams.
• Hands‑on expertise in large‑scale evaluation of LLMs or frontier ML systems.
• Strong written and verbal communication skills for technical and non‑technical audiences.

Required skills

• Design and execution of large‑scale LLM evaluations.
• Dataset and benchmark creation for safety‑relevant properties.
• Red‑team operations (automated and manual) against AI systems.
• Authoring model and system cards, technical reports, and peer‑reviewed publications.

What we offer

• Opportunity to contribute to a mission with major impact.
• Comprehensive health benefits.
• Minimum 20 days vacation per year.
• Employer retirement contribution of 4%.
• Flexible benefits for well‑being.
• Collaborative, inclusive work environment in Montreal’s Mile‑Ex district.