Director, Evaluations
LawZero · Montreal
Job description
About the role
LawZero is a non‑profit building safe‑by‑design AI systems. The Director of Evaluations will define world‑class evaluation practices, build and lead the Evaluations Team, and ensure independent assessment of the Scientist AI and frontier LLMs.
Key responsibilities
- Define the evaluations strategy and roadmap in coordination with research and product teams.
- Build and scale the Evaluations Team to 8–10 members across research, engineering, dataset and benchmark design, and red‑teaming.
- Design novel benchmarks and datasets to measure capabilities, safety, explainability, and adversarial robustness.
- Lead automated and manual red‑teaming programmes, both in‑house and with external partners.
- Develop internal tooling and infrastructure to run evaluations at scale.
- Own public communication of evaluation results through model cards, technical reports, publications and blog posts.
Required profile
- Advanced degree (MSc or higher) in machine learning, computer science or a related field.
- 10+ years of ML experience with at least 5 years in a leadership role building technical teams.
- Hands‑on expertise in large‑scale evaluation of LLMs or frontier ML systems.
- Strong written and verbal communication skills for technical and non‑technical audiences.
Required skills
- Design and execution of large‑scale LLM evaluations.
- Dataset and benchmark creation for safety‑relevant properties.
- Red‑team operations (automated and manual) against AI systems.
- Authoring model and system cards, technical reports, and peer‑reviewed publications.
What we offer
- Opportunity to contribute to a mission with major impact.
- Comprehensive health benefits.
- Minimum 20 days vacation per year.
- Employer retirement contribution of 4%.
- Flexible benefits for well‑being.
- Collaborative, inclusive work environment in Montreal’s Mile‑Ex district.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Canada.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 1 month ago
Expires 1 week from now
36 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
LawZero
Montreal
Related job offers
-
Programmeur d'outils seniors, Unreal Editor
Epic Games (French) Montreal -
Programmeur senior, apprentissage automatique
Epic Games (French) Montreal -
Développeur sénior, outils de jouabilité, UI et IA
Epic Games (French) Montreal -
Principal Engineer, Identity & Access Management
Questrade Financial Group -
Technical Support Specialist - Telecom
servicetitan Canada Manitoba Remote