Staff Data Scientist, AI Evaluations Platform
rbc · 6666 Rue St Urbain
Job description
About the role
RBC's AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of its core controls. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk and performance. You will design, build and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks and measurement frameworks, while mentoring other data scientists. The role involves frequent interaction with partners and employees across Canada and the world.
Key responsibilities
- Design and drive advanced model and agent evaluation methodologies, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation and measurement frameworks, and guide junior team members.
- Define evaluation science standards that turn model risk, responsible AI, product quality, safety and business expectations into measurable criteria and repeatable methods.
- Own the end-to-end lifecycle of evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse and improvement.
- Design scalable evaluation approaches for generative AI and agentic systems, covering task, workflow, trajectory and runtime levels.
- Partner with AI research, platform engineering, product, risk, governance and business teams to embed evaluations into build, release, certification, monitoring and recertification workflows.
- Establish human evaluation and review protocols with reliable labels, reviewer guidance, adjudication, quality controls and audit-ready evidence.
- Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage and explainability.
- Provide technical leadership, executive-ready communication and mentorship to junior data scientists.
Required profile
- 8+ years of experience in data science, applied machine learning, AI evaluation, ML quality or a related field, including technical leadership and mentorship.
- Strong experience designing evaluation frameworks for ML, generative AI or agentic AI systems, including metrics, datasets, benchmarks, rubrics and quality measurement.
- Proven ability to turn governance, model risk, responsible AI and business requirements into measurable controls and decision-ready evidence.
- Nice to have: experience with agentic AI, tool-calling workflows, runtime traces and trajectory scoring; financial services, regulated AI, model risk or audit-ready evidence experience; familiarity with MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines; publications, patents or open-source contributions; Ph.D. or Master's degree in a quantitative or technical field.
Required skills
- Data science, statistics, machine learning, experimentation and data curation
- Python and SQL
- LLM evaluation methods such as LLM-as-judge, deterministic scoring, hallucination and factuality assessment, safety evaluation and model quality benchmarking
- Communication and stakeholder management with senior leaders across research, engineering, product, governance, risk and business
What we offer
- Total rewards program with bonuses, flexible benefits, competitive compensation, commissions and stock where applicable
- Coaching and development opportunities, a world-class training program in financial services and a dynamic, collaborative team
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Canada.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 4 days ago
Expires 1 month from now
4 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
rbc
6666 Rue St Urbain
Related job offers
-
Technical Artist (Game Development)
People Can Fly Montréal -
Senior AI Programmer
People Can Fly Montréal -
Analyste en assurance qualité (Maintenance applicative) – Hybride
ISAAC Instruments Saint-Bruno-de-Montarville -
Quality Assurance Analyst (Application Maintenance) - Hybrid
ISAAC Instruments Saint-Bruno-de-Montarville -
Développeur(euse) intelligence artificielle (IA) – Hybride
ISAAC Instruments Saint-Bruno-de-Montarville