Staff Data Scientist, AI Evaluations Platform
rbc · 6666 Rue St Urbain
Description du poste
About the role
RBC's AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of its core controls. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk and performance. You will design, build and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks and measurement frameworks, while mentoring other data scientists. The role involves frequent interaction with partners and employees across Canada and the world.
Key responsibilities
- Design and drive advanced model and agent evaluation methodologies, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation and measurement frameworks, and guide junior team members.
- Define evaluation science standards that turn model risk, responsible AI, product quality, safety and business expectations into measurable criteria and repeatable methods.
- Own the end-to-end lifecycle of evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse and improvement.
- Design scalable evaluation approaches for generative AI and agentic systems, covering task, workflow, trajectory and runtime levels.
- Partner with AI research, platform engineering, product, risk, governance and business teams to embed evaluations into build, release, certification, monitoring and recertification workflows.
- Establish human evaluation and review protocols with reliable labels, reviewer guidance, adjudication, quality controls and audit-ready evidence.
- Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage and explainability.
- Provide technical leadership, executive-ready communication and mentorship to junior data scientists.
Required profile
- 8+ years of experience in data science, applied machine learning, AI evaluation, ML quality or a related field, including technical leadership and mentorship.
- Strong experience designing evaluation frameworks for ML, generative AI or agentic AI systems, including metrics, datasets, benchmarks, rubrics and quality measurement.
- Proven ability to turn governance, model risk, responsible AI and business requirements into measurable controls and decision-ready evidence.
- Nice to have: experience with agentic AI, tool-calling workflows, runtime traces and trajectory scoring; financial services, regulated AI, model risk or audit-ready evidence experience; familiarity with MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines; publications, patents or open-source contributions; Ph.D. or Master's degree in a quantitative or technical field.
Required skills
- Data science, statistics, machine learning, experimentation and data curation
- Python and SQL
- LLM evaluation methods such as LLM-as-judge, deterministic scoring, hallucination and factuality assessment, safety evaluation and model quality benchmarking
- Communication and stakeholder management with senior leaders across research, engineering, product, governance, risk and business
What we offer
- Total rewards program with bonuses, flexible benefits, competitive compensation, commissions and stock where applicable
- Coaching and development opportunities, a world-class training program in financial services and a dynamic, collaborative team
Questions fréquentes
Pourquoi signalez-vous cette offre ?
Aller plus loin
Salaires, guides et recherches au Canada.
Salaires par métier
Postulez en 30 secondes
Entrez votre email pour postuler. Un compte sera cree automatiquement.
En continuant, vous acceptez nos conditions d'utilisation.
Deja un compte ? Connexion
Une question sur cette offre ?
Posez-la ici : vous recevrez le récapitulatif de l'offre par e-mail, tout de suite.
Publie il y a 4 jours
Expire dans 1 mois
3 vues · 0 interesses
Boostez vos chances
Importez votre CV : nous vous proposons les offres qui matchent votre profil.
Analyse de votre CV en cours...
rbc
6666 Rue St Urbain
Offres similaires
-
Technical Artist (Game Development)
People Can Fly Montréal -
Senior AI Programmer
People Can Fly Montréal -
Analyste en assurance qualité (Maintenance applicative) – Hybride
ISAAC Instruments Saint-Bruno-de-Montarville -
Quality Assurance Analyst (Application Maintenance) - Hybrid
ISAAC Instruments Saint-Bruno-de-Montarville -
Développeur(euse) intelligence artificielle (IA) – Hybride
ISAAC Instruments Saint-Bruno-de-Montarville