📢 Nouveau : recevez les offres du jour sur notre canal WhatsApp
Jobiglo

Aucun resultat.

Staff Data Scientist, AI Evaluations Platform

rbc · 6666 Rue St Urbain

Nouveau
Senior 🇬🇧 English
Python SQL Machine Learning LLM evaluation Data curation MLflow Langfuse LangSmith OpenTelemetry Grafana CI/CD

Description du poste

About the role

RBC's AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of its core controls. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk and performance. You will design, build and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks and measurement frameworks, while mentoring other data scientists. The role involves frequent interaction with partners and employees across Canada and the world.

Key responsibilities

  • Design and drive advanced model and agent evaluation methodologies, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation and measurement frameworks, and guide junior team members.
  • Define evaluation science standards that turn model risk, responsible AI, product quality, safety and business expectations into measurable criteria and repeatable methods.
  • Own the end-to-end lifecycle of evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse and improvement.
  • Design scalable evaluation approaches for generative AI and agentic systems, covering task, workflow, trajectory and runtime levels.
  • Partner with AI research, platform engineering, product, risk, governance and business teams to embed evaluations into build, release, certification, monitoring and recertification workflows.
  • Establish human evaluation and review protocols with reliable labels, reviewer guidance, adjudication, quality controls and audit-ready evidence.
  • Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage and explainability.
  • Provide technical leadership, executive-ready communication and mentorship to junior data scientists.

Required profile

  • 8+ years of experience in data science, applied machine learning, AI evaluation, ML quality or a related field, including technical leadership and mentorship.
  • Strong experience designing evaluation frameworks for ML, generative AI or agentic AI systems, including metrics, datasets, benchmarks, rubrics and quality measurement.
  • Proven ability to turn governance, model risk, responsible AI and business requirements into measurable controls and decision-ready evidence.
  • Nice to have: experience with agentic AI, tool-calling workflows, runtime traces and trajectory scoring; financial services, regulated AI, model risk or audit-ready evidence experience; familiarity with MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines; publications, patents or open-source contributions; Ph.D. or Master's degree in a quantitative or technical field.

Required skills

  • Data science, statistics, machine learning, experimentation and data curation
  • Python and SQL
  • LLM evaluation methods such as LLM-as-judge, deterministic scoring, hallucination and factuality assessment, safety evaluation and model quality benchmarking
  • Communication and stakeholder management with senior leaders across research, engineering, product, governance, risk and business

What we offer

  • Total rewards program with bonuses, flexible benefits, competitive compensation, commissions and stock where applicable
  • Coaching and development opportunities, a world-class training program in financial services and a dynamic, collaborative team

Questions fréquentes

Le salaire n'est pas communiqué publiquement par le recruteur. Vous pouvez postuler et négocier directement avec rbc.
Cliquez sur "Postuler maintenant" en haut de la page. Vous pouvez importer votre CV en 1 clic — Jobiglo extrait automatiquement vos informations et postule pour vous.
Source : ats:workday

Pourquoi signalez-vous cette offre ?

Merci pour votre signalement. Nous allons examiner cette offre.

Postulez en 30 secondes

Entrez votre email pour postuler. Un compte sera cree automatiquement.

Postuler maintenant →

En continuant, vous acceptez nos conditions d'utilisation.

Deja un compte ? Connexion

Une question sur cette offre ?

Posez-la ici : vous recevrez le récapitulatif de l'offre par e-mail, tout de suite.

Publie il y a 4 jours

Expire dans 1 mois

3 vues · 0 interesses

Boostez vos chances

Importez votre CV : nous vous proposons les offres qui matchent votre profil.

Analyse de votre CV en cours...

rbc

6666 Rue St Urbain