LogoFresh Jobs Ahead
RBC
RBC
H1B Visa SponsorBankingFinanceInsurance
RBC is a global financial services institution headquartered in Canada, with businesses spanning banking, finance, insurance, investing, wealth management, financial planning, and capital markets. Serving more than 17 million clients across Canada, the United States, and 27 other countries, RBC operates with a workforce of more than 94,000 employees. Its diversified model combines established financial services with a focus on innovation and client experience.
LinkedIn
Company Jobs

Staff Data Scientist, AI Evaluation Platform

Lead the design of evaluation frameworks, datasets, and quality controls for RBC’s enterprise AI systems. Set technical standards, support governance evidence, and mentor data science teams across Canada.

CACanadaOn-siteFull-timeLeadEnglish RequiredData Scientist
GrafanaPythonSQL

Description

  • Develop advanced evaluation methods for models and agents using datasets, rubrics, LLM-as-judge techniques, deterministic scoring, human review, and measurement frameworks.
  • Establish evaluation science standards that convert model risk, responsible AI, product quality, safety, and business needs into measurable criteria and repeatable processes.
  • Manage the full lifecycle of evaluation datasets and scorecards, from sourcing and curation through validation, quality assurance, versioning, lineage, reuse, and continuous improvement.
  • Create scalable evaluation strategies for generative and agentic AI across task, workflow, trajectory, and runtime levels.
  • Collaborate with AI research, platform engineering, product, risk, governance, and business teams to integrate evaluations into development, release, certification, monitoring, and recertification processes.
  • Build human evaluation and review protocols that deliver dependable labels, reviewer instructions, adjudication procedures, quality controls, and audit-ready evidence.
  • Assess and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation methods.
  • Provide technical direction, communicate effectively with executives, and mentor junior data scientists.

Requirements

  • Bring at least eight years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical discipline, including technical leadership and team mentorship.
  • Demonstrate strong experience creating evaluation frameworks for machine learning, generative AI, or agentic AI systems, covering metrics, datasets, benchmarks, rubrics, and quality measurement.
  • Have practical knowledge of LLM evaluation approaches such as LLM-as-judge, deterministic scoring, human review, hallucination and factuality assessment, safety evaluation, and model-quality benchmarking.
  • Possess a strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and current AI/ML development practices.
  • Translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and evidence suitable for decision-making.
  • Communicate clearly and manage stakeholders effectively, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business functions.
  • Bring experience assessing agentic AI, tool-calling workflows, multi-step reasoning, runtime traces, trajectory scoring, or workflow-level performance.
  • Have experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit-ready evidence processes.
  • Be familiar with MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tools.
  • Publications, patents, open-source contributions, or industry experience in AI evaluation, ML quality, AI safety, applied research, or responsible AI are relevant.

Benefits

  • Receive a Total Rewards Program with bonuses, flexible benefits, competitive compensation, commissions, and applicable stock awards.
  • Develop your career with leaders who provide coaching and management opportunities.
  • Contribute to work designed to create meaningful, lasting impact.
  • Join a collaborative, progressive, dynamic, and high-performing team.
  • Access a comprehensive training program focused on financial services.
  • Take on challenging work with opportunities for professional growth.

Related Jobs