AI Evaluation Engineer

About the role

Zafin is seeking an AI Evaluation Engineer to ensure AI agent solutions are accurate, reliable, safe, and production‑ready within regulated banking environments. The role provides objective, evidence‑based evaluation of AI behaviour against business, quality, risk, performance, and regulatory criteria.

Key responsibilities

• Design and execute structured evaluation scenarios that validate AI agent accuracy, reliability, safety, compliance, and business outcomes.
• Validate AI agent behaviour against approved business rules, policies, technical requirements, source knowledge, and expected outcomes.
• Conduct regression evaluation across releases, monitor behavioural drift, performance degradation, and newly introduced failure modes.
• Identify, document, prioritize, and track quality issues, defects, and production risks.
• Maintain reusable evaluation scenarios, benchmark datasets, expected outcomes, and regression suites.

Required profile

• 3–10+ years of experience in software quality engineering, AI evaluation, machine learning evaluation, or related disciplines.
• Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or equivalent practical experience.
• Strong understanding of large language model behaviour, hallucination detection, safety, instruction adherence, factual accuracy, and business correctness.

Required skills

• Python
• SQL
• Experience with AI evaluation platforms such as LangSmith or OpenAI Evals

What we offer

• Fully company‑paid health benefits
• Annual bonus opportunities
• Generous vacation and holidays
• RRSP matching
• Employer‑paid insurance coverage