Senior Software Engineer, ML Evaluations

About the role

Artificial Agency, a player-facing agentic AI company for games, is looking for a Senior Software Engineer to own the platform its researchers and engineers use to run experiments and evaluations at scale. Reporting to the Head of Platform Engineering, the role is hybrid in Edmonton, Alberta, with remote possible for exceptional candidates. It combines distributed-systems design, cloud infrastructure and hands-on work with ML engineers.

Key responsibilities

• Own the ML evaluation platform end to end: the Python CLI and library, queueing service and API, database schema and the researcher web UI.
• Set technical direction and design the job execution layer: queueing, worker leasing and recovery, concurrency limits, cancellation, timeouts, retries, artifacts and logs.
• Make evaluation results traceable and trustworthy by recording test versions, game builds, models, agent configurations and run settings, and preserving execution history.
• Extend the data model, APIs and UI to support baseline comparisons, regression investigation and access to logs and artifacts, and implement evaluation and reporting methods such as repeated trials and result aggregation.
• Own the machines running the jobs: an auto-scaling GPU Windows fleet in AWS, containerized Linux workers on Kubernetes and a few in-office machines, covering Terraform, machine images, GPU drivers, bootstrap scripts, secrets handling and cost/capacity decisions.
• Partner with the Senior Test Engineer on interfaces for workload configuration, execution requirements, structured results and diagnostic artifacts, and integrate evaluations into developer tools and CI workflows.
• Work with ML, agents, game engineering and research teams to diagnose failed runs and turn one-off requests into features, prioritizing across teams and keeping runbooks and documentation up to date.

Required profile

• Bachelor's or advanced degree in Computer Science, Software Engineering or a related field, or equivalent practical experience.
• 5+ years of professional backend development, with a strong focus on Python.
• An AI-first approach to engineering, with hands-on experience using AI tools for development, testing, debugging or analysis.

Required skills

• Distributed job or workflow systems: queueing, scheduling, worker coordination, concurrency control, cancellation and failure recovery.
• Relational database design (e.g. PostgreSQL) and schema migration.
• AWS (EC2, IAM, Secrets Manager, S3), Windows administration and PowerShell scripting.
• Designing CLIs, libraries or internal tools; GitHub workflows and CI/CD; Docker and Kubernetes.
• Experience with ML experiment tracking or evaluation harnesses, and with automating game engines (Unity, Unreal) in batch mode, is a plus.