Software Engineer – Data & AI

About the role

Scotiabank’s Data & AI Technology team is looking for a Software Engineer to design, build, and scale an enterprise data catalog and related data‑management services. You will work with the Databricks ecosystem to enable data discovery, governance, and secure access across the organization.

Key responsibilities

• Design and develop scalable components of an enterprise data catalog, automating data‑governance workflows.
• Integrate the catalog with Databricks, Azure/GCP services, and other data platforms.
• Build reusable frameworks for data quality, profiling, and classification.
• Lead technical design discussions, make architectural decisions, and evaluate trade‑offs between correctness, performance, and customer impact.
• Identify technical risks and propose long‑term, scalable solutions.
• Mentor junior engineers and champion best engineering practices.
• Collaborate with product managers and stakeholders to align data infrastructure with product goals.

Required profile

• Bachelor’s or master’s degree in Computer Science, Engineering or a related field.
• Strong problem‑solving mindset with a focus on quality and detail.
• Experience guiding teams and communicating technical concepts to non‑technical audiences.

Required skills

• Python and/or Java programming.
• Databricks, Unity Catalog, Delta Lake, Apache Spark.
• Enterprise data‑catalog tools (e.g., Atlan, Alation, Collibra, DataHub, OpenMetadata).
• Data‑quality, profiling and classification frameworks.
• Data security, access‑control models, and data‑governance concepts.
• Cloud platforms (Azure, GCP) and IaC tools (Terraform).
• RESTful APIs, microservice architecture, distributed system design.
• Streaming technologies such as Kafka and Structured Streaming.