Technology Lead – PySpark Developer

About the role

Infosys is looking for an experienced Technology Lead specialized in PySpark to design, develop, and optimise scalable big‑data solutions. The role focuses on building high‑performance batch and real‑time data pipelines within the Hadoop ecosystem and Spark framework.

Key responsibilities

• Design and implement robust ETL and streaming pipelines for large‑scale datasets.
• Collaborate with data engineers, architects, and business stakeholders to deliver end‑to‑end data solutions.
• Optimise Spark jobs for performance, reliability and cost efficiency.
• Maintain and enhance data processing workflows using Hadoop, Kafka and NoSQL stores.
• Ensure data quality, security and compliance throughout the pipeline lifecycle.

Required profile

• Authorized to work in Canada without employer‑based visa sponsorship.
• Bachelor’s degree or equivalent; or three years of relevant experience in lieu of each year of education.
• Minimum 4 years of IT experience and 4 years in big‑data technologies.
• Strong expertise in Apache Spark, Scala and PySpark.
• Hands‑on experience with Kafka, Hadoop ecosystem (HDFS, Hive, Impala) and NoSQL databases (HBase, MongoDB, Couchbase).

Required skills

• Apache Spark (Core, SQL, DataFrames, RDDs)
• Scala programming
• PySpark
• Kafka (real‑time streaming)
• Hadoop ecosystem – HDFS, Hive, Impala
• NoSQL databases – HBase, MongoDB, Couchbase
• SQL and data modelling