Technology Analyst – PySpark Developer

About the role

Infosys is seeking an experienced Spark/Scala developer to design, develop, and optimise scalable big‑data solutions. The role focuses on building high‑performance batch and real‑time data pipelines using the Hadoop ecosystem and distributed computing frameworks such as Apache Spark.

Key responsibilities

• Design and implement scalable big‑data processing systems.
• Develop batch and streaming data pipelines with Spark, PySpark and Kafka.
• Integrate and manage data stored in Hadoop, Hive, Impala and NoSQL databases.
• Collaborate with data engineers, architects and business stakeholders to deliver robust solutions.
• Maintain and optimise ETL processes for large‑scale datasets.

Required profile

• Authorized to work in Canada without sponsorship.
• Bachelor’s degree or equivalent, or three years of progressive experience per year of education.
• At least 4 years of IT experience, including 4 years in big‑data technologies.
• Strong expertise in Apache Spark (Core, SQL, DataFrames, RDDs), Scala and PySpark.
• Hands‑on experience with Kafka, Hadoop ecosystem (HDFS, Hive, Impala) and NoSQL databases (HBase, MongoDB, Couchbase).
• Proficiency in SQL and data modelling, and experience building ETL pipelines.

Required skills

• Apache Spark
• Scala
• PySpark
• Kafka
• Hadoop (HDFS, Hive, Impala)
• NoSQL databases (HBase, MongoDB, Couchbase)
• SQL
• Data modelling
• ETL pipeline development