Apache Spark Engineer

Posted 01 August 2023
LocationAuckland
Job type Temporary
DisciplineData, BI & Analytics
Reference797792

Job description

ABOUT THE COMPANY:
This is a great opportunity to join a market-leading organisation in the Financial Services sector. They are going through a period of exciting change, and are working on many new initiatives with a big focus on IT and Digital. 

With offices based in Auckland CBD, they offer the flexibility to work from home or in their office when best suits the team and the projects that people are working on. 
  
ABOUT THE ROLE:
We’re looking for an Apache Spark Engineer to build the pipes from our CDM1, Ingestion Data Zone & IFRS17 data within Confluent Kafka onto MS Fabric where it will be used for analysis and reporting via Power BI
  • Design, develop, deploy and maintain robust data pipelines and ETL (Extract, Transform, Load) processes to transform data into actionable insights
  • Collaborate with stakeholders to understand reporting requirements and translate them into scalable pipelines, queries and Power BI reports
  • Identify and implement best practices for Spark application development, data ingestion, and data transformation
  • Implement data quality checks and ensure data accuracy, consistency, and integrity
  • Optimize Spark jobs for performance, scalability, and reliability.
  • Collaborate with DevOps teams to automate deployment and management of Spark notebooks and pipelines

WHO ARE YOU AND WHAT IS YOUR PROFILE:
  • Proven experience as an Apache Spark Engineer or similar role, working with large-scale Spark deployments.
  • Strong knowledge of Apache Spark architecture, including Spark Core, Spark SQL and Spark Structured Streaming
  • Proficiency in programming languages such as Pyspark or Scala.
  • Experience with building data insights using Power BI or equivalent tool.
  • Proven experience of DevOps and CI/CD to develop and deploy data pipelines
  • Knowledge of data ingestion and processing tools and technologies, such as Apache Kafka, Confluent Kafka, Databricks,
  • Strong communication and collaboration skills, with the ability to work effectively in cross-functional teams.
  • Experience with cloud platforms and services like Azure or AWS is a plus.
  • Insurance experience is desirable but is not essential.


WHAT'S NEXT?
Please apply online, or for more information contact Sarah Malcolm on smalcom@brightspark.io or call 0272150562