Data Engineer

As a Data Engineer, you will be responsible for building and maintaining the data integrations and processing pipelines that fuel Multiverse’s analytics and data science capabilities. You will consolidate, organise and optimise data flows from/to multiple systems that serve every part of the business.

The role will be within the Data & Insight (D&I) team, reporting to the Director of Data Engineering. Within D&I, you will work closely with Analytics Engineers and our DevOps team, as well as Data Product Managers, Analytics Developers and Data Scientists. You will also need to collaborate with our Product & Engineering teams – as well as domain experts from across the business. You will need to be methodical, analytical, creative and collaborative, with very high attention to detail.

What you’ll focus on:

Data Integrations

Developing and maintaining bidirectional data integrations with first and third-party systems:

ingesting and transforming data to enable our Analytics, Data Science and AI teams
exposing the outputs of their work back to all Multiverse systems
Integrations utilising a range of REST APIs, GraphQL APIs, databases and event-driven systems
Combining custom development (in Python) with re-use of third-party commercial or OSS technologies where appropriate

Cloud Infrastructure

Supporting and extending our AWS infrastructure, deployed using Terraform and Serverless, which includes:
A Data Lake, using S3, Glue and Athena

Processing pipelines, using Lambda, ECS, EventBridge, SQS, DynamoDB, Airflow (AWS MWAA)
A Data Warehouse, currently RDS (but likely to move to Redshift/Snowflake/BigQuery etc. in future)
We prefer managed services, serverless technologies and event-driven processing, where possible

Orchestration, Monitoring & Observability

Building scalable, efficient DAGs to manage our processing pipelines
Ensuring that we have best-in-class monitoring and observability tooling:
we know when there are issues before anybody else

we are able to diagnose and fix issues quickly
the tooling has high satisfaction scores from the D&I team