Senior Data Engineer

About us

Fuelled by our passion for using data to solve customer problems, we’re working to be the top-tier data-driven organisation in the UK’s General Insurance market. With Analysts, Engineers and Developers working together in an agile environment, our Data Enabling Tribe is at the heart of our mission. Building a new, enterprise-scale, cloud-hosted, secure and trusted analytics platform, we’re using the economics of big data, cloud elasticity, machine learning and AI automation. And we’re turning information into business insights to provide even greater customer support.

The Data Chapter

Our team is at the heart of this mission, and we are looking for people passionate about using data to solve customer problems and making the organisation data driven. We are working with our teams to build our new enterprise-scale, cloud hosted, secure and trusted analytics platform. The platform uses the economics of big data, cloud elasticity, machine learning (ML)/Artificial Intelligence (AI) automation and to turn information into business insights so that we can support our customers better.

Who you’ll be working with

You will be working closely with data engineers, analysts, and other stakeholders to understand their data requirements and provide the necessary data infrastructure and support. You will be reporting to the Data Chapter Lead.

What you’ll need:

• Proven experience as a Senior Data Engineer with expertise in Python, PySpark, AWS Glue.
• Extensive experience in designing ETL (Extract, Transform, Load) processes for data ingestion and integration.
• Proficiency in Python and Spark for data processing & optimization techniques.
• Hands-on experience in dbt models
• Hands-on experience in orchestrations tools such as Airflow
• Strong experience with AWS services, including but not limited to S3, Glue, Lambda, EC2, and IAM.
• Hands-on experience with CI/CD tools and practices, such as Jenkins & Git
• Experience with data warehousing solutions (e.g., Redshift, Snowflake) and data lake architecture.
• Strong problem-solving skills, attention to detail, and the ability to work in a fast-paced, collaborative environment.
• Excellent communication and teamwork skills, with the ability to explain complex technical concepts to non-technical stakeholders.

What you'll be doing:

As a PySpark & AWS Glue Data Engineer at DLG you will be responsible for the below:

Data Pipeline Development:

Design, develop, and maintain robust data pipelines using Python/ PySpark and AWS Glue to extract, transform, and load data from various sources into our data lake or warehouse.Create and maintain dbt models and transformations to ensure data consistency, accuracy, and integrity.

Orchestration:

Create, configure, and maintain Apache Airflow workflows to automate data pipelines and other operational processes.

Define dependencies and schedules for tasks within workflows to ensure smooth execution.

Data Quality Assurance:

Implement data quality checks and validation processes to maintain data integrity, accuracy, and consistency.

AWS Infrastructure Management:

Manage and maintain AWS infrastructure resources, including EC2 instances, Lambda functions, Glue jobs, and data storage solutions.

Continuous Integration and Deployment (CI/CD):

Implement and maintain CI/CD pipelines to automate and streamline the deployment of data pipelines, code changes, and infrastructure updates.

Monitoring and Optimisation:

Monitor the performance of data pipelines using Spark UI and otherwise, AWS resources, and data storage, and proactively optimise for efficiency and scalability.

Performance Tuning:

Continuously optimize data pipelines, SQL queries, and dbt models to improve data processing efficiency.

Security and Compliance:

Implement security best practices and ensure compliance with data protection regulations, working with the security and compliance teams.

Documentation:

Maintain detailed documentation for data pipelines, infrastructure,   

configurations, and DevOps processes.

What we’ll give you in return:

We wouldn’t be where we are today without our people and the wide variety of perspectives and life experiences they bring. That’s why we offer excellent benefits to suit your lifestyle and a flexible working model combining the best parts of home and office-working, varying with the nature of your role. Core benefits include:

• 9% employer contributed pension
• 50% off home, motor and pet insurance plus free travel insurance and Green Flag breakdown cover
• Up to 10% annual bonus
• 25 days holiday (rising by 1 each year to 28) + bank holidays and option to buy or sell up to 5 days
• Additional optional Health and Dental insurance
• EV car scheme allows all colleagues to lease a brand new electric or plug-in hybrid car in a tax efficient way.
• Buy as you earn share scheme
• Employee discounts and cashback
• Plus many more

Ways of Working

Our mixed model way of working offers a 'best of both worlds' approach combining the best parts of home and office-working, offering flexibility for everyone. How much you'll be in the office depends on your role, and we'll consider the flexible working options that work best for you.

Read our flexible working approach here.

We recognise and embrace people that work in different ways so if you need any reasonable adjustments within this recruitment process, please reach out to us and we can discuss how we can support you with this.

There’s no-one else like you. No-one with the exact same mix of strengths, quirks, skills and thoughts. That’s why you could belong here. As part of a team of brilliant individuals, in a place that empowers you to be the best you can be. We’re proud of who we are, of what we do, and what every single one of us brings. Join us. Help us keep innovating and putting customers at the heart of everything. To be an insurance company of the future. When we work together, we can all achieve great things. Inspiring, challenging, and supporting each other to aim higher.

Together we’re one of a kind.

#LI-Hybrid

#LI-CG1