Data Engineer

Skills and Competencies

• 3-5 years of experience building scalable data applications using Apache Spark (Scala preferred; Python also acceptable) and hands-on work with Databricks in cloud environments (AWS preferred), combined with strong SQL skills for data transformation and querying

• Solid understanding of distributed computing concepts, including data partitioning and performance optimisation, along with experience handling modern data formats such as Parquet, Delta Lake, and JSON

• Knowledge of graph data structures, clustering techniques, similarity approaches, and familiarity with statistical methods such as TF-IDF and similarity metrics

• Proficiency in version control and build tools, including Git, Gradle, Maven, or SBT

• Strong awareness of code quality, maintainability, and testing practices, with hands-on experience using frameworks such as ScalaTest or JUnit, and contributing to unit and integration testing approaches

• Ability to leverage AI-assisted development tools to enhance productivity and support development tasks, supported by strong analytical and problem-solving skills

• Effective communication and collaboration skills within cross-functional teams, along with a willingness to learn new technologies and continuously develop engineering expertise

• Basic understanding of artificial intelligence concepts, with curiosity and enthusiasm for learning how AI tools can be used to improve processes and drive efficiency. Interest in exploring AI systems and a willingness to develop awareness of responsible AI practices, including risk management and ethical use

Education 

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field

• Equivalent practical experience will also be considered 

Responsibilities 

The Data Engineer will support the design, development, and optimization of data pipelines and data processing solutions within a scalable data engineering environment

• Support the design and development of scalable Spark-based data pipelines for processing large volumes of data, including implementing parsing, normalization, and standardization logic across datasets

• Build and maintain ETL workflows ingesting data from sources such as Delta Lake, Parquet, and S3, ensuring efficient and reliable data processing

• Contribute to data quality checks, validation processes, and incremental processing approaches to maintain data integrity

• Assist in developing graph-based analytics and entity resolution workflows, applying clustering, similarity approaches, and statistical methods

• Write and maintain unit and integration tests, supporting overall reliability, code quality, and adherence to engineering best practices

• Support monitoring and troubleshooting of data pipelines, while participating in code reviews and contributing to continuous improvement initiatives

• Collaborate effectively with cross-functional teams (Product, Data Science, Engineering), document data flows and technical solutions, and demonstrate strong analytical, problem-solving, communication skills, and willingness to learn new technologies

About the Team 

Join our Data Engineering team, responsible for building scalable data pipelines across Change Data Capture, Entity Resolution, and Data Mastering. You will work on large-scale data challenges in a collaborative environment, contributing to high-quality data solutions and developing your technical capabilities. The team is embracing a more AI-enabled way of working, with a focus on automation, insight generation, and continuous innovation