Data Engineer

Job description

About

Design, develop and maintain scalable data systems and data engineering solutions to meet business and technical requirements.

Build and maintain data pipelines to collect, process, transform and integrate data from multiple internal and external sources.

Analyse business and technical requirements and translate them into appropriate data architecture and engineering solutions.

Design and implement processes for extracting, transforming and loading data into databases, data warehouses and other data platforms.

Develop and maintain data models, database structures and data integration processes to support organisational requirements.

Integrate structured and unstructured data from multiple systems and ensure data is available in appropriate formats for analysis and reporting.

Develop automated processes for data ingestion, transformation, validation and processing.

Monitor data pipelines and data platforms to identify performance, reliability and data-quality issues.

Investigate and resolve technical problems relating to data processing, integration, storage and availability.

Implement appropriate data validation and quality controls to maintain the accuracy, consistency and reliability of organisational data.

Optimise data pipelines, database queries and processing workflows to improve system performance and efficiency.

Work with business analysts, data analysts, software developers and other technical professionals to understand data requirements and deliver appropriate solutions.

Assess existing data systems and recommend improvements to data architecture, integration methods and processing capabilities.

Support the implementation and integration of new data technologies and platforms in accordance with organisational requirements.

Develop and maintain technical documentation covering data architecture, data models, data pipelines, system configurations and integration processes.

Apply appropriate data security, access-control and data-management requirements when designing and maintaining data solutions.

Test data engineering solutions and validate data outputs to ensure they meet defined technical and business requirements.

Monitor and maintain data engineering solutions to ensure reliable and efficient operation of organisational data systems.