Data Engineer at ADvTECH

Job Description

The data engineer is responsible for ingesting and transforming a wide range of data across the business into formats that can be used by end-users to develop differentiating business insights.  By structuring  data requirements into logical data constructs, the Data Engineer enables optimal, physical implementation of the data structure by generating data models. A key deliverable is the development of visual reports, dashboards and KPI scorecards.

The role will involve collaborating with internal and external resources to ensure the data analytics team have the required data from the source systems as well as be responsible for the accuracy of data loaded into the Data Warehouse

Data analysis / data modelling/  report building

• Creates data model designs in accordance with recommended best practice
• Collaborates with team members and business to translate business requirements into data requirements
• Analyzes and profiles the source data to understand data quality issues, relationships, patterns, and rules in the data
• Structures data requirements into logical data constructs based on recommended best practice including ERD models and dimensional models to ensure optimal implementation
• Compiles Source to Target Mapping Specifications with the Business Intelligence Business Analyst (BIBA)
• Ensures complete and correct data by identifying definitive or authoritative sources of data; analyzing source data; and identifying gaps
• Enables physical implementation of the data structure by generating the physical data model from the logical data model
• Cleans and maintains databases by restructuring and deleting old data/stored procedures
• Develops visual reports, dashboards and KPI scorecards using Power BI desktop.
• Connects to data sources, importing and transforming data for Business Intelligence
• Implements row level security on data, demonstrating an understanding of application security layer
• Writes DAX queries in Power BI.
• Develops Power BI reports as per the business requirements.

ETL and pipeline development

• Assists in designing and implementing scalable and robust processes for ingesting and transforming complex datasets.
• Develops the work packages as per the design to deliver elements of the data warehouse architecture (data ingestion, ETL, etc.,)
• Provides technical support, troubleshooting and upgrades setup or support on ETL and data warehouse related issues
• Analyzes and maps data from source systems to target data warehouse
• Participates in all aspects of quality assurance
• Monitors and maintains ETL pipelines and data warehouse
• Develops the data dictionary

Task tracking and Support Resolution

• Makes use of DevOps to track tasks, effort estimates, effort completed and repos
• Responds timeously to support calls and tracks issues to completion

Continuous improvement

• Owns the development of elements of centralised, automated data improvement initiatives under the guidance of the Team Lead
• Makes increased contributions by broadening and enhancing individual skills
• Keeps abreast of industry trends and best practice and makes recommendations to management
• Investigates new technologies to improve and optimize standard reporting.

Education (Formal Qualification required)

• Bachelor’s degree in (Information Systems, Informatics, Computer Systems or Information Technology) – NQF 7  Or National Diploma / Diploma in Software Development /IT- NQF 6
• Understanding of data warehousing principles (e.g. Kimball ) is recommended SQL/ Data engineering certification
• Azure Data Certification
• Diploma in IT or related field