Senior Data Engineer - Umhlanga at Ignition Group

Description

The Senior Data Engineer is accountable for creating and maintaining the technological infrastructure of a data platform and implementing methods to improve data reliability and quality. This position is responsible for ensuring the consistent delivery of data architecture throughout ongoing projects by creating data models, building data pipelines and coordinating ETL (extract, transform, load)

Responsibilities :

Technical Development and Programming

• Developing technical tools and programming that leverage artificial intelligence.
• Building and maintaining the infrastructure that collects and processes large amounts of data.
• Cleaning and developing data pulling methods used in data models.
• Propose solutions and strategies to business challenges.
• Optimize query performance in Snowflake.
• Collaborate with engineering and product development teams

Data Pipeline Development and Optimization

• Stay abreast of new technologies, frameworks, and market trends in the Big Data space.
• Continuously upskill to remain proficient in the latest advancements in data engineering.
• Work with Snowflake and is proficient in loading and unloading data from Snowflake and building data ingestion pipelines.
• Identify valuable data sources and automate collection processes.

Data Management and Administration

• Loading data from a wide variety of data sources using SQL technologies.
• Working with internal stakeholders by assisting with data-related technical issues and supporting data infrastructure needs.
• Administering and optimizing data storage systems and databases to ensure performance and scalability.
• Performing data cleansing and transformation processes to prepare raw data for analysis.
• Design, build, and develop data warehouses.
• Enabling data integration and migration across various systems, sources, databases, and servers.
• Developing and optimizing ETL (extract, transform, load) processes to support analytics and reporting needs.
• Perform data modeling (Relational and Star Schema).

Documentation and Process Improvement

• Providing support in the identification, design, and implementation of internal process improvements.
• Manage real-time analytics and batch processing.
• Maintaining clear and current documentation of data pipelines, processes, and data dictionaries.

Data Retention Policy Definition

• Define and implement data retention policies in collaboration with stakeholders.
• Ensure compliance with data governance and regulatory requirements.

Optimization through Automation

• Identify opportunities to optimize existing data processes.
• Implement automation solutions to streamline data workflows and increase efficiency.

Data Reporting and Documentation

• Assist in creating comprehensive data-related reports and documentation.
• Ensure documentation is accurate, up-to-date, and easily accessible for the team.

Analytics Data Source Maintenance and Optimization

• Maintain and optimize data sources used for analytics purposes.
• Conduct regular reviews to ensure data sources align with business needs.

Data Reporting

• Generate regular data reports as per business requirements.
• Customize and automate reporting processes to enhance efficiency.

Supporting Data-Related Queries:

• Assist the team in resolving data-related support queries.
• Provide expertise and guidance on data-related issues as needed.

Requirements

Knowledge, skills and attributes:

• Good attention to detail.
• Analytical mind with an inclination for problem-solving.
• Excellent verbal and written communication skills.
• Ability to work in a team-orientated, collaborative environment.
• Planning and organising skills with the ability to prioritise tasks and adopt a logical approach to work.
• Process orientated with comprehensive documentation skills.
• Working and conceptual knowledge of building and maintain physical and logical data models.
• Proficient in SQL, Python and ETL tools.
• Understanding of database design principle.
• Working knowledge of data warehousing concepts and practices.
• Ability to create and maintain good working relationships with colleagues and customers.
• A willingness to keep up to date with developments in new technology.
• An understanding of information legislation, such as the Data Protection Act
• Ability to manipulate, process and extract value from large, disconnected datasets.
• Conversant with data modelling and data integration techniques.
• Knowledge of data quality processes and master data management.
• Proficiency in data integration and ETL solutions using SSIS, and/or SQL Server stored procedures.
• Strong analytical mind and business acumen.
• Proficiency in database query languages such as T-SQL,  PL/SQL.
• Expertise in data mining, large-scale data modeling, and business requirements gathering/analysis.

Education and training:

• Diploma in Computer Science, Information Technology, Engineering, or equivalent.
• Bachelor’s degree in computer science, Information Technology, Engineering, or equivalent advantageous.
• Relevant certification (Snowflake Certified Data Engineer, Certificate Professional, etc).

Experience:

• At least 5 years’ working as a data engineering or a similar role.
• Experience with DBMS such as Snowflake, SQL Server, Mongo DB.
• Experience with architecture and design of secure, highly available and scalable systems.
• Experience with writing scripts using scripting language.
• Experience working with data sets and ordering data through various functions.
• Experience working with internet technologies.
• Experience dealing with internal and external stakeholders.
• Experience working with and/or on Power BI.
• Software development experience with Python required.
• Experience with SQL Server Management Studio and Visual Studio.
• Experience with Azure SQL databases, Azure Storage Accounts, and/or Big Query.
• Experience in data integration and transformation.