Senior Data Engineer at Africa Health Research Institute (AHRI)

Overview of Duties:

Overview of Duties

• Design, develop, and maintain end-to-end technical aspects of all data pipelines required to support the research scientists and data managers
• Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
• Optimize data performance, scalability, and security
• Provide technical guidance and support to data analysts and research scientists.
• Design data integrations and data quality frameworks
• Work and collaborate with the rest of the IT department to help develop the strategy for long-term scientific Big Data platform architecture
• Document and effectively communicate data engineering processes and solutions.
• Make use of and help define the right cutting-edge technology, processes and tools needed to drive technology within our science and research data
• management departments.

Minimum Qualifications

• Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
• ndustry recognized IT related certification and technology qualification such as Databases and Data related certifications.
• This is a technical role so a strong focus needs to be on technical skills and experience

Minimum Experience

• 7+ years’ experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
• Strong experience with high performance computing environments including Unix, Docker, Kubernetes, Hadoop, Kafka, Nifi, or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics
• At least 5 years’ advanced experience and very strong proficiency in UNIX, Linux, Windows
• Knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.

Knowledge and Abilities

• Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc.
• Experience of Big Data technologies such as Hadoop, Spark and Hive
• Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
• Experience working with containerization is advantageous
• Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
• Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous
• Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous
• Strong Experience designing end-to-end data pipelines.
• Strong knowledge of data modeling, architecture, and governance principles
• Strong Linux Administration skills
• Programming skills in various languages advantageous
• Strong data security and compliancy experience
• Excellent communication, collaboration, and problem-solving skills
• Ability to work independently and as part of a cross-functional team
• Interest and enthusiasm for medical scientific research and its application