The Data Engineer is responsible for the maintenance, improvement, cleaning, and manipulation of data in the business’s operational and analytics databases. The Data Engineer works with the business’s software engineers, data analytics teams, data scientists, and data warehouse engineers in order to understand and aid in the implementation of database requirements, analyze performance, and troubleshoot any existent issues.
The Data Engineer has to be an expert in SQL development further providing support to the Data and Analytics in database design, data flow and analysis activities. The position of the Data Engineer also plays a key role in the development and deployment of innovative big data platforms for advanced analytics and data processing.
The Data Engineer defines and builds the data pipelines that will enable faster, better, data-informed decision-making within the business.
Objectives and Responsibilities of the Data Engineer
Management: The Data Engineer manages his position and junior data engineering support personnel position by creating databases optimized for performance, implementing schema changes, and maintaining data architecture standards across all of the business’s databases.
In this position, the Data Engineer leads innovation through exploration, benchmarking, making recommendations, and implementing big data technologies for platforms. He is also tasked with the development and implementation of scripts for database maintenance, monitoring, performance tuning, and so forth.
The Data Engineer is additionally tasked with designing and developing scalable ETL packages from the business source systems and the development of ETL routines in order to populate databases from sources and also to create aggregates. It is also the role of the Data Engineer to oversee large-scale data Hadoop platforms and to support the fast-growing data within the business.
In this capacity, the Data Engineer is responsible for enabling and running data migrations across different databases and different servers, for example, data migration from SQL servers to MySQL. He defines and implements data stores based on system requirements and consumer requirements.
The Data Engineer is also responsible for performing thorough testing and validation in order to support the accuracy of data transformations and data verification used in machine learning models. The Data Engineer strives to ensure proper data governance and quality across the Data and Analytics department and the business as a whole.
Analytics: The Data Engineer plays an analytical role where he performs ad-hoc analyses of data stored in the business’s MySQL/MS SQL databases and writes SQL scripts, stored procedures, functions, and views. In this position, the Data Engineer troubleshoots data issues within the business and across the business and presents solutions to these issues. He proactively analyzes and evaluates the business’s databases in order to identify and recommend improvements and optimization.
He also prepares activity and progress reports regarding the business database status and health, which he presents to senior data engineers. In this position, the Data Engineer will additionally analyze complex data elements and systems, data flow, dependencies, and relationships in order to contribute to conceptual physical and logical data models.
Support/Collaborative Role: The Data Engineer plays a collaborative role where, in collaboration with senior data engineering management, he develops and implements scripts for database maintenance, monitoring, and performance tuning to be applied across the business. The Data Engineer plays a supporting role to various departments across the business where he supports and provides useful insights on the implementation of workflows.
The Data Engineer also plays a supporting role to the data warehousing department in the implementation of the data warehouse for the new big data platforms. He works collaboratively with the entire Data and Analytics team, providing support to the entire department for its data centric needs.
Knowledge: It is also the Data Engineers duty to keep up with industry trends and best practices, advising senior management on new and improved data engineering strategies that will drive departmental performance leading to improvement in overall improvement in data governance across the business, promoting informed decision-making, and ultimately improving overall business performance.
Other Duties: The Data Engineer performs similar duties and duties as delegated by the Senior Data Engineer, Head of Data Science, Director Data Science, and Chief Data Officer.
Required Qualifications of the Data Engineer
Education: The Data Engineer must have a bachelor’s degree in Computer Science, Applied Mathematics, Engineering, or any other technology related field. An equivalent of the same in working experience is also accepted for the position.
Experience: A candidate for the position will have as at least 3 years of working experience as a database engineering support personnel or a database engineering administrator within a fast-paced a complex business setting. The candidate will also have had experience working with MySQL databases and Microsoft SQL.
The candidate must demonstrate experience working with large and complex data sets as well as experience analyzing volumes of data through basic Microsoft Excel functions, for example, macros and pivot tables. A candidate for this position will also have had experience working in internet technologies, for example, SaaS, IaaS, and PaaS.
A suitable candidate will also have had experience in the creation and debugging of databases critical to the business’s mission. The candidate will have strong working and conceptual knowledge of building and maintaining physical and logical data models and experience with Tableau, Domo or other business intelligence tools.
A suitable candidate for the position will also have had system management expertise with monitoring, disaster recovery, backup, automated testing, automated schema migration, and continuous deployment.
Communication Skills: Communication skills for the Data Engineer are non-negotiable. Communication skills will be needed in his managerial role where he has to convey messages and instructions clearly to the supporting personnel in order to ensure efficient execution of duties within the junior department.
Communication skills will also be necessary in his collaborative capacity. Due to the technical nature of the Data Engineers role, he will be required to have excellent communication skills enabling him to convey technical messages to collaborative non-technical departments and personnel, leading to a positive outcome of the Data Engineers partnerships and the overall improvement in business performance.
The Data Engineer will also be required to draft regular performance and progress reports and prepare presentation for senior data engineering management and senior data science leadership, reports that have to be clear, concise, engaging, and convincing, which will require exceptional communication skills to deliver.
Ms Office/Software: A suitable candidate for the position must also demonstrate proficiency in the use of Ms Word, Ms Excel, PowerPoint, and Outlook, all necessary for the creation of both visually and verbally engaging reports and presentations, for senior data engineering management and senior data science leadership.
The Data Engineer must further have demonstrated skills in SQL server reporting services, Tableau, analysis services, Salesforce, integration services, or any other data visualization tools.
Technological Savvy/Analytical Skills: The Data Engineer must also have exceptional analytical skills, showing fluency in the use of tools such as MySQL and strong Python, Shell, Java, PHP, and T-SQL programming skills. He must also be technologically adept, demonstrating strong computer skills. The candidate must additionally be capable of developing databases using SSIS packages, T-SQL, MSSQL, and MySQL scripts.
The candidate will also have an ability to design, build, and maintain the business’s ETL pipeline and data warehouse. The candidate will also demonstrate expertise in data modeling and query performance tuning on SQL Server, MySQL, Redshift, Postgres or similar platforms.
Interpersonal Skills: A candidate for the position must also possess certain personal attributes that make him more suited for the position. The candidate must be result-driven, be an analytical and creative thinker, be an innovative problem solver, be self-motivated and proactive, be highly organized, have ability to handle-multiple and simultaneous tasks meeting aggressive deadlines, be a team player, and demonstrate an exceptional ability to stay calm and composed in the face of adversity.
People Skills: People skills like communication skills are a necessity for this position. The Data Engineer will be an approachable and relatable individual who forms strong connections with others and inspires trust and confidence in both his juniors and seniors.
This will enable junior support personnel to follow more easily in his directives and for senior management and leadership to give credit and consideration to his insights and judgments, enabling smooth performance of his duties.