About all the magical creatures living in the data forest
Introduction
The new slogan, "data is the new oil", is motivating organisations across the globe to take action: there is now a multitude of so-called "data-to-insight" projects bridging business and IT units.
Each such project is made up of specific technical jargon, dedicated workflows and various types of data sources. All this together can seem like a deep, dark forest for the unseasoned business person. However, the good news is that this dark forest is home to a team of extraordinary professionals, who, just like magical creatures from Tolkien's books, can help fulfil even the most demanding of guests.
This article is split into two parts. The first half describes a regular data-related project in an organisation and acts as a map. The second half concentrates on the different professionals that operate only in the specific stages of the data-related scheme.
A few words about the typical data-to-insight project
While each initiative is different, with varying complexity and business objectives, the assumption is that they all are built around the same acyclic structure.
Business identification is traditionally the sole responsibility of business professionals. This stage is all about naming the objective of the project. In software development, like in the case of smartphone applications, it would specify the functionalities of the app and the overall customer experience. The "what" is more about the questions that need to be answered in data-related projects. For instance, the business team would like to know how a brand new product will do or the most effective marketing channels. The quality of the definition behind business needs will directly impact the rest of the process.
Refinement of requirements involves dissecting the business need into more specific elements, for example, so-called user stories and tasks. Refinement is the first stage when the business user communicates to the broader team what needs to be done. Traditionally, in refinement, some concrete and difficult questions are asked, and the original business idea morphs into something much cleaner – hence the name of this stage. Unlike in software development, questions will primarily relate to data in possession, missing data, and how it will be used. Sometimes the entire project team is present during refinement. Still, it is predominantly the area of product owners and business analysts from both the business and technical sides.
The Ingestion of raw data stage is when specific infrastructure is put in place to deal with the request. One of the most important matters here is the availability of infrastructure to store and process the data. Most of the big enterprises use what is called an "Enterprise Data Lake" on third party Cloud providers, such as AWS (Amazon), Azure (Microsoft) or GCP (Google). Simply put, a data lake is a central repository of an organisation's data that enables easy access for stakeholders. Again, the assumption is that the typical data related project can already take advantage of a data lake. Otherwise, building a data lake is a project on its own and not a straightforward one. New data sources identified in the previous stage will require brand new pipelines so the data in its raw format can flow to a specific reservoir in the data lake, either regularly or via ad-hoc demand.
Validation to a trusted format is all about cleaning up the raw data and preparing it for further stages. The assumption is that, at the end of this stage, data stored in a dedicated reservoir of a data lake can be fully trusted by the whole organisation. This might sound pretty straightforward, but data from both internal and external sources come in all shapes and forms. Data validation to a trusted format is about standardising data and cross-checking the actual content against the documentation provided by the data source owner. Data in a trusted format won't carry any insight beyond its face value but has much-refined potential.
The data transformation and enrichment stage is arguably the one where most of the "magic" happens. The actual change is that all existing and brand new data sources come together, fostering synergy outside the traditional spectrum. This stage is dominated by analytics engineers and data scientists, but business analysts and product owners often accompany them. Valuable insight is forged in the process of enrichment and transformation. The output is still in a form that is relatively difficult to comprehend. Depending on the business case, the uncut form alone might be enough because it feeds into an established reporting system or it might still need "explaining" to the business users.
The presentation layer stage is the final part of the data-to-insight project. The sole objective of the presentation layer is the visualisation of the insight and the explanation of relations. This might sound like the "gift wrapping" on the work from previous stages, but it is much more than that. Selecting the correct type of chart graphs to provide the best user experience requires natural talent and devotion. The data visualisation expert is the main person in this stage. However, because the end product will be directly "consumed" by the business, the product owner will also be nearby.
Quality assurance generally happens when everything else is done: the entire setup needs to be thoroughly tested. However, QA will also execute small checks called "unit testing" and prepare documentation during earlier stages. Quality assurance engineers are under a lot of pressure: the numbers they produce will likely be seen by the senior leaders in their organisation. A technique called "User Acceptance Testing" is used to make a small number of business users in the team responsible for testing the final delivery.
The people who make the "magic" happen.
The Product Owner (PO) is, at the same time, the customer of the development team and the first link in the chain responsible for describing the requirements. Before engaging the project team, the product owner should have a clear hypothesis and a good understanding of what is needed and what data might be necessary, especially if it is a brand new source. In organisations with a well-established data setup, it is not uncommon for the product owner role to be split into business and technical sub-roles.
The Business Analyst (BA) is a typical role in many web projects, but it also has a solid position in data-related projects. The central role of the business analyst is during refinement when the exact requirements must be captured and forged into specific user stories. The better described and fragmented the stories are, the better tracking of the progress and outcome is required. The BA's role is also to challenge the product owner with questions until all requirements are crystal clear. The business analyst will be present at all stages of the project as the guardian of the PO's needs.
The DevOps Engineer (DO) is most needed when new or additional infrastructure is implemented during the raw data ingestion phase. These professionals provide data engineers with development tools and infrastructure. In most cases, all of the ingestion code will, at some point, have to pass through the hands of the DevOps Engineer. Cyber security is also traditionally the domain of DOEs.
The Data Engineer (DE) plays the most significant part during the ingestion of raw data, and later during validation to the trusted layer. The DE is the professional who develops a database and connects any new data sources. The strong software background of data engineers enables them to work with multiple languages and connect new data, irrespective of protocol API, SFTPs, etc. However, moving large quantities of data in a performance-optimised fashion is only part of the equation. Data engineers also test raw data in preparation for further enrichment.
Analytics Engineers (AE), similar to data engineers, are highly involved during the validation of raw data to trusted layers of the data lake. However, while the data engineer concentrates more on infrastructure, analytics engineers look more at the "content" of data. AE examine data for internal consistency, and if that is not the case, investigate the issue and solve it. In the transformation and enrichment stage, the analytics engineer acts as the "right hand" to the data scientist by sharing the knowledge gathered during validation. Both analytics engineers and data scientists share the same passion for applied mathematics.
The Data Scientist (DS) operates almost exclusively in the area of data transformation and enrichment. This is the person that will mine the insights hidden deep in data and turn them into a meaningful story. Data scientists can handle large-scale processing, but they are above all masters of mathematics and use statistical techniques to get the desired insights daily. The best data scientists will be able to use data to describe reality or even predict the near future. They will also be able to prescribe specific actions to help their organisation get to the desired outcome. In the age of digital and AI, data scientists often "train" computers to gain precious insight. Data scientists' professionalism can also extend to visualisation. However, their powers are best applied during transformation and enrichment.
Business Intelligence Engineer (BI) take over from analysts and scientists close to the final product and turn data into easy-to-understand charts, tables and dashboards. Insight, in its uncut form, might be challenging to follow, and if that is the case, it might go to waste altogether. The BI's responsibility is to support the stories outlined by analysts and scientists in the best possible way. Overall, customer experience associated with using reports and dashboards is firmly rooted in the business intelligence engineer role. When a project requires a bespoke platform to present results, BIs will join forces with web developers. However, web developers are a rare sight in typical data-to-insight projects.
Quality Assurance Engineers (QA) generally spring to action when everyone else is already done. In web development, the role of the QA is to break the functionality, document it, and replicate this in front of the dev team. In data-to-insight projects, this is about looking for data inconsistency across the different layers of transformation, documenting it and demonstrating it to the analytics team. QA experts also participate in other project stages by gathering the information needed to design exhaustive testing scripts. QA experts are the last line of defence before the project gets released to a broader audience outside of the project team.
Final words
A team of professionals comprised of the roles described above can help fulfil even the hardest of requests. However, no matter how "magical" some of the defined roles seem, they are real professionals operating in the same reality as marketing, supply chain and sales colleagues.
In the absence of meaningful data, even the best won't be able to get actionable insight. Organisations aiming for the prestigious title of "data-driven company" have to strike the right balance between building generic and proprietary databases and building a team of the right professionals.
