Business Intelligence (BI) Data Pipeline

 Business Intelligence Data Pipeline

We must ensure the reports are delivered correctly. But Business Intelligence report is just the final output of a very long data pipeline. So, if the report is wrong, it is not necessary that the defect is in the report; the root cause of the defect might be somewhere along with the breadth of the data pipeline. 

 The Business Intelligence Data Pipeline can be divided into two distinct stages:

 Part A: Data Processing & Storage

a. Source data

b. ETL

c. Database/Data warehouse

 Part B: BI Testing

a. Data Layer

b. Reports

c. Dashboards

    

BI System Testing process has 3 stages:

Data Layer: 

Data Layer or Metadata Layer is about data. It describes where data is being used and stored, the source of data, and how one piece of data relates to information. It includes description of data structure (schema, dimensions and hierarchies) and definitions of conformed dimensions and facts.  Users can just query and retrieve these metadata from repositories. This data needs to be compared with the equivalent database. We have to make sure that the data is the same across all environments. 

Here We have to make sure that the data from one data layer (production) is in sync with another in a non-production environment. Say now all these dimensions and measures are to be reconciled with the source data.

  

 BI Report:

 The data table is the most basic building block of business intelligence. It consists of a series of columns and rows that intersect in cells, plus a header row in which the names of the columns are stated, to make the content of the table understandable to the end user. A tabular report is used primarily to record information. Data tables are the most basic component of BI, and still one of the most useful. Regardless, building a report is a development activity and must be tested to ensure quality. For example, this table displays quarterly sales revenue of the e-fashion store.

   

BI Dashboards: 

 BI Dashboards are information management and data visualization solutions used to analyze your data. Dashboards are one of the most popular capabilities of BI platforms because they present easily understandable data analysis, allow you to customize which information you want to view , and provide a way to share the results of your analysis with others. It is used to track KPIs, metrics, and other key data points relevant to a business, department, or specific process.