Investigational Computing and Business Intelligence Explained

Introduction to Business Intelligence and Big Data

Business intelligence is the process of using large data sets, processed using statistical techniques, to aid in business decision-making. The Internet has provided vast quantities of data that can be used to gain insight. For example, a large retailer gathers data on every purchase — such as what items were purchased together, when they were bought, how they were bought, and where possible, customer characteristics as well. This information can provide valuable insights. For example, a business might learn about cross-price elasticities by identifying products that are frequently purchased together and then adjusting the prices of those items to observe the impact on sales of the other. A company can learn what the elasticity is for fans, for instance, with respect to changes in temperature. It can also learn more about the target markets for specific goods. All of this information helps businesses gain an edge in the market and over competitors (Chen, Chiang & Storey, 2012). As data sets become larger and data quality improves, technology continues to lower the cost of obtaining these insights.

What Is Investigational Computing?

The Data Warehouse Initiative (2013) proposes that investigative analytics fills a gap between traditional analytics and predictive analytics. Predictive analytics "consume data from structured and semi-structured sources," as in the examples above. Predictive analytics is commonly used to aid in business decision-making because it can help managers examine the effects of changes to independent variables on whatever dependent variable they have sufficient data to support significant analysis on. One of the drawbacks of predictive analytics, it is argued, is that it requires a high level of statistical ability to interpret the results. As a tool, it is limited because its outputs must be translated into plain language that managers can understand. While software exists to assist with this, actually building models and knowing what data to capture requires a degree of statistical expertise that makes predictive analytics an inherently specialized field.

In contrast with the closed-ended nature of predictive analytics — which is essentially the ability to answer a question you have already asked — investigative computing is a more open-ended concept. With investigative computing, managers can use it to discover what questions they should be asking; in other words, it serves as a tool for hypothesis generation (TDWI, 2013). Investigative computing achieves this by looking for patterns, anomalies, and clusters. This will eventually lead to testing hypotheses via predictive or traditional analytics, but it is in this initial discovery stage where tremendous value is to be found.

For example, consider a retailer that wants to know the elasticity of demand for fans on hot days. Predictive analytics can deliver a direct answer to that specific question. Investigative computing would instead ask a broader probing question: which products spike in demand on hot days? The idea is that the manager can gain insights that otherwise might have been overlooked. For instance, if there is a difference in which products spike in sales on hot days between northern and southern states, that pattern could be identified through investigative analytics. Perhaps the data reveals something worth investigating further — such as differences in hot-day sales anomalies between stores located within two miles of the coast and those further inland. There are many questions managers could ask if only they thought to ask them.

What all of this means is that investigational computing could be described as brainstorming on steroids. While it replicates the age-old technique of noticing things about a business and then investigating the phenomenon for insights, it does so in a way that harnesses the power of big data. When a business holds a tremendous amount of data, investigational computing runs through that data to surface insights that might otherwise remain overlooked due to sheer volume. It would be impossible, without investigational computing, to notice all meaningful trends or anomalies in a data set as large as those held by companies like Google or Amazon. Any one manager can notice a handful of trends, but even the most diligent manager would miss some, simply because the data set being analyzed is too large. Investigational computing allows all trends and anomalies to be identified. Thus, it is more powerful than any manual process for the same reasons predictive analytics is so powerful: processing speed and the comprehensive nature of the technology, which allows it to examine every single data point rather than only those that happen to be noticed by a human observer. Indeed, processing speed has been cited as a major factor in the rise of predictive analytics, and it follows naturally that increased processing speeds have been equally critical to the development of investigational computing.

Infrastructure and Iterative Processing

As of 2013, TDWI reported that there were two main components to the physical infrastructure of investigative computing. Hadoop is an open-source platform that distributes data storage. Originally conceived as a means of defending against storage hardware failure, Hadoop has a significant role to play in investigative computing. One of its key benefits is that it allows for much faster data processing because it uses multiple nodes — similar to the way BitTorrent works. It delivers substantially greater capacity for data storage and processing, allowing companies to handle much larger data volumes; this is critical to deriving value from data sets in the petabyte class (Swoyer, 2013).

Investigative computing is also iterative in nature — in other words, it learns. With each iteration of data processing, the algorithms become more refined, resulting in better analysis over time. As a consequence, the insights gained from the system improve as well. By running models more quickly and having those models refined more frequently, investigational computing not only increases the speed of analysis but also its accuracy. This, in turn, facilitates more open-ended queries. Another critical benefit is that the entire process is efficient enough to be embedded in operational processes. Typically, business analytics requires presenting data to a human who then analyzes it and makes decisions. The infrastructure underlying investigational computing goes a step further by being embedded directly into processes, using analytical output to make decisions without a human intermediary. The result is potentially very powerful, as a key bottleneck — human intervention — can be bypassed in many instances.

Conclusion

Investigational computing aims to fill the gap between traditional analytics and predictive analytics. The role it plays is to work with massive data sets while enabling real-time investigation and ad hoc queries. Rather than relying on SQL-based methods, it seeks out anomalies and trends in the data, allowing managers to determine what is worth investigating further. It is, in essence, an idea generator. Once ideas are generated and refined, the task is often handed to predictive analytics — but investigational computing is the technique by which the question was identified in the first place.

Managers have always done this — looking for trends and anomalies — but with modern big data sets of mixed types on the petabyte scale, investigational computing is the only reasonable means by which managers can learn of all the trends and anomalies their data can reveal. Investigational computing is, ultimately, a more complete and refined approach to gaining critical business insights.

References

Chen, H., Chiang, R., & Storey, V. (2012). Business intelligence and analytics: From big data to big impact. MIS Quarterly, 36(4), 1165–1188.

Swoyer, S. (2013). Investigative computing: The new frontier in BI. TDWI. Retrieved November 14, 2015, from http://assets.teradata.com/resourceCenter/downloads/Brochures/TDWI_Hot_Topic_Teradata_No-2_Investigative-Computing.pdf

TDWI (2013). Investigative analytics: The new BI frontier. TDWI. Retrieved November 14, 2015, from http://www.enterprisemanagement360.com/wp-content/files_mf/1383730412TDWI_EBook_InvestigativeAnalytics_Infobright_Web.pdf