Data warehousing technology: history, architecture, and organizational applications

¶ … data warehouse has been of tremendous benefit to organizations highly dependent on data management. This technology allows for greater access to data originating from different platforms and areas within an entity. It also allows greater ease of data extraction.

Data warehouse technology became popular in the 1990's. Before this, many felt that significant amounts of data were simply "in jail," out of reach for examination or manipulation. Further, data management was time consuming and left to data management "departments" rather than individuals.

Data warehouses are composed of an "architecture" that is discussed. Further, a brief description of possible future developments, as well as restrictions is presented.

Introduction

There is little question that many critical enterprises in the world of today are dependent on quick and dependable access to information. From the halls of academia, to the world of business -- science to medicine -- the ability to readily access critical information within any particular organization or working entity is essential to survival and growth. However, even in today's technology-driven industries, it is often difficult for companies and other organizations to effectively provide the most comprehensive and critical internal information to those who need it.

Although it may seem to be quite simple, the ability to store (and later access) can be quite elusive. This is especially true when organizations fail to effectively "store" the countless bytes they accumulate within their spheres of activity, and instead allow them to be spread between unconnected applications and systems. When this occurs, many find that one of their most valuable resources -- information -- is for all intents and purposes "locked away" from use, becoming what Alexander Nepomnjashiy effectively named "Data in Jail" (2002). Recently, however, there has emerged a relatively new technology by which organizations can begin to effectively store and access critical information. This technology is known as "data warehousing."

Data Warehouse Architecture

Before discussing just how data warehouses are built, it is important to understand that there are two main categories of information use. Not only do these two categories include information that is vastly different in type, but they may also differ in other important ways (including age and reliability). In broad terms, these can be classified as operational verses informational (Lambert, 1996).

Operational systems rely on data related to the nuts and bolts of running a company or organization. These may include such areas of operation as accounting, inventory, manufacturing output, billing, etc. In contrast, informational systems are those concerned with the "management" of the organization. This may include the use of data for planning, strategizing, estimating, or predicting possible outcomes to name a few (Poe, Klauer, Brobst, 1997). Not only are the types of information involved in these two systems very different, but the data warehouse must be capable of handling both. This is of particular importance when planning the "architecture" of the warehouse.

According to Ken Orr in his 2000 work, "Data Warehouse Technology," the data warehouse architecture, or DWA is simply a way of "representing the overall structure of data, communication, processing and presentation that exists for end-user computing within the enterprise." Further, this structure is made up of the following interconnected parts:

Operational Database / External Database Layer

This includes the processing structure for transactions such as payroll, billing, accounting, etc.

Information Access Layer

The layer that the user "sees" and uses directly. This includes commonly used applications (Excel, Access, or Lotus 1-2-3, for example). This is also where hardware is represented (printers, monitors, etc.). One of the key focal points of data warehousing development is improving the ease with which the data is available to this layer.

Data Access Layer

This is where the information access layer "communicates" with the operational layer. Most frequently, a data interchange language known as SQL is used as a common data language -- this allows for a greater compatibility between different systems (rendering them far more powerful). Further, in addition to allowing the possibility of data retrieval from different systems continued development promises a kind of universal data access through which any individual can use any location or access tool or application to gain access to required information.

Data Directory (Metadata) Layer

As strange as it may sound, it is extremely necessary within data warehousing development and management to have "data about data." This may include records of user view of data. This data can assist in making the warehouse more effective as well as time-efficient.

Process Management Layer

This is the layer that organizes and "schedules" the steps that must be taken to build, and preserve the data warehouse. Without this layer the data warehouse may become outdated fairly quickly.

Application Messaging Layer

In this layer one finds the resources necessary for messaging or "transporting" information throughout the organization's network of computer resources. This layer is also often known as "middleware," much like a middleman facilitates transactions between two separate entities.

Data Warehouse Layer

This is often referred to as the "physical" layer of the architecture. Here, the virtual data may exist for "viewing." Here, the data is readily available for the user to access. Further, it is in this layer that one finds the physical "location" of the warehouse -- often a client/server platform or mainframe.

Data Staging Layer.

Also known as "copy" or "replication management," it is this layer that encompasses all of the actual processes involved in retrieving and manipulating data (including selection, editing, summarization, and combining). It is also here that analysis programs may come into play, designed to improve data "quality" (2000).

Restrictions

As with any technology (particularly newer ones), there are significant barriers that may prevent its full market utilization. Of course, one of the first barriers is the relative ease or difficulty of use. Unfortunately, some organizations simply find leading data warehouses to be too complex or even cumbersome to use. Still others fail to utilize the technology to its full potential out of a lack of understanding of its capabilities.

Perhaps one of the most common barriers, however, remains the high cost of developing a good data warehouse (CIOL, 2001). Although data warehouse technology has utilized among many of the Fortune 1000 companies for more than twenty years, for most small or mid-size companies the cost simply places it out of reach (PEP, 2004). In fact, "...when considering the additional costs of installation, data file mapping, customization, and other services, industry experts have estimated the minimum cost of a complete data-warehouse solution at close to $100,000 (2004)."

The Present and the Future

Although the data warehouse field has dramatically improved the way in which many businesses and organizations perform data-dependent functions and analysis, one of the most often cited pitfalls of their present use is the widespread problem of obsolete information. In fact, according to one study conducted by Dynamic Markets Limited, "research reveals that, while data warehouse managers acknowledge that the value of the data stored changes over time, few have the technology architecture in place to address this (CRM, 2004). Although some data warehouses are beginning to incorporate strategies for dealing with the problem of obsolete data (and in many enterprises, data can become obsolete in less than a second). However, many, if not most data warehouses have yet to deal effectively with this very real problem.

Additionally, in the near future new parallel database servers are likely to improve the power of data warehousing exponentially.

Not only will these servers allow the user to access immense pools of data, but they will also allow them to do so in a fraction of the time (ICFAI, 2004).

Finally, it is also very likely that in the near future the very "number driven" data warehouses (numbers being the de facto definition of data), may become capable of "recognizing" text and even multi-media as data (Orr, 2000). Clearly, the implications of this are enormous -- both theoretically, as well as in the practical realm.

Conclusions:

Data warehouses are without question one of the most powerful technologies to impact the world of data management. The simple fact is that knowledge is power -- especially in the business world. Without a good way to access all of the available data produced or related to a given enterprise, many key decisions and operations may be made in grave error. Although most data warehouses have yet to fully exploit the world of "real time" data, or even today's expanding definition of data, they are still tremendously powerful, if not indispensable. One can only imaging that even smaller companies may soon be able to utilize this amazing technology as development prices eventually fall. It may even be possible one day for data to be accessible by any authorized individual at any time. Should that day come, data warehouses would have definitely paved the way.