Data Warehousing as the Senior

Data Warehousing

As the senior analyst responsible for data staging, you are responsible for the design of the data staging area. If your data warehouse gets input from several legacy systems on multiple platforms, and also regular feeds from two external sources, how will you organize your data staging areas? Describe the data repositories you will have for data staging. (one page answer)

The need for data integration at the platform level first and secondly, at the taxonomy level necessitates having a staging server for each inbound legacy systems first and then a consolidated staging server to ensure data structures, taxonomies and resulting queries can be successfully accomplished. This approach preserves data structures that are most likely specific to each of the legacy systems first, and second, allow for cross-integration of taxonomies on a text server to ensure actual production data is not affected. These data repositories would then be used for regression testing of the data across all other applications to ensure the new data staging area can be reliably used.

Of the many aspects of data integrity that will need to be audited on the test server, the most critical is the data element integration across all systems of record to ensure applications work as they are intended to. This is a critical element of any data migration through the use of data staging areas to ensure that data validity and mapping stay consistent and the taxonomies created on legacy systems are still valid on an enterprise-wide database. In effect this system of record must concentrate on making sure there is data reliability and validity across the entire data set. Using periodic audits and the performance of the databases over time to ensure they are delivering query-based performance is also crucial and needs to be part of any release strategy.

Examine the concepts that metadata is like a nerve center. Describe how the concept applies to the data warehouse environment. (one page answer)

In the context of a data warehouse environment, metadata will take on added complexity and significance as in many cases a data warehouse is constructed of multiple legacy database systems. Of the many challenges of creating a data warehouse, foremost among all of them are the variations in metadata definitions and resulting taxonomies. As data warehouse architects will attempt to create a common taxonomy across all data records, there is the need to ensure a consistency to ensure legacy applications based on legacy databases can still function. Creating a new single system of record that is as taxonomically correct and robust as possible yet also as specifically focused on the individualized requirements of each legacy application requires data warehouse architects to make trade-offs regarding functionality relative to performance. There are also the aspects of data dependence, orthogonally, and process workflows that define the context, quality, security and relevance of the data over time. Data warehouse architectures must take all these factors into account in order to ensure a high degree in order to ensure any given data warehouse will be able to successfully support the many applications it is called upon to provide data for. In addition, the support of multiple taxonomies is also critical for a data warehouse, and to the extent the architects have created a database architecture that will provide for metadata definition and re-defining of taxonomies is the extent to which the data warehouse will have greater use in the organization. Without a strong focus on these aspects of data agility, a data warehouse can quickly become outmoded and only marginally successful.

Assume that you are the data quality expert on the data warehouse project team for a large financial institution with many legacy systems dating back to the 1970s. Review the types of data quality problems you are likely to have and make suggestions on how to deal with those.