Complexities of Data Warehousing

Technology, Resource, Financial, and Selection Issues Are Complex

Oct 9, 2009

Large organizations routinely collect vast amounts of data from customer transactions and managing this data involves several complex corporate issues and processes.

There are a number of complex technology, resource, financial and vendor selection issues and consideration involved in the design and development of a data warehouse.

Some companies build large data warehouses to crunch information about their customers to determine buying habits or product preferences. If correlation among customer purchasing habits is not properly done the relationship between one set of data and another will not be valid from a business viewpoint. Technology and business processes must be applied in a logical context to ensure that customer data is used effectively to meet CRM objectives.

Fundamental Issues

Modeling Business Processes

A considerable amount of time in the development of a data warehouse is devoted to extracting, cleaning and loading data, because problems that may have been undetected for years can surface during the design phase. Finding stored data that had not been accessed previously, or data that has been altered and stored, are examples of some of the data-oriented problems that may arise during the data warehouse development stage. It is important for the development team to understand the businesses and all the processes that need to be modeled.

CRM is based on a set of technology tools, especially data information-based tools that allow organizations to capture, analyze and apply the large volumes of detailed customer data needed to achieve a better understanding of their customers, and to make more informed business decisions.

Data Warehousing vs. Other Client/Server Applications

Another major consideration that is important to up-front planning is the difference between the data warehouse and most other client/server applications. First, there is the issue of batch orientation for much of the processing. The complexity of processes – which may be executed on multiple platforms – as well as data volumes, and resulting data synchronization issues, must be correctly analyzed and resolved.

Next, the data volume in a data warehouse, which can be in the terabyte range, has to be considered. On the hardware side, new purchases of large amounts of disk storage space and magnetic tape for backup should be expected.

It is also vital to plan and provide for the transport of large amounts of data over the network. The capability of data warehousing to support a wide range of queries, from simple ones that return only a limited amount of information, to complex ones that might access several million rows of data, can cause complications.

Corporate metadata needs to be incorporate into this thought process. The designers of the data warehouse have to remember that metadata is likely to be replicated at multiple sites, requiring synchronization across different platforms to avoid inconsistencies.

Security and Networking Issues

Finally, security and networking issues must be considered. In terms of location and data security, data warehousing and non-data warehousing applications must appear seamless. For example, users should not need different IDs to sign on to different systems, but the application should be smart enough to allow access with only one password

The long-term relationships enabled by CRM use technology to manage business processes, so organizations can use historical customer transaction data to manage customer relationships better, providing them with the benefits of a formal approach to customer relations.

The copyright of the article Complexities of Data Warehousing in Customer Relations is owned by Duane Sharp. Permission to republish Complexities of Data Warehousing in print or online must be granted by the author in writing.