Making Sense Out of Gibberish

  Business Intelligence   Strategy  

Text is everywhere. From surveys to diagnoses to repair orders, comments are often captured about a transaction or set of transactions. If you look at the past 10 years of business intelligence history, you'll see that we have done little to incorporate text into the business intelligence process. This is primarily for two reasons: 1) We have struggled considerably with the non-textual components of business intelligence and 2) The technologies and algorithms to sensibly analyze text have not been readily available. Recent products have made the analysis of text significantly more feasible and have begun to bring text into the business intelligence mainstream.

Text can be invaluable from a number of different aspects in the business intelligence continuum. From a data quality standpoint, it is the only check and balance to ensure that a record has been classified appropriately by an analyst prior to being loaded into the warehouse.

For example, many of our business intelligence solutions rely on pre-existing codes. In the medical industry, diagnostic codes called ICD-9s capture a patient's ailments. They are the primary basis for analyzing disease patterns in a medical claims data warehouse. However, it is possible that many of those claims were improperly coded since they are susceptible to human coding mistakes.

Between the time the …

Please log in or sign up below to read the rest of the article.