Top Ten Data Quality Problems: Part I
By: Sally McCormack, Data Quality Competency Director, Datasource Consulting, LLC
Problems with data quality are costly to an enterprise. When facing the potential for missed opportunities, uninformed decision-making, non-compliance sanctions, and low customer satisfaction, today’s business leaders are making data quality a priority in their organizations’ data management programs. An Experian report found that 88 percent of companies see a direct effect of inaccurate data on their bottom line, losing an average of 12 percent of their revenue. In a similar study by Database Marketing, organizations estimate that they could increase sales by nearly a third (29%) with corrected customer data. (Source: Internal Results)
Understanding the root causes behind common data quality issues is an essential first step in an effective data quality initiative. By evaluating common data problems, organizations can begin to turn off the continuous flow of bad data. Businesses looking to improve data quality should be aware of potential ‘red flags’ to avoid drawbacks and enhance analytical insights and decision support processes across the enterprise.
1. Inaccurate Customer and Contact Data
The ability to efficiently and systematically reach current and potential customers is crucial to any business. The challenge is that customer contact data touches nearly every aspect of an organization, and contact data flows through each phase of the customer lifecycle. This includes data collected at the time of purchase/order placement to the moment cross-sell offers are auto-generated and social media marketing campaigns are launched.
Solution: When it comes to cleansing contact data, the IT department and marketing leaders need to first understand the data to find the source of common errors (i.e. address records with missing digits or secondary information, incomplete phone numbers). Once identified, the team can focus on cleaning up those areas. Next, data capture standards should be evaluated and all parties need to reach a consensus on standardized formatting and data quality controls during the data collection process.
2. Big data makes it hard to focus
Big data is often a challenge for CIOs, IT departments, and marketers because of the nature of traditional relational databases. These complex enterprise data systems often struggle to keep pace with the volume of data collected and the disparity of information sources. When bad data enters these systems— even when it’s a small error such as an incorrect courtesy title (Ms. vs Mr.) —there’s a compounded effect as data is compiled, analyzed, and filtered throughout the organization’s CRM, ERP, billing, and other enterprise transactional systems.
Solution: The goal of a big data initiative is to capture rich and diverse data points that provide a deeper understanding of customers. Captured data can augment competitive research and performance analytics, and in many cases capture ‘customer sentiment' metrics. Big data captured generally falls into three categories: unstructured (text, videos, audio), semi-structured data (email, reports, spreadsheets), and structured data (sensor data, machine data, financial models, algorithms, etc.). Understanding the context and relevance of each data type will ensure data quality is preserved and the data shows a more complete picture of the customer, presented in a meaningful and relevant way.
3. Duplicate or Obsolete Data
Since contact data is rarely static, obsolete data and duplicate records are common data quality challenges. Every day, people move, marry, and change their names and contact preferences, resulting in the business need for effective data verification methods at each collection point. This is particularly true for organizations that collect information at multiple stages during the customer lifecycle and from multiple channels (i.e. call centers, websites, and retail locations). Duplicate or obsolete records make it nearly impossible for companies to effectively communicate with prospects and customers. Should there be a miscommunication during the marketing process due to these inaccuracies (for example, using a divorced customer’s married name) the company risks a decrease in customer satisfaction. It can also lead to distrust and frustration from potential buyers if they receive multiple mailings, each with different names or prefixes.
Solution: A commitment to data quality stewardship early on can go a long way in preventing duplicate or inaccurate data records. If a company commonly has multiple records for the same customer in a database or data warehouse, duplicate identification tools can help match and link records based on individual standards, allowing for greater accuracy over time.
4. Compliance issues
Data security and compliance requirements come from various sources and may include corporate requirements in addition to industry and government mandates such as HIPAA or PCI Data Security Standards (PCI DSS). Failure to meet these rules can lead to heavy fines, and, perhaps even more costly, loss of customer loyalty. Often, requirements outlined by mandates such as HIPAA and PCI make a strong case for implementing a comprehensive data quality management program.
Solution: Consolidating privacy and security compliance management as part of an overall data governance initiative gives a business a significant advantage. This could include centralized data security and data quality monitoring procedures validated by an auditor, giving business leaders and IT assurance that their company meets critical privacy standards and safeguards against potential data leaks. By protecting customer data integrity with a unified data quality program, customers are encouraged to build strong and lasting connections to the brand.
5. Reconciling data quality issues during a merger
During any merger or acquisition there is the need to integrate multiple disparate source systems into a centralized or extended data warehouse environment. After integration, the data is generally filtered downstream into a comprehensive business analytics solution, or a series of BI solutions for each business segment. The major data quality challenge here is that with multiple data source systems, transactions cannot be properly attributed to a single source. For example, if a company is reconciling two ERP systems, a supplier would show up twice, as two different suppliers. The business could face significant problems if that supplier fulfills an order and the transaction is recorded twice. As in the example of a customer with numerous addresses, reconciling inconsistent data poses another challenge. The address for the customer in one system likely won’t match the address in another, and the same could be true with supplier addresses, phone numbers, etc. This compounds the data quality problem throughout billing and ERP systems.
Solution: Most data quality issues can be significantly reduced through careful planning during the design phase and data integration effort. At the start of a merger, for example, a company should take time to regroup and create standards, guidelines, and procedures for corporate data processes. Business analytics are determined by the quality of the data, and the best way to ensure accurate business representation and future trend prediction is with a consistent set of business rules applied consistently to source data. Proper data cleansing and data profiling early in the process familiarizes users with the data and prepares the data for clean integration, storage, and usage across the organization.
Leveraging data as a strategic asset is a hallmark of a successful company. When individual business users have confidence in corporate data and supporting business analytics and reporting, they can focus on achieving strategic business imperatives. A complete data quality initiative with data quality improvement activities will help to deliver a single source of trusted data. When data quality is an integral part of each business process, an organization optimizes performance and can move quickly on new opportunities. If your organization is interested implementing a data quality program, email us (add link to email address), we’d love to chat!
Want to have a PDF version of this blog to print or save? Click to download below.
Are you ready to discuss your project?
Let's chat about it, we look forward to helping and becoming your data partner.