Dirty data. It’s a term that can mean very different things to different people. Depending on who you speak to, you could end up with many different definitions. But, at its most basic level, dirty data is any kind of data that’s incorrect.
What is dirty data?
OK, so we’ve cleared up the basic definition. But let’s be honest… it’s a bit loose. So, it’s time to get more specific. Here’s an example, let’s say you work in procurement. In this case, dirty data could be:
- misspelt vendors
- incorrect invoice descriptions
- missing product codes
- a lack of standard units of measure (e.g. ltr, l, litres, or miles vs. kilometres)
- currency issues
- duplicate invoices
- incorrect/partially classified data.
See what we mean? It’s anything that gets in the way of sharing a clear and accurate picture with your data. And it can affect the whole organisation.
We all have an impact on and are responsible for the data we work with. Accurate data should be everyone’s responsibility, but all too often, it’s the sole responsibility of a person or department. Everyone trusts them to make sure the data is accurate. Talk about pressure!
How does dirty data arise?
Just like dust on a bookshelf, dirty data creeps in unseen, and then – for many reasons – isn’t immediately dealt with. How many times have you been working with a data set and noticed a small error but not said anything? Perhaps you manually correct something from an automated report, just to get it out of the door on time. They’re little errors, but can filter all the way to the top of an organisation through to reports and dashboards essential for critical decision-making.
How does dirty data affect my organisation?
One of the most widespread and noticeable impacts of dirty data is in reporting and analytics. If you’re in senior management, you will probably receive a dashboard from your team that you could be using to review cost savings, supplier negotiations, rationalisation, forecasting or budgets.
Now imagine for a moment, that your dashboard contains £25k of cleaning spend under IBM. We can already hear you saying, “That’s ridiculous”. And yes, we’d agree. But it’s only obvious when pointed out. Yet, as we have seen with our own eyes, dirty data like this happens easily and occurs more frequently than you might think.
When there are tens or hundreds of thousands of rows of data, errors occur multiple times across many suppliers. This could affect demand, planning, sales, marketing and financial decisions.
Let’s head back to the IBM example. Each quarter, your data is automatically refreshed (with the faulty cleaning classification). As a result, that £25k grows to £50k, then £75k the following quarter… It’s only when the value becomes significant that someone notices the issue. By this stage, how many decisions has your organisation made based on this incorrect information?
So, how do I fix dirty data?
There’s no magic bullet to improve your data’s accuracy. You have to use your team or an experienced professional. Our advice? Get your team to familiarise themselves with the data. This way, if they regularly review and maintain it, they’ll soon be able to spot errors in the data quickly and efficiently.
The best way to ensure clean data is to make sure it’s wearing its COAT and is:
- Consistent – everyone working to the same standards
- Organised – categorised properly
- Accurate – correct
- Trustworthy – you wouldn’t drive around in a car without a regular inspection, would you?
A data COAT – the key to cleaning up your dirty data
To clean your dirty data, you’ll need a spreadsheet of spend transactions over a period of time, such as 12 to 24 months. With this in hand, you take the following three steps:
- Supplier normalisation. Supplier normalisation is where you consolidate several versions of the same company to get a true picture of spend with that supplier. For example, I.B.M, IBM Ltd, I.B.M. would all be normalised to IBM.
- Spend data classification. Data can be classified using minimum information, such as Supplier Name, Invoice/PO line description and value. To get more from the data, you can then add other factors, such as unit price. Where unit price information is unavailable, the quantity can be divided by the overall value.
- Choose a taxonomy. A suitable taxonomy will allow you to classify the data. This can be an off-the-shelf product such as ProClass, UNSPSC, PROC-HE, or a customised taxonomy specific to your organisation or industry.
This initial stage may take months if you are working with large volumes of data. That’s why outsourcing to experienced professionals (such as The Classification Guru) is helpful – allowing you to complete the project more quickly and more accurately.
Cleaning dirty data saves you money in the long run
Data accuracy is an investment, not a cost and it pays to address your dirty data issues at the beginning. While it might seem like a costly exercise, you will undoubtedly spend less than if you have to resolve an issue further down the line with a time-consuming and costly data clean-up operation. And, by involving the whole team or organisation, it will be much easier to manage and maintain the most accurate data possible.
The result? A true view of your spend, allowing improved cost savings, better contract compliance and possibly the most important – preventing costly mistakes before they happen.
Enjoyed this post? Read this one next – Should I maintain my data?
