Of Course Your Data Is Sh*t
It’s not your fault. It’s not anyone’s fault. It’s just hard. It takes a ton of time and energy and attention and work to keep a clean house.
Your data is trying to get bad, always. Remember learning about “entropy” in science class? How the universe is trying to get messy?
Take your kitchen, for example. Your kitchen is always trying to get messy. If you’re managing to keep it clean, hats off to you. That doesn’t happen by itself. It happens with a lot of repeated work. Some large, focused fits of work, and some smaller, steady, ever-present tasks that you may not even realize you’re doing. But it’s not keeping itself clean, that’s for damn sure.
Photo by Nathan Dumlao
Well it’s the same with your data. Your data is always trying to get messy.
And how unjust that is. Because without accurate, trustworthy data, you can’t really get any useful data intelligence about what you’re doing, and what is and isn’t working. You can’t ask questions and get accurate answers. You’re flying blind. You’re working harder than needed and doing a bunch of guesswork.
This sounds unacceptable but it’s overwhelmingly the norm.
Do you trust the data in your systems? Do you really trust it?
If you have clean, useful, trustworthy data, then congratulations, that is a monumental achievement. In that case you probably have a number of important things in place:
Customization and iteration
You have a system in place that supports your work beautifully. What you actually do in the world is well supported and tracked by the system, so you don’t have to maintain your own system of shadow spreadsheets.
When you encounter a problem or have a new idea, there is a team to whom you can explain the problem, and get it solved quickly.
Support and Training
You have a series of trainings in place for new staff and staff who change roles.
You also have ongoing trainings to help existing users constantly up-skill.
You have easy-to-access, on-the-spot support available. So when people forget what they learned in your trainings, or encounter a new situation, they can get a quick answer rather than having to start inventing their own sub-optimal workarounds.
Data integrity monitoring and improvement
Everybody in all departments feel ownership and empowerment over maintaining data quality and integrity in their area.
A specialized team helps maintain a system of dashboards and reports that empower each team to monitor and correct their own data.
When a data problem is identified, it is not only corrected but preventive measures are also put in place to prevent that specific problem recurring in the future, leading to an ever-improving data integrity baseline.
Imports, integrations, and migrations
Data imports are done with obsessive care and detailed attention, likely by a data specialist.
Larger data migrations from other systems are done with robust time, attention, detail, and care, rather than being rushed or squeezed into limited project budgets. (Raise your hand if you were happy with your last data migration. Nobody?)
Ongoing data integrations are carefully mapped and built, but then relentlessly monitored, adjusted, fixed, and improved, again by a specialized team.
That’s not even a full list. But it’s a lot, and it’s a constant battle that must be fought. Most of you will readily admit that your organization doesn’t do all of these things. Most don’t even come close.
So yes, currently your data may be sh*t. Of course it is. It’s not anyone’s fault, it’s just that clean, trustable data is a lot harder to accomplish than your organization is admitting to itself. Your data effort isn’t probably near as robust or well-resourced as it needs to be, and it’s probably no one’s job yet to change that.
But this kind of robust effort is how you get to high quality, trustable data. Data that will give you what you need, when you need it. Data that will accurately tell you what your organization has accomplished. Data that will allow you to ask questions, and pose hypotheses, and attempt experiments, and see how they work. Data that will allow you to innovate.
You need clean, trustworthy, accurate data if you want to get where you’re trying to go. So you have to be willing to fight for it.