If you want the best techology that serves the interest of both your customers and your team. Following a structured process matters.
I have a process that I follow as best I can that has been heavily influenced by two things in my background:
- I was formally trained in Urban Planning and think strategically ever since
- I am a Certified Data Management Professional so I view data comprehensively
Having gotten that out of the way I think developing technology that consumes data requires 3 distinct steps which I like to say this way:
Good Data Strategy enables robust Data Engineering which in turn enables easy to use Technology Products.
Lets explore each of these a bit.
Data Strategy
Data Strategy is probably the most skipped component of the process. But it is really important. This determines how you will use data in your organization.
- What data do you need?
- What are your data standards?
- How do your define Quality Data?
- How are you going to do Quality Control?
- What are your definitions?
These are the things that if not address will cause a litany of downstream data issue. I have seen it repeatedly.
I like to use a sports analogy. Many companies are like a football team. They want to show up on game day and play. But good teams do off-season conditioning, study the play book and watch film so they are ready to win on game day.
Be the second team. Put in the work, do it right and plan first.
Data Engineering
Data Engineering is the backend work that most end users will never see. It relies on data strategy to be done well and if data engineering is done well your end uses will love your product.
Data Engineering is concerned with the movement of data, integrations, infrastrure and so forth. However, if you didn't plan a strategy and your data engineers have to wing it how are they going to know what good data looks like. Keep in mind these are the peole that will clean your data so they better know what clean data looks like. I hope they don't have to make it up as they go.
Technology Products
This is the fun part everyone like. The cool app that makes your team's job much easier. It is really good if you did good planning and good data engineering. If you didn't it is how you end up with a half dozen (or more) disconneted systems that can't talk to each other. So a customer is updated in your CRM for sales but somehow that never made it to shipping and the order went to their old house and they aren't thrilled.
That is what a good data strategy and good data engineering are designed to prevent. A disappointed customer.
Be vigilant. Follow the strategy.