There are several values that are key to your success when transforming to a leaner, more agile approach to Data Management. Taking a cue from the Disciplined Agile Manifesto, we’ve captured these values in the form of X over Y. While both X and Y are important, X proves to be far more important than Y in practice. These values are:
- Evolution over definition. The ability to safely and quickly evolve an existing data source, either to extend it to support new functionality or to fix quality issues with it, is absolutely paramount in today’s hyper-competitive environment. Yes, having defined data models and metadata describing them is also important, but nowhere near as important as being able to react to new business opportunities. Luckily agile database techniques, long proven in practice, exist that enable the safe evolution of production data stores.
- Holistic organization over Data Management. Earlier we said that data is the lifeblood of your organization. Yes, blood is important but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.” Traditional Data Management approaches often run aground because they locally optimize for data concerns, whereas a DA approach to Data Management recognizes that we must optimize the overall whole. This implies that sometimes we may need to sub-optimize our strategy from a data point of view, for the sake of organizational level optimization.
- Sufficiency over perfection. Data sources, like many other IT assets, need to be good enough for the task at hand. The old saw “perfect is the enemy of good” clearly applies in the data realm – too much time has been lost, and opportunities squandered, while development teams were forced to wait on Data Management teams to create (near) perfect models before being allowed to move forward. Traditional data professionals mistakenly assume that production databases are difficult to evolve and as a result strive to get their designs right the first time so as to avoid very painful database changes in the future. The Agile Data method has of course shown this assumption to be wrong, that it is very straightforward and desirable to evolve production databases. A side effect of this revelation is that we no longer need to strive for perfect, detailed models up front. Instead we can do just enough up-front thinking to get going in the right direction and then evolve our implementation (including data sources) over time as our understanding of our stakeholder needs evolve.
- Collaboration over documentation. We’ve known for decades that the most effective way to communicate information is face-to-face around a shared sketching environment, and that the least effective is to provide detailed documentation to people. The implication is that we need to refocus our efforts to be more collaborative in nature. As data professionals we need to get actively involved with solution delivery teams: to share our knowledge and skills with those teams, and to enable them to become more effective in working with data. Yes, we will still need to develop and sustain data-related artifacts, but those artifacts should be lightweight and better yet executable in nature.
- Cross-functional people over specialized staff. Agilists have come to realize that people are more effective when they are cross-functional (also known as T-skilled or generalizing specialists). Although specialists are very skilled in a narrow aspect of the overall process, the problem is that you need a lot of specialists to perform anything of value and as a result the overall workflow tends to be error prone, slow, and expensive. The other extreme would be to be a generalist, someone who knows a little bit about all aspects of the overall process. But, the challenge with these people is that although they’re good at collaborating with others they don’t actually have the skills to produce concrete value. We need the best of both worlds – a generalizing specialist with one or more specialties so that they can add value AND a general knowledge so that they can collaborate effectively with others and streamline the overall effort.
- Automation over manual processes. The only way that we can respond quickly to marketplace opportunities is to automate as much of the bureaucracy as we possibly can. Instead of creating detailed models and documents and then reviewing potential changes against them we capture our detailed specifications in the form of executable tests. This is quickly becoming the norm for specifying both the requirements and designs of code, and the same test-driven techniques are now being applied to data sources. Continuous integration (CI) and continuous deployment (CD) are also being applied to data sources, contributing to improving overall data quality and decreasing the time to safely deploy database updates into production.
As you can see, we’re not talking about your grandfather’s approach to Data Management. Organizations are now shifting from the slow and documentation-heavy bureaucratic strategies of traditional Data Management towards the collaborative, streamlined, and quality-driven agile/lean strategies that focus on enabling others rather than controlling them.
- To read more, we have a detailed article overviewing a Disciplined Agile approach to Data Management
- Need training in agile skills for DW/BI? Consider the workshop Disciplined Agile Data Warehousing(DW)/Business Intelligence (BI).