Uncertainty has today become a buzzword. Maybe today is more uncertain than yesterday, and the future will be more uncertain than today, but we can definitely not blame all our problems in the organizations on the need of handling an increasingly extreme uncertainty. Another buzzword today is complexity, where many people states that we need “to embrace complexity”, which is only correct if we have complexity. But unfortunately, we, despite this statement, do neither reduce the transdisciplinary complexity, nor the transdisciplinary complicatedness enough in product development at scale, especially for software, agile or not.
First of all, we need define what is meant with uncertainty and complexity in System Collaboration, where; except from the normal meaning about uncertainty relates to when not really knowing what is the right product to develop, due to needs outside the organisation, and complexity that relates to developing the product right which is a need inside the organisation (and product). Uncertainty we for sure also have even when we know what product to develop. When we developing our product, we can under estimate the time an activity takes, misjudge the complexity and complicatedness of an activity, make mistakes, communication errors, which means that many activities can go wrong even with the best of intentions and planning.
Different domains, can have very different uncertainty, where software normally have much higher uncertainty than hardware when starting an initiative. But, within a context and its different domains, we always have the same complexity, no matter if the uncertainty can differ between the domains within the context. For example, in product development there is no difference regarding transdisciplinary complexity/ complicatedness between the domains when making a new initiative (we have passed the disciplinary research phase present only for hardware), where making a new product or system, always is very complex, no matter if the domain is hardware, software, or hybrids. It is worth to add, that the uncertainty level we have from start when making an initiative, will be the same until we have delivered the product or system to the customer, even if we through the initiative has reduced the transdisciplinary complexity/ complicatedness to a minimum, when we start to produce and deliver the product or system to the customers. That means that the uncertainty level is context independent, while the level of complexity always is context dependent.
Now it is time to dig into the details of uncertainty and complexity, to be able to understand the solutions, the HOW, to able to reduce each of them.
Uncertainty – do the right things (products/systems)
We first start to dissect uncertainty, since there is a big difference in the level of uncertainty about what product our customers want, where the uncertainty often depends on the domain.
Of course, it would be great if we can have a “one size fits all”-way of working to take care about uncertainty, but if we use our common sense, we understand that is not realistic. And if we think about it a bit, we have situations which are very different: 1) we have low probability to know if the customers even want the product, we are developing 2) we have quite high probability that our customers want it, which can be our next generation of our popular product, or 3) our customers (users) do not have an option. So, we have the span from the former that has high uncertainty, via the middle one that has low uncertainty, to the latter one with no uncertainty in what the customer wants. The former one really need to have loops of early feedback from the functionality to understand if the customers want the product in the end or not. Of course, also the two latter ones need early interactions with the customers, in order to get fast feedback, since we still want to make an awesome user-friendly product, but can we separate that, we will give ourselves an extreme advantage.
In a world that is uncertain, being able to predict the exact date for a delivery, is many times impossible. This has led to that in the agile world, planning has really become a non-mantra. Planning has almost become something ugly.
But planning is not synonymous with only time planning, it is so much more than just the time plan. Because, planning also gives us our future needs, that we need in order to be able to deliver at all, even though we cannot specify exactly when we will use our needs. The later we will understand our needs, the more reactive we will be. This means that if we only refer to the world as always having high uncertainty, and therefore cannot be planned, we will become reactive compared to traditional waterfall way of working or Lean product development. So, to throw out all babies with the bath water is never a good idea, not this time either. And when we look at our examples with high and low uncertainty above, the former one makes time planning very difficult. But still, even a short-term planning gives us indications of our needs, meaning we will get less reactive. The two latter ones, are also the ones that we quite straightforward are able to do a reasonably good planning for, to be able to understand our future needs.
Another deficiency related to insufficient planning, and that will increase our organizational problems, is when we are using some of the agile scaling frameworks that use methods originating from a production context. They often divide the releases into increments of functionality as small-scale agile does, which many times means that the details for the middle-term and long-term planning is not done. In turn this means that it is very high risk that every increment starts with a planning from zero, as the memory of the planning with all needs, hinders and risks from the last increment, has been reset. This kind of non-long-term-planning gives a very reactive behaviour of needs through the whole organisation; resources, competences, education, technology, etc., which of course gives a bad flow, which in the end means late/missed releases, frankly low efficiency. Not to mention, that the very same needs can be brought up again and again at every planning of a new increment. This differ enormously from small-scale agile where the teams, that also are the whole organisation, have the Big Picture and all needs in reach, as well as that small products have a small Big Picture.
Complexity – do the things (products/systems) right
Complexity, and in this case, transdisciplinary complexity/ complicatedness, has a completely other feedback loop than uncertainty. The feedback loop for transdisciplinary complexity/ complicatedness we of course also need to close as soon as possible, so we are sure that we are developing our product right. The bigger system product we have, the more complex our solution to our system product will be to find, and the longer the feedback loop will be. This automatically gives that it will be more difficult to use incremental methods for adding more and more functionality (as well as non-functional requirements) to the customer, that was mentioned above as a solution for the feedback loop of uncertainty. The reason is that the systems design is how we reduce the complexity for our system product, to be able to understand which parts we need to be able to integrate them to a coherent and cohesive whole. The deductions made, to understand the necessity of systems design, to be able to reduce the transdisciplinary complexity/complicatedness, can be find in this article.
Due to the high complexity when making new products, we can neither specify the perfect systems design even if we have all requirements, where the non-functional requirements are vital for the systems design. This means that we need early feedback also for the systems design, where we can think about an endoskeleton of the systems design, which is completely different feedback compare to the early feedback from the customers. Without this early feedback of the coherence and cohesiveness of the total system, we are risking years of work to achieve a big product that may not even work, since we did not develop the product right due to ignorance of the systems design. This means that for uncertainty we need early (rather just-in-time) feedback from the customers that we are building the right product, and for complexity, we need early (rather just-in-time) feedback from our own system, so we are building our product right.
Let us now look into how we with new deductions, can divide uncertainty regarding what product to make, from the complexity in how to make the product right. If the initiatives are new big systems or are bigger enhancements of existing big systems, we always need to iteratively make the systems design or make appropriate updates to the existing one. Since we in big systems most certainly know what product or system to do, the backend can simply be separated from the frontend with an UX component (or any industrial design), which means that we are separating the two feedback loops for uncertainty (right product) and complexity (product right). If our collaboration with the customer gives the opportunity for new functionality, we have most probably iterated the enhancements many times by using other tools with very short feedback loops, like pen and paper, power point, etc., before implementing the code directly in the UX component. Either way, having a new system or enhancements, by using a UX component, we can be systematic and structured when we are planning our work for backend and frontend. There is therefore no need for rush, or for sending half-finished functionality to the customer or end user. This is especially valid for the backend parts of any initiative, since it is where the main part of the difficulties with setting a proper systems design are. By separating this backend work from the frontend work, i.e., using a UX component in the architecture for the frontend, we can be systematic and reduce the transdisciplinary/ complicatedness complexity by explorations of the systems design for backend, at the same time as we together with the customer and end users can explore and develop an awesome user experience. And of course, frontend also to some extent will give input to the systems design, to have the full transdisciplinary view from start. By having a UX component, we most of all, to a great deal, have separated the two different feedback loops, the one for reducing the transdisciplinary/ complicatedness complexity by iteratively making the systems design for the backend, and the feedback loop for the uncertainty on how to fulfil the customer needs. This is also valid for any domain in product development, where hardware would consider it as industrial design, the appearance of the product. We can from this also conclude that if we are going to make a big product, but we do not have a clue regarding what the customer need, that is the only time that an MVP (Minimum Viable Product) should ever be considered. Remember though, that not knowing what big product or system to do for the customer, fortunately is a very odd case, and that through its extremely expensive initial costs, really should be reconsidered.
But, let us also look into why an agile way of working sometimes can work out very well in a small company. In a small company, with one product, developed by a few agile teams, they will for sure have control of their needs due to the short distance (both hierarchical and physical, even though we can never forget about Conway’s law) between the teams and the product owner, they are really sitting in the same boat. And they must have control, because there is no other one to ask, they need to plan and fulfil their own needs themselves. It is worth mentioning that the T-shape of these teams are high (and need to be since they are the only ones in this small company). They also need to have the overall Big Picture of the product, and they have themselves done the systems design to some extent in order to reduce the complexity in a suitable way, with architecture, design and everything regarding test as output. If this small company find themselves to be too reactive, they will take care of that together, by finetuning their way of working, and refactor both the structure of the code, and the code itself.
We stated earlier that in many agile scaling frameworks, the functionality is planned increment per increment, often only at the start of the increment. For the complexity side this means that the requirements for the total product is seldom considered, leading to that proper systems design cannot be done from the beginning. This means a very high risk that, especially all the non-functional requirements, have not been part of the systems design, which means that the transdisciplinary complexity/ complicatedness have not been reduced properly. Actually, this means that the parts are not designed to integrate to a unified whole, even if they fit together like the pieces in a jigsaw puzzle. This also means that the test environment activities for the whole product will not start, due to that this aggregation thinking in itself, leads to a believe that all the parts instead can have built-in quality. This is an erroneous thinking which is leading to an extreme risk taking; first we not reduce the complexity properly, then we do not even plan for testing the whole product, or the big parts of it, so we could be able to find our blunder, until very, very late, maybe years later.
This in turn means, that regarding quality issues on the wholeness, the whole product, no one can see them coming, since the teams add tests that only test their added functionality that is incrementally implemented. In worst case, quality issues will not be until the release in production, since no one has been responsible for the whole. This can be avoided, preferably with mocked components on the whole, to be able to start early system tests in order to close the “develop the product right” feedback loop as early as possible. As we can see, it is not only bad planning that make us reactive, because if we omit the systems design, we will later, probably much later, have big problems too.
As we can see in this article, uncertainty and complexity are by themselves, tricky subjects, and easily made erroneously intertwined as well. Sometimes it can be hard to understand the difference between problems from our own way of working that are not due to the initiative itself, and normal problems that always will occur in any initiative. We always need to be flexible due to uncertainty of the latter, and are able to adapt to the new situation, but for the former, we cannot just be comfortably happy that we found a problem to solve and that we can learn from.
Of course, there are many unexpected things that can happen when we are not detangling our mess of problems, where reactiveness is one, which is deep-dived in this blog post.
If we put uncertainty and complexity together, it results in something interesting, but also quite funny, and that is that the complaints on the waterfall way of working in software, was that the output was the wrong product, but it was (maybe) developed correct. But, now with the agile scaling frameworks, it is the other way; we (maybe) develop the right product, but we develop it wrong. It becomes therefore rather clear, that by throwing away all old way of workings, it means that we did not learn anything from what we have done in the past, which never is a wanted mindset, especially not in agile.