Complexity has today become a buzzword. There are many articles and blog posts stating the need “to embrace complexity”, which is only correct if we have complexity. This means in short that we can neither only use methods taking care of complexity as a solution for everything, nor blame all organizational problems on complexity.
Another buzzword today is uncertainty. Maybe today is more uncertain than yesterday, and the future will be more uncertain than today, but we can definitely not blame all our problems in the organizations on the need of handling an increasingly extreme uncertainty. Blaming all problems on uncertainty is unfortunately a big problem today, even a greater problem than trying to treat everything as complex. For example, the automotive industry is now facing the need for autonomous vehicles, electrification and at the same time moving from products to services, which is judged as high uncertainty. But they seem to neglect that the inherit complexity, which requires the ability to find an integrated solution, i.e., a proper way of working, that can eliminate this inherit complexity.
But, before we digging further into this matter, we first of all need to define what is meant with uncertainty and complexity, which in System Collaboration is; uncertainty means not really knowing WHAT product to develop due to uncertain external customer needs outside our organization, and where complexity relates to developing the product right, the HOW, which is an internal need inside a product developing organisation. Uncertainty we for sure also have even when we know what product to develop, i.e., a complicated or clear context, which in System Collaboration is denoted as variability. Because, when we are developing our product, we can under estimate the time an activity takes, misjudge the complicatedness of an activity, make mistakes, have misunderstandings in our communication, and many other errors, which means that many activities can go wrong even with the best of intentions and proper planning. The higher complicatedness we have in our planned activities, the higher the variability will be. The variability when producing or developing a product, is the reason for having margins in the planning, since we can never in advance know when things go wrong, but over time we will get better and better in our estimations. We simply know that there always will be errors in our estimations, and margins have been used by mankind most probably for more than thousands of years. It is worth noting that complexity means that we do not have the knowledge at all for example about an activity, and because we have never found the solution to this activity or a similar one before, there is simply nothing that can vary. Therefore, System Collaboration does not mix complexity and variability, i.e., variability is used solely for activities in a complicated or clear context, and are the cause for the needed margins in time plans; by reducing the degree of utilization, or adding resources. This also means that if the only problem is that the margin is too low, we can just add resources or add over time, as too low margin is the root cause (and then we of course need to learn this to our next planning 🙂 ). A reminder is that if we are not planning at all, or we are not reducing the transdisciplinary complexity properly, we will be extremely reactive due to all generated organizational symptoms originating from these two root causes. Proper planning includes also considering the whole system from start, for multiple length of “terms”, will make use proactive instead of reactive, since we will “see” things that will happen sometimes in the future, even if we cannot say exactly when they occur. Planning without considering the whole system we are building, will only lead to a catastrophic failure for the system, since the transdisciplinary complexity/ complicatedness has not been eliminated. Having organizational symptoms shall not be confused with variability problems due to too small margin in the time plan, since reducing the degree of utilization or adding more resources are never the answer. Symptoms are always impossible to directly solve, and will always give more reactiveness if we try to solve them. It is neither meaningful to talk about variability if we are never planning, since we never learn how long time it takes to solve different types of activities. Since organizational problems normally are handled at meetings, a clear indicator of having unsolved root causes, are a continuously increase of the number of meetings. In order to understand what symptoms that are in play when having organizational problems, it is always necessary to make a problem picture analysis, with help of SPPA.
It is also important to note that the terms in VUCA; Volatility, Uncertainty, Complexity and Ambiguity all points at the environment, i.e., outside an organization, which means that System Collaboration label them all as uncertainty. This is an important distinction, since the aim of System Collaboration is to solve organizational problems and find ways of working (the HOW) that eliminates these problems. In order to do that, it is always important to distinguish between problems that are due to external factors, and which problems that are internally caused. The external factors are related to the organizational strategy, (the high-level WHAT), where in turn the internal problems are due to a mal-functioning way of working (the HOW), having built-in root causes that generates many symptoms within the organization.
Different domains can have very different uncertainty in what to develop, where software normally have much higher uncertainty than hardware when starting an initiative, due to the fact that software is a recently added domain compared to hardware, as well as the greater uncertainty in the customer need. But, within a complex context, no matter domain, we can always have high complexity, no matter if the uncertainty of WHAT to deliver, can differ between the domains within the complex context. For example, in product development there is no difference regarding transdisciplinary complexity/ complicatedness between the domains of hardware and software when making a novel/new project or initiative, since making a novel/new product or system, always is very complex, no matter if the domain is hardware, software, or hybrids. When eliminating the disciplinary complexity in hardware, this is always done in the research phase, proceeding the phase where we starting the project or initiative. This is rather natural for us, since a research phase cannot be planned in time, due to the waste amount of new knowledge that need to be gained. Unfortunately, the elimination needed for transdisciplinary complexity is not treated as research in any software development method of today, but for hardware we have Toyota´s Product Development taking care of that. In software neither the transdisciplinary complicatedness is eliminated, giving a big bang for waterfall or gig bang for scaled agile in the end of the project or initiative, since testing the wholeness is not done until the end. This is taken care of by the TDSD method, that is valid for software, (hardware) and hybrids, by planning that this transdisciplinary research need to be done first, and properly of course, giving that many parallel activities can be performed simultaneously.
It is worth to add, that the uncertainty level we have from start in what product to do, when making an initiative, will be the same until we have delivered the product or system to the customer, even if we through the initiative has reduced the transdisciplinary complexity/ complicatedness to a minimum, when we start to produce and deliver the product or system to the customers. That means that the uncertainty level is context independent, while the level of complexity always is context dependent, and where System Collaboration only talks about variability for plannable activities, i.e., for activities that are complicated or clear.
Now it is time to dig into the details of uncertainty and complexity, to be able to understand the solutions, the HOW, to able to reduce each of them.
Uncertainty – do the right things (products/systems)
We first start to dissect uncertainty, since there is a big difference in the level of uncertainty about what product our customers want, where the uncertainty often depends on the domain.
Of course, it would be great if we can have a “one size fits all”-way of working to take care about uncertainty, but if we use our common sense, we understand that is not realistic. And if we think about it a bit more, we have situations which are very different: 1) we have low probability to know if the customers even want the product, we are developing 2) we have quite high probability that our customers want it, which can be our next generation of our popular product, or 3) our customers (users) do not have an option. So, we have the span from the former that has high uncertainty, via the middle one that has low uncertainty, to the latter one with no uncertainty in what the customer wants. The former one really need to have loops of early feedback from the functionality to understand if the customers want the product in the end or not. Of course, also the two latter ones need early interactions with the customers, in order to get fast feedback, since we still want to make an awesome user-friendly product. But these are different feedback loops, so can we separate the more exact WHAT to do (the right product) in the UX, from the HOW to make (the product right) the backend parts, we will give ourselves an extreme advantage. This is also extremely important if we want to have speed in our development process when developing new functionality, especially smaller functionality. Because, if we can differentiate functionality that will make our product go through re-certification or not, we can shorten the timeline from months to hours without re-certification. This also goes for some of the bug corrections. So, by being systematic in our development process from start that give us speed (as well as other qualities; modularity, serviceability, quality, etc.) in our development process in the normal case, also gives us further advantages when adding functionality or correcting bugs.
In a world where all is claimed to have high uncertainty, it will be impossible to predict the exact date for a delivery. This has led to that in the agile world, planning has really become a non-mantra. Planning has almost become something ugly.
But planning is not synonymous with only time planning, it is so much more than just the time plan. Because, planning also gives us our future needs, that we need in order to be able to deliver at all, even though we cannot specify exactly when we will use our needs. The later we will understand our needs, the more reactive we will be. This means that if we only refer to the world as always having high uncertainty, and therefore cannot be planned, we will become reactive compared to traditional waterfall way of working or Lean product development. So, to throw out all babies with the bath water is never a good idea, not this time either. And when we look at our examples with high and low uncertainty above, the first one makes time planning very difficult. But still, even a short-term planning gives us indications of our needs, meaning we will get less reactive. The two latter ones after the complexity have been eliminated, are also the ones where we quite straightforward are able to do a reasonably good planning for, to be able to understand our future needs.
Another deficiency related to insufficient planning, and that will increase our organizational problems, is when we are using some of the agile scaling frameworks that use methods originating from a production context. They often divide the releases into increments of functionality as small-scale agile does, which many times means that the details for the middle-term and long-term planning is not done. In turn this means that it is very high risk that every increment starts with a planning from zero, as the memory of the planning with all needs, hinders and risks from the last increment, has been reset. This kind of non-long-term-planning gives a very reactive behaviour of needs through the whole organisation; resources, competences, education, technology, etc., which of course gives a bad flow, which in the end means late/missed releases, frankly low efficiency. Not to mention, that the very same needs can be brought up again and again at every planning of a new increment. This differ enormously from small-scale agile where the teams, that also are the whole organisation, have the Big Picture and all needs in reach, as well as that small products have a small Big Picture.
Complexity – do the things (products/systems) right
Despite the statement that we need “to embrace complexity”, we do neither reduce the transdisciplinary complexity, nor the transdisciplinary complicatedness enough product development at scale, especially for software, agile or not. The reason for this is that, transdisciplinary complexity/ complicatedness, has a completely other feedback loop than uncertainty. The feedback loop for transdisciplinary complexity/ complicatedness we of course also need to close as soon as possible, so we are sure that we are developing our product right. The bigger system product we have, the more complex our solution to our system product will be to find and therefore also time-consuming, so the longer the feedback loop will be. This automatically gives that it will be more difficult to use incremental methods for adding more and more functionality (as well as non-functional requirements) to the customer, that was mentioned above as a solution for the feedback loop of uncertainty. The reason is that the systems design is how we reduce the complexity for our system product, to be able to understand which parts we need to be able to integrate them to a coherent and cohesive whole. The deductions made, to understand the necessity of systems design, to be able to reduce the transdisciplinary complexity/complicatedness, can be find in this article.
Due to the high complexity when making new products, we can neither specify the perfect systems design even if we have all requirements, where the non-functional requirements are vital for the systems design. This means that we need early feedback also for the systems design, where we can think about an endoskeleton of the systems design, which is completely different feedback compare to the early feedback from the customers. Without this early feedback of the coherence and cohesiveness of the total system, we are risking years of work to achieve a big product that may not even work, since we did not develop the product right due to ignorance of the systems design. This means that for uncertainty we need early (rather just-in-time) feedback from the customers that we are building the right product, and for complexity, we need early (rather just-in-time) feedback from our own system, so we are building our product right.
Let us now look into how we with new deductions, can divide uncertainty regarding what product to make, from the complexity in how to make the product right. If the initiatives are new big systems or are bigger enhancements of existing big systems, we always need to iteratively make the systems design or make appropriate updates to the existing one. Since we in big systems most certainly know what product or system to do, the backend can simply be separated from the frontend with an UX component (or any industrial design), which means that we are separating the two feedback loops for uncertainty (right product) and complexity (product right). If our collaboration with the customer gives the opportunity for new functionality, we have most probably iterated the enhancements many times by using other tools with very short feedback loops, like pen and paper, power point, etc., before implementing the code directly in the UX component. Either way, having a new system or enhancements, by using a UX component connected to the backend through APIs, we can be systematic and structured when we are planning our work for backend and frontend. There is therefore no need for rush, or for sending half-finished functionality to the customer or end user. This is especially valid for the backend parts of any initiative, since it is where the main part of the difficulties with setting a proper systems design are. By separating this backend work from the frontend work, i.e., using a UX component in the architecture for the frontend, we can be systematic and reduce the transdisciplinary/ complicatedness complexity by explorations of the systems design for backend, at the same time as we together with the customer and end users can explore and develop an awesome user experience. And of course, frontend also to some extent will give input to the systems design, to have the full transdisciplinary view from start. By having a UX component, we most of all, to a great deal, have separated the two different feedback loops, the one for reducing the transdisciplinary/ complicatedness complexity by iteratively making the systems design for the backend, and the feedback loop for the uncertainty on how to fulfil the customer needs. This is also valid for any domain in product development, where hardware would consider it as industrial design, the appearance and first touch of the product. We can from this also conclude that if we are going to make a big product, but we do not have a clue regarding what the customer need, that is the only time that an MVP (Minimum Viable Product) should ever be considered. Remember though, that not knowing what big product or system to do for the customer, fortunately is a very odd case, and adding to that its extremely expensive initial costs, it should really be reconsidered.
But, let us also look into why an agile way of working sometimes can work out very well in a small company. In a small company, with one product, developed by a few agile teams, they will for sure have control of their needs due to the short distance (both hierarchical and physical, even though we can never forget about Conway’s law) between the teams and the product owner, they are really sitting in the same boat. And they must have control, because there is no other one to ask, they need to plan and fulfil their own needs themselves. It is worth mentioning that the T-shape of these teams are high (and need to be since they are the only ones in this small company). They also need to have the overall Big Picture of the product, and they have themselves done the systems design to some extent in order to reduce the complexity in a suitable way, with architecture, design and everything regarding test as output. If this small company find themselves to be too reactive, they will take care of that together, by finetuning their way of working, and refactor both the structure of the code, and the code itself.
We stated earlier that in many agile scaling frameworks, the functionality is planned increment per increment, often only at the start of the increment. For the complexity side this means that the requirements for the total product is seldom considered, leading to that proper systems design cannot be done from the beginning. This means a very high risk that, especially all the non-functional requirements, have not been part of the systems design, which means that the transdisciplinary complexity/ complicatedness have not been reduced properly. Actually, this means that the parts are not designed to integrate to a unified whole, even if they fit together like the pieces in a jigsaw puzzle. This also means that the test environment activities for the whole product will not start, due to that this aggregation thinking in itself, leads to a believe that all the parts instead can have built-in quality. This is an erroneous thinking which is leading to an extreme risk taking; first we not reduce the complexity properly, then we do not even plan for testing the whole product, or the big parts of it, so we could be able to find our blunder, until very, very late, maybe years later.
This in turn means, that regarding quality issues on the wholeness, the whole product, no one can see them coming, since the teams add tests that only test their added functionality that is incrementally implemented. In worst case, quality issues will not be until the release in production, since no one has been responsible for the whole. This can be avoided, preferably with mocked components on the whole, to be able to start early system tests in order to close the “develop the product right” feedback loop as early as possible. As we can see, it is not only bad planning that make us reactive, because if we omit the systems design, we will later, probably much later, have big problems too.
As we can see in this article, uncertainty and complexity are by themselves, tricky subjects, and easily made erroneously intertwined as well. Sometimes it can be hard to understand the difference between problems from our own way of working that are not due to the initiative itself, and normal problems that always will occur in any initiative. We always need to be flexible due to uncertainty of the latter, and are able to adapt to the new situation, but for the former, we cannot just be comfortably happy that we found a problem to solve and that we can learn from.
Of course, there are many unexpected things that can happen when we are not detangling our mess of problems, where reactiveness is one, which is deep-dived in this blog post.
If we put uncertainty and complexity together, it results in something interesting, but also quite funny, and that is that the complaints on the waterfall way of working in software, was that the output was the wrong product, but it was (maybe) developed correct. But, now with the agile scaling frameworks, it is the other way; we (maybe) develop the right product, but we develop it wrong. It becomes therefore rather clear, that by throwing away all old way of workings, it means that we did not learn anything from what we have done in the past, which never is a wanted mindset, especially not in agile.