Welcome to this blog post about the Systemic Organizational Systems Design – SOSD method, which is the third blog post in the series about Dissolution of Organizational Problems. Today’s blog post will go through some theory necessary for understanding why the actual SOSD method looks like it does, and of course also the method itself, which focus mainly on product development organizations. And with product development is meant the whole life cycle of product development, from ideas to maintenance, i.e., new product development and not only maintenance or some small enhancements of existing products to start with. There can be two different outputs from SOSD: one is an appropriate way of working for a product development organization in the cases there are existing methods for the context that are following the organizational principles. The other output proposes how to dissolve the organizational problems and their root causes, which depends not only on the context, but also on the domain the organization is operating in. This second output is actually valid for any human complex adaptive system, giving the root causes to the problems.
This blog post generally discusses the portfolio needed for all product development in bigger organizations, the strategical and tactical management within it and above it. It is done to grip the concept of SOSD, that it is all about the whole organization, and to make a systems design that takes care of the transdisciplinary complexity. It is especially done to be able to understand the method TDSD – Test-Driven Systems Design, needed when developing big software systems, see this blog post about TDSD. For other kind of product development of big systems, like waterfall and Lean Product Development, they will from start take care of the total organization, as well as the portfolio. The main focus is instead on the complexity of product development of an initiative, the operative part of an initiative, and the theory behind that. But as already stated, since all organizations are human complex adaptive systems, it means a top-down systems design of the way of working of the organization in order to reduce both complexity depending on for example the size and transdisciplinary complexity when making completely new or enhancing products. The systems design of course needs to be systemic, and that is what the name of SOSD implies. This means that we cannot start in the middle of the way of working, with the product development, and add the portfolio later. This is because the portfolio stands for planning and prioritization, which is on a strategic and tactical level, and then gives the operative responsibility to the product development. But, in order to be able to be responsible for structure, planning and prioritization, the portfolio also has responsibility for the money used for each initiative. This means that the portfolio will get most of the organization’s money in a product development company, even in a hardware company, where the portfolio gets much more money than the research organization.
In the same way we cannot start directly with the portfolio. The reason is that we cannot plan and prioritize until we know the way of working for the product development as part of the total way of working, which implies how our initiatives will be broken down in order to reduce the complexity for the product we are going to build. Unfortunately, many scaling frameworks for agile do not consider the complexity in either the product or the organization, and without consideration just divides the work and the organization into parts, a construction sometimes called value streams. This gives an enormous risk for sub-optimization of both the initiative and the organization. This sub-optimization is even worse than for projectized organization (see this blog post), which at least only sub-optimized on the organization, since the total scope of the project/product/initiative was not split. This believe that the whole can be divided into smaller parts as a first step without consideration of the whole, also leads to the thinking that making the top-levels including the portfolio can be done as the last step. But, since product development always has high transdisciplinary complexity that need to be reduced, especially new product development, this thinking is putting the transformation and the whole organization at an extremely big risk for failure.
Every organization need to prioritize which work that should be done and also to plan the work, finance the work etc., which is derived from the purpose of the organization. Planning and prioritization are valid for all organizations and will be strictly formal with a portfolio for big organizations and more informal in smaller organizations, where the initiatives are few. So, the reason for discussing the portfolio is therefore two-fold; we need to know the way of working for the product development first, until we can know the implementation of the portfolio, and also that we need to reduce the risk taking in many agile transformations at scale.
We start with the picture of the total of System Collaboration, which this blog book is all about, from the purpose to its fulfilment – SOSD.
As you can see above, SOSD is within the Dissolution of Problems part of System Collaboration, with a tight connection to Systemic Problem Picture Analysis – SPPA. The output from SOSD itself will be an appropriate way of working for the organization, which includes also structures like the line hierarchy and the virtual (cross-functional) delivery structures, where the delivering teams are operative and can be found on different levels in the latter structure.
If the organization needs help from scratch or there are too many root causes to dissolve, SOSD will, depending on many different parameters, propose what to think about regarding the portfolio structure, and the virtual structures, as well as propose an appropriate way of working for the actual product development; waterfall with projects, Lean Product Development, Agile and Test-Driven Systems Design – TDSD (under construction). Why these are chosen, depends on that they neatly handle all our Organizational Principles for their respective context and domain where they are appropriate, which we will explain further below. When an organization is redesigned from scratch, the recommendation is still to do a SPPA, since there can be root causes that can be easily fixed, making the organization more efficient to some degree right away.
If the organizational root causes of the organization should be dissolved instead, a proposal for how to fix them will be presented. As can be seen in the picture, all Organizational Principles valid for the context of Complex, Complicated or Clear, which in turn are generated from the purpose, need to be fulfilled. Moreover, other parameters of the purpose needed for our system as; security level, size of the system, size of the organization, length of the life cycle for the system, etc. are also very important input in order to achieve a flourishing organization. Depending on these different parameters, other necessary qualities of the system need to be taken into the equation, for example requirements traceability if our system starts to get big, which will be further discussed below.
This also leads to that it is difference between qualities and qualities, as well as qualities and non-functional requirements. All put together will give some different ways of working, as already presented, which actually is not so many as one can expect, even if every one of them can have some flavour depending on the organization. We can then see that the solution for the top level of big systems is kind of domain independent as well. We will also be able to understand that talking about emergent architecture and design, i.e., bottom-up software development, is easier said than done when it comes to big software systems, or many teams in a big organization that is working within the same system. Worth mention is also that for hardware systems, emergent architecture/design is not possible, so a proper systems design is needed, no matter the size of the system. The line between small and big systems depends on many parameters. But, a few teams and a total of maximum 50 persons, is where an organization many times is said to go from small to bigger, needing a formal setup with a line hierarchy and virtual delivery structures, a proper systems design and documented information. For big systems, built from scratch, it is seldom the uncertainty of WHAT the customer needs that is the hard part. Rather it is reducing the complexity on HOW to make the big product, which is the case for many service organizations, like banks, governments, insurance companies or retail. It is rather how to reduce the complexity of especially the backend part of the big system developed, all in order to avoid risk taking, by making a proper systems design. This also means that most of the requirements for big systems, both the functional and non-functional requirements, normally can be acquired early on, meaning that we do not need to take the risk with emergent architecture and emergent design. With that said, we can understand that systems design is a vital part when doing big systems, which we will come back to further down.
Different contexts give different solutions – example with people structures
Even before we have chosen the context, we need to look over the line hierarchy, which is a people structure (context independent) used to reduce the complexity for us humans and has been used for thousands of years, simply to keep track of all the people in the organization, see this blog post for more information. The line hierarchy is not only a people structure, but also responsible for the strategical and the tactical, and also stability regarding the development of competency, number of resources, career, education, guild, salary and bonus, etc., i.e., like a big resource pool.
We start with an example regarding the line hierarchy in production. For a clear context, like production, every unit, no matter abstraction level, know exactly what to deliver. This means that every unit themself and its manager, can deliver their purpose and over time they can improve its HOW they manufacture the WHAT. The unit shall be seen as a team which can be seen as a kind of virtual structure. With our principle of Respect of People, everybody in the team will grow in their learning by T-shaping within the team. To even further increase Respect of People, the team leader and the manager shall be different persons. The risk for suboptimization on the whole is low, since the WHAT on the whole is already set, they do not compete with other teams to deliver more WHAT, and do not need to interact for a new solution with other teams. The total virtual structure can be seen as one delivery team per group of people, and the result from each team are aggregated to the whole. The reason why this is possible is because the product development already has guaranteed that the parts are coherent and cohesive, when aggregated to the whole. The money needed for the organization, can also easily be retrieved and divided according to the line hierarchy and its units.
But, for product development, which is in the Complex domain of the Cynefin framework, the line hierarchy and the possible team structure presented above are by themselves not enough. This depends on the high probability for suboptimization with silo-KPIs on the managers on different levels in the line hierarchy, and is the reason for the need of a cross-functional virtual delivery structure. And having the manager responsible for also the team as is possible in production, will be counter-productive in product development, due that there will be KPIs on the managers as well as sub-optimising measurements that will compare teams. We of course also have the increased risk of Respect for people problem when having different opinions within the team, when the manager and team leader are the same person, mostly due to the complexity in product development.
Many organizations nowadays implementing agile at scale, have problems doing a proper implementation of the virtual structure, where the root cause is that there is an improper portfolio from start. But instead, the belief is that these problems instead will be solved if the line hierarchy is changed to fit the different team structures. Unfortunately, symptoms are impossible to solve, which leads to a sub-optimization that in fact goes backwards in our evolution to get rid of the sub-optimization. The root causes need to be solved, where the solution once again will be to introduce a virtual (cross-functional) delivery structure, which means that this “agile” organization need to be removed. Because, together with having one responsible for any initiative and to avoid the suboptimization when having only the line hierarchy, are the two main reasons why projects incorporated with a virtual cross-functional delivery structure, were introduced in the 1950s, see this blog post, for more information about when projects saw their first light.
Worth adding is also that in product development, the line hierarchy not only consists of capabilities for taking care of a predefined WHAT, but it also needs to consist of capabilities for HOW, to be able to make the systems design. As a result, we also must add additional silos taking care of this HOW, not only regarding systems design, but also regarding systems test. This is the top level of the initiative, and of course requires a top-level team that takes care about the wholeness of the initiative. This is especially missing on the top level when agile is scaled in a bottom-up transformation of big organizations, which also implies that if resources for HOW did exist, they will be removed before the organization even understands that they were still needed. The top-level can never be found inside the agile teams, more than in small organizations and small systems, since refactoring can be done gradually. We also need to be aware that the portfolio is another top-level requiring support functions, that never can be part of the teams and will be discussed further down.
When having virtual structures, these structures can be static or dynamic, which need to be considered, where normally hardware organizations have dynamic ones and software organizations tend to more and more go to the static ones, especially for agile software development. And as always there are pros and cons when having static or dynamic virtual structures. Dynamic virtual structures have built-in flexibility in setting up new initiatives, but will generate extra work in the portfolio to keep track of the initiatives and the money, and for management where we need to be vigilant with too much focus on resource allocation, which is sub-optimising per se. This is especially valid for big organizations doing waterfall development with waterfall for small initiatives, where the need of T-shaping is a necessity in order to be able to compete with smaller competitors. The administration waste, originating from having small percent of I-shaped employees participating in many different small projects is devastating for the efficiency and also to some degree for the effectiveness. This is also the reason for the difficulty to compete for big organizations making small projects (small systems), where an inefficiency of more than 50% is not unusual, see this blog post for examples and a deep-dive and into Project Administration waste and the theory behind.
It can be added that for smaller companies, there is normally not yet a line hierarchy due to the smallness, and the company itself can be considered as one whole virtual cross-functional delivery structure. The employees in a small company will work with many different functions, giving the right I-competence, T-competence and cross-functional broad competence needed for developing the product(s). Not only the organization itself gives less complexity, but the small products(s) also give less complexity. This means that there is a possibility to have less of documented information, less structure and order from start of an initiative. All this together means that more information can be in the head of the employees and that the architecture and design may be emergent, even though a lot of new functions then can dramatically reduce the life span of the product(s).
The need of a portfolio
Due to the need of virtual cross-functional delivery structures for product development in bigger organizations, there is therefore automatically a need of a portfolio for the initiatives for the organization as a whole. The portfolio is responsible for and keeping track of the thorough planning and prioritization of all the different on-going initiatives in the organization. From the road map, the portfolio gives the responsibility for operating the initiatives to the virtual structures. The initiatives started by the portfolio need to follow the portfolio strategy, that in turn ensures strategic alignment with company goals, derived from the purpose of the organization. With a portfolio, the executives, team leaders, team members, and stakeholders get an open and overall view of the initiatives run by the organization. This also includes how the initiatives fit into the organizations vision, mission and strategy, which in turn leads to insights into potential returns and what risks that can be seen. The need of a portfolio goes for both software and hardware organizations, where the latter also have a research organization, that normally is outside the portfolio management. This is because research is highly complex and therefore impossible to say what the result will be and when it comes. This makes research inappropriate for the portfolio and its initiatives, since initiatives are something promised to the customer and therefore must be less complex, see this blog post for a deep-dive. For smaller organizations it is normally enough with a portfolio handling function. It is worth to emphasize once again that the portfolio needs to be able to handle new product development, which is really the complex part of product development, already from start. With only maintenance and small functional enhancements in an existing product only, which is rather a clear context in an organizational point of view, the risk is high that the portfolio will not be fit for purpose, and only shovel around money. This is unfortunately common in transformations to agile at scale, which will be further discussed below.
All competence needed for developing big products cannot be hold within the team, T-shaped or I-shaped does not matter, even if that would be nice to have. The reasons are many, but some of them are; the wholeness is not within the parts, every individual does not think everything is fun to do, some competencies require a very long education, it is not possible to learn everything (brain overflow), and finally plan and prioritize the wholeness within the organization. This means that we need support functions, or centralized functions, with resources that is not full-time resources, with special education or individuals spanning over many initiatives, i.e., the wholeness within initiatives or within the whole organization. These are resources like, CM, release, systems testing, test coordinating, law, legal, risk & compliance, security, safety, enterprise architecture/design, etc. But as long as we understand that, it is only about planning, so we avoid that the teams use the resources at exactly the same time.
As said earlier, systems design is done in order to reduce the complexity, as well as the risk, when we later are integrating our parts to a united and well-functioning whole, or a coherent and cohesive whole. This means a structure way in order to reduce the whole system into its parts, often with the need of prototypes. Here is an attempt to explain the difficulty with systems design:
The specified requirements on the respective parts, to be able to fulfil their respective functionality, as well as their respective non-functionality, i.e., the parts’ interactions to become a unified and well-functioning whole, all in order to fulfil the specified functional and non-functional requirements on the whole, can never be set in advance. This is what is meant with transdisciplinary (integrative) complexity, we cannot specify the solution of the respective parts from start. Instead, we need to integrate the parts to a whole, and verify and validate the whole, in order to see if our assumption or hypothesis, the systems design regarding the parts, were correct. And regarding this transdisciplinary complexity, there is no difference between hardware and software. Reducing transdisciplinary complexity often means prototypes in order to gain new knowledge from the tests on the whole, until we have all the knowledge needed for developing our product. This is very different from disciplinary complexity, which is about digging down into a discipline in order find new science or new technology, and is therefore mostly a part of the way of working for hardware. This is normally called research, where we do not even know if we will find what we are looking for, and if we find it, we will not know when.
So, we can never fully specify the requirements for the parts in a highly complex context, since it is impossible to have that knowledge in advance. This is what Alicia Juarrero mentions as new cohesiveness. This is why all new product development, no matter domain, is highly complex. Simply, we can only achieve new knowledge at the integrations of the parts to a whole and including the tests of the whole, since we can never specify us out of high complexity. This is why prototypes are self-evident in hardware product development, an integral part for us humans since probably thousands of years in “hard” domains. If we are developing a completely new product, we need a lot of experimentation, but from well-reasoned hypotheses, to gain new knowledge, probably we have both disciplinary and transdisciplinary complexity. Changes on an existing product or variant of it, only need exploitation since it is only more like complicated, i.e., the results from one or two prototypes analysed by experts, will give us the needed knowledge so we can integrate to a unified and well-functioning whole. But, when we for example are combining monoliths to a common modular platform, we probably have all the disciplinary science and technology, but we have high transdisciplinary complexity, since we have never done the platform before. This is why systems design is key when we are developing products regarding software, hardware, new (architectures for) buildings, etc.
We start the systems design, by first analysing to come up with a hypothesis on how we can divide the unknown solution on the whole into parts, our first abstraction level. This is then repeated in an iterative manner. For every abstraction level, from the highest to the lowest level of appropriate granularity, the complexity is reduced by a systems design for every component on the abstraction level, which is also the start of the architecture of the product. The systems design is not only about dividing into components and What each component will do, but also most important, HOW they will interact. The latter step is extremely important and is where all the non-functional requirements on the product are taken care of, so that each component also derives its respective requirements from the outcome of this step. Systems design is all together very tricky and many iterations are most probably needed. Preferably, the system test development activities are also started, so that this blueprint, this architectural skeleton, can be verified as soon as possible. Without this latter step of HOW the components will interact, we will not get an architecture artifact, but instead a sketch artifact*, since we have not handled the non-functional requirements correct, see this blog post for a deep-dive.
A warning here is apt for agile software product development nowadays, where the risk is high, that this latter step is being neglected on the highest level, meaning we only get a sketch artifact. Especially if the organization is making a bottom-up transformation when going towards agile at scale and big systems, since the teams then will start working only on parts of the functional requirements of the total product. This means that these parts are only containers (like value streams) with functional requirements within a sketch, and not real software components within a real architecture of the total product. Often in agile development, the terms emergent architecture/design and refactoring are mentioned. But somewhere, depending on for example the size of a system (and that do not require a high level of security), there will be an invisible line between where refactoring can be done in order to restructure the architecture and the code, and where it cannot. As long as refactoring can be done, we can also talk about the possibility of an emergent architecture and emergent design, but when the system is too big, an emergent architecture and emergent design will exponentially increase the risk. And for big systems we most often know WHAT system we need to do, so why should we then even take this kind of risk on HOW to do our system. Without doing a proper systems design, the HOW, we also increase the risk exponentially regarding our ability for requirements traceability – RT.
RT in short means that we need to keep track of the initial ideas, via the requirements and the systems design, all the way down to the realisation of the system and the possibility to go backwards as well. We can clearly see that there will be levels of different kinds of artefacts, and we can also see the need of a top-down approach to keep track of these levels. RT is always important to consider regarding systems, especially big systems, due to their higher complexity, which in turn requires better overview, more structure and order. RT is even more important when a new system is developed from scratch. The more security requirements the system have, for example legal, risk and compliance in bank systems, and that goes for any size of the system, the importance of the RT rises as well.
The neglection of making a systems design, which in turn leads to an improper RT, is instead a top risk for not making the product right, and is for big systems never interchangeable with having flexibility to make the right product. This becomes even more obvious when looking at all aspects of achieving a good way of working by SOSD – Systemic Organizational Systems Design, see this blog post for further information about requirements traceability.
Purpose gives us other important parameters for our way of working
Now we will go through parts of the purpose that will give us valuable input to how our virtual delivery structure will look like, which in turn gives valuable information to what kind of initiatives the portfolio team, responsible for the portfolio, needs to be able to manage. The portfolio team is responsible for the strategical and tactical parts of the portfolio, and is mostly represented by higher level managers, since the line hierarchy is responsible for the strategy and tactics. The portfolio team plans the road map and prioritizes the initiative and hands over the operative responsibility to the virtual delivery structure.
It is very important that the portfolio can handle the most complex initiatives that the organization will have, otherwise the organization will run into huge problems sooner or later. This goes hand in hand with the agile example above, because if the bottom-up transformation starts with less complexity initiatives like maintenance or a small functionality lift in existing structures and test environment, and we adapt the portfolio thereafter, we will only divide the money into kind of autonomous parts. This problem is especially true for organizations that do not only fully understand complexity, or not taking care of dependencies and interdependencies, but also those organisations that only plans with only the length of the nose, which tries to use scaling agile frameworks. Without these kinds of understanding, the portfolio will more look like some kind of ATM, dividing money to parts with no connections at all, like people taking out money from the ATM. Instead, the portfolio always must take the most complex initiative into consideration, which means systems design, that give interconnected parts, which does not mean just to divide money to parts without connection or where the connections already is set, since that can only be done in a Clear context, where we know that the parts are coherent and cohesive. The highest risk for portfolio failure is organizations that come from a Clear context; banks, insurance companies, governments, and other service companies, since they are used to a portfolio that are more like an ATM machine, since they have been used to buy in the products.
First of all, we can understand from the purpose if it is software or hardware development, or a mix which of course also is possible, which is valuable information regarding maintenance setup. For hardware organizations, the maintenance is normally a separate organization that the development organization hands over to directly, or after a year or two after the customer release in order to correct the first faults in the hardware. The main reason is to free development resources, for next complex initiative to handle, leaving the only complicated released hardware with documented information to maintenance. With a high level of requirements traceability including baselines, the development can continue with next hardware variant any time, since it is separated from the current hardware release. Also, the customer connections are important, and they are also separate due to that every hardware shipped to the customer is unique, also combined with many other unique hardware at the different customers. For software organizations, we have normally only one software for multiple customers, and with some variants due to the continuation of next release, having development and maintenance in the same virtual structure, is the most suitable. The drawback with this setup is the risk of losing control over the maintenance cost, due that the stakeholders believe that they can choose more development and not listen to the teams. But high maintenance cost has always a tight connection to bad quality. This in combination with a superstition on refactoring from the teams also for big software systems, gives high risk for a running maintenance costs. By pushing the problems in front of us, makes us to become too reactive instead of proactive, giving us more and more problems. The root cause to this is most probably a bad or non-existing systems design from start of the initiative.
From the purpose, it is important to most of all consider the following for the products that are developed; safety criticality, security level, organization size, product size (a big system is normally backend heavy), life cycle, modular platform or not, sure what to deliver or not and time to market, since they all effect the way of working. Even if every initiative is unique, we can from the purpose and the initiatives we have in the road map, understand what we need to require on our way of working. As we already stated before, if the system is big, it gets too complex for us humans to get it right, which means at least these things; we lose our ability to refactor the code for our system, we lose our ability to understand when tests on the parts and different flows are not any longer representing tests on the whole, we lose our ability to track the bugs to the requirements on higher levels, and we also lose our ability to keep all information about the system and its parts in our heads and present it to colleagues only orally. All put together means that we need documented information and requirements traceability, which in turn means that we need a proper systems design for big systems. If we have safety critical systems, need security, or require a platform, then we always need structure and order, which means a systems design and requirements traceability also for smaller systems. Safety, security and platform, specifically requires system test specifications for the whole system, the subsystems, and even the sub-subsystems, all which are the results from the systems design(s), which make agile with emergent architecture/design more than inappropriate.
We can also add that big systems also have a tendency to have a long life-cycle, which makes the systems design even more important, so we not end up with a dead product after some years of adding on new functionality, which always will happen. A dead product is a product that no one wants to touch, because no one knows what will happen we make changes to it; new functionality or bug fixing. The life-cycle must also be considered for small systems and organizations, meaning that agile development with emergent architecture/design can easily become a Walk in the Dark.
Dissolving our problems
From the method SPPA, we will get the root causes, but if we now start to think of our root causes, we realise that we have a problem, since we cannot directly solve even them. Our only root causes are often built-in limitations within ourselves, science about us humans, or that activities need to be done in a certain order, or the that we sometimes need to understand complexity in order to be able to solve an activity, i.e., science for the activities. This means that we simply cannot solve the root causes we found regarding us humans and activities, since the science is already there and we simply just need to fulfil it. And this is exactly what Dr. Russell Ackoff meant when he talked about dissolve a problem, we need to redesign the system (organisation) instead. This is why understanding systems design is vital, not only for products, but for organizational design as well. And as you also realise, change the way of working in organisations is exactly what many methods have tried the last 70 years, but no one of the commercial ones has been successful in making flourishing organizations, which Ackoff calls manager’s panaceas, and which is described in this blog post. And as you can see in the example above, there are some organisational hiccups in Agile too.
So, what do these methods do wrong. Let’s try to explain it with a picture, here as a .pdf file, no analysis – no design – false synthesis. The way of working in the organization can never be built up of the wrong components, since we people are what we are, that is the grey boxes to the left, where a solution cannot be found, it is invalid. People can of course make mistakes and we have Murphy’s law, which for the former indirectly means that we always can make our processes clearer, but we people are never the wrong components. A red box means a failing organization and a green box means success.
The picture above is only the final picture. From start we thought our organization was doing well, with our system right, but actually we were in the Chaotic domain according to the Cynefin framework, so the initial start condition is a red explosion of chaos in the right top corner. In the Confused domain of the Cynefin framework is the place where we do our analysis.
As you can see in the picture, today’s methods and frameworks seldom analyse, and almost never deeply enough, an organisation’s problems (since they never even ask for the problems within the organization), and therefore cannot find the organisational root cause(s). This directly means that the methods and frameworks do not analyse themselves, meaning that the problems of the organization can still reman, and even worse is that the methods and frameworks can introduce more problems, without (further) notice. This means that an organization will be constantly on the edge to chaos, without really understanding it, see pictures in this blog post. And without finding the root cause(s), we have no new input for a systems design of our organization, that eliminates the root cause(s) and still fulfils all other organizational principles. We can therefore only make a false (random) synthesis (not an integration, only a kind of “aggregation” in this case, since no systems design is made) on chance for our new way of working in the organisation. This means we are once again making a system that is not working, but this time we know that it will not work, so we are from now on lost in the right bottom box. This means a Walk in the Dark, where we can loop for eternity, we will never leave the bottom right red box. This means that the top right box also will remain invalid (grey) forever, since we are trying to do the impossible, to solve symptoms, meaning that we can never reach it.
The only way to leave the chaos and instead successfully reach the top right corner, is to do according to the below pictures, by as in the first picture head for the root cause(s), the reason for an organisation’s problems. Here as a .pdf file, analysis with SPPA.
The picture above shows the same start conditions, we think our organization is in the top right corner, but actually we are in the Chaotic domain according to the Cynefin framework, with an explosion of chaos. In the Confused domain of the Cynefin framework is the place where we do our analysis, and this time we start with our proper Systemic Problem Picture Analysis- SPPA. This makes it possible to find the root cause(s), and we understand that our system was wrong and no mistakes had been done. So, now we make our systems design and synthesis (integration) to eliminate the root cause(s), and of course still fulfil all other organizational principles, and we successfully end up in the top right corner with our systemic solution of a new way of working. We end up with this picture, design with SOSD.
This is the only way to do it, to redesign the system by making a new systems design of our way of working, which instead will give us a sunny Walk in the Park.
Our adaptiveness as humans, makes our organisation a human complex adaptive system, which means that we will always get some degraded outcome from even a low-performing organization with many built-in problems. But do not forget, at the same time our adaptiveness can of course never solve our organisational problems (symptoms and consequences), because it is impossible to solve them. So, not even our adaptiveness can solve the symptoms, consequences and even the root causes, since the root causes in turn are limitations within ourselves or other science, so we need to change our organisation instead. This means that every time our organization does not fulfil our organizational principles, we have built-in symptoms and consequences, that only will get worse, if we try to solve the them directly.
As earlier stated, from the purpose of the organization we will get the context, often referred to as one of three domains in the Cynefin framework; Clear, Complicated or Complex. This context will give us the organizational principles that is specifically valid for this context. But all principles always need to be fulfilled for a flourishing organization, so we need to be vigilant for context shifts, or special cases occurring within in our current context. This can mean that we sometimes may need to apply methods from other contexts, which can mean other organizational principles to fulfil. With a continual use of SPPA, we will be able to be proactive and recognize context shifts early, even if they are rare. If they still occur, they are most probably only temporary.
Even though we now have acquired the theory behind SOSD, as well as some important aspects about product development for especially big systems, implementing a systemic dissolution for the root causes can sometimes be a bit tricky, depending on the impact of the changes. Here are therefore some examples of some common combination of root causes from waterfall with projects respective organizations that has transformed to agile at scale, and how to dissolve the root causes. This series of blog posts, can also be recommended; a deep-dive into Dr. Russell Ackoff’s “panaceas”, antisystemic methods that still are in use, but still having the same pain points. The reason why it is possible to give examples, are because there are not so many root causes. This means that even if the symptoms and consequences are different between two organizations doing product development with waterfall projects, the root causes are most probably the same. This is why the examples below are apt.
Common organizational root causes for big organizations doing waterfall with projects
Here is a file, pain points in silo organizations, with some big pain points in an organization with a line hierarchy doing the waterfall methodology with projects.
Before we are getting into the pain points, we can note the following, which is also what we should expect! We can see that there are no problems having a line hierarchy and also adding a virtual cross-functional delivery structure (projects) when the purpose is to do product development. The reason is that the line hierarchy is necessary for reducing the complexity when having many people working, with all the normal responsibilities of a line hierarchy. The line hierarchy takes care of all the problems that emanating from an unstructured and unordered bunch of people. To dissolve these problems, the line hierarchy is therefore a must, and is valid for all organizations, it is context independent. In product development we need to remove the delivery responsibility of a decided initiative (project) from the line hierarchy and introduce a virtual cross-functional delivery structure that takes care of the delivery, HOW to deliver it. The line hierarchy can now be seen as a dynamic resource pool, a flexible solution for getting the needed resources to an initiative (project). A resource pool can also be static as many times with agile at scale, but where we still need the line hierarchy to avoid sub-optimization as stated above. The cross-functional delivery structure, dynamic or static, is thus a necessity, the dissolution of the problems regarding sub-optimisation in the line hierarchy, where the sub-optimisation is about the lack of the planning and prioritization for the initiative (project). But, the management in the line hierarchy still have the responsibility for WHAT to deliver. And since there in big organizations will be more than one initiative, when need to plan and prioritize also these multiple initiatives, and that is why we need a portfolio, which is the responsibility of the management.
The pain point in the middle (left) gives common problems in hardware product development organizations doing waterfall with projects. Especially when the size of projects is reduced due to small competitors intruding on parts of the business, that is normally not the core business. The small companies can have an efficiency of 2-3 times higher than the big organization, mostly depending on the too low T-shape competence profile for the employees and the long chains of interactions between the employees (which also gives the Chinese Whispers effect, which means information forgotten and/or corrupted) at the big organization. The dissolution of these problems and their root cause is to make flatter organizations for the virtual cross-functional delivery structures, which will induce a lot of T-shaping for the employees, making the broader, and by this, it is possible to have a lot more full-time employees in small projects even in big organizations. This also means that the managers need to let go their requests on their employees only working with their respective managers area of competence responsibility. This will also give enormous benefits to big projects, since fewer part-time employees will be needed in the projects. Having more T-shaped employees therefore lead to less Project Administration waste, see this blog post for more information.
The two middle (right) pain points are about that there sometimes can be too many stakeholders and with too low agreement between them, which also can relate a lot to uncertainty of what product to make for the customer. This in turn gives late requirements or late start date, which in turn gives too little time to reduce the complexity in the product. The latter is also connected to the sometimes too big time-boxes in waterfall projects, unable to reduce the complexity fast enough even though prototypes are used. This is all about making a proper pre-study and systems design, and not rush into the execution phase too fast, which will easily happen when we lack the time that we really need. If we start too early with execution, but we have not really reduced the complexity, it will only take a longer time. Stakeholders that cannot agree due to uncertainty, means that we need more interactions with the customers, to understand what the really want, which requires iterations in order to acquire more knowledge about WHAT product to make. Here we need to add that waterfall with projects sometimes have too loose connection to the production, which is a very important stakeholder, since we need to be able to manufacture the product easily. Lean Product Development has since long figured that out, and a production team is always within the product development from start.
The right most pain point is two-fold; the inability to make platforms, which gives the possibility to make incremental deliveries of new “better” modules continually, as well as that a phase need to be completed before next phase can continue, which gives long lead times. One part of how to dissolve these problems, are as mentioned above to have a flat organization, to induce more T-shaping, but also the needed iterations for being able to make the platform by reducing the transdisciplinary (integrative) complexity, which is the strength of Lean Product Development, a phase that does not exist in the waterfall method.
The final problem and root cause, is the one to the most left in the bottom of the picture. The lack of an organizational problem-solving method inside the way of working, that solves organizational problems within the method or framework itself, goes for all methods and frameworks. Except Toyota’s Toyota Production System and Toyota Product Development System, which are built non only on the Kaizen pillar, but also on Toyota’s Jidoka pillar, where the latter means to automatically find problems and solve them as well.
Common organizational root causes for organizations transformed(ing) to agile at scale
Here is a file , pain points for agile at scale, with some big pain points in an organization that has transformed or is under transformation to agile at scale.
Continuous Improvement was a method used for organizations already three decades ago, and was brought up by Dr. Russel Ackoff as manager’s panaceas, methods that Ackoff considered antisystemic. Continuous Improvement is dissected in this blog post, where it is stated that a clear sub-optimization will be the effect when used in organizations, especially product development organizations. It is simply not possible to optimize the whole, by optimizing the parts, which is exactly what is happening when doing Continuous Improvement, for example with the PDCA method, commonly referred to when doing agile product development.
The word continuous in Continuous Improvement gives the impression that it is good to do improvements as fast as possible on any part in a system. This has led to tremendous misconceptions about the meaning of Continuous Improvement and when and how it can be performed. This can be seen especially in Agile product development using the Continuous Improvement from Toyota/Lean production, with their, close to autonomous, production processes, without consideration about the change of context; from a clear to a complex context. This together with a focus on queues in Agile, due to the misunderstanding of WIP inventories in Toyota/Lean Production, has many times led to a culture of non-planning. Planning is context free and is something our specie has known since ancient times, a necessity to avoid a mess. Since queues are only a symptom, neither WIP Limits, nor Process Cycle Efficiency calculations nor other measurements, nor any other method, can solve the problem with queues directly. Only taking care of the root causes to queues can and no Continuous Improvement in the world can make it better either, it will only sub-optimize on the whole. All problems are generated by non-existent mid- and long-term planning, and the focus on that teams with their Continuous Improvement for the respective team, that they can fix these problems (symptoms), can be seen exactly with two pain points in the middle of the picture.
But, as always, the smaller the system is, the easier it is and sometimes possible to take short-cuts. This means that for a few teams doing a small software system it is possible to understand when PDCA can be done and when it cannot be done, in the same way that we can have emergent architecture/design as well as very little documented information for small systems. But, when the software system is big, the complexity and all the interdependencies between the teams and the dependencies within the software code, makes it impossible to differ an organizational problem, from something that is working, but which we can make better. It is then no longer possible to judge when Continuous Improvements can be done or not.
The two pain points to the right in the picture, shows the problems occurring when a proper systems design is neglected. This leads to many problems with the requirements traceability, lack of documented information, lack of a proper architecture, many bugs, etc., frankly to a tremendous lot of problems, as we discussed earlier in the blog post. And if we do not have any proper systems test on the whole product as well, we will find our problems within the product very late, if we are not doing SPPA continually. Without SPPA the risk is instead that we are trying to solve symptoms, which leads to that all teams must plan at the same time (due to non-existing mid- and long-term planning), that everybody in the team need to know everything (due to non-existing mid- and long-term planning, as well as that the team need to do all the work, since no people working with the whole is left in the organization anymore, etc.). That all teams need to plan at the same time is interesting, since that is a context independent root cause (false cadence), valid both for production and product development, and apparently it cannot be the solution. It is obvious that this is an attempt to solve the symptom of non-existing mid- and long-term planning. And all symptom solving is sub-optimization which gives nasty unintended consequences, that can come anywhere and anytime. This sub-optimization is a really clear one, since false cadence on some parts of the organization really makes things worse on the total.
Unfortunately, we will also get problems with our flow efficiency, as well as the risk of making the product wrong, which are the top two pain points in the picture, which can take longer time for the organization to realize. These problems are still many times apparent and visible, which will start making a lot of friction within the organization between people that need to continue on the agile track, and people that are objective, stating that symptoms cannot be solved. That is a clear fact. But, by doing a SPPA, it will be clear that agile at scale have many built-in problems, that we need to hunt out.
The final problem and root cause, is the one to the most left in the bottom of the picture, which was already presented above, but worth to repeat. The lack of an organizational problem-solving method inside the way of working, that solves organizational problems within the method or framework itself, goes for all methods and frameworks. Except Toyota’s Toyota Production System and Toyota Product Development System, which are built non only on the Kaizen pillar, but also on Toyota’s Jidoka pillar, where the latter means to automatically find problems and solve them as well.
What way of working is most appropriate?
Here is a diagram with questions, achieving an appropriate way of working with SOSD, to show what way of working of already existing methods today that will be most appropriate depending on the fulfilment of the Organizational Principles in combination with fulfilling the purpose of the organization. As we can see, there is not yet a method or framework fulfilling the Organizational Principles good enough in many cases for software, especially big software systems, i.e., agile at scale is still a nut to crack. That is why Test-Driven Systems Design – TDSD both for software and the hybrid for hardware and software, is under construction.
Any building of an organization is really a prototyping resulting in a lot of feedback of the way of working until we get it right. But thanks to all experience already built-up during thousands of years, common sense and the organizational principles we are able, if we are willing, to make a really good organization already from start. So, in the diagram above, all natural feedback loops have been removed in order to not make the picture too busy and some things have been taking for granted.
As we understand from the diagram and all these parameters to consider and the fact that product development needs to follow all the Organizational Principles, narrows down the available ways of working for product development, to the ones taken as examples in the beginning of the blog post; waterfall projects with prototypes (mainly for hardware), Lean Product Development for hardware, Agile for small systems and Test-Driven Systems Design – TDSD (under construction) for hybrids and big software systems. Why these are chosen, and not others, depends a great deal on that they all rely on common sense, which can be read about in this blog post series about transforming with common sense. But, as can be understood from that, sometimes common sense is not enough, and in worst case commercial forces start trying to make dubious things, to get something to become a new common sense, to fit their purpose, see this blog post about how sometimes common sense is supressed and why. That is why we need the Organizational Principles as fact, so we can keep our common sense clear. Another good series that shows the way towards a good way of working is the implications our Organizational Principles, which can be seen in this blog post series. Putting common sense, the Organizational Principles and the implications together can be read about in this this blog post series about trembling steps towards our method, which also includes the very important scaling aspects. To understand more deeply about why these respective methods and how they neatly handle all our Organizational Principles for their respective context and domain where they are appropriate, and also their weaknesses, the following blog posts are recommended:
Lean Product Development; The pillars of Lean Thinking series, The New New Product Development Game article in our set of principles series, Lean Product Development’s anti-clogging system, WIP Inventories are great Flow Efficiency enablers in Lean Production.
Agile (only for small software systems); Sequential or parallel work – which is best? – part 4/6 – Agile way of working, WIP Limits in Agile development are sub-optimising our organisation, Flow efficiency – part 3/5 – Value Stream Mapping done on an Agile team’s sprint, Flow efficiency – part 4/5 – Agile’s calculation of flow, Process Cycle Efficiency, is sub-optimising our organisation, Are you stuck in a rabbit hole invaded by “experienced” artifacts?, Are you stuck in a rabbit hole of blinding new methods?, Are you stuck in a rabbit hole of Gig Bang integration testing?, Aggregation vs Integration vs False integration.
Test-Driven Systems Design – TDSD (under construction) (for big software systems and for software/hardware hybrids; for organizations which purpose gives the context and domain pointing at TDSD as the solution, the strong recommendation is to do the SPPA in order to understand the root causes. The dissolution of these root causes, with the examples mentioned earlier in this blog post, will anyway be the best approximation that can be achieved before TDSD is ready. This is due to the fact that no other method or framework for bigger software systems follow the Organizational Principles to a higher degree.
Already now I think you understand that a good way of working will be a true mix of the different methods presented, which becomes obvious when looking at the pain point pictures above, where there are no over-lapping pain points! Agile product development, Lean product development and the traditional organisations with projects and the waterfall method, the first good at the soft people side and the last good at the hard activity side and scaling, and the middle is quite good at both, but originates from a hardware context. And it does not matter if we build the organisation from scratch, or by having input from organisational problem-solving by SPPA. Both ways are possible when redesigning human complex adaptive systems, since they both follow the organizational principles. This means that they both are terrific ways to make an awesome organisation with the SOSD method.
And as you were promised from start, no rocket science in sight, only this:
Without changing our pattern of thought, we will not be able to solve the problems we created with our current patterns of thought.
– Albert Einstein
We simply need to change our pattern of thought about human complex adaptive systems, so we treat the way of working as ordered, but where the multiple interactions used for solving our initiatives with our way of working cannot be foreseen in advance. Together with the Organizational Principles that every way of working need to follow, this leads to that if our way of working is not following the Organizational Principles, we will have symptoms and consequences, that in turn of course will have root causes happened in the past. And since the Organizational Principles are science and the root causes frankly are non-fulfilled science, the root causes can only be dissolved. This means that the way of working (also includes people structures) needs to be redesigned, which is what Dissolution of (organizational) Problems is all about. It also means that if we try to solve symptoms and consequences which is impossible, it is not until then we have improperly intervened with our human complex adaptive system, leading to severe unintended consequences per se, or “a mess – a system of problems”, as Ackoff put it.
The next blog post in this series is a deep dive into the theory behind Dissolution of problems, the SPPA and SOSD methods, and will also start to bring up the difference between self-caused and non-self-caused problems. This difference is extremely important to understand, since the theory behind SPPA and SOSD further brings light on why they work and the necessity of using them for all organizational problem-solving and redesign of our organizations.
C u soon.
*A common worry today is that our organisations are viewed with the eyes of the engineering metaphor. This means that first the problem to be solved, is reduced into smaller parts to be solved and then the result is aggregated together to a whole, see this blog post by Dave Snowden for more details about this problematic thinking. The worry is at least twofold. One is that the reduction goes into too small parts because of wrong reasons a) to try to reduce complexity and b) “less is more”, making too small activities in the ordered domains, that results in both low flow efficiency and bad resource efficiency, which is both inefficient and ineffective, where b) also making us lose the Big Picture. The other is that aggregation is only possible when we know that the parts fit together, for example in production. Aggregation can only be used by production, which means the Obvious domain, and aggregation is a subset of integration. Even though we have done a proper systems design for our product development initiative, we do not know if the parts will fit together at our integration event, which is the reason for using prototypes. If we have not done a systems design in product development, we are actually doing a false integration, and not a proper integration, since we have not even tried to reduce the complexity.