Are you stuck in a rabbit hole invaded by “experienced” artifacts?

This blog post will be about tacit vs explicit knowledge, the difference between them, and why it is a necessity that we are attentive to their difference.

But first we start with a definition of the word artifact. An artifact in product development, software or hardware, is a by-product* to the actual development; intermediate artifacts like a backlog, a prototype, a user story and input/output artifacts like a system architecture, requirements, etc. Artifacts in product development are intentionally used since they are a necessity for us humans, and can be seen as a way to reduce complexity or keeping track of things to remember, all done to match our cognitive ability. Artifacts are explicit knowledge, both regarding that we can read about them as parts of methods and frameworks, and also when an artifact is updated with information during its usage.

Let us take an example with architecture. The word architecture is more than 2000 years old. Its meaning today, is both the process and the product (the artifact) of designing and building, as distinguished from the skills associated with construction [1]. An architecture as a product (input/output artifact) can be an architecture for as different domains as a building or a software product and it is achieved through the process of architecture. In product development for both hardware and software we would call this process system design/engineering, a transdisciplinary process where the actual complexity is reduced, resulting in the architecture artifact**.

Tacit knowledge is such as skills and experience [2], which can be very hard or impossible to explicitly transfer to other persons, such as riding a bicycle, playing a musical instrument, or to do a systems design. It is also important to add that to acquire tacit knowledge for a method or framework and adding that to the CV, means to be part of the whole process until the method or framework actually is fully implemented and proven to be working as well. This means that we cannot just be part of the start-up phase ten times, and then in the CV state that we have skills and experience of the whole process, and definitely not state anything about if the method or framework would have worked at all. Explicit knowledge on the other hand is knowledge that easily can be transferred to other people [3], and can mostly be stored in different media. Explicit knowledge is often seen as complementary to tacit knowledge. In our example with the architecture above, the architecture process is where tacit knowledge is really required, resulting in the architecture product (the artifact). That means that just because we read about the artifact, does not mean that we understand the process behind it, since that still requires skills and experience. The need to spread tacit knowledge around the organisation, was one of the main conclusions in the article “The New New Product Development Game” from 1986 [4], which the authors referred to as “organizational transfer of learning”***. Only having Lessons Learned from earlier product development (projects), or joining educations, is simply not good enough, since that is only about artifacts, i.e. explicit knowledge.

The problem arises when we think that we can get rid of the tacit knowledge, peoples’ skills and experience, just by documenting it so we get explicit knowledge, and we are fine we that. If this was true, it would mean that we could attend a course, and the next day we could teach other people that course, since the explicit knowledge could just keep going on. And just because we teach others, we are suddenly referring to ourselves as experts. As you understand that is only silliness. Dave Snowden often refers to this erroneous thinking, since knowledge management already in the late 1990s failed to show that tacit knowledge was available for other people after documentation [5]. Even more troubling is if the explicit knowledge actually is wrong or important parts are missing. Due to the lack of tacit knowledge of the teacher only learnt from the explicit knowledge, it means that many people will be educated, before the incorrectness even is revealed. This is especially treacherous in new product development of large system products, that can have very long feedback cycles of the total system. And here is where the artifacts come into play, since artifacts are only explicit knowledge, that we read about for example in product development in a method or framework. This means that we can never only rely on the artifacts instead of the problem we actually want to solve. If we do so, we are unfortunately starting to treat the artifacts as tacit knowledge, a one size fits all structure of artifacts, building up to a process, that will solve any problem for us. This means that we will be totally blind when a new way of working is required, here are some different situations:

  • context shift, when we go from an easy context to a complex context
  • scaling, when we go from small systems to big systems.
  • a shift to a lower uncertainty, when we know the customer need

Instead, we many times need transdisciplinary tacit knowledge in order to reduce the complexity of our problem, which makes it easy to see what artifacts that we really need.

A very interesting and apt example, is a systematic case study that was made in Jan 2013 [6], with the aim to “develop an artifact model for agile development”. This was done by searching on internet for publications consisting of typical terms of agile development combined with the term artifact, which resulted in hundred found publications. The search interval was from the date of the Agile Manifesto 2001 up to Sep 2012. The developed artifact model, with its 19 commonly used artifacts and their relations, describes very well the process of agile development, where Scrum, XP and Kanban were among the top five processes found. The study also states that the most common practices are TDD (Test-Driven Development) and Refactoring****. Going back to the architecture artifact discussed above, it is missing in the artifact model. But this is not strange at all, if we refer to the Agile manifesto principle number 11; “The best architectures, requirements, and designs emerge from self-organizing teams.”, in combination with the Agile Manifesto value “Working software over comprehensive documentation”. At the same time, it is neither strange that refactoring then is one of the most common practices in agile software development, since without an architecture artifact, it means low focus on the architecture from start and also during the agile development, meaning that the software code needs to be continually restructured. The need of refactoring is especially valid for agile development due to the absence of the architecture artifact, compared to traditional (waterfall) software development. With a few agile teams, making a smaller system, the system validation (the right product) and system verification (the product right) loops is feedbacked every increment. This makes it possible to understand not only when refactoring is needed, but also be successful with the refactoring, due to the smallness of the system.

Since we do not have an architecture artifact in agile development, we will then not have the architecture process to achieve it either. Over time, the risk is then high that the craftmanship of the architecture process necessary for large systems in software development will be lost, the skills and the experience built up. This due to the fact that we only build on methods and frameworks using artifacts and processes, that was originally developed for smaller systems. Without this transdisciplinary craftmanship, we will only get a sketch artifact instead of an architecture artifact. Even though they may look the same, it is only the latter one that is (system) designed. If we also think that all work will be done in the agile teams also in large software system products, we will have an even higher risk that this transdisciplinary craftmanship will get lost.

When developing large software systems, no matter how the teams work together, the system verification feedback does not go together with the system validation feedback loop for every increment or part of the development. This means that we need and can have a good architectural skeleton (our system) also from start, or at least some idea of the architecture that we should verify as soon as possible as well as continually during the development. We can think about it as TDD for the big system, meaning we need to think Test-Driven Systems Design (TDSD) as well. In this was we will achieve fast feedback also on our total system with its architecture, which is close to thinking of Set-Based Design and multiple concepts at Toyota (Lean Product Development) or prototypes in hardware product development.

To summarise this blog post, we can state that the artifacts will never do the work for us, and without also understanding why and how we make the artifact. Otherwise, the risk is high that we will miss a necessary artifact, or that the artifact only becomes a shallow sketch. Instead, as always, we need skilled and experienced people to do the job, on all different “levels”. We must also put extra attention on the transdisciplinary top-down work, especially for bigger products, which requires tacit knowledge for systems design and not only coordination of parts, which sometimes seems to be the belief. This in order to be sure that our teams, agile or not, start to work with their tacit knowledge on the right parts, that as soon as possible will integrate to a unified and well-functioning whole. Sometimes, though, it is easy to focus too narrowly on the implementation of commonly used best practice artifacts, instead of figuring out how our own way of working shall look like, so we can solve our own problems, in our context and our domain.

That was all for this time. C u soon again.

 

*We can divide the different by-products in two categories, intermediate by-products and input/output by-products. The intermediate artifacts, like backlogs and prototypes, we can discard when the actual product is released. The input/output artifacts, like the system architecture and requirements are necessary for both starting up the development, as well as for coming additional requirements and bug fixes after the release. This in order to continually up until the release, as well as after the release, have good control over the system.

**we need to consider our first architecture like a prototype, a first skeleton of the system, meaning that we need feedback as soon as possible, so we know that our system architecture is on the right track or not.

***There conclusion also challenges the thinking about always having static teams. But maybe static agile software teams actually doing the work, is not the problem, instead the problem is that the transdisciplinary team needed for reducing the complexity top-down cannot be static. This depends on that the initial problem to solve top-down is mostly different from last time, then the actually coding of the functionality is for the agile team.

****Refactoring as a term in software development is from 1990 (a decade before Agile), even though code refactoring has been done for decades earlier than that [7]. Decomposition is another word for factoring [8].

[1] Encyclopedia Britannica. Link copied 2021-05-31.
“architecture”.

[2] Wikipedia. Tacit knowledge. Link copied 2021-05-31.
Tacit knowledge – Wikipedia

[3] Wikipedia. Explicit knowledge. Link copied 2021-05-31.
Explicit knowledge – Wikipedia

[4] Takeuchi, Hirotaka and Nonaka, Ikujiro, ”The New New Product Development Game”. Harvard Business Review, Jan 1986 issue. Link copied 2018-09-05.
https://hbr.org/1986/01/the-new-new-product-development-game

[5] Snowden, Dave. Blogg post. Link copied 2021-05-31.
ASHEN revisited – Cognitive Edge (cognitive-edge.com)

[6] Gröber, Matthias. Master’s Thesis Jan 2013. Link copied 2021-05-31
mg-thesis.pdf (tum.de)

[7] Wikipedia. Code refactoring. Link copied 2021-05-31.
Code refactoring – Wikipedia

[8] Wikipedia. Decomposition. Link copied 2021-05-31
Decomposition (computer science) – Wikipedia

Leave a Reply