Are Projects Still Needed or an Old-Fashioned Relic of Yesterday?
Part 2: Why Software Development Is Inherently Unplannable Yet Still Requires Disciplined Processes.
Today I’m walking onto very thin ice!
It’s been almost 40 years since I wrote software for my thesis that could calculate energy consumption in a factory.
The topic of the thesis wasn’t the software itself, but improving the energy efficiency of the factory. The software was ‘just’ a tool, a means to achieve a result.
Using IT for such problems was considered highly innovative back then. And to be honest, I must admit that I had to request an extension on the submission deadline because I didn’t finish on time.
Does anyone still remember Fortran?
Does anyone still know what a punch tape looked like?
How you searched the holes in the punch tape for errors when compilation failed and corrected them with a hand punch?
Later in my professional career, I had the pleasure of watching programmers desperately trying to get the software running at least halfway bug-free by the delivery deadline, with desperation and fear breathing down their necks. In this respect, things haven’t improved all that much apparently.
For me, it's a perspective from which it's relatively comfortable to talk smart.
I want to do that today.
I’m aware that software development today works fundamentally differently than it did back when object-oriented programming was still unknown.
For today’s software developers, this article offers the opportunity to read an outsider’s perspective. That might be an interesting viewpoint as well.
Please write to me whether you can relate to my view of things or whether I’m mistaken.
This article focuses on pure software development; I’m setting aside the problems that arise when software and hardware must work together.
That will come in the next article, which will deal with mechatronics development.
What Actually Makes Software Development So Difficult?
I think it’s important to ask this question if you want to understand and improve the approach to software development.
It follows the old but still valid principle that you must first identify the problem and understand its causes before you can find an effective solution.
So let me write down some theses about these possible problems.
You Don’t Know Exactly What the Development Goal Is
With software, it’s often not exactly clear from the start what the software must be able to do in order to fulfill the purpose for which it’s being developed.
This is a multifaceted topic, but I believe my thesis is a pretty good example to demonstrate the problem.
At the beginning of my work, it seemed reasonable to assume that the greatest energy-saving potential would lie in the electrical drive energy of the work machines in production. That’s why I initially focused on how the efficiency of the machines could be calculated.
However, during the analysis it turned out that a major waste of energy occurred through the extraction of warm room air in the chip extraction systems at the machines. Heat was being blown into the environment in large quantities and had to be replaced by reheating.
This dramatically changed the focus for the software. Suddenly it was no longer just about analyzing electrical and mechanical efficiencies, but about calculating thermal energy, which had to be determined from temperature and pressure differences in the airflows of the chip extraction systems.
The phenomenon of volatile goal definition is frequently encountered in software development. I’ve observed this repeatedly.
While with hardware the requirements for the product are relatively stable, as I described in the last article, with software the focus and thus the requirements for the product often change during development.
You Don’t Know Exactly What’s Inside the Black Box
At its core, software development is still based on manipulating binary states.
However, since it’s impossible to solve complex problems at the level of logic gates, we work with pre-built software structures.
While languages like Fortran first reduced complexity through procedural commands and control structures (like IF-THEN or loops), today we use object-oriented programming and microservices.
These function as ‘black boxes’: complex logic is encapsulated in objects or modules, where only the interface – that is, the input and expected output – needs to be known, and of course, what the module can be used for.
This gain in efficiency and development speed has its price, however: you lose full transparency over the internal processes. When a system shows unexpected behavior, a systematic root cause analysis begins:
Is there a GIGO effect (Garbage In, Garbage Out) because the input parameters weren’t validated?
Does the chosen module even fulfill the functional requirements, or is there an architecture error?
Is the abstraction reaching its limits, so that we need to implement a custom solution?
Does the underlying mathematical or physical model even correspond to the reality of the problem?
So the work isn’t done just by selecting the right object or module and building it into the code, but errors that occur must be identified and eliminated until the software ‘runs satisfactorily.’
I deliberately don’t say error-free, because I’ve given up on that expectation.
In each use case, it must be carefully examined how much and which errors are unacceptable and must be fixed and what we can live with.
Milliseconds Decide Whether It Works or Not
In addition to purely logical complexity, modern systems have another insidious level of complexity: time.
Software today rarely acts in isolation. It runs in multicore environments or communicates asynchronously via APIs.
Errors here often arise from so-called race conditions.
When two processes access the same resource simultaneously, the result depends on who ‘arrives first.’
Such a system is no longer deterministic – that is, with exactly the same input, sometimes it delivers the right result and sometimes a completely wrong one, just because a background process shifted the timing by milliseconds.
These ‘Heisenbugs’ (errors that disappear when you try to investigate them) are extremely time-consuming to identify during development.
And there we have it, the complexity that makes classical project management impossible.
That’s Not All…
These three phenomena are not yet all the aspects that make software development difficult.
But this should be enough for today to convey an impression of the nature of modern software development.
I think it has become clear that there are completely different problem areas here than with hardware development, which I discussed in the last article.
If you want to point out additional aspects, please write them in the comments.
What Is Easy About Software Development?
Now I also want to look at the flip side of the coin. There are also welcome aspects.
Software Development Works Incrementally
A special characteristic of software development is that individual functions or features can be developed and completed in isolation, without the entire product having to be completely finished.
This enables the software product or service to be worked on incrementally.
You can deliver a first version with few functions and then gradually expand the product with more and more functions and features.
This first variant is often referred to as a Minimum Viable Product (MVP).
It’s the version that has the minimum functional scope with which the software can be used.
This offers the possibility of quickly delivering important functions to the customer without having to wait for the rest.
This shortens the time to market, which is a very important criterion in today’s world because the environment is also developing very quickly.
There Is No Bathtub Curve
Software has no wear and no aging.
Hardware is characterized by a bathtub curve, where early failures due to development errors occur at the beginning, then a long phase of stable functioning, until later failures occur due to wear and aging.
With software we have the burndown chart. Here we see the errors from development that are resolved until a stable state is reached that no longer changes.
(Well, until the next change, but that’s another topic.)
Finding all errors is not easy and also takes time, but once you’ve worked through the errors, you’re also done.
Coding Is Relatively Universal
A decisive advantage of software development over hardware development lies in greater flexibility in personnel deployment.
While hardware projects often require highly specialized experts for different disciplines, the competencies in software development are significantly less differentiated.
This doesn’t mean there are no specialists – the crucial point is rather that the fields are so closely related that specialists can also help in adjacent areas.
Developers can usually be deployed across domains and can familiarize themselves relatively quickly with adjacent project areas.
Additionally, development tools and testing facilities in software are largely standardized and can be used across teams, which further shortens the familiarization time.
This more universal deployability makes it possible to react flexibly to capacity peaks, compensate for bottlenecks through resource redistribution, and scale teams as needed.
Software Is Relatively Cost-Effective
When I see the costs of a software release, the shock is often initially great.
Somehow you intuitively expect software development to be cheaper than hardware development – and when large sums are quoted, the disillusionment is great.
However, this initial reaction shouldn’t obscure the fact that software development is actually much more cost-effective than comparable hardware projects.
The decisive difference lies in the significantly shorter capital payback period.
While hardware products often require years to amortize, software projects can be refinanced more quickly.
This enables shorter product cycles and more agile adaptation to market requirements – a strategic advantage that more than pays off economically in the overall view.
Should I Set Up a Project or Work Agile?
I’m now doing the same exercise as in the last article and going through the criteria for a project and agile development individually. This time I’m looking at the characteristics of software development.
Let’s see what comes out.
Time-limited – works in principle, but since the result is often uncertain, the plan is just as uncertain. This speaks against projects and for agile, because agile can better handle uncertainties.
Interdisciplinary – Not such a big issue, since it’s essentially about relatively closely related fields.
Fixed result – Usually not guaranteed. This speaks against a project.
Defined requirements and constraints – Also not given, so also speaks against projects.
Iterative approach – Helps keep development loops short in case of errors and inaccuracies, so speaks for agile.
Self-organized teams – An efficient way to effectively prioritize available resources. Resources are relatively flexibly deployable. Clearly a plus for agile.
Quick delivery to the customer – Quick availability of features and functions enables quick delivery. So agile.
Short-cycle improvement based on customer feedback – Short capital payback period. Customer feedback from use can also be quickly implemented commercially and strengthens competitiveness. Agile!
Since predominantly complex cause-and-effect relationships exist, the occurrence of problems cannot be effectively prevented with risk management – speaks against project management.
As was not otherwise to be expected, the agile approach is clearly better suited for software development than classical project management. Agile is specifically tailored to the nature of software development and therefore fits like a glove.
Can We Then Completely Forget Classical Project Management?
Well, when projects become very large, the agile way of working also reaches its limits.
That’s the point where we talk about ‘scaled agile.’
Scaled Agile is a method to maintain the advantages of agile development at the core and to get a grip on the complexity that increases with functional scope through superordinate structures.
The best-known scaled framework is probably SAFe.
When I look at these models with my project experience, much seems familiar.
There are coordination processes that strongly remind me of interdisciplinary work in projects. Agile Release Trains have striking parallels to program management.
In my assessment, the agile approach is unbeatable in actual function and feature development and clearly superior to a project approach.
If the software project is large and requires many developers, then the agile approach gets into difficulties and must be scaled.
The scaling, in turn, can borrow very well from classical project management. Nevertheless, an adaptation to the characteristics of software development is preferable to a purely classical project management approach.
So Now We Have a Dilemma
For hardware development, classical project management with agile elements fits.
For software development, an agile approach scaled with borrowings from project management fits.
We now need to bring these together in the next chapter when it comes to developing mechatronic products, which consist of both.
That's exactly what I'll explore in the next article. Make sure to subscribe so you don't miss it.
If you have any comments or feedback on this topic, please write them in the comments.
or send me a direct message.
We can also chat about it.
I’m looking forward to hearing from you!
If you found this article helpful, don’t forget to share it with others who might enjoy it too!
And if my newsletter resonates with you, please consider sharing it with others.