Softmake's Software Development System - Simplified History of Software Development

The story so far

Softmake favours what might be called a "fourth generation" software development system.

Having distinguished software development methodology, system, and method we can divine a simplified history of software development systems. It is "simplified" in that it is not meant to be historically comprehensive or even accurate. This simplified history is merely a means to illustrate a few key issues in software development.

Once upon a time there was programming languages and no methodology, no system, and no methods with which to employ them. When software was required to do something programmers dove in and started programming. We can now, in retrospect, understand this to be the classic mistake of code-like-fuck.

Then, in the late 60's there was the idea that to make software do something useful an engineering system might be applied. So the "Software Engineering" System was born. This system privileged particular methods including: specification documents to a low level of detail, right down to the programming procedure stub; a strict waterfall lifecycle; a division between analysts and programmers.

Despite the use of the Software Engineering System projects in the 70s, 80s, and 90s where frequently of poor quality, cancelled, or overran their estimates. This has been called the "Software Crisis."

Consider the 1994 Standish Group Chaos report. Only 16% of projects where successful:

1994 data from Standish Group Chaos report quoted in Interview: Jim Johnson of the Standish Group., accessed 2021-06-23.

Key:

• Success: delivered on time, on budget, with required features and functions.
• Challenged: late, over budget and/or with less than the required features and functions.
• Failed: cancelled prior to completion or delivered and never used.

There has been some improvement in the 10 years to 2004:

2004 Data and Key from Standish Group Chaos report quoted in 2004 Third Quarter Research Report Chaos Demographics, (Original URL http://www.standishgroup.com/sample_research/PDFpages/q3-spotlight.pdf accessed 28^th Oct 2006).

So, there are about twice as many successful projects and failures have almost halved. However, that less than a third of projects can count as successful means it is still reasonable to characterise Software Development in general as "in crisis".

The cause of the crisis can be in part demonstrated to be the inadequacy of the Software Engineering System. Often enough, though, the crisis can be shown to be the result of any classic mistake, starting with the classic mistake of having no system at all (code-like-fuck).

In 2000 "Agile" was coined (see the discussion of the Snowbird workshop by Martin Fowler in The New Methodology) to refer to a group of systems (Fowler and others use "Methodology" where we speak of "System") differentiated from Software Engineering Systems. "Extreme Programming" is the most prominent example of an Agile System.

An Agile System, in crude terms, uses an iterative lifecycle (Release some functionality early and repeat) and leaner or no documentation (The code is the specification). An agile system is alleged to emphasize people rather than process and better to cope with requirements change.

Fourth Generation System

From this differentiation between Software Engineering Systems and Agile Systems useful debates have emerged. Agile Systems have been right to identify problems with Software Engineering Systems but the substitute proposals go too far.

So, for example, there is a debate about whether you ought do Big-Design-Up-Front. Agile rightly points out that the Big-design-up-front done by traditional Software Engineering takes attempts to nail down all issues in heavy documentation. There are two problems with this. Some issues can only really arise out of implementation rather than document centred design. That is, sometimes you can't discover a flaw until you have your hands dirty with code. Secondly, the heavy documentation specification leads to resisting a change in requirements. Change and flexibility is part of the world and software development needs to accommodate this reality.

The proposed fix to Big-design-up-front, in Agile, is to reduce or eliminate the documentation and have frequently released cycles of functionality where users feedback as the project is incrementally delivered. The code, rather than documentation, is said to constitute the spec.

Joe Spolsky is correct to argue against Agile and for Big-design-up-front:

The reason you write a spec is not to solve every possible problem in advance: the reason you write a spec is to solve as many problems as you possibly can in advance so that you minimize the number of surprises that come up during development. Many times, thinking things out in advance saved us serious development headaches later on. [Joe gives an example] Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can't tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I'm proud to use it, no matter what the XP fanatics claim. They're just wrong on this point and I can't be any clearer than that.

The Project Aardvark Spec. Joel on Software. Joel Spolsky (2005-08-17). Retrieved on 2006-09-27.

This is borne out by the research:

Projects that skimp on upstream activities [requirements analysis, architecture, & design] have to do the same work downstream at anywhere from 10 to 100 times the cost of doing it properly in the first place.

Fagan 1976; Boehm and Papaccio 1988. Referenced in McConnell 1996, p45

Getting the requirement right in the first place typically costs 50 to 200 times less than waiting until construction or maintenance to get it right.

Boehm and Papaccio 1988. Referenced in McConnell 1996, p62

A design defect that takes an hour-and-a-half to fix at design time will take anywhere from two days to a month to fix if it isn't detected until system testing.

Fagan 1976. Referenced in McConnell 1996, p123

Arguing against Agile in favour of Big-design-up-front is not to defend the traditional Software Engineering Systems. There is what we might call: a fourth generation System. At least if we can be allowed to identify: the first generation as no system but for code-like-fuck; the second Software Engineering Systems; and the third Agile.

The forth generation system does not seek a return to heavy, inflexible and an unnecessarily detailed documentation from Software Engineering. Nor does it embrace the code-as-spec practice of Agile, which too easily becomes code-like-fuck. The forth generation system is middle of the road (I'm loath to defend any position that is middle of the road, but alas in this case I seem compelled by reason to do so). It is characterised in part by:

• An emphasis on the use of throwaway prototyping for specification solicitation and to perform design spikes to eliminate surprises. This gives you the best of both worlds:
  1. From the Agile like use of client feedback to a coded system: You get your hands dirty with code and so a more likely to discover issues early; the client and users have something concrete to look at and give feedback on.
  2. From the Software Engineering System - insistence on a complete specification before coding: You still producing a specification document but this might, in part, just refer to the prototype screenshots.
• The specification documentation is only to the level of detail to capture design and to start coding (but not down to procedure level). We can minimize or eliminate many of those formal specification documents.

References

• McConnell, Steve. 1996. *Rapid Development: Taming Wild Software Schedules*. 1st edition. http://www.amazon.com/Rapid-Development-Steve-McConnell/dp/1556159005. Microsoft Press, 1996-07-02.