The author’s argument is also correct: in safety-critical applications, you *cannot* pay later. It’s like paying for a product where they don’t take credit cards, only cash. And in regards to testing, I don’t buy the argument that software should be like bridges but that bridges are not tested but are reliable due to specifications. That may be true for bridges, but software cannot have quality constructions that way. The reason is simple: *unlike* a bridge, you only ever write software once! There’s no point in writing the same software code again and again, you simply copy it! Bridges can’t be duplicated out of thin air, they have to be rebuilt again and again. Every software coded is new in some way, and THAT is why you must test it.

And the day you can rewrite Linux in pure haskell is the day you’ll have an analogy that works.

Forrest, to look at it more clearly, if you have a bar of properly formed steel, you know it will have a tensile strength of X from prior destructive testing on other bars. Your testing, therefore, is not to see if X is the tensile strength; but if the bar is properly formed. It is indirect testing.

With unit testing, you exploit the fact that you can directly test software by seeing if it does what it’s meant to – whereas with a bar of steel, if you test its tensile strength directly, you have to do it destructively.

There’s nothing running counter to the flow of causality here. It’s a different form of testing, that’s all.

Note that testing to _validate_ that the material is in fact steel and is correctly formed is an inherently /different/ kind of testing — it is a validation of what something is, rather than what it does. This is also very different than testing a new, otherwise unknown material, to identify what sort of characteristics it has. Neither of these other types of ‘testing’ has anything to do with the basic idea that it is *only* possible to argue causality in the direction from what something _is_ to what it _does_. IF we are working with a bar of known size, material, and mechanical correctness THEN we can be certain of its having known functional properties. *All* engineering modalities, regardless of kind, depend explicitly and completely on causal relations of exactly this type: they all are causal statements that go from what something is to what it can be expected to do.

The basic problem with software ‘testing’ is that it attempts to go backwards, against this necessary and fundamental flow of causality. Software testing attempts to go from testing what some code /does/ (its functions) backwards towards assertions about what it /is/ (its quality/correctness). Not only is this terrible engineering practice, it goes against the very nature of logic itself. Software testing is the logical fallacy of affirming the consequent. It is the syllogism of attempting to argue from statements: 1) “IF A is true THEN B is true” and 2) “B is true”, towards the inherently faulty conclusion of “Therefore, A is true”. How different is this from the idea of software testing which writes tests in from of “IF our algorithm is correct (of high quality), THEN it will perform function Z”? When the code ‘passes’ the Z functional tests, then the QA department, engineers, management, etc, feel/think/claim that they can then assert “the algorithm is correct” — yet this is not so!

The basic problem here is inherent the very idea of how most people conceive of the function of software testing. Testing of system function (what something does) does will never allow for a way to argue backwards “that the system is of high quality” (an claim of what something is). Conflating the basic notions of validation testing (testing what something is) with property/characteristic testing (testing what it does) — as many people do — will not help (posts above make this mistake). How often do you find yourself or your peers thinking: ‘because the unit/system tests have all passed, that/therefore the system is of high quality (ie, reliable)”? It is a surprisingly insidious and common idea — yet even having a majority of practicing software professionals believe it does not make it logically correct — it is still complete bullshit to think that ‘software testing’ as it is currently conceived has anything at all do do with software/system quality.

Software design practices will remain at a infantile stage forever, unless and until the basic logic and principles of structural casual process are recognized and incorporated into the code writing methodologies at a primary and fundamental level. Chasing after ‘improved testing practices’ (TDD, etc) as a ‘silver bullet fix’ is inherently broken. It is a fad, and it will pass. The rise of TDD is more the result of management types trying to get useful code work from masses of unskilled ‘developers’ (cheap labor) than it is a reflection of any basic principle of ‘good software engineering practices’. If you reflect on it, you may notice that TDD proponents all tend to be associated with development groups that are much more cost-driven than quality-driven: TDD is seen as a way to avoid hiring truly skilled engineers, instead replacing them with vast numbers of code monkeys. As a truly skilled engineer (the minority) would you wish to contribute to the promotion of this basic fallacy?

If Joel is writing for managers without technical knowledge, then he is doing no favours to us working stiffs by recommending stupid ideas like losing tools from the toolbox. It’s hard enough to produce something without having a PHB tell us what tools to use when they have no idea what those tools are or do. And as to recommending dropping testing to PHBs because it doesn’t add value, well that’s just plain vindictive.

And I’m pretty sure you’ve got the same perspective issue going on here that I mentioned below. There’s coding for customers (where you want to ship, ship, ship because there’s money to be made!) and coding for users (who are often not the same people and whose interests conflict with those of customers, and quite often, it’s the user who takes it in the neck instead of the customer when something goes sideways). Engineering is very insistent that engineers code for users – business and management are very insistent that engineers code for customers.

Thing is, every engineer out there is (or should be) immediately remembering the phrase “take off your engineering hat and put on your management hat” when faced with that choice.

Historically speaking, we’re in a unique place right now, right at the start of all our fun stuff becoming part of the infrastructure. This has happened to all the other branches of engineering over the years. Mechanical starts off as a bunch of guys trying to build something and sell it, there’s little in the way of ethical considerations, it’s ship-ship-ship all the way; over time, as society becomes dependant on what they’re building, the requirements of the actual user, and the ethical duties to that user became prevalent. It’s not so much a point you hit in the development of the industry as it is a phase the industry slides into, and we’re mid-slide right now from what I can see.

When we start seeing the first serious legal penalties coming in for poor software engineering, that’s when we’ll know we’re nearing the end of that slide. Thing is, waiting until then to adjust our techniques is somewhat unethical.

As to the amount of embedded code out there, Ganssle estimated off back-of-envelope calculations that there were some five million embedded projects out there, with around 240,000 more coming in every year. He wound up with a 50-50 split between the embedded and non-embedded codebase sizes. I’ve seen papers from ACM conferences which go further than that, to the point where it’s projected that 60% or more of all code written everywhere next year will be for an embedded system.