How one line of code quadruples testing – in hardware at least!

A sentence in an article by Joel Spolsky caught my attention: “Can’t we just default to IE7 mode? One line of code … Zip! Solved!”. This is the typical necessary solution that quadruples the testing effort. If you though it only doubled the testing, think again.

Let’s examine this in detail. First we have the code:

...
// When fizBoo is enabled, call it for additional sparkling
if (fizBooEnabled) {
  fizBoo();
}
...

All right, so how does this quadruple the testing? Well, first thing is that in hardware verification, there is no such thing as conditional compilation to remove FizBoo from the silicon: the fizBoo() function is always there. This is because only ONE chip is manufactured to supports all hardware architectures. In software, you can #ifdef things out of the executable based on architecture, but in hardware, everything is always there to support all of them. This way you can use the same hardware with a variety of operating systems (Windows, Linux) and a variety of architectures (Sun, IBM, low-end PCs, etc).

So now we have to test the fizBoo() in 4 different cases:

  1. fizBooEnabled == 0, in systems where fizBoo() does not need to be called
  2. fizBooEnabled == 1, in systems where fizBoo() adds some sparkling
  3. fizBooEnabled == 0, in systems supporting fizBoo(), but if not called, only the additional sparkling is missing
  4. fizBooEnabled == 1, in systems not supporting fizBoo(), to ensure there is no evil things like memory corruption or a bus error

One line of code quadruples testing. But this example is easy. It’s easy for the designer to put it in, easy for the tester to spot, and easy to cover off in a unit test.

But there are thousands of features in hardware, and end users are chaotic: they will plug in the hardware in the wrong architecture, and will load the wrong driver in the right architecture. The last thing they want though is the system to go up in flames. The hardware is expected to neither electrocute the user, nor cause memory corruption, regardless of how chaotic the end user is. End users are not the only chaotic people: your own software team is chaotic, even the team writing the driver for the hardware is chaotic!

In my opinion, this makes hardware verification a much more challenging job than hardware or software design. And because the fabrication cost of chips is so high, as much as possible has to be verified BEFORE fabrication, (no one likes a coaster, esp. when it costs in excess of 7-figure digits to fab). Without actual hardware to verify the hardware, all verification is done in software, through simulations, including the simulation of end user behavior, and the simulation of all the relevant pieces of the architectures under which the hardware is expected to be used. This is some serious testing. And this testing better simulate not only correct use with its humonguous state space, but also chaos pretty well. In fact, the only way to verify correctness is to introduce chaotic behavior in the test environment itself. By building randomness at each step of the input stimulus generation, chaos is introduced in the verification, and finds more bugs in more areas.

This is way more complicated than just using the same seed to reproduce a bug. This means that constraint random predictive stimulus generators, self checking scoreboards, automated test benches, and computer farms are needed to automate functional, formal and runtime verification. Actually, I don’t understand why these types of verification are classified under “software development” in the Wikipedia article on verification as they are used extensively in hardware verification.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: