How one line of code quadruples testing – in hardware at least!

A sentence in an article by Joel Spolsky caught my attention: “Can’t we just default to IE7 mode? One line of code … Zip! Solved!”. This is the typical necessary solution that quadruples the testing effort. If you though it only doubled the testing, think again.

Let’s examine this in detail. First we have the code:

...
// When fizBoo is enabled, call it for additional sparkling
if (fizBooEnabled) {
  fizBoo();
}
...

All right, so how does this quadruple the testing? Well, first thing is that in hardware verification, there is no such thing as conditional compilation to remove FizBoo from the silicon: the fizBoo() function is always there. This is because only ONE chip is manufactured to supports all hardware architectures. In software, you can #ifdef things out of the executable based on architecture, but in hardware, everything is always there to support all of them. This way you can use the same hardware with a variety of operating systems (Windows, Linux) and a variety of architectures (Sun, IBM, low-end PCs, etc).

So now we have to test the fizBoo() in 4 different cases:

  1. fizBooEnabled == 0, in systems where fizBoo() does not need to be called
  2. fizBooEnabled == 1, in systems where fizBoo() adds some sparkling
  3. fizBooEnabled == 0, in systems supporting fizBoo(), but if not called, only the additional sparkling is missing
  4. fizBooEnabled == 1, in systems not supporting fizBoo(), to ensure there is no evil things like memory corruption or a bus error

One line of code quadruples testing. But this example is easy. It’s easy for the designer to put it in, easy for the tester to spot, and easy to cover off in a unit test.

But there are thousands of features in hardware, and end users are chaotic: they will plug in the hardware in the wrong architecture, and will load the wrong driver in the right architecture. The last thing they want though is the system to go up in flames. The hardware is expected to neither electrocute the user, nor cause memory corruption, regardless of how chaotic the end user is. End users are not the only chaotic people: your own software team is chaotic, even the team writing the driver for the hardware is chaotic!

In my opinion, this makes hardware verification a much more challenging job than hardware or software design. And because the fabrication cost of chips is so high, as much as possible has to be verified BEFORE fabrication, (no one likes a coaster, esp. when it costs in excess of 7-figure digits to fab). Without actual hardware to verify the hardware, all verification is done in software, through simulations, including the simulation of end user behavior, and the simulation of all the relevant pieces of the architectures under which the hardware is expected to be used. This is some serious testing. And this testing better simulate not only correct use with its humonguous state space, but also chaos pretty well. In fact, the only way to verify correctness is to introduce chaotic behavior in the test environment itself. By building randomness at each step of the input stimulus generation, chaos is introduced in the verification, and finds more bugs in more areas.

This is way more complicated than just using the same seed to reproduce a bug. This means that constraint random predictive stimulus generators, self checking scoreboards, automated test benches, and computer farms are needed to automate functional, formal and runtime verification. Actually, I don’t understand why these types of verification are classified under “software development” in the Wikipedia article on verification as they are used extensively in hardware verification.

Advertisements

OVM World Seminar, Ottawa – of database vs. object oriented code, and open source

Today I attended an OVM World seminar in Ottawa. There were three presentations. The first presentation was by Hans van der Schoot. Hans is a very interesting, brilliant and passionate engineer, which made this a fun presentation to watch. Shame on me for not having written down the names of the two other persons presenting after Hans: there was an engineer from Mentor and one from Doulos. All had presentations with lots of code examples in them, which to me is a plus.

Of database vs. object oriented code

A very common problem in ASIC verification is the modeling of configuration information. In typical object oriented code, you will build a nested structure of configuration objects, through which you access relevant information for different components of your environment, or for the DUT. For instance, when you access configuration, it may look like this:

  // Object Oriented approach
  Config config = new;
  ...
  config.pci_transactor[4].bfm.en_err_injection = 1;
  config.packet_scoreboard.tolerate_discards = 1;

In OVM, they propose an alternative to this form of nested strucure. It’s a database, indexed by strings:

  // Database-like approach
  ...
  set_config_int("pci_transactor_4.bfm.en_err_injection",1);
  set_config_int("packet_scoreboard.tolerate_discards",1);

You can also use wildcards in the argument strings. We are not far from using SQL syntax here! The main advantage of the database approach is that you do not need compile time binding of configuration objects. Say for instance you work on a block level environment where you do not have pci_transactors. Ideally, you do not even want pci_transactor configuration objects in your block level environment because they consume memory, create unnecessary dependencies, and extend you edit-recompile cycle. In Object Oriented code, it’s very hard to come up with the right structure to swap in and out those undesired members of the Config class. You can always do it with compiler #ifdef directives, but it’s hard. The database approach is more flexible. However, I would not be surprised if it cost more in terms of simulation speed.

A similar approach with the OVM Factory allows great flexibility when building an environment: instead of calling the constructor of your verification components directly, you can call a factory function, asking it for an object type with a simple string! The factory function returns an object of the specified type… or an overloaded version of your choice, which you specify in your testcase through another function. Let’s see this in code:

  // Call a factory function, cast the returned
  // object to a compatible instance
  Bfm bfm;
  $cast(bfm,create_component("Bfm","normal_bfm"));

Now the interesting part is you can instruct the factory function to replace the returned object of every call requesting a normal_bfm, to return a error_injection_bfm instead, like this:

// in the testcase
set_type_override("Bfm","error_injection_bfm");

Of course, all this requires registration of types and instances in a database, but it saves having to extend and overload instances in testcases. IMO it is always a big pain to overload existing instances at runtime because you must do it at the right time, between the build phase and the run phase, plus you have to redo the connections and the constructor (the new) arguments. It’s even a pain trying to explain it. I don’t know if SystemVerilog classes have static constructors, but I imagine this would be how their sole presence anywhere in a compilation unit resulted in their registration into a global database.

This concept can easily be extended to other aspects of verification environments, wherever deeply nested data structures are used.

OVM licensing questions

At the end of the seminar, there were many questions about the open source nature of the OVM. The answers in a nutshell: yes you can copy and redistribute it under the terms of the Apache 2.0 license (i.e. do not claim it’s yours, leave the copyright notice intact, etc. – read the license it’s all in there); no you do not have to make all your proprietary code publicly available just because you use an open source library if all you do is use the library internally for your own development.

Opening the OVM bug database

There was also insistance by some of the audience on the importance of making the OVM bug database open as well. Mentor and Cadence said they were tracking OVM bugs through their respective internal databases, but they were also trying to find a way to make these bugs public. One of the EDA vendor said they were going to publish known bugs using release notes at each OVM release, but that is not real time information about current bugs. But the nice thing is representatives from all three companies, Doulos, Mentor and Cadence all agreed that it was an important thing to do.

As far as reporting bugs through a public database, this is a delicate as anyone who has worked on project with a great number of users can attest: misunderstandings, duplicate bugs, and feature requests may eventually pollute the bug database. Not necessarily a big deal if users search through bugs before entering what they believe is a new bug, but most times users that hit the bug database have reach a high level of frustration, and they want their issue resolved now. Nevertheless, a public bug database is better than nothing at all. And the OVM user community is quite small compared say to a large open source project like gentoo, so I don’t think users will overwhelm the OVM developers with false/duplicate requests/bugs in the bug database.

Nothing stops anyone from opening a public bug database for the OVM and start using it… and same for creating a public repository for the source code as well. It is so simple now with the internet, I would chose savannah simply because it supports git, but any public server with a decent versioning tool would work.

Accepting contributions from the user community

There were two people asking on how they plan to accept contributions from the community. Their answer was that both Mentor and Cadence have people participating in the forums, and contributions made through the forums would be considered for inclusion in the OVM. The EDA representatives pointed out that contributions cannot be accepted as-is since they run tests on OVM before releasing it to insure its quality. I guess they could open source the OVM test suite as well. They could also use Linus’ philosophy: only grant write access to the source code database to trusted users with a proven track record of useful contributions or bug fixes.

ASIC register modeling

There were also talks about open sourcing the asic register modeling library, but no commitments were made. (I might have gotten the name wrong here since I am not a user of such library.)

OVM support

There were questions about support for the OVM: the EDA user community is accustomed to email support with EDA vendors. But the EDA guys really seemed hesitant to commit to email support for helping users with the OVM… I mean they are giving out the OVM source code under and open source license, and people expect free support? I think the EDA vendors do not have to provide free support for the OVM. I have paid for support of open source tools in the past and it’s the normal way of compensating someone for their time. Companies use linux and they still pay their IT staff to support the “free” OS!

One of my favorite way to obtain support is through public IRC channels. I get support for gentoo and Tango in their respective IRC channels. Of course, one cannot go to an IRC channel, get help, and never give back, because it is simply not fair: you are expected to give when you want to be part of a community, it’s not just about taking for yourself. Nevertheless, this maybe a good way for EDA vendors to provide support for the OVM: users helping users on an IRC channel on freenode, with OVM developers on the channel from time to time. After a while, users might be able to support themselves.

All right, that’s what I have to say about the OVM. I congratulate Mentor and Cadence on releasing the OVM under an open source license, a first for an EDA company!

Book review: Learn to Tango with D

Learn to Tango with DOn my never ending quest to find the utlimate language for ASIC Verification, I decided to learn DigitalMars D. Yes, it’s called D, as in A, B, C, C++, D. It is a compiled, object-oriented, garbage collected programming language which runs as fast as C/C++, without the hassles of C/C++ and without the verbosity of Java.

So I picked up a bit of D on my spare time, and then a book came out, Learn to Tango with D. D is the language, Tango is a systems library. My main worry about the book was that I am not a Java, nor a C/C++ programmer. But to understand the book, you do not need to know C/C++ or Java. The authors take the time to explain the concepts as they apply to D, which is really great. Most of the book is about D, and that’s perfect for me. Starting from chapter 6 is where the authors present the packages of the Tango library.

If you want to learn D, you could read the D manual, but it would be like reading a dictionary: long and dry. To me reading the online D manual was not enough, because there were things that I did not understand, such as the imports of modules, the use of the static keyword (what it really does), procedural lifetimes, and some intricate details of templates. All those and more are explained in the the book. The book also does a great job at explaining arrays. Arrays in D are awesome, and go above and beyond what you find in Vera. For instance, D dynamic arrays support slicing, which means you can have multiple references to the same data in memory, or portions of that data. Isn’t this great? Imagine how this could simplify and speed up nested packing/unpacking of data structures: your data would not move in the computer memory!

So can D be used for ASIC Verification? Well, you’d need a few things, such as threading and fixed sized integers, aka bit vectors of arbitrary size. The Tango library has Threads and Fibers, and with Fibers you can build coroutines, and with coroutines, you can build a simulator with mailboxes, semaphores, mutexes, etc. The first time I saw a demonstration that a simulator could be built with coroutines was in MyHDL but I am sure there are others. So I have no doubt a HVL simulator can be built with Tango Fibers. Tango also has a nice Logger package, meaning you don’t have to invent one.

D has assertions, which are equivalent to the boolean layer of SystemVerilog assertions. This is really an unfair comparison, since D was not designed as an HDL, nor as an HVL. If you need the full spectrum of HDL assertions, you’ll have a lot of work to do. Ideally, you’d also want a constraint solver, a constraint language, and something for functional coverage too. That’s a lot of things to be added before D can be used for ASIC Verification.

Nevertheless, D is great and I definitely recommend the book if you want to learn D and the important packages of the Tango library. The book has many code examples which greatly help understand the concepts.