May 7, 2008 Leave a comment
A friend pointed me to Buildbot today. Every ASIC project I have worked on has more or less always required some form of automated system to insure new code did not break old code, and bugs were found on a continuous basis through regression testing.
The state space is so large in ASIC verification that running continuous regressions that apply random stimulus is necessary to find as many bugs as possible before releasing the code for fabrication. Usually, the person responsible for running those regressions and reporting the results to developers gets bored to death of this repetitive job and ends up building an entire system that runs these tests automatically.
However, after reading most of the documentation, it seems Buildbot is not directly usable for ASIC regression automation. This probably comes from the fact that software builds are not the same as ASIC builds.
A software build means compiling the code specifically for an architecture (x86, ia_64, etc), and running a set of tests, all on the same computer, to prove that the software can run flawlessly on a specific computer architecture.
An ASIC build means compiling the ASIC submodules and top level to executables, then run series of tests specific to those submodules and top level builds to find bugs in the ASIC. The underlying hardware architecture that is used to run those executables is irrelevant to the tests being run. The compiler that compiled the ASIC into executables must work on all the different computer architectures, but the resulting ASIC executables will behaves identically, regardless of the architecture on which they runs (x86, ia_64, etc).
This fundamental difference makes Buildbot unsuitable for ASIC verification unless some effort in made in customizing it. Specifically, I expect there would be customization required in the Builders. According to the control flow,
a Build is started on each of a set of configured Builders, all compiling/testing the same source code
and this leads me to believe that all tests that a Builder is configured to run will be farmed out to the same BuildSlave. There appears to be no load sharing for the tests once the main executable is build. There is probably a way to configure multiple Builders to share the load, but that would require manually balancing the tests on each Builder, taking the benefits of automation away.
Hopefully I am wrong and there is a way to load balance the tests, but until then, I think using Buildbot for ASIC verification is not a straightforward solution.
In any case, this show that constructing a system for regression automation is not a trivial problem that a script kiddie can do over a weekend.