I've begun writing a test suite for software and hardware emulations. My plan is for it to test that an emulation is accurate to Williams hardware. To start, I've included a 6809 test suite that I found, some blitter timing tests, blitter validation tests, 6809 instruction timing tests, and hardware register read/write results and timing.
Many people have discovered that emulation of the blitters is flawed in that the time that it takes to complete a blit on the emulator is much quicker than on an actual machine. MAME has improved this substantially, but there is still room for improvement: when the blitter is used to move data from RAM, rather than from ROM, it has to set mode bit 2 to synchronize with the E clock to avoid contention with the video hardware. If that's not done, the data is corrupted. This slows down those blits to essentially 1/2 speed. I did a quick test with MAME by running Robotron in logging mode, and nearly 10% of the blits have mode bit 2 set, so it seems like a significant issue.
I originally crammed all the tests into one program, thinking that would be more convenient. But after using it, I decided to break each test into a separate file. The original code is here.
Update- Aaron Giles added a small patch to MAME0145u2 that doubles the timing correction when mode bit 2 is set. With this change, the blitter timing just about perfectly matches my real machine. There still appears to be some timing issue that needs to be found and corrected to make game play more comparable to a real game, though.
Update 2- I wrote a small program to read the 6-bit vertical position register repeatedly and store the results in memory. I compared what I got on my actual boardset to what MAME produced, and noticed a difference. It turns out that MAME specifies 260 scan lines for the Williams games, but the read handler for the vertical position register is only 8 bits. So when the count gets to 256, the register returns 0x00. On my actual game, the register returns 0xFC for lines 256-259, before rolling over to 0x00. This difference would cause any code that waits for the top of the screen to return early.
Update 3- JROK wrote a similar program to compare his new multi-Williams board to the real thing: board benchmarking I ran his code under MAME and made one more tweak to the blitter code.
I patched Robotron to reset the random number generator at the start of a game, then connected the control panel harness to both the game and my laptop running MAME. I also patched Robotron so I could start the game on any wave. On most waves, starting a game simultaneously on a real Robotron machine and MAME, then not moving or shooting, resulted in exactly the same events, even if the # of men was increased to 20. But wave 40 has different results, so something must still need to be tweaked. Also, if I move or shoot, the results are different, but this might be due to different control latency, since I'm using a USB adapter for MAME.
Here are dumps of RAM (0000-BFF7) from an actual boardset after each blit validation pass. Note that the CRCs in the program are bit inverted and byte reversed as compared to CRC-32.