Virtually everything we do is server based on XML services. The majority is custom XML messaging versus using SOAP (in many cases we can easily outperform the standard SOAP stack by a factor of 10-35%). We are in the process of upgrading one of our major services and the existing test cases are sitting in individual HTML pages and using the Microsoft XML components to submit the incoming data and the user visually compares the results to the displayed expected outcomes (not very automated but it derived from the development itself). We are performing a major upgrade to this technology and one of the areas that we wanted to improve on was in the automated testing area. We have over 130 test cases for this particular server and it really isn't enough but in order to engage the user we need something that is friendlier than html and xml with a smattering of JScript. Perhaps stupidly, I asked for this particular task since it is something I wanted to further on a more general basis for the rest of our services.
Hmmm, lots of background for a lesson about even old dogs get caught by simple things. I have been using Pascal/Delphi since tp5.
Anyway, I have put together a specialized editor to allow the entry of the test case materials broken down by category, procedures, and individual test cases. This data is all managed using XML and client data sets (which are perhaps one of my most loved Delphi features). I use that data to generate a test file in a format that the primary tool I created, the HTTP/XML test runner, takes as input. It is with the runner tool that I had a very stupid mistake (no one has seen it, so at least some face has been saved ;)).
Effectively, I need to loop through the structured data (categories, procedures, and cases) and I do it using a simple while loop while maintaining the pass/failure status. The root problem itself is that this is a command line tool and I wanted the exit code to represent whether failures occurred to support automated build/test processes. As such, the processing loops are contained within functions which return false in the event that any inner tests had failed. This was being maintained by the deceptively simple "
Result := Result and ProcessCase();". Sure, that works, looks like it works,
but it doesn't. While testing I found that the process would give up the ghost as soon as the first failure would occur but continue to appear to work, but no more test cases would be run (hmmm).
Hey, wait, my side effect, the call to ProcessCase(), was not happening. Well, a big DUH! Back to the grind and thank goodness for failing test cases at the right time.
Labels: Delphi