Some Lessons In Software Engineering

It's been an interesting semester of ICS 314 so far, and since it's about the middle of the semester (and time for our first midterm) I decided to summarize some of what I've learned over the past few weeks.

Validation and Verification

1. In software review, validation is the process of checking that code is being made for the correct purpose, and verification is the process of checking that code is made correctly.
   Here I define these four validation and verification methods, and specify whether they are static or dynamic: load testing, defect testing, software inspections, and static analysis of source code.
   
• Load testing (Dynamic): Dynamic tests, unlike static tests, test the program by running it. Checks for errors in scalability, performance, and reliability.
  
• Defect testing (Dynamic): This is another form of static testing that checks for functionality errors.
• Software inspections (Static): Manual inspections of code by other developers. This refers to the formal process of "inspection" developed by Michael Fagan (described in more detail by "The Software Experts" here), as well as more informal code reviews and peer reviews. In all of these processes, other trained developers examine a project's code for implementation errors and requirement satisfaction.
• Static analysis of source code (Static): Examines the source code by looking for control and data flow issues, without running it.

Testing and Branch Coverage

2. One of the factors used by tools like Jacoco to determine the coverage of a program's test cases is control flow coverage - whether and how many of your control statements (if, else, etc.) are ever executed. I explored many issues of control flow coverage as I designed my test cases for the Robocode project.
   So, what are the three types of control flow coverage?
   
   • Branch coverage: Testing evaluates each conditional as true and false.
     • I made an attempt to provide branch coverage in my method unit tests. In what was probably at best a confusing design decision, I created and tested a method called whichWallOrCorner that returned one of nine integer values based on whether the robot was near a wall, near a corner, or near neither. What made this complicated to test was that, in accordance with branch coverage, the JUnit test needed to check all possible return values, as did every other method which called this method. In effect, each assertion in the JUnit test case evaluated one of the conditionals as true and all of the other conditionals as false. The distanceToWallOrCorner method, which depended on whichWallOrCorner, was just as long as the method it relied on; in effect, every piece of code that used the whichWallOrCorner method had to handle all nine possible return values in some way.
   • Loop coverage: Testing executes every loop 0 times, once, and more than once.
     • Apart from the main while(true) loop, the robot did not have many other loops in its code. The only way to check the while(true) loop and its sub-loops was to run acceptance tests which pitted the robot against other robots. Even so, these could only verify victories or proper method execution, not loop execution.
   • Path coverage: Testing executes all possible paths through the program.
     • Once again, as most loops were nested within the the while(true) code, it was not possible to test each nested loop directly, or even indirectly via the acceptance tests. For the method unit tests, it was much easier to force all possible paths to execute by passing appropriate variables to the method.
   My Robocode experience is described in more detail in this previous post.
   For at least the next few months, a more current version of my robot can be found here.
   
   

   Unit Testing for Input which is a Value

3. How should unit tests be written for methods which take values as input?
   
   Unit tests for methods that take values as input should, at minimum, test the method's behavior for the maximum legal value, the minimum legal value, an empty value, a value somewhere within the range of legal values, and an illegal value.
   
   Alternately, the minimum, according to equivalence partitioning, is as follows: once the values have been divided into partitions (ranges of input), only one test is needed per partition (e.g., one partition holding illegal values that are too low, one partition holding all the legal values, and a third partition after it which holds illegal values that are too high). In equivalence testing, it is thought that all the test cases for values within a given partition are the same - so only one value in each partition needs to be tested.
   



Pitfalls of Automated Quality Assurance

Just as a car company would use a wind tunnel to test aerodynamics but would use a focus group to test visual appeal, in software engineering there are some things manual testing does much better than automated testing.


4. Why is it a bad idea to use automated quality assurance to verify product requirements?
   
   Automated quality assurance is not smart enough to know whether something implements design requirements. It can only check if code is syntactically correct (e.g., Checkstyle, PMD) or creates usable bytecode at build-time (e.g., FindBugs).
   
   Only manual quality assurance can find problems related to the project requirements - a machine can check whether code is error-free, but only humans can reliably judge whether or not code is useful. My Robocode robot is a good example of this - despite its being very ineffective, even against most of the sample robots, the current version passes all of the FindBugs, Checkstyle, and PMD tests with no errors.
5. Why does automated quality assurance sometimes generate false positives?
   
   Automated quality assurance often flags code as a "problem" because it thinks it is a stylistic error, even if you intended it to work this way.
   
   I encountered this issue often when writing my JUnit tests. PMD would repeatedly flag my tests with UseAssertSameInsteadOfAssertTrue errors. In an ill-advised attempt to make this "go away", I replaced all the assertTrue methods with AssertSame. However, because this checked object equality, all of my tests that compared the values of numbers failed. As I found out from Stack Overflow, assertSame fails for any integers larger than 128, and based on personal experience, it does not work with doubles at all. I then converted all the assertions to assertEquals and added statements to print the purpose of each test. This caused all the tests to throw JUnitAssertionsShouldIncludeMessage errors in PMD. The errors finally went away when I converted the assert statements to the three-argument form of assertEquals, which includes a message. The code worked as expected before, but PMD refused to validate it until the "correct" test was used. Of course, one of the benefits of automated quality assurance is that it can detect problems the developer is too inexperienced to notice, so it is more probable that PMD was correct and my code was written inefficiently.