A Week of Configuration Management

Configuration Management

Configuration management software attempts to identify and track the elements of a system's configuration so that errors can be found and fixed. Version control is a special sub-case of this which focuses on maintaining multiple branches (development versions used for experimentation or developing new features) and the central trunk (the main version which all branches must eventually be merged with). Defect tracking systems track error reports from identification to the point at which the errors are fixed, and are part of configuration management because fixing the errors usually requires configuration changes.

To use a configuration management system, developers must first check out (download) the project from the SVN repository (the directory on a remote server where the project files are stored). Once this is done, they can edit their local copy of the project files, then commit (upload) the changed files or directories. If a developer adds new files to the project, they must be manually added to the files under version control before being committed to the repository. If others make changes, a developer must update (download new versions) the local copies of the changed files. William Wake explains it in more detail in this graphic.

As we were instructed in class, best practices for configuration management systems include verifying that the system runs and builds correctly before committing any changes. Otherwise, you run the risk of having to hurriedly fix previously undetected mistakes one after the other, as reflected in this log of commit messages:

Rev 53571: "This is a basic implementation that works."
Rev 53572: "By works, I meant 'doesnt work'. Works now.."
Rev 53573: "Last time I said it works? I was kidding. Try this."
Rev 53574: "Just stop reading these for a while, ok.. "
Rev 53575: "Give me a break, it's 2am. But it works now."
Rev 53576: "Make that it works in 90% of the cases. 3:30."
Rev 53577: "Ok, 5am, it works. For real.
Back when I said basic implementation? Scratch that."

(Source: This Stack Overflow thread)

Subversion

Subversion, the successor to the earlier CVS, is an open-source configuration management system that runs on the Berkeley Database, manages tags and branches similarly to folders, and handles all file types in the same way. It is currently an industry standard due to its centralized nature, which helps with access control issues and allows a single project leader to coordinate development; however, many in the open-source community prefer Git's flexibility and multiple workflows over Subversion' centralization.

Many GUI and command-line clients exist to access SVN repositories; I used TortoiseSVN, a Windows client. The GUI was fairly easy to use for my first assignment of revising and committing a preexisting project. For the second project, in which I had to upload my project to Google Code (about which more is written in the next section), I had a little more trouble with the command line interface, and what was ultimately more helpful than the project documentation in figuring out how to add files and do other operations was this general SVN manual, which did a good job of describing SVN's general logic.

Google Project Hosting

To complete my training in configuration management, after uploading my code to Google's remotely hosted repository via TortoiseSVN, I set about configuring the front page and some instructions for users and developers working with my robocode-tjo-meteor robot. The page I set up can be found here, and the source files can be checked out using the instructions here.

Google Project Hosting is fairly easy to add new pages to and configure settings for, using a system of simple menus similar to other Google interfaces I have used in Blogger or Google Sites.

Google Project Hosting allows users to choose Subversion, Git, or Mercurial as their configuration management system, and handles defect tracking by listing open issues on the "Issues" page. The current SVN trunk is viewable in the Source page, and the revision history is also available.

The only real issue I have with Google Project Hosting mostly deals with with element positioning:

• The tables are made with sets of ||s, not tags:Google only detects a table cell if both the opening || and closing || are on the same line (i.e., you never press enter at any point between them).
  For example, typing:
  
||Contents||
||I want this entire sentence to appear
in one table cell||

  results in:
  
  
  If you break this rule, you may end up with several scattered partial table cells. The workaround, typing the text you want to enclose as one long sentence, means that enclosing long sentences or paragraphs in table cells looks ugly in code and is almost unreadable if you use any other HTML tags or wiki markup within the table. I spent hours figuring out how to force Google to recognize the table cells before I realized that the bug was related to newline characters in this way. Google does allow the use of <table> tags, however, so there is a workaround for this.
• The anchor links:There are several things to be aware of when writing these. The support page helpfully explains that spaces need to be replaced with underscores in anchor links. Essentially, what I discovered was this:
  
// In this case I am trying to make an anchor link to header == Foo Bar ==
[# Foo Bar ] // This creates a link to the main URL of the page itself, and reloads it instead of going to the anchor. In some cases it does not work at all. The preceding and trailing spaces are not needed even though they are in the header.
[#Foo Bar] // This will not work either. As mentioned, the space needs to be replaced with an underscore.
[#Foo_Bar] // This will work.

  If you are mostly using this to make a table of contents as I did, there is a faster piece of built-in wiki code that does the job:
  //Set the depth to specify how many layers deep you want the contents to go.
  
• You cannot import images directly:Blogger allows you to import pictures from within the editor and host them on Picasa, in addition to embedding image links to pictures you have hosted elsewhere. This is not possible in Google Code. Though I there are definitely reasons to prefer this approach (Picasa's limitations, privacy concerns, or Blogger's arbitrary display sizes of "large", "medium", and "small"), a similar feature might be nice as an additional option.
  

Despite these minor issues (some of which were aggravated by my misreading of Google's support page), Google Code seemed to provide a decent level of services for being a free service and was an easy way for me to learn the basics of version control in a controlled setting, without the risk of damaging a real enterprise project - and demonstrated the ease with which I could use Subversion systems to store personal projects as well, though perhaps not quite at the level of Joey Hess.

Some Lessons In Software Engineering

It's been an interesting semester of ICS 314 so far, and since it's about the middle of the semester (and time for our first midterm) I decided to summarize some of what I've learned over the past few weeks.

Validation and Verification

1. In software review, validation is the process of checking that code is being made for the correct purpose, and verification is the process of checking that code is made correctly.
   Here I define these four validation and verification methods, and specify whether they are static or dynamic: load testing, defect testing, software inspections, and static analysis of source code.
   
• Load testing (Dynamic): Dynamic tests, unlike static tests, test the program by running it. Checks for errors in scalability, performance, and reliability.
  
• Defect testing (Dynamic): This is another form of static testing that checks for functionality errors.
• Software inspections (Static): Manual inspections of code by other developers. This refers to the formal process of "inspection" developed by Michael Fagan (described in more detail by "The Software Experts" here), as well as more informal code reviews and peer reviews. In all of these processes, other trained developers examine a project's code for implementation errors and requirement satisfaction.
• Static analysis of source code (Static): Examines the source code by looking for control and data flow issues, without running it.

Testing and Branch Coverage

2. One of the factors used by tools like Jacoco to determine the coverage of a program's test cases is control flow coverage - whether and how many of your control statements (if, else, etc.) are ever executed. I explored many issues of control flow coverage as I designed my test cases for the Robocode project.
   So, what are the three types of control flow coverage?
   
   • Branch coverage: Testing evaluates each conditional as true and false.
     • I made an attempt to provide branch coverage in my method unit tests. In what was probably at best a confusing design decision, I created and tested a method called whichWallOrCorner that returned one of nine integer values based on whether the robot was near a wall, near a corner, or near neither. What made this complicated to test was that, in accordance with branch coverage, the JUnit test needed to check all possible return values, as did every other method which called this method. In effect, each assertion in the JUnit test case evaluated one of the conditionals as true and all of the other conditionals as false. The distanceToWallOrCorner method, which depended on whichWallOrCorner, was just as long as the method it relied on; in effect, every piece of code that used the whichWallOrCorner method had to handle all nine possible return values in some way.
   • Loop coverage: Testing executes every loop 0 times, once, and more than once.
     • Apart from the main while(true) loop, the robot did not have many other loops in its code. The only way to check the while(true) loop and its sub-loops was to run acceptance tests which pitted the robot against other robots. Even so, these could only verify victories or proper method execution, not loop execution.
   • Path coverage: Testing executes all possible paths through the program.
     • Once again, as most loops were nested within the the while(true) code, it was not possible to test each nested loop directly, or even indirectly via the acceptance tests. For the method unit tests, it was much easier to force all possible paths to execute by passing appropriate variables to the method.
   My Robocode experience is described in more detail in this previous post.
   For at least the next few months, a more current version of my robot can be found here.
   
   

   Unit Testing for Input which is a Value

3. How should unit tests be written for methods which take values as input?
   
   Unit tests for methods that take values as input should, at minimum, test the method's behavior for the maximum legal value, the minimum legal value, an empty value, a value somewhere within the range of legal values, and an illegal value.
   
   Alternately, the minimum, according to equivalence partitioning, is as follows: once the values have been divided into partitions (ranges of input), only one test is needed per partition (e.g., one partition holding illegal values that are too low, one partition holding all the legal values, and a third partition after it which holds illegal values that are too high). In equivalence testing, it is thought that all the test cases for values within a given partition are the same - so only one value in each partition needs to be tested.
   



Pitfalls of Automated Quality Assurance

Just as a car company would use a wind tunnel to test aerodynamics but would use a focus group to test visual appeal, in software engineering there are some things manual testing does much better than automated testing.


4. Why is it a bad idea to use automated quality assurance to verify product requirements?
   
   Automated quality assurance is not smart enough to know whether something implements design requirements. It can only check if code is syntactically correct (e.g., Checkstyle, PMD) or creates usable bytecode at build-time (e.g., FindBugs).
   
   Only manual quality assurance can find problems related to the project requirements - a machine can check whether code is error-free, but only humans can reliably judge whether or not code is useful. My Robocode robot is a good example of this - despite its being very ineffective, even against most of the sample robots, the current version passes all of the FindBugs, Checkstyle, and PMD tests with no errors.
5. Why does automated quality assurance sometimes generate false positives?
   
   Automated quality assurance often flags code as a "problem" because it thinks it is a stylistic error, even if you intended it to work this way.
   
   I encountered this issue often when writing my JUnit tests. PMD would repeatedly flag my tests with UseAssertSameInsteadOfAssertTrue errors. In an ill-advised attempt to make this "go away", I replaced all the assertTrue methods with AssertSame. However, because this checked object equality, all of my tests that compared the values of numbers failed. As I found out from Stack Overflow, assertSame fails for any integers larger than 128, and based on personal experience, it does not work with doubles at all. I then converted all the assertions to assertEquals and added statements to print the purpose of each test. This caused all the tests to throw JUnitAssertionsShouldIncludeMessage errors in PMD. The errors finally went away when I converted the assert statements to the three-argument form of assertEquals, which includes a message. The code worked as expected before, but PMD refused to validate it until the "correct" test was used. Of course, one of the benefits of automated quality assurance is that it can detect problems the developer is too inexperienced to notice, so it is more probable that PMD was correct and my code was written inefficiently.

Going In Circles And Then Some: Preparing for a Robocode Tournament

Project Overview

The robot I was attempting to design for the Robocode tournament would have avoided hitting walls altogether, and would have been able to maintain a basic lock on its target. Neither of these goals was accomplished to any useful extent.

Design

• Movement

Ideally, the robot was to have moved 100 pixels each turn, turning itself perpendicular to the headings of any enemies encountered and turning back before hitting walls where possible. I attempted to do this via the avoidBounds method, which was intended to turn the robot away from walls and thus avoid collisions with the walls entirely. As far as I can tell, this instead caused freezing problems whenever the robot managed to be positioned perpendicular to a corner, and in some other cases I was unable to define.

• Targeting

Ideally, the robot was to have rotated its radar 180.0 degrees per turn, turned its body to be perpendicular to an enemy robot once it was scanned, logged its opponent's name, then begun scanning in smaller 22.5-degree intervals as long as it continued to detect that same opponent. Once that opponent was destroyed or had not been seen for two scans, it would resume scanning in increments of 180.0 degrees. In reality, the robot lost targeting locks almost constantly, as its other behaviors caused it to turn too frequently to maintain a consistent heading or back-and-forth pattern of strafing.

• Firing

The robot implements a basic bullet-power calculation algorithm based solely on distance, starting at the maximum of 3.0 for anything less than 100 pixels away, dropping from 2.9 to 2.0 for anything 100-300 pixels away, then to 1.9 to 1.0 for anything 300-500 pixels away, then to 0.9 to 0.5 for anything more than 500 pixels away. It fires whenever it detects the robot it is currently targeting, and targets the same robot for the duration of at least three 22.5-degree scans before giving up and looking for a new target.

Results

(Each robot was fought for 10 rounds.)


Meteor's lack of predictive targeting means that Walls has moved out of range by the time Meteor aims its cannon at where Walls used to be; Meteor has no way of accounting for drastic shifts in direction or velocity on the part of an enemy robot.

Robot Name	Wins Against	Losses Against
SittingDuck	10/10	0/10
Virtually all robots can beat SittingDuck.
Tracker	1/10	9/10
Meteor's primary weakness against Tracker is that it takes too long to scan for changes in motion (Tracker hardly scans at all once it has a lock), and thus fires much less often than Tracker. In addition, when this robot loses track of a target, it completes a 180.0-degree radar rotation, leaving it vulnerable to attack.
Corners	0/10	0/10
Meteor's avoidBounds method backfired here, frequently causing it to spin in circles without firing or moving while Corners swept its radar across the battlefield in increments.
Fire	0/10	0/10
Once again, the problem was that Meteor took too long to aim and fire,getting only a fifth the bullet damage of Fire. Meteor also froze repeatedly in this match.
Crazy	3/10	7/10
Meteor did not perform predictive movement analysis, and had no way of estimating Crazy's movement pattern, whatever it was. I am not sure why Meteor won against Crazy at all.
SpinBot	0/10	10/10
Meteor actually dodged several of SpinBot's projectiles, but it failed to score any hits of its own and usually got stuck against a wall after about thirty seconds, thanks to its freezing bug.
RamFire	2/10	8/10
Once again, Meteor's need to make large turns and radar sweeps each time it lost a targeting lock did it in; this made it easy for RamFire to push Meteor into walls or otherwise obstruct its movement.
Walls	0/10	10/10

Testing

My tests consist of two acceptance tests, TestMeteorVersusCorners and TestMeteorVersusFire, both of which the robot fails (see the Results section); one behavioral test, TestMeteorFiring, that checks whether Meteor.calculateShotPower is functioning correctly; and three unit tests (UnitTestWhichWallOrCorner, UnitTestDistanceToWallOrCorner, UnitTestNewXAndY) for the methods Meteor.whichWallOrCorner (which returns an integer code based on whether the robot is near a wall or corner), Meteor.distanceToWallOrCorner (which calls whichWallOrCorner itself, and determines the distance to a wall or corner) and a combination unit test for the very similar Meteor.newX and Meteor.newY methods, which calculate the point that a particular motion will cause the robot to drive to. UnitTestNewXandNewY does not work, however, due to what are probably errors in how I assumed it was supposed to work.

Lessons

The Robocode design process has revealed a harsh truth about my coding style: specifically, that it tends to produce a lot of code, most of which performs as expected, but most of which goes towards performing some extremely complex task (avoiding walls) which in the long run is not very important. In addition, the sheer length of my code virtually guarantees problems like Meteor's mysterious freezing episodes, which could come from any number of points in the code which control movement and which do not seem to share any common factors. Thus, I have a robot with about 400 lines of code (excluding comments) that cannot beat any of the sample robots with any sort of reliability, is still incapable of detecting when it is near a wall and stopping, and shoots straight but on average takes longer than most of its opponents to do so. The major problem is the freezing bug, which I have repeatedly failed to fix over the course of the last two days. If I had tried to do this over again, I would have spent less time trying to reinvent the wheel and more time modifying example code, as my attempt to implement a wall-avoiding function took so much time there was little left to actually test anything else.