The WattDepot Katas Project: Thoughts on an Open Source API

This week, I worked on a series of code katas for the WattDepot API, an open-source interface used to retrieve data on daily power and energy generation and use from remote sensors. The WattDepot API is intended to give users the tools to process technical, obtuse, automatically generated energy data and put it into a form that means something to the people who will use it.

I was able to complete the first Kata, SourceListing, in one or two hours; it asked for an alphabetical listing of all the Sources on the WattDepot server and their descriptions. Most of the code for this can be attributed to the SimpleApplication.java example program included in the distribution, as a Source object includes a name and a description.

The second Kata, SourceLatency, which required a program that calculated latency for each Source and printed the sources in ascending order based on latency, took two days. I had somehow managed to finish ICS 211 without learning how to use Hashtable, or else I had just forgotten that it existed. As a result, I spend the next two days trying to figure out how to sort one array according to the ordering of another array, without any success whatsoever. The solution I eventually found, and the way in which I found it, demonstrate why it is better to work on complex projects in groups. The reason I finally gave up on my insane attempt to sort two arrays simultaneously was because one of my classmates suggested during peer-programming exercises that I use a Hashtable. It turned out that a Hashtable with array-list values to store Sources with the same latency values worked just fine using the latencies as keys. As in some of my other projects, I have found that solving programming problems is a strange mix of happiness that something works, and frustration that it took me several sleep-deprived hours to figure out a simple mistake.

The third Kata, SourceHierarchy, required a program which printed Sources and their SubSources with increasing indentation for each level of sub-source. This kata also took two days, mostly because I sought clarification on how to parse the SubSources. The major issue I had at first, which turned out to be a non-issue when I figured out the actual format of the assignment, was that the SubSources returned by Source.getSubSources are not a recursive data structure (that is, a SubSource does not contain the Source data for its SubSources, only their URIs), and it was not possible to construct the hierarchy recursively. The trick, as Professor Johnson said, was to make a single pass through the list to figure out which sources were SubSources, and then use that to create the hierarchy. The solution I eventually implemented parsed the URIs to retrieve the SubSource names and did not print any Source which was listed as another source's SubSource.

The fourth Kata, EnergyYesterday, required a program which totaled up the energy used by each source on the previous day and printed each Source and its energy use in ascending order by energy use. This was similar in design to SourceLatency; in both cases, I used Hashtables to store Lists of Sources with identical energy or latency values, using the energy or latency values as the keys to retrieve them, then iterating through the lists and printing each Source with its associated key. Once I had fixed my SourceLatency issues, it did not take much editing to create EnergyYesterday; the main additional problem was the simple task of creating a timestamp which was rolled back by one day.

The fifth Kata, HighestRecordedPowerYesterday, required a program to print the highest recorded power for each source for the previous day, sorted in ascending order. It took up almost all of Monday, and several hours of Tuesday morning; this blog entry is being written at 6:00 AM on Tuesday morning, November 8, and the last kata still isn't finished. I had to write several utility methods for this class, mostly to generate the timestamps for power data requests and to parse the timestamps back into AM/PM format. The kata appears to be working correctly, except for one thing: I was never able to figure out whether it was a bug, or just a reflection of college study habits, but the peak power measured for every single sensor was at 12:00 AM midnight:

// My output for Kata 5 is reproduced below:
Connected successfully to: http://server.wattdepot.org:8190/wattdepot/
Server: http://server.wattdepot.org:8190/wattdepot/

Source              Highest recorded power in watts ( 07-Nov-2011 )
Lokelani-12-telco   1,203  12:00am
Lehua-10-telco      1,263  12:00am
Mokihana-08-telco   1,390  12:00am
Mokihana-04-telco   1,612  12:00am
Ilima-12-telco      1,818  12:00am
Lehua-04-telco      1,868  12:00am
Ilima-10-telco      2,227  12:00am
Lokelani-04-telco   2,432  12:00am
Mokihana-10-telco   2,662  12:00am
Lokelani-06-telco   2,890  12:00am
Ilima-08-telco      3,276  12:00am
Ilima-08-lounge     4,036  12:00am
Lokelani-10-lounge  4,204  12:00am
Ilima-10-lounge     4,285  12:00am
Lokelani-06-lounge  4,667  12:00am
Ilima-06-lounge     5,208  12:00am
Ilima-A             5,237  12:00am
Lokelani-04-lounge  5,458  12:00am
Mokihana-12-lounge  5,665  12:00am
Lokelani-D          5,845  12:00am
Mokihana-A          5,862  12:00am
Lehua-08-lounge     6,233  12:00am
Ilima-D             6,512  12:00am
Mokihana-10-lounge  6,612  12:00am
Mokihana-C          6,877  12:00am
Lehua-B             7,258  12:00am
Mokihana-E          7,484  12:00am
Lehua-C             7,760  12:00am
Ilima-B             8,022  12:00am
Lokelani-08-lounge  9,564  12:00am
Ilima               33,961  12:00am
Mokihana            35,322  12:00am

The sixth kata, MondayAverageEnergy, required me to write out the average energy consumed by a Source over the previous two Mondays. It is probable that I did not finish this kata in the hours remaining before the deadline. The main problem is in figuring out if today is Monday, then subtracting the number of days since the last Monday (to generate the first data set), then rolling back the date seven days to the previous Monday (to generate the second data set), and if I had had more time I would most likely have been able to reuse some of the code from EnergyYesterday to do this. I would have to put most of the blame for my poor scheduling on the two days of time I used to try to sort two arrays simultaneously for the second and fourth katas.

Though I may have frequently been frustrated with my own lack of knowledge or the difficulties of certain parts of the assignment, the WattDepot katas generally served as another example of how code katas can force developers to learn about a language or API's capabilities very quickly. Parsing WattDepot energy data into human-readable form is mostly a matter of timestamp generation, sorting among multiple sets of data elements, and parsing strings, and I was forced to learn new capabilities of the Java language and the WattDepot API for all of these tasks. The difficulties of making energy conservation relevant to consumers and enterprise users begin with putting information about their energy use into a form they can understand, and the WattDepot API provides a flexible platform for this purpose.

Apache Ant Code Katas: Learning A Build System In One Week

Introduction

Build systems are software packages which help software developers by automating project tasks that need to be done repeatedly, such as compiling source code, running compiled code, generating documentation, and packaging a system for distribution. All of these tasks do not take much time to do by hand for small projects, but for large academic or enterprise projects with hundreds or thousands of files, doing anything manually is virtually impossible. Apache Ant, a free build system commonly used in enterprise, carries out all of these tasks and more by using Java libraries to process instructions in user-created XML files. Users can extend Ant's functionality by writing their own libraries or by combining it with dependency-management tools like Ivy, which download whatever third-party libraries are needed to run a given piece of code, then save them locally. For this exercise, I completed several code katas designed to teach me some of the basic functionality of Ant 1.8.2. Each kata required me to create a specific filename.build.xml file, which would contain processing instructions for Ant.

Kata 1: "Hello World"

In the time-honored tradition of everything in computer science, I started here. The objective here was to teach use of the <echo> element to print informative text to the console. This did not take very long.

Kata 2: Properties Are Immutable

This kata was intended to teach procedures for defining properties, and to demonstrate that properties cannot be changed once defined within a build file. I defined a property, then attempted to define another property with the same name. As intended, this had no effect whatsoever, and the <target> which printed the property value printed the first value which was defined.

Kata 3: Dependencies

In Ant, the "depends" attribute of a <target> element specifies which targets must be completed before a target's instructions will be executed. This is used to make sure that any prerequisites for a target to run correctly (e.g., compiling code before trying to run it) can be carried out. This kata required us to print the name of the target as it was being executed, and I spent a few hours fruitlessly trying to print each target's "name" attribute before discovering that it might not be possible without Javascript workarounds. In the end, I defined properties with the same names and values as the targets they were defined within, and settled for <echo>ing those.

Kata 4: Calling the Java Compiler on HelloAnt

This exercise used a simple Java program which printed "Hello Ant" to the console. Here, within a "compile" target, I defined two properties: paths to a source directory (where the .java file was) and a build directory (where the .class files would be output to). I then used the <javac> element, which would tell Ant to run the javac compiler given the specified directories as parameters. This kata was still fairly easy. More importantly, running this script repeatedly was faster than having to specify the directories each time I ran javac from the command line.

Kata 5: Running HelloAnt

This exercise called for importing the .xml file from Kata 4 and using its "compile" target to compile the system before running it. As it turned out, defining the directory paths in Kata 4 meant I could easily reuse them here, as importing a file also imports all of its defined properties. Using the same principles outlined by the Dependencies kata, I made the "run" target dependent on "compile". The hardest part of this was guessing at how to write the <java> element's "classname" attribute for java files in a package, which I couldn't find an example of in the Apache documentation. At first, I tried to treat the class name as a series of nested folders (/edu/hawaii/ics314/filename.extension), which is how it appears in the file system. After several variations on this, I learned that ant requires that the classname be written using its Java format (edu.hawaii.ics314.filename, no file extension).

Kata 6: Making Documentation for HelloAnt

This kata demonstrated the use of the <javadoc> element, within a target that was dependent on the <compile> target, which was imported. This ensured that the Javadoc files would only be generated if the source code actually compiled. The <javadoc> element also allows you to control what information is included in the javadoc by setting attributes such as "author" or "overview" to "true". Generating documentation manually is always tedious, and the attribute options are less tedious to use than typing each option individually (every time you needed to revise the documentation) and running Javadoc from the command line.

Kata 7: Cleaning the Software Package

This exercise required me to modify the .xml file from Kata 4 to add a target which deleted the "build" directory generated by the "compile" target. This was intended to teach the use of the <delete> element, which instructs Ant to delete the directory specified in the "dir" attribute.

Kata 8: Package HelloAnt for Distribution

I had a recurring problem when I first tried to implement this kata: as required, it would delete the build directory, but it would mysteriously include the build directory in the .zip file it created for distribution. Eventually, I realized that because I had to create the build directory again to store the .zip file in build/dist, it was always saving the empty build directory into the .zip file. I fixed this problem by modifying some code from a class example to first save the .zip file to a temporary directory, copy it over to the desired directory, then delete the temporary directory. The main lesson here was to use the "excludes" attribute to exclude the build directory from the distribution. Regardless, manually excluding unnecessary files before creating the distribution directory would have been an annoyance for a small project like HelloAnt, but impossible for any enterprise project of moderate size.

The second requirement was to make sure that unzipping the directory from the command line would preserve the original directory structure. This was simpler and involved creating a directory with the same name as the current directory within the distribution directory. Removing the build files from the distribution not only reduces its file size, but makes sense in light of Prime Directive 2: forcing other developers or users to build the system using the build files you've provided is a good way to verify that others can install and use your system.

Ant / Ivy and the Three Prime Directives

As free software packages which work together, Ant and Ivy are fairly easy to install as long as one follows the site instructions (Prime Directive 2), and automating otherwise tedious tasks easily fulfills a useful task (Prime Directive 1). While I can't say that I have personally tried to extend the system, I was able to complete most of the katas without going outside Apache's documentation, which does go some of the way towards fulfilling the third prime directive as well. It is also worth noting that despite my initial problems on some of the later katas, all of the tasks I was trying to instruct Ant to perform would have taken just as long, if not longer, if they had to be performed by hand and/or directly from the command line over and over again. Build systems may take time to learn, but a little time invested up front here will always save more time in the end.

Final Thoughts

Though I wouldn't claim to understand everything about Ant at this point, the code katas did force me to learn enough to understand its basic capabilities and encourage me to look further into its other features. The extensive use of custom XML markup for interpretation by Ant and Ivy's Java libraries demonstrates to me that XML, as a user-defined markup language, is as useful as a developer wants it to be. Learning new software tools or APIs quickly is always a little like jumping into a pool; code katas won't get you into the deep end right away, but they can teach you enough to slowly move into deeper water and not drown.