February 2012 Update

Thanks to the Hackathon, my coursework, and various Greyhats-related officer duties, it has been a while since I was last able to write anything. Here is what has happened since my last entry for ICS 314:

CityCamp Honolulu Hackathon

On January 20-21, I participated in the CityCamp Honolulu Hackathon, a 24-hour programming competition to design a web-based or mobile application based on data provided by the state or city government. Our team designed a Javascript-based application that used the Google Maps API to pull data from a GPS coordinates-storing site called CartoDB. Our GPS data and site descriptions were pulled from state government-maintained hiking-related websites or directly from Google Maps as much as possible, with the intent that anyone unfamiliar with local hiking trails and beaches would be able to use the application as a starting point.

Unfortunately, I cannot claim credit for any of the web application visible in the above screenshot.

My lack of Javascript knowledge made me somewhat useless on this project except as an informal beta tester and data gatherer for the CartoDB database, combing through state websites and Google Maps results to try to gather previously separated information in one place. It was definitely an educational experience to watch the more experienced web designers and Javascript coders at work, however, and I hope to participate in this kind of competition in the future, when I have hopefully filled in some more of the gaps in my knowledge.

Though we did not place at the Hackathon, I saw some very polished applications made by the other teams, including a pothole-reporting application and a bus route application that measured the distance to nearby bus stops and recalculated arrival schedules in real time. It really is amazing that people are able to create fully functioning, error-free mobile and web applications in only 24 hours, and makes for a good demonstration of what is possible through the skilled recombination of open-source libraries. Though there are definitely security and performance risks in blindly integrating third-party code into an application, being able to do so does allow applications to be made with a speed that one would probably never have seen even five years ago.

University of Hawaii at Manoa Greyhats

The Greyhats, a cybersecurity-oriented Registered Independent Organization at the University of Hawaii at Manoa, are going to be competing in the at-large regional for the 2012 Collegiate Cyber Defense Competition in March. I joined the club after the last CCDC had ended, so this is the first one I will be eligible to participate in. I am not sure how it will work out; I sort of ended up on the team by default this year, so I'll do my best to get up to speed.

I have also become the club's Treasurer. We are trying to secure funding from UH's Student Activity Program Fee Board to buy a server for virtualization purposes, so that those of us who are not graduating will have an easier time training next year's team, and anyone else who wants to learn information security skills at our meetings.

Algorithms and Database Software

The two ICS courses I am taking at the University of Hawaii this semester are ICS 311 (Algorithms) and ICS 321 (Data Storage and Retrieval). Now that I'm taking junior-level courses, it is obvious that the real work has begun. (In a year or so, when I start in on 400-level courses, I am sure that I will be saying the exact same thing.)

ICS 311 is a successor to my previous classes in discrete mathematics, which has unfortunately not been one of my strong points during my time in the ICS program. I know that having at least a basic understanding of discrete mathematics is the key to writing cleaner, faster-running code and being able to justify its efficiency through analysis, so hopefully I will be able to make up for past mistakes and misunderstandings and make the most of this course.

ICS 321 teaches the syntax and relational algebra of SQL, both as logical operations and through the use of IBM's free DB2 Express-C software. Up until now, my understanding of SQL has been limited to a superficial security-related understanding of its weaknesses. Accessing and modifying databases is an important part of many enterprise applications, and this course should definitely fill in a glaring gap in my resume.

Time allowing, I will definitely pick up in my C++ education where ICS 212 left off. Due to time constraints and delays during the units on C, the class was only able to briefly cover C++ in the last two weeks of class. Though C, as a widely used and efficient language and as the predecessor to most later object-oriented languages, was definitely important enough to warrant the time it took, I was still a little disappointed.

Like any skill, learning about programming only gets harder (or at least, more complicated). The important thing is to keep going.

The WattDepot Katas Project: Thoughts on an Open Source API

This week, I worked on a series of code katas for the WattDepot API, an open-source interface used to retrieve data on daily power and energy generation and use from remote sensors. The WattDepot API is intended to give users the tools to process technical, obtuse, automatically generated energy data and put it into a form that means something to the people who will use it.

I was able to complete the first Kata, SourceListing, in one or two hours; it asked for an alphabetical listing of all the Sources on the WattDepot server and their descriptions. Most of the code for this can be attributed to the SimpleApplication.java example program included in the distribution, as a Source object includes a name and a description.

The second Kata, SourceLatency, which required a program that calculated latency for each Source and printed the sources in ascending order based on latency, took two days. I had somehow managed to finish ICS 211 without learning how to use Hashtable, or else I had just forgotten that it existed. As a result, I spend the next two days trying to figure out how to sort one array according to the ordering of another array, without any success whatsoever. The solution I eventually found, and the way in which I found it, demonstrate why it is better to work on complex projects in groups. The reason I finally gave up on my insane attempt to sort two arrays simultaneously was because one of my classmates suggested during peer-programming exercises that I use a Hashtable. It turned out that a Hashtable with array-list values to store Sources with the same latency values worked just fine using the latencies as keys. As in some of my other projects, I have found that solving programming problems is a strange mix of happiness that something works, and frustration that it took me several sleep-deprived hours to figure out a simple mistake.

The third Kata, SourceHierarchy, required a program which printed Sources and their SubSources with increasing indentation for each level of sub-source. This kata also took two days, mostly because I sought clarification on how to parse the SubSources. The major issue I had at first, which turned out to be a non-issue when I figured out the actual format of the assignment, was that the SubSources returned by Source.getSubSources are not a recursive data structure (that is, a SubSource does not contain the Source data for its SubSources, only their URIs), and it was not possible to construct the hierarchy recursively. The trick, as Professor Johnson said, was to make a single pass through the list to figure out which sources were SubSources, and then use that to create the hierarchy. The solution I eventually implemented parsed the URIs to retrieve the SubSource names and did not print any Source which was listed as another source's SubSource.

The fourth Kata, EnergyYesterday, required a program which totaled up the energy used by each source on the previous day and printed each Source and its energy use in ascending order by energy use. This was similar in design to SourceLatency; in both cases, I used Hashtables to store Lists of Sources with identical energy or latency values, using the energy or latency values as the keys to retrieve them, then iterating through the lists and printing each Source with its associated key. Once I had fixed my SourceLatency issues, it did not take much editing to create EnergyYesterday; the main additional problem was the simple task of creating a timestamp which was rolled back by one day.

The fifth Kata, HighestRecordedPowerYesterday, required a program to print the highest recorded power for each source for the previous day, sorted in ascending order. It took up almost all of Monday, and several hours of Tuesday morning; this blog entry is being written at 6:00 AM on Tuesday morning, November 8, and the last kata still isn't finished. I had to write several utility methods for this class, mostly to generate the timestamps for power data requests and to parse the timestamps back into AM/PM format. The kata appears to be working correctly, except for one thing: I was never able to figure out whether it was a bug, or just a reflection of college study habits, but the peak power measured for every single sensor was at 12:00 AM midnight:

// My output for Kata 5 is reproduced below:
Connected successfully to: http://server.wattdepot.org:8190/wattdepot/
Server: http://server.wattdepot.org:8190/wattdepot/

Source              Highest recorded power in watts ( 07-Nov-2011 )
Lokelani-12-telco   1,203  12:00am
Lehua-10-telco      1,263  12:00am
Mokihana-08-telco   1,390  12:00am
Mokihana-04-telco   1,612  12:00am
Ilima-12-telco      1,818  12:00am
Lehua-04-telco      1,868  12:00am
Ilima-10-telco      2,227  12:00am
Lokelani-04-telco   2,432  12:00am
Mokihana-10-telco   2,662  12:00am
Lokelani-06-telco   2,890  12:00am
Ilima-08-telco      3,276  12:00am
Ilima-08-lounge     4,036  12:00am
Lokelani-10-lounge  4,204  12:00am
Ilima-10-lounge     4,285  12:00am
Lokelani-06-lounge  4,667  12:00am
Ilima-06-lounge     5,208  12:00am
Ilima-A             5,237  12:00am
Lokelani-04-lounge  5,458  12:00am
Mokihana-12-lounge  5,665  12:00am
Lokelani-D          5,845  12:00am
Mokihana-A          5,862  12:00am
Lehua-08-lounge     6,233  12:00am
Ilima-D             6,512  12:00am
Mokihana-10-lounge  6,612  12:00am
Mokihana-C          6,877  12:00am
Lehua-B             7,258  12:00am
Mokihana-E          7,484  12:00am
Lehua-C             7,760  12:00am
Ilima-B             8,022  12:00am
Lokelani-08-lounge  9,564  12:00am
Ilima               33,961  12:00am
Mokihana            35,322  12:00am

The sixth kata, MondayAverageEnergy, required me to write out the average energy consumed by a Source over the previous two Mondays. It is probable that I did not finish this kata in the hours remaining before the deadline. The main problem is in figuring out if today is Monday, then subtracting the number of days since the last Monday (to generate the first data set), then rolling back the date seven days to the previous Monday (to generate the second data set), and if I had had more time I would most likely have been able to reuse some of the code from EnergyYesterday to do this. I would have to put most of the blame for my poor scheduling on the two days of time I used to try to sort two arrays simultaneously for the second and fourth katas.

Though I may have frequently been frustrated with my own lack of knowledge or the difficulties of certain parts of the assignment, the WattDepot katas generally served as another example of how code katas can force developers to learn about a language or API's capabilities very quickly. Parsing WattDepot energy data into human-readable form is mostly a matter of timestamp generation, sorting among multiple sets of data elements, and parsing strings, and I was forced to learn new capabilities of the Java language and the WattDepot API for all of these tasks. The difficulties of making energy conservation relevant to consumers and enterprise users begin with putting information about their energy use into a form they can understand, and the WattDepot API provides a flexible platform for this purpose.

The Robocode API: Rapid Learning With "Code Katas"

Introduction

Learning and practicing new computer skills (a programming language, an IDE, an operating system, etc.) is a task that is often frustrating at best and intimidating at worst, especially when one is having trouble learning the system at its most basic levels. The philosophy of code katas, a form of practice with a name borrowed from Japanese martial arts, seeks to break the learning process down into a series of repeated drills, each one slightly more complex than the last. As the developer completes each drill, he or she gains essential skills along the way. This kind of "effortful practice" produces results faster than unfocused practice because it is focused around specific tasks, ideally just beyond one's current level of competence.

With this in mind, I set out to complete thirteen code katas for the Robocode API:

1. Position01: The minimal robot. Does absolutely nothing at all.
2. Position02: Move forward a total of 100 pixels per turn. When you hit a wall, reverse direction.
3. Position03: Each turn, move forward a total of N pixels per turn, then turn right. N is initialized to 15, and increases by 15 per turn.
4. Position04: Move to the center of the playing field, spin around in a circle, and stop.
5. Position05: Move to the upper right corner. Then move to the lower left corner. Then move to the upper left corner. Then move to the lower right corner.
6. Position06: Move to the center, then move in a circle with a radius of approximately 100 pixels, ending up where you started.
7. Follow01: Pick one enemy and follow them.
8. Follow02: Pick one enemy and follow them, but stop if your robot gets within 50 pixels of them.
9. Follow03: Each turn, Find the closest enemy, and move in the opposite direction by 100 pixels, then stop.
10. Boom01: Sit still. Rotate gun. When it is pointing at an enemy, fire.
11. Boom02: Sit still. Pick one enemy. Only fire your gun when it is pointing at the chosen enemy.
12. Boom03: Sit still. Rotate gun. When it is pointing at an enemy, use bullet power proportional to the distance of the enemy from you. The farther away the enemy, the less power your bullet should use (since far targets increase the odds that the bullet will miss).
13. Boom04: Sit still. Pick one enemy and attempt to track it with your gun. In other words, try to have your gun always pointing at that enemy. Don't fire (you don't want to kill it).

The Robocode project, once managed by IBM and now an open-source, community-driven effort, provides a way to program armed virtual robots and pit them against each other. Robocode robots are controlled by programming three types of behavior: movement, radar scanning, and firing. Though the core concepts are simple, the API is versatile enough that in the hands of experienced developers, Robocode battles can become a serious business indeed.

Look at this genetic algorithm, now look back at mine. Sadly, my code lacks complex algorithms. But the idea behind the code katas is that, with practice, that code is what my code could look like. Each group of katas emphasized a specific skill, whether finding specific points on a map, circling a particular area, tracking and picking targets from among multiple enemies, or controlling the power of the robot's shots based on information acquired about enemy robots.

The first three position methods were easy to implement. The fourth took a little more work, but finding the center was fairly easy after writing methods to calculate the angle between the robot and the center, and the distance of the straightest path to the center, and the sixth was just a variation of the fourth. The fifth method was the hardest of the group for me, and it took a while before I realized that it was returning the wrong angles because the robot was not accounting for its own previous turning when calculating the turn angle. I made heavy use of the Math.atan2 method here, as it was easy to retrieve the x- and y-distances between two points for use in calculating the angle.

The Follow robots were easier to write, since most of the data needed to match the headings of enemy robots could be obtained from pre-made methods, especially the method which retrieves a scanned robot's name for easy tracking. Follow03 was easily implemented by causing the robot to scan continuously for enemies, store their data, and compare the results after each scan.

Most of the Boom robots were also quickly written; Boom01 and 03 only needed to fire automatically. Boom02 was also easily handled using the getName() function. I am still not sure whether or not I implemented Boom04 correctly; it lost its "lock" on the target frequently,
and occasionally aimed in the opposite direction of its target's motion. The problem was probably related to correcting for differences between the gun angle and the robot's heading, something I did not quite get the hang of; since each exercise in a group was based on previous ones, Boom05 could be the exercise where sloppily designed but still previously workable methods finally became unusable.

Overall, the "code katas" were easy ways to pick up basic programming skills, and even some of the errors along the way pointed the way to competitive strategies I might be able to use in the future.

The Three Prime Directives of Open Source Software: PDF Split and Merge

Introduction

In the world of open-source software, the "Three Prime Directives" define the characteristics of a true open-source project. To fulfill the first directive, "the system must successfully accomplish a useful task." To fulfill the second prime directive, it must be true that "an external user can successfully install and use the system." The third directive is fulfilled when "an external developer can successfully understand and enhance the system." Fulfilling the three prime directives is important for an open-source project because it ensures that users and other developers can contribute bug reports, feature requests, and their own add-ons to the project, thus enabling the developers to better understand the needs of their customers and improve their system.

Sourceforge

Sourceforge is a well-known source code repository for open-source software, offering hosting and version-control services and providing a means for end users to directly contact development teams with bug reports and other feedback. ICS 314 is a Java-oriented class, so we were asked to find and review a Java-based project.

PDF Split and Merge

PDF Split and Merge's homepage describes itself as "... an open source tool (GPL license) designed to handle pdf files." Its GUI is written in Java Swing, and the command line interface is also based on Java.

The Windows installer for PDF Split and Merge can be found at http://sourceforge.net/projects/pdfsam/, and the other installers can be found at http://www.pdfsam.org/?page_id=32.

Prime Directive 1

PDF Split and Merge (referred to as PDFSam from here on) provides a wide suite of tasks related to the manipulation of PDF files. These include splitting and merging the pages of existing PDF files, rotating PDF pages, and combining multiple sections extracted from PDFs into a single new document. Most other programs which provide these functions are proprietary (e.g., Adobe Acrobat or Adolix Split Merge PDF), so consumer demand does exist for the useful task which PDFSam accomplishes.

Prime Directive 2

Ease of Installation

PDFSam was easy to install, and though there were problems with the installation, they were quickly fixed. The Downloads page features "basic" versions for Windows, Mac OS X, and a .zip file, which contains all the files that the .exe installer creates. The program requires Java SE 2 version 1.6 or higher to run. The site also offers an "enhanced" version that is free (if you compile the code yourself) or available as a regular installer for "a single donation of any amount." I installed the basic version.

Installing From A .zip File

Once the files are extracted, the program is easily run from the pdfsam-2.2.1.jar file. For some reason, the folder still contains a useless "pdfsam-starter.exe" file, exactly like the folder created by the .exe installer. The .exe installer had its own set of problems, which are covered in the next section.

Using the Windows Installer

The .exe installer ran with no problems on my Windows 7 machine, but the application failed to launch from the startup menu. I ran a compatibility check, which recommended running it using Windows XP SP2 settings. This caused the program to open and crash, saying it had failed to find "pdfsam-2.2.1.jar." The installer had created pdfsam-2.2.1.jar in the installation directory, but for whatever reason pdfsam-starter.exe couldn't detect it.

The readme.txt file included with the installation stated the following:

"Installation: Unzip the archive into a directory. Double click pdfsam-2.2.1.jar or open a console a type the command "java -jar /pathwhereyouunzipped/pdfsam-2.2.1.jar"".

Following this instruction successfully launched the program. Though the program runs from the .jar file, and readme.txt does tell the user to run the .jar file and not the nonfunctioning launcher "pdfsam-starter.exe," there are problems with this approach. Any user who used the .exe installer would not be able to easily access the program from the start menu, which only provides a link to the broken "pdfsam-starter.exe" file and no link to the .jar file that actually runs the program.

The program isn't unusable by any means, but the need to read documentation to get around a broken launcher implies that the problem is known but still not addressed, and doesn't satisfy the first prime directive's requirement for ease of installation.

Ease of Use

I was able to use most of PDFSam's PDF manipulation features without consulting the included instruction manual. It has five basic functions: "Alternate mix" (which can reverse the order of a document's pages or mix pages from documents at specified intervals), "Merge/Extract" (which lets you merge specific parts of PDFs), "Rotate" (which rotates all pages in a document in increments of 90 degrees), "Split" (which divides a document into pieces), "Visual document composer" (which lets you take individual pages of multiple documents and merge them into a single document) and Visual reorder (which lets you change the order of pages in one document). I tested PDFSam's basic functions with multi-page PDFs of lorem ipsum text and found the features easy to use.

The PDFSam GUI.

The command-line features are less easy to use, but still usable by following the instructions in the tutorial PDF or its wiki (command line instructions shown):

The mistake to avoid making here is to assume that "options" and "required" mean the "options" and "required" arguments that apply to "command." Actually, "required" covers all the arguments exclusive to "command," whether the arguments are specified as optional or not. After an hour or so of working through my confusion, I successfully did this:

The command line output produced by merging two PDFs.

To be fair, my mistake was a careless one, and command-line tools aren't usually intended for casual users (who will mostly use the GUI), so any difficulty on the part of the command-line tools doesn't detract much from the program's overall usability. PDFSam still mostly fulfills the second Prime Directive as it relates to ease of use.

PDFSam does a mixed job of fulfilling the second Prime Directive, and is easier to use than it is to fix it after it installs.

Prime Directive 3

PDFSam does a good job of fulfilling the third Prime Directive. Source code and developer-level documentation is easy to find on the project's official (non-Sourceforge) site. The source code is available on the Downloads page. Each Java source code file begins by making developers aware of their modification rights under the GPL, and each file includes cross-references to other Java files. This is demonstrated by the code excerpt below:

Developer-level documentation is provided on the Resources page. Java documentation is provided for both the basic and enhanced version's APIs, and changelogs and software requirements are also available. The Resources page also provides links to request features and access its SVN repository. Bugs are reported in the forums. Developer documentation is easy to find, the source code is readily available, and changes are publicly proposed and discussed, fulfilling the Third Prime Directive.

Conclusion

Despite my problems with PDFSam's Windows installer, its GUI is easy to understand and provides many useful ways of manipulating PDFs. Developer documentation, source code, and forums for communicating with developers are easily accessed. Overall, PDFSam fulfills many of the requirements of the Three Prime Directives.