Technical Review and Evaluation of the Hale-aloha-cli-grads Open-Source Software Project

In this blog entry I post about my experience reviewing and evaluating the open-source software system named Hale-aloha-cli-grads. 

Introduction and Background

To understand the system as well as and the evaluation criteria, I will first provide summary data of some background information necessary to understand the issues examined in the review.

Hale-aloha-cli-grads



The Hale-aloha-cli-grads system under review is an open-source project that implements a 
command line interface (CLI) for WattDepot, created by Andrea Connell, Leo DeCandia, and Sergey Negrashov. It retrieves and displays energy and power data for the four Hale Aloha residential towers at the University of Hawaii Manoa campus.

WattDepot

The wattdepot open-source project developed at the University of Hawaii is a collection of technologies and tools for collecting and storing data from electricity meters for smart grid research and experimentation. From the wattdepot Google hosting site: “WattDepot is an open source, RESTful web service that collects electricity data (such as current power utilization or cumulative power utilization) from meters and stores it in a database. The data can then be retrieved by other tools for visualization and analysis. It is designed to provide infrastructure for experimentation and development of "Smart Grid" applications.”

The 3 Prime Directives of open-source Software Development

For the purposes of this evaluation, the three prime directive of open-source software development are summarized as follows:

1. Does the system accomplish a useful task? 
2. Can an external user can successfully install and use the system?
3. Can an external developer successfully understand and enhance the system? 

Issue-Driven Project Management

The basics of Issue Driven Project Management are summarized as follows:

• Divides work into tasks
• Tasks take no longer than 2 days
• Tasks are specified by issues
• Every person always has an open task
• Every commit specifies an issue in its log comment 
• Each person has a task
• Each person knows what it is
• Each person knows what the next task is
• Each person is rarely blocked
• Completing a task brings the project closer to completion
• Project state is visible at all times to all team members
• Breakdowns in PM are recognized by everyone, quickly

Continuous Integration



The basic concepts of Continuous Integration are:

• The System must be able to be built and tested automatically
• The System must be under CM
• Everyone commits every day or two
• Upon commit, The System is immediately and automatically compiled, tested, and developers notified

Review and Evaluation

My review and evaluation will cover important aspects of open-source software development, as well as answer detailed questions about the system including: build process, developer contribution, documentation, testing, QA, process, software quality, extensiblity, and integration. These elements will be grouped into three main sections that correspond to the three prime directives of open-source software development.

Prime Directive 1: Does the system accomplish a useful task? 

Following the directions which were 'spread out'  among the home page, installation guide, and user guide, I downloaded, installed, and ran the system under Windows 7. 
I exercised the system using various commands and command line arguments.

The functionality that is present in the system corresponds to what is documented on the project home page, and the messages displayed for bad commands and command arguments are informative.

I was surprised to see the entire help menu printed after every command, and the help menu does not specify the valid tower and lounge names. 

The help command does not show that 'help' and 'quit' are available commands. 
Also, there is no command line prompt, just a blinking cursor, and as mentioned above, I don't think 3 pages are necessary to get the information required to download, install and run the system.


However, I conclude that prime directive #1 is satisfied because a useful task is performed and the system basically behaved as outlined in the documentation. There were no difficulties in downloading, installing, and running the system, and it displayed expected output based on input parameters.

Prime Directive 2: Can an external user successfully install and use the system?

I carefully reviewed the project site,  Including the home page, user guide wiki, and downloads page.

The home page provides a clear understanding of what the system is supposed to accomplish, and is accompanied by a list of user commands recognized by the system. The user guide has a screen snap of the program in action, showing both sample input and output. The user guide also explains that both physical and virtual ‘sources’ are supported. However it does not indicate what version of the Java JRE is required to run the system.

The user guide links to an installation page that provides clear details on how to download, install, and execute the system from the perspective of a user, but does not include a link to the downloadable zip file. The downloadable zip file is stamped with a version number, and the zip contains an executable jar file. I did not have to compile and build the system in order to use it. I exercised the system under both valid and invalid inputs as documented below.

Valid commands and command line arguments:

energy-since Ilima 2011-11-28

Total energy consumption by Ilima from 2011-11-28 00:00:00 to 2011-11-30 14:42:26 is: 1616.9 kWh

energy-since Mokihana 2011-11-28

Total energy consumption by Mokihana from 2011-11-28 00:00:00 to 2011-11-30 14:43:01 is: 1616.7 kWh

current-power Ilima

Ilima's power as of 2011-11-30 14:45:12 was 26.0kW

daily-energy Ilima 2011-11-28

Ilima's energy consumption for 2011-11-28 was: 611.7 kWh

daily-energy Ilima-A 2011-11-28

Ilima-A's energy consumption for 2011-11-28 was: 104.7 kWh

rank-towers  2011-11-28 2011-11-30

For the interval 2011-11-28 to 2011-11-30, energy consumption by tower was:

Ilima                            1255 kWh

Mokihana                         1261 kWh

Lehua                            1268 kWh

Lokelani                         1468 kWh

rank-towers  2011-11-24 2011-11-30

For the interval 2011-11-24 to 2011-11-30, energy consumption by tower was:

Lehua                            3364 kWh

Mokihana                         3379 kWh

Ilima                            3464 kWh

Lokelani                         3942 kWh

help

--Available commands are:

energy-since: [tower | lounge] [Start]

Returns the energy used since the date (yyyy-mm-dd) to now.

current-power [tower | lounge]

Returns the current power in kW for the associated tower or lounge.

daily-energy: [tower | lounge] [Date]

Returns the energy in kWh used by the tower or lounge for the specified date (yyyy-mm-dd).

rank-towers:  [start date] [end date]

Returns a list in sorted order from least to most energy consumed between the [start] and [end] date (yyyy-mm-dd)

--Enter a command (type 'quit' to exit):

quit

quitting...

Invalid commands and command line arguments:

"Just pressing the return key"

‘ ‘ is not a valid command

energy-since Ilima 2011-11

Bad xml 400: Range extends beyond sensor data, startTime 2011-11-01T00:00:00.000-10:00, endTime 2011-11-30T15:01:54.660-10:00:   Request: GET http://server.wattdepot.org:8190/wattdepot/sources/Ilima/e
nergy/?startTime=2011-11-01T00:00:00.000-10:00&endTime=2011-11-30T15:01:54.660-10:00&samplingInterval=15

energy-since Mokulua

Not enough arguments.

current-power

Invalid arguments for current-power.

current-power Mokulua

Mokulua is not a valid source name.

daily-energy

Not enough arguments.

daily-energy Ilima 2011-12-99

Argument "2011-12-99" is invalid :Invalid value 99 for Day field.

daily-energy Mokulua

Not enough arguments.

rank-towers  2011-11-28

Invalid arguments for rank-towers.

rank-towers  2011-11-24 2011-11-99
Argument "2011-11-99" is invalid.

rank-towers  2011-11-30 2011-11-24

End date must be greater than start date.

exit

'exit' is not a command!

The system responded in a useful and helpful way under both valid and non-valid commands and command line arguments with a few exceptions.  My conclusion is that the system satisfies prime directive 2 by providing clear, easy to understand instructions and documentation that is basically correct. The system runs as expected, and properly handles both valid and invalid input commands and command line arguments, except in a few cases. I have 'beautified' the output: in reality, the help message is displayed after every command, valid or invalid, and makes it very difficult to see the valid command output or even the error messages among all the help text.

Prime Directive 3: Can an external developer successfully understand and enhance the system?

To answer this question, I began by carefully reviewing the developer's guide wiki. It states that coding standards and quality assurance standards should be followed for this project, and briefly describes tasks a new developer can use. However, it says to use an xml file format for development without noting that the xml file is just an Eclipse code format file. It says to use the Elements of Java Style without saying what it is.

The developer wiki also says to use Issue-Driven Project Management and includes a brief explanation, even though this is clearly optional, just as Eclipse is optional. It says to check the CI build and includes a link, indicating the process followed by the developers.  The developer wiki does not explain how to generate JavaDoc documentation, and although it provides clear instructions on how to verify the system, detailed instructions for obtaining and building the system from source are not explicitly provided.

I checked out the sources from SVN (read only) using Tortoise SVN, generated and reviewed all the JavaDoc pages. They are well-written and informative, providing a basic understanding of the system architecture. Additionally, the names of the system packages, classes, methods, and fields are well chosen, and indicate their underlying purpose.  

Building the system from sources without errors was simple, just invoking a single command, Ant. However, verifying the system (Ant -verify verify.build.xml) failed about half the time while running the JUnit tests. The JUnit tests also printed alot of text to the console, indicating that the test were probably being run manually.

I generated coverage information regarding the system using Jacoco and it reported 80%, 60%, and 0% for the three packages in the system, for a total of 69% coverage. The test cases exercises 1,018 of 3,321 lines of code.

The test case source code should show how the developers are assuring the correctness of the functionality of the system.  Even though documentation is present, I found that in 50% of the test cases, it is difficult to understand exactly what functionality is being tested, and/or the methodology used. I believe more comments, or even better tests, would help. There are also spelling and grammar errors.

Although coverage is 69%, the command and processor test cases clearly do test the system under both valid and invalid input. I believe that in most cases, these existing test cases would prevent a new developer from making changes that break the existing code. Of course this depends on the level of programmer expertise and understanding of how to extend the current system.

A thorough examination of the source code shows that coding standards are followed and the code is commented appropriately.  The code is relatively easy to understand, and although the amount of commenting seems appropriate, some comments are clearly superfluous, and in other cases, insufficient. There are some classes such as FakeCommand and ReallyFakeCommand that seem to have been built for testing purposes. Not only are these the most confusing part of the system, they have the poorest documentation as well.

Examining the issues pages on Google project hosting clearly shows what parts of the system were built by each developer on the team, and it could be used to determine which developer would be the best person to ask questions regarding a particular component of the system. Also it reveals that roughly speaking, an equal amount of work (7,10, and 12 issues) from each of the developers went into the development of the system. Examining the issues pages does not account for the amount of work each issue required.

By inspecting the project page on the Jenkins CI server, it appears that any build failures were corrected promptly; the longest failure between successful builds was only 20 minutes.

From Google project hosting source changes page, I can see that only 5 of about 50 commits was not associated with a specific issue, and the total number of commits was 58 including the editing of wiki pages. By examining the commit log messages, issues page, and CI server, I can observe that the system was built in a consistent, methodical way.

My conclusion is that the system does satisfy prime directive 3, and a new external developer could easily and successfully understand the current system in its current state, and successfully enhance the system in the future.