In this blog entry I post about my experience reviewing and evaluating the open-source software system named Hale-aloha-cli-grads.
Introduction and
To understand the system as well as and the evaluation criteria, I will first provide summary data of some background information necessary to understand the issues examined in the review.
Hale-aloha-cli-grads
The Hale-aloha-cli-grads system under review is an open-source project that implements a
command line interface (CLI) for WattDepot, created by Andrea Connell, Leo DeCandia, and Sergey Negrashov. It retrieves and displays energy and power data for the four Hale Aloha residential towers at the University of Hawaii Manoa campus.
The Hale-aloha-cli-grads system under review is an open-source project that implements a
command line interface (CLI) for WattDepot, created by Andrea Connell, Leo DeCandia, and Sergey Negrashov. It retrieves and displays energy and power data for the four Hale Aloha residential towers at the University of Hawaii Manoa campus.
WattDepot
The wattdepot open-source project developed at the University of Hawaii is a collection of technologies and tools for collecting and storing data from electricity meters for smart grid research and experimentation. From the wattdepot Google hosting site: “WattDepot is an open source, RESTful web service that collects electricity data (such as current power utilization or cumulative power utilization) from meters and stores it in a database. The data can then be retrieved by other tools for visualization and analysis. It is designed to provide infrastructure for experimentation and development of "Smart Grid" applications.”
For the purposes of this evaluation, the three prime directive of open-source software development are summarized as follows:
Issue-Driven Project Management
The basics of Issue Driven Project Management are summarized as follows:
The 3 Prime Directives of open-source Software Development
For the purposes of this evaluation, the three prime directive of open-source software development are summarized as follows:
- Does the system accomplish a useful task?
- Can an external user can successfully install and use the system?
- Can an external developer successfully understand and enhance the system?
The basics of Issue Driven Project Management are summarized as follows:
- Divides work into tasks
- Tasks take no longer than 2 days
- Tasks are specified by issues
- Every person always has an open task
- Every commit specifies an issue in its log comment
- Each person has a task
- Each person knows what it is
- Each person knows what the next task is
- Each person is rarely blocked
- Completing a task brings the project closer to completion
- Project state is visible at all times to all team members
- Breakdowns in PM are recognized by everyone, quickly
Continuous Integration
The basic concepts of Continuous Integration are:
The basic concepts of Continuous Integration are:
- The System must be able to be built and tested automatically
- The System must be under CM
- Everyone commits every day or two
- Upon commit, The System is immediately and automatically compiled, tested, and developers notified
Review and Evaluation
My review and evaluation will cover important
aspects of open-source software development, as well as answer detailed questions about the system including: build process, developer contribution, documentation, testing, QA, process, software quality, extensiblity, and integration. These elements will be grouped into three main sections that correspond to the three prime directives of open-source software development.
Prime Directive 1: Does the system accomplish a useful
task?
Following the directions which were 'spread out' among the home page, installation guide, and user guide, I downloaded, installed, and ran the system under Windows
7.
I exercised the system using various commands and command line arguments.
I exercised the system using various commands and command line arguments.
The functionality that is present in the system corresponds
to what is documented on the project home page, and the messages displayed for bad commands and command arguments are informative.
I was surprised to see the entire help menu printed after every command, and the help menu does not specify the valid tower and lounge names.
The help command does not show that 'help' and 'quit' are available commands. Also, there is no command line prompt, just a blinking cursor, and as mentioned above, I don't think 3 pages are necessary to get the information required to download, install and run the system.
However, I conclude that prime directive #1 is satisfied because a useful task is performed and the system basically behaved as outlined in the documentation. There were no difficulties in downloading, installing, and running the system, and it displayed expected output based on input parameters.
I carefully reviewed the project site, Including the home
page, user guide wiki, and downloads page.
The home page provides a clear understanding of what the
system is supposed to accomplish, and is accompanied by a list of user commands
recognized by the system. The user guide has a screen snap of the program in
action, showing both sample input and output. The user guide also explains that
both physical and virtual ‘sources’ are supported. However it does not indicate
what version of the Java JRE is required to run the system.
The user guide links to an installation page that
provides clear details on how to download, install, and execute the system from
the perspective of a user, but does not include a link to the downloadable zip
file. The downloadable zip file is stamped with a version number, and the zip contains
an executable jar file. I did not have to compile and build the system in order
to use it. I exercised the system under both valid and invalid
inputs as documented below.
Valid commands and command line arguments:
energy-since
Ilima 2011-11-28
Total
energy consumption by Ilima from 2011-11-28 00:00:00 to 2011-11-30 14:42:26 is:
1616.9 kWh
energy-since
Mokihana 2011-11-28
Total
energy consumption by Mokihana from 2011-11-28 00:00:00 to 2011-11-30 14:43:01
is: 1616.7 kWh
current-power
Ilima
Ilima's
power as of 2011-11-30 14:45:12 was 26.0kW
daily-energy
Ilima 2011-11-28
Ilima's
energy consumption for 2011-11-28 was: 611.7 kWh
daily-energy
Ilima-A 2011-11-28
Ilima-A's
energy consumption for 2011-11-28 was: 104.7 kWh
rank-towers 2011-11-28 2011-11-30
For
the interval 2011-11-28 to 2011-11-30, energy consumption by tower was:
Ilima 1255 kWh
Mokihana 1261 kWh
Lehua 1268 kWh
Lokelani 1468 kWh
rank-towers 2011-11-24 2011-11-30
For
the interval 2011-11-24 to 2011-11-30, energy consumption by tower was:
Lehua 3364 kWh
Mokihana 3379 kWh
Ilima 3464 kWh
Lokelani 3942 kWh
help
--Available
commands are:
energy-since:
[tower | lounge] [Start]
Returns
the energy used since the date (yyyy-mm-dd) to now.
current-power
[tower | lounge]
Returns
the current power in kW for the associated tower or lounge.
daily-energy:
[tower | lounge] [Date]
Returns
the energy in kWh used by the tower or lounge for the specified date
(yyyy-mm-dd).
rank-towers: [start date] [end date]
Returns
a list in sorted order from least to most energy consumed between the [start]
and [end] date (yyyy-mm-dd)
--Enter
a command (type 'quit' to exit):
quit
quitting...
Invalid commands and command line arguments:
"Just pressing the return key"
‘ ‘ is not a valid command
energy-since Ilima 2011-11
Bad xml 400: Range extends beyond sensor data, startTime 2011-11-01T00:00:00.000-10:00, endTime 2011-11-30T15:01:54.660-10:00: Request: GET http://server.wattdepot.org:8190/wattdepot/sources/Ilima/energy/?startTime=2011-11-01T00:00:00.000-10:00&endTime=2011-11-30T15:01:54.660-10:00&samplingInterval=15
energy-since Mokulua
Not enough arguments.
current-power
Invalid arguments for current-power.
current-power Mokulua
Mokulua is not a valid source name.
daily-energy
Not enough arguments.
daily-energy Ilima 2011-12-99
Argument "2011-12-99" is invalid :Invalid value 99 for Day field.
daily-energy Mokulua
Not enough arguments.
rank-towers 2011-11-28
Invalid arguments for rank-towers.
rank-towers 2011-11-24 2011-11-99
Argument "2011-11-99" is invalid.
Argument "2011-11-99" is invalid.
rank-towers 2011-11-30 2011-11-24
End date must be greater than start date.
exit
'exit' is not a command!
The system responded in a useful and helpful way under both
valid and non-valid commands and command line arguments with a few exceptions. My conclusion is
that the system satisfies prime directive 2 by providing clear, easy to
understand instructions and documentation that is basically correct. The system runs as
expected, and properly handles both valid and invalid input commands and
command line arguments, except in a few cases. I have 'beautified' the output: in reality, the help message is displayed after every command, valid or invalid, and makes it very difficult to see the valid command output or even the error messages among all the help text.
Prime Directive 3: Can an external developer successfully
understand and enhance the system?
To answer this question, I began by carefully reviewing
the developer's guide wiki. It states that coding standards and quality
assurance standards should be followed for this project, and briefly describes tasks
a new developer can use. However, it says to use an xml file format for
development without noting that the xml file is just an Eclipse code format
file. It says to use the Elements of Java Style without saying what it is.
The developer wiki also says to use Issue-Driven Project Management and includes a brief explanation,
even though this is clearly optional, just as Eclipse is optional. It says to
check the CI build and includes a link, indicating the process followed by the developers. The developer wiki does not explain how
to generate JavaDoc documentation, and although it provides clear instructions on
how to verify the system, detailed instructions for obtaining and building the
system from source are not explicitly provided.
I checked out the sources from SVN (read only) using
Tortoise SVN, generated and reviewed all the JavaDoc pages. They are well-written
and informative, providing a basic understanding of the system architecture. Additionally,
the names of the system packages, classes, methods, and fields are well chosen,
and indicate their underlying purpose.
Building the system from sources without errors was
simple, just invoking a single command, Ant. However, verifying the system (Ant -verify verify.build.xml) failed about half the time while running the JUnit tests. The JUnit tests also printed alot of text to the console, indicating that the test were probably being run manually.
I generated coverage information regarding the system using
Jacoco and it reported 80%, 60%, and 0% for the three packages in the system,
for a total of 69% coverage. The test cases exercises 1,018 of 3,321 lines of
code.
The test case source code should show how the developers
are assuring the correctness of the functionality of the system. Even
though documentation is present, I found that in 50% of the test cases, it is difficult
to understand exactly what functionality is being tested, and/or the methodology
used. I believe more comments, or even better tests, would help. There are also
spelling and grammar errors.
Although coverage is 69%, the command and processor test cases
clearly do test the system under both valid and invalid input. I believe that in
most cases, these existing test cases would prevent a new developer from making
changes that break the existing code. Of course this depends on the level
of programmer expertise and understanding of how to extend the current system.
A thorough examination of the source code shows that
coding standards are followed and the code is commented appropriately. The
code is relatively easy to understand, and although the amount of commenting
seems appropriate, some comments are clearly superfluous, and in other cases,
insufficient. There are some classes such as FakeCommand and ReallyFakeCommand that seem to have been built for testing purposes. Not only are these the most confusing part of the system, they have the poorest documentation as well.
Examining the issues pages on Google project hosting
clearly shows what parts of the system were built by each developer on the
team, and it could be used to determine which developer would be the best
person to ask questions regarding a particular component of the system. Also it
reveals that roughly speaking, an equal amount of work (7,10, and 12 issues)
from each of the developers went into the development of the system. Examining
the issues pages does not account for the amount of work each issue required.
By inspecting the project page on the Jenkins CI server,
it appears that any build failures were corrected promptly; the longest failure
between successful builds was only 20 minutes.
From Google project hosting source changes page, I can see
that only 5 of about 50 commits was not associated with a specific issue, and
the total number of commits was 58 including the editing of wiki pages. By examining
the commit log messages, issues page, and CI server, I can observe that the system
was built in a consistent, methodical way.
My conclusion is that the system does satisfy prime
directive 3, and a new external developer could easily and successfully
understand the current system in its current state, and successfully enhance
the system in the future.
