Reflection on Best Practice Software Engineering

In this post I ask and answer 5 quick questions about software testing that any software engineer should know. Try to answer the questions before looking at the answers!

1. Why are coding standards and tools like Checkstyle important? Is running Checkstyle an 'expensive' operation?

Coding standards help by enforcing consistency that makes code easier to read and understand across an organization. Enforcing standards saves time by allowing developers to spend more time coding and less time formatting it. Once the standards are agreed upon, tools like Checkstyle can be run often, from a build script, with almost no 'cost'.

2. Why is "exhaustive" testing difficult?

There are too many possible program inputs and outputs to 'exhaustively' prove that a program is "correct".

3. What is equivalence partitioning?

Equivalence partitioning divides all possible input data into classes or partitions that represent each input 'partition', such that testing one value is equivalent to testing all values in that 'partition'. For example, if the input is a number, a test design heuristic might suggest the following test values: typical, empty or null, illegal, minimum, and maximum, too small a value, and too large a value.

4. What type of errors cannot be detected with white-box testing?

Errors of omission, i.e. code that should have been written - missing exception handlers, missing conditional tests, missing tests for validating command line argument formats.

5. What is the difference between validation and verification testing?

Validation testing basically tests that "we are we building the right software to get the job done" and verification testing tests that "we are building it correctly". In addition, validation testing involves the customer, and verification testing involves the code-base.

Configuration Management with Subversion and Google Project Hosting

CM using Subversion

It is almost impossible to build large, scalable, collaborative software systems without a (good) configuration management system. We can find evidence of this as the demand for better CM tools over time has driven the availability and adoption of more robust and easier to use CM systems. Some of these are relatively new - like Git - and are gaining popularity quickly, with the promise of speed, reliability, security, and a host of other features.

Any Configuration Management approach should address the three classic CM problems:

• The double maintenance problem (must prevent multiple copies of the same file)

• The shared data problem (must allow developers to work on the same file simultaneously)

• The simultaneous update problem (must allow the developers working on the same file to update it without clobbering each others' changes)

Subversion solves these problems by using file versioning, file checkout and commit, and resolving commit conflicts using merging. In addition there are many more features that I won't go into in this post, but I encourage you to check out on the Apache Subversion site.

For my local Subversion client, I used the Tortoise SVN client based on Subversion 1.7, which integrates seamlessly into the Windows 7 environment using context-sensitive menus. It decorates files and folders under Subversion control with easy-to-understand icons.

Using Tortoise SVN, it was very easy to put my project (source tree) under Subverion control, make my initial commit, and thereby place my project into the repository on Google project hosting. Almost immediately I realized I needed to delete some files I really didn't want in the repository, and again, this was surprisingly easy to do.

Google Project Hosting

Project Hosting using Google Code provides an easy to understand and easy to use environment for collaborative development of open source projects. It was a piece of cake to create a new project backed by Subversion for the repository. My open source project hosts my competitive Robocode robot code. It provides issue tracker, wiki pages, admin, download, and other useful pages/links on a per-project basis. I was able to document changes to the repository, view revisions/changes, add 'commit' access for project collaborators, and create both a UserGuide wiki and a DeveloperGuide wiki using the builtin editors.

I accomplished all tasks I set out to do in this first attempt at open source project hosting using a Google project hosting and a local Subversion client.

I had a few moments of misunderstanding when .svn folders were not appearing in my Tortoise SVN client. By using a shell window, I made sure they weren't there and it was not a Tortoise SVN issue. However I was able to checkout files, make changes and commits with correct expected results.

I learned from the Tortoise SVN forum (registration required to post) that Subversion 1.7 no longer maintains a .svn directory for each subdirectory of a project's files. Instead it uses a single .svn directory located at the project root. I verified this on the Apache Subversion site.

I was surprised when I was able to checkout files from the repository without supplying any credentials. I soon discovered that Google Project Hosting asks for credentials when comitting changes, not when checking out files from the repository. And Tortoise SVN can store those credentials resulting in an even quicker, more seamless workflow.

Robocode, Automated Build Systems, and Automated Testing

 Overview

As an exercise in developing systems that scale, I continued development of a competitive Robocode robot, integrating it with an automated build system that builds the system, runs the robot simulation, and tests the code using JUnit tests, acceptance tests, behavioral tests, and quality assurance tools. This build system uses a collection of Ant scripts; coding is still done using Eclipse. The big difference is that now, with a few simple commands from a shell, I can accomplish all the tasks outlined above. The real benefit is that as the code-base grows, the same scripts can be run. Of course, new tests and script modifications can be added, but the impact of maintaining a few scripts or adding tests should be negligible compared to the time savings if I was doing all the build, run, test, and  quality assurance manually.

The Robocode “Surfer” Robot

Movement: My strategy for moving the Surfer robot is to conserve energy and perhaps fool tracking robots. Surfer follows a counterclockwise circuit of the right half of the battlefield, then reverses its direction and follows a clockwise circuit. Additionally, Surfer conserves energy by not firing often, and varying the bullet strength.

Targeting:  Surfer stops and rotates it’s radar to scan for other robots at 4 places during each clockwise or counterclockwise circuit: top center, top right corner, lower right corner, and bottom center.

Firing: Any time Surfer scans an enemy robot, it fires at it with a bullet strength based on both the distance from the enemy and the size of the battlefield itself. Surfer also fires more bullets (up to 3) if it is closer. The thresholds are: fire 3 bullets if 200 pixels or closer, fire 2 bullets if 300 pixels or closer, or just fire 1 bullet when farther away.

Results:  Here are the results of 5 battles of ten rounds each against some sample robots:

Walls: Surfer wins 0/5.

RamFire: Surfer wins 5/5.

SpinBot: Surfer wins 1/5.

Crazy: Surfer wins 5/5.

Fire: Surfer wins 5/5.

Corners: Surfer wins 3/5.

Tracker: Surfer wins 5/5.

SittingDuck: Surfer wins 5/5.

So Surfer can reliably beat most of these sample bots except Walls, SpinBot, and Corners. Given Surfer’s conservative nature, I think these results are pretty good. To beat Walls, SpinBot, and Corners, I think Surfer needs a more advanced targeting algorithm and perhaps a more advanced strategy for bullet firing.

What I would improve in the Surfer robot initially, includes locking the radar once an enemy is scanned as well as devising a strategy when colliding with another robot. Beyond that, after becoming more familiar with the game physics and the Robocode API, literally dozens of changes and improvements could be made to Surfer.

Testing

My testing approach begins with JUnit tests to test the methods I use to calculate the number of bullets to fire, as well as the bullet strength. I also perform behavioral tests that check if my robot visits the positions on the battlefield that I expect it to, and that the bullets it fires vary in strength. My acceptance tests are implemented by having my robot battle against other robots. In addition I used Jacoco to get feedback on "coverage". This is very useful because it reveals lines of code are not being executed during testing, and if they are not executed, they may well contain bugs.

Quality Assurance

Above and beyond JUnit, behavioral, acceptance tests, and Jacoco, QA was done using checkstyle, findbugs, and PMD. Although these three products have some degree of overlap, they can find many kinds of errors, including 'best practice' coding style errors, unused variables, code that can never be reached, etc... Once they are built into the automated build system they pay big dividends that increase as the project grows and the lines of code increase. To do these type of tests manually would be painstakingly difficult and time consuming.

Lessons learned

From this project I learned that running automated testing and QA tools can be done in matter of seconds, and that these tools will scale to larger systems where manual techniques will not scale. Bugs, incorrect behavior, and errors in coding style, can be discovered more quickly than performing these processes manually - by perhaps an order of magnitude or more. Virtually no large software development effort would be undertaken without tools and techniques similar to the ones I used. Even though the QA tools may take some time to configure and understand, and scripts may take some time to develop, in the long run they will save an enormous amount of money, time, and effort.