Home › Blogs › charlie.collins's blog

Testing a Java "executable" (class with a main method, command line options, and std output)

Mon, 08/11/2008 - 06:28 — charlie.collins
I was recently tasked with helping to create a Java class that had the same command line options and operation as a C executable. We already had a Java API that handled the actual work, so the goal was really just to handle the command line options and exit codes, hook into the API correctly for all the options, and output the correct stdout and stderr responses.

Sounds pretty easy at first, but, once you make a few options and then try to write a test for a class that only produces stdout (System.out) and stderr (System.err) output, and then exits the VM every time it does something, you realize there is a bit more to it than first glance reveals.

After I got done doing this for a fairly involved class with a lot of options, I made a small example application - App.java - for future reference of the key points. In this post I will discuss the example application, beginning with the easy stuff, parsing the command line options.

I used commons-cli for the command line option handling. There are other libraries out there that do this too, and it's also not out of the question to just roll your own command line parsing, but I was familiar with CLI from days past, and it has always worked well for me, so I just went back to it. The command line processing looks like the following: 
   /**
    * Parse the command line options, and produce usage information - uses Commons-CLI.
    *
    * @param args input
    * @throws ParseException if problem encountered parsing options
    */
   private void parseCommandLine(final String[] args) throws ParseException {

      CommandLineParser parser = new PosixParser();

      // create the boolean options
      Options options = new Options();
      options.addOption("h", "help", false, "print this message");
      options.addOption("v", "version", false, "show version");

      // create the name/value pair options - d,r,l
      options.addOption(OptionBuilder.hasOptionalArg().withArgName("value").withDescription(
         "echo the value passed in (required)").create("e"));

      // parse the options
      CommandLine line = parser.parse(options, args);

      if (line.hasOption("h")) {
         HelpFormatter help = new HelpFormatter();
         help.printHelp(80, "TestMainExample [options]", "demonstrates a Main java class and options handling/testing",
                  options, null);
         System.exit(App.EXIT_NO_ERROR);
      }
      else if (line.hasOption("v")) {
         System.out.println("Version 1.0");
         System.exit(App.EXIT_NO_ERROR);
      }

      // if it's the e option, set the echo instance var value
      if (line.hasOption("e")) {
         String value = line.getOptionValue("e");
         if (value != null) {
            this.echo = value;
         }
      }
      else {
         System.err.println("ERROR: -e option with value is required");
         System.exit(App.EXIT_USAGE_ERROR);
      }
   }
With CLI you have a CommandLineParser, and Option[] array. You may notice that I created a separate method just for the CLI stuff. I didn't start out this way, but in order to make it more testable, as we shall come to, I refactored to avoid stuffing everything into the main method (in addition to being more testable, this also makes the class just easier to read and maintain overall). Also, you can see that when a USAGE_ERROR or other problem occurs, the VM is exited with the corresponding exit code.

Next, after the command line handling was in place, I moved the logic into a "process" method, like so: 
   /**
    * Process method to keep logic/etc out of main and help testability. 
    * 
    * @param args
    */
   private void process(String[] args) {

      // first parse command line to set state 
      try {
         this.parseCommandLine(args);
      }
      catch (ParseException e) {
         System.err.println("ERROR processing command line args - " + e.getMessage());
         System.exit(App.EXIT_UNKNOWN_ERROR);
      }

      // normal processing 
      System.out.println("ECHO  . . . " + this.echo);
      System.exit(App.EXIT_NO_ERROR);
   }
The process method invokes the command line parser first, to ensure that instance variables are setup based on the selected options. This is one of the key points. I avoided statics for this stuff, which you often see in "main" method type classes, and just made sure the command line processing was the first step, after which the instance variables are used to "do stuff." (In this case just a simple echo, but whatever you need can be handled at this point.)

After the process method the next few things I needed were the main method itself, and a constructor. These are shown below: 
   /**
    * Constructor that requires args, and then invokes process 
    * (guarantees that process occurs). 
    * 
    * @param args
    */
   protected App(String args[]) {
      this.process(args);
   }

   /**
    * Command line main runner, makes an instance.
    * 
    * @param args
    */
   public static void main(String[] args) {
      new App(args);
   }
The main method just makes a new instance of the class, and hands it the command line args[]. The constructor calls the process method (which goes through the command line parsing and then performs the logic, as we saw previously). One thing that may seem curious about this is that the constructor is not private. I did that to help out the testing, which we will look at next.

Changing ANY aspect of your class exclusively to facilitate testing always seems like a bad smell to me. Don't get me wrong, I think testing is very important, essential even, but I also think you should have more than just that in mind when you refactor (or add new dependencies for injection frameworks or such). Usually you can justify changes with enhanced readability, or better decoupling, etc, and not say, well, "it's easier to test" this way (just something to keep in mind, I like to see people justify changes with more than "it's easier to test this way" - well, *why* is it easier to test, there is another benefit right?).

At any rate, I made the constructor protected access, just for the tests. Sure there are a few other ways to create private objects that I could have hooped through, but the simplest way was just make it protected and put the test in the same package (as really should be the case anyway). This leads into the tests themselves, where the real challenges in this project were.

So how well does JUnit work when the VM exits, and when the only output is System.out/err? Out of the box, it really doesn't work. But, after searching around a bit I found a lot of people asking about this (testing a main method class), and only a few helpful responses (no articles though, hence this one ;)). Specifically the most helpful response was Lasse Reichstein Nielsen on a velocityreviews forum.

Nielsen mentioned using a security manager that prevents calling System.exit and then catching SecurityException. I went with that approach, modifying the code supplied in the forum a bit to create an AbstractSecurityTestCase: 
package com.totsp.example;
import java.security.Permission;
import junit.framework.TestCase;

public class AbstractSecurityTestCase extends TestCase {

   public AbstractSecurityTestCase(String name) {
      super(name);
   }

   protected static class ExitException extends SecurityException {
      public final int status;
      public ExitException(int status) {
         super("There is no escape!");
         this.status = status;
      }
   }

   private static class NoExitSecurityManager extends SecurityManager {
      @Override
      public void checkPermission(Permission perm) {
         // allow anything.
      }

      @Override
      public void checkPermission(Permission perm, Object context) {
         // allow anything.
      }

      @Override
      public void checkExit(int status) {
         super.checkExit(status);
         throw new ExitException(status);
      }
   }

   @Override
   protected void setUp() throws Exception {
      super.setUp();
      System.setSecurityManager(new NoExitSecurityManager());
   }

   @Override
   protected void tearDown() throws Exception {
      System.setSecurityManager(null); 
      super.tearDown();
   }   
}
With that testing support in place, it then became pretty easy to write tests for a main[] method class, not only checking the exit codes, but also redirecting the System.out and System.err output and asserting against it as well. My AppTest test class demonstrates this: 
package com.totsp.example;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import junit.framework.Assert;

public class AppTest extends AbstractSecurityTestCase {

   private ByteArrayOutputStream out;
   private ByteArrayOutputStream err;
   private PrintStream stdout;
   private PrintStream stderr;

   public AppTest(String testName) {
      super(testName);
   }

   public void setUp() throws Exception {
      super.setUp();
      this.stdout = System.out;
      this.stderr = System.err;
      this.out = new ByteArrayOutputStream();
      this.err = new ByteArrayOutputStream();      
      System.setOut(new PrintStream(this.out));
      System.setErr(new PrintStream(this.err));
   }

   public void tearDown() throws Exception {
      super.tearDown();
      System.setOut(this.stdout);
      System.setErr(this.stderr);
   }

   public void testOptionH()  {
      try {
         String[] args = new String[] {"-h"};
         new App(args);
         Assert.fail("should have exited with status 0");
      }

      catch (ExitException e) {
         String output = new String(this.out.toByteArray());
         Assert.assertTrue(output.indexOf("usage: TestMainExample [options]") != -1);
         Assert.assertEquals("Exit status", 0, e.status);
      }
   }
   
   public void testOptionV()  {
      try {
         String[] args = new String[] {"-v"};
         new App(args);
         Assert.fail("should have exited with status 0");
      }

      catch (ExitException e) {
         String output = new String(this.out.toByteArray());
         Assert.assertTrue(output.indexOf("Version 1.0") != -1);
         Assert.assertEquals("Exit status", 0, e.status);
      }
   }

   public void testInvalidCommandLine()  {
      try {
         String[] args = new String[] {"some", "crap", "with", "no", "e", "option"};
         new App(args);
         Assert.fail("should have exited with status 2");
      }

      catch (ExitException e) {         
         Assert.assertEquals("Exit status", 2, e.status);
      }
   }

   public void testProcessE()  {
      try {
         String[] args = new String[] {"-e", "TESTING"};
         new App(args);
         Assert.fail("should have exited with status 0");
      }

      catch (ExitException e) {
         String output = new String(this.out.toByteArray());
         Assert.assertTrue(output.indexOf("ECHO  . . . TESTING") != -1);
         Assert.assertEquals("Exit status", 0, e.status);
      }
   }
}
So, this was a quick example, but has a few pretty key, and handy, points: you can handle command line options easily with CLI, you can refactor to keep things out of main, and you can test classes that exit the VM with the security manager approach.

This full example is available as a Maven project in SVN here: TestMainExample.