CWB offers a ConsoleRunner class that can be used to run Cucumber features from the command line.  Compared to the standard Cucumber Main class, if offers the following advantages:

• parallel execution of features in separate processes
• automatic retry of failed scenarios
• default logging and html report generation

Getting started

We usually configure running the features from Maven.  The simplest pom xml might look like this:

<build>
   	<plugins>
    	<plugin>
         	<artifactId>maven-antrun-plugin</artifactId>
         	<version>1.7</version>
         	<executions>
            	<execution>
               		<phase>integration-test</phase>
               		<configuration>
                  		<target>
					 		<java classname="com.foreach.cuke.core.cli.ConsoleRunner" fork="yes" failonerror="true">
					   			<classpath refid="maven.test.classpath"/>
						   		<sysproperty key="file.encoding" value="UTF-8"/>
						   		<arg line="--output ${project.build.directory}/cucumber-results"/>
						   		<arg line="--threads 3"/>
						   		<arg line="--retries 1"/>
						   		<arg line="--glue com.foreach.cuke"/>
						   	   	<arg line="--tags ~@ignore"/>
								<arg line="${project.basedir}/src/test/resources/features"/>
							</java>
						</target>
					</configuration>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

The base format is:

java com.foreach.cuke.core.cli.ConsoleRunner [arguments] <directory>

The single <directory> contains all features files that should be inspected for execution.  That directory and all sub-directories will be scanned for .feature files.

The arguments in the example above are:

More arguments are available to finetune the execution, for a full list please see the section below or execute the ConsoleRunner without any arguments.

Parallel execution

The ConsoleRunner allows you to run tests in parallel.  This is done by starting a separate Java process for every batch of tests you want executed.  Upon start the ConsoleRunner will scan for all scenarios that need running and divide them into batches of approximately the same size.  For every batch a separate process will be launched.  The maximum number of processes is controlled by the threads argument.

Running tests in parallel can gain you a lot of time when executing tests, at the cost of system resources.  Also because every batch is a separate process, there is some overhead involved and only worth it for longer running tests (taking several seconds per scenario).

Passing properties to scenario execution

The ConsoleRunner will set all properties configured using -P <propertyname>=<propertyvalue> arguments as system properties on the batch process.

Configuring batch process classpath

By default every batch process will inherit the classpath of the initial ConsoleRunner application.  Sometimes that classpath is detected incorrectly, in which you can configure the classpath to use with the batchClassPath argument.

Controlling scenario distribution

When dividing scenarios in batches the ConsoleRunner will do so aggressively: every Scenario or Scenario Outline example will count as a separate item and might end up in a different batch.  There is absolutely no guarantee of the order of execution and if scenarios are not completely independent this will cause problems.

Keeping scenarios together

Even though ill-advised, it can sometimes be handy to have scenarios depend on each other.  Especially when transitioning from sequential execution to parallel execution you might end up with unforeseen problems.  You can tell the ConsoleRunner not to split scenarios by using the @keepTogether tag.

If you use it on a Scenario Outline, the examples of that outline will be executed by the same process, in their respective order.  If you use it on a Feature, all scenarios in that feature will be executed in sequence within the same process.

Be aware that tag filtering or scenario retries might exclude some scenarios or examples from running, however the order within the feature will always be respected if @keepTogether is present.

Mutliple processes and SAHI

You can use parallel execution with SAHI based tests, however it's advised to use at least SAHI 5.0.  Note that not all configured browsers might support multiple instances on the proxy, and it is also usually better to launch the browser instances in private browsing mode.

Process timeout

Sometimes a process can hang for a very long time, especially with SAHI tests we see this happen once in a while.  In that case we consider the process timed out and kill it explicitly.  By default this happens after 900 seconds, but you can configure the number of seconds with the processTimeout argument.

Batch threshold

Due to the overhead of a separate process it is sometimes not worth parallel execution if you only have to run a couple of tests.  You can tune this with the batchThreshold argument.  No matter the number of threads allowed, parallel execution will only happen if there are at least that number of scenarios to run.  The default value for this argument is 1 meaning a separate process might be launched for just a single scenario.

Automatic retry of failing tests

Sometimes a test can fail and then succeed on a simple rerun. I see this happen especially with SAHI based tests in complex websites.  It's possible to have the ConsoleRunner automatically retry failing tests any number of times.  After the initial execution, the application will take all failed scenarios and rerun only those.  The application will then repeat this process until all tests have succeeded or the maximum number of retries has occurred.  How many times tests can be retried is controller by the retries argument.

Scenario retry limit

Automatic retries are only useful if a couple of scenarios are suffering from random failures.  To tune this you can set the scenarioRetryLimit argument.  If more than this number of scenarios are failing, the ConsoleRunner will fail and no retries will occur.  The default value is 10.

Report generation and logs

The ConsoleRunner writes logs and reports to the directory determined by the output argument.  After a full run you would find a structure very similar to the following:

Migrating from using Cucumber Main

The previous pom.xml of CWB configured Cucumber Main and cucumber-reporting as two consecutive plugins.  The ConsoleRunner generates the HTML reports by default, so the separate cucumber-reporting task can usually be removed.

 <plugin>
   <artifactId>maven-antrun-plugin</artifactId>
   <version>1.7</version>
   <executions>
      <execution>
         <phase>integration-test</phase>
         <configuration>
            <target>
               <java classname="cucumber.api.cli.Main" fork="yes">
                  <sysproperty key="file.encoding" value="UTF-8"/>
                  <sysproperty key="environment" value="${environment}"/>
                  <sysproperty key="browser" value="${browser}"/>
                  <sysproperty key="nopause" value="${nopause}"/>
                  <sysproperty key="highlight" value="${highlight}"/>
                  <classpath refid="maven.test.classpath"/>
                  <arg line="--format com.foreach.cuke.core.formatter.ConsoleReporter"/>
                  <arg line="--format html:${project.build.directory}/cucumber/${report.output.dir}/plain-html-reports"/>
                  <arg line="--format junit:${project.build.directory}/cucumber/${report.output.dir}/cucumber-report.xml"/>
                  <arg line="--format json:${project.build.directory}/cucumber/${report.output.dir}/cucumber-report.json"/>
                  <arg line="--glue com.foreach.cuke"/>
                  <arg line="--glue mypackage"/>
                  <arg line="--tags ~@ignore"/>
                  <arg line="--tags ${tags}"/>
                  <arg line="${project.basedir}/features/${features}/"/>
               </java>
            </target>
         </configuration>
         <goals>
            <goal>run</goal>
         </goals>
      </execution>
   </executions>
</plugin>
<plugin>
   <groupId>net.masterthought</groupId>
   <artifactId>maven-cucumber-reporting</artifactId>
   <version>0.0.4</version>
   <executions>
      <execution>
         <id>cucumber-reports</id>
         <phase>integration-test</phase>
         <goals>
            <goal>generate</goal>
         </goals>
         <configuration>
            <projectName>${project.name}</projectName>
            <outputDirectory>
               ${project.build.directory}/cucumber/${report.output.dir}/pretty-html-reports
            </outputDirectory>
            <cucumberOutput>
               ${project.build.directory}/cucumber/${report.output.dir}/cucumber-report.json
            </cucumberOutput>
            <enableFlashCharts>true</enableFlashCharts>
         </configuration>
      </execution>
   </executions>
</plugin>

 <plugin>
   <artifactId>maven-antrun-plugin</artifactId>
   <version>1.7</version>
   <executions>
      <execution>
         <phase>integration-test</phase>
         <configuration>
            <target>
               <java classname="com.foreach.cuke.core.cli.ConsoleRunner" fork="yes" failonerror="true">
                  <classpath refid="maven.test.classpath"/>
     			  <sysproperty key="file.encoding" value="UTF-8"/>
                  <arg line="--output ${project.build.directory}/cucumber/"/>
                  <arg line="--threads 1"/>
                  <arg line="--retries 0"/>
                  <arg line="--glue com.foreach.cuke"/>
				  <arg line="--glue mypackage"/>
                  <arg line="--tags ~@ignore"/>
			      <arg line="--tags ${tags}"/>
                  <arg line="-P browser=${browser}"/>
				  <arg line="-P highlight=${highlight}"/>
				  <arg line="-P environment=${environment}"/>
                  <arg line="${project.basedir}/src/test/resources/features"/>
               </java>
            </target>
         </configuration>
         <goals>
            <goal>run</goal>
         </goals>
      </execution>
   </executions>
</plugin>

Skipping the HTML reports

If you want to skip the HTML reports generation entirely, you can do so by passing the disableHtmlReports argument.

Customizing the HTML reports

It's not possible to customize the HTML reports generated by the ConsoleRunner.  If you pass a buildName system property to the ConsoleRunner application, that build name will be displayed on the HTML reports instead of the execution time.  If you need more customization your should skip the report generation and revert to generating them manually using the JSON report.

Command line overview

The following is a list of all command line options the ConsoleRunner supports:

Tips and tricks

I want my maven build (not) to fail when the ConsoleRunner has failed tests

The ConsoleRunner will return exit code -1 if any tests were still failing in the end.  In maven/ANT it is the failonerror attribute on the java task that will determine if the entire build will fail or not.

ConsoleRunner starts fine but all batch processes fail/Batch processes do not find my test code

The problem is a badly configured classpath.  The ConsoleRunner probably cannot detect the classpath correctly and pass it to the batch processes.  Solution is to manually configure the classpath using batchClassPath argument.

 <execution>
   <phase>integration-test</phase>
   <configuration>
      <target>
		 <!-- ensure the classpath is available as an ANT property -->
         <property name="maven.test.classpath" refid="maven.test.classpath" />
         <java classname="com.foreach.cuke.core.cli.ConsoleRunner" fork="yes"
               failonerror="true">
            <classpath refid="maven.test.classpath"/>
            <sysproperty key="file.encoding" value="UTF-8"/>
            <arg line="--output ${project.build.directory}/cucumbers"/>
            <arg line="--threads 3"/>
            <arg line="--retries 1"/>
            <arg line="--processTimeout 10"/>
            <arg line="--glue com.foreach.cuke"/>
            <arg line="--tags ~@ignore"/>
            <arg line="--batchClassPath ${maven.test.classpath}"/>
            <arg line="-P nopause=true"/>
            <arg line="${project.basedir}/src/test/resources/features"/>
         </java>
      </target>
   </configuration>
   <goals>
      <goal>run</goal>
   </goals>
</execution>

I get an error that the command is too long when executing the ConsoleRunner

The problem is most likely a very long classpath that you are passing as an argument.  A solution to this could be to copy all dependencies into a single directory, and then passing that directory as a classpath using <dir>/*.