Dec 30, 2015

WireSpock - Testing REST service client components with Spock and WireMock

In a previous post I have written about using the Spock framework for the exploratory testing of open source software. In this post I want to showcase a neat technology integration between Spock and the WireMock framework for testing your REST service client components. This is especially useful when testing micro service based architectures, since you want to test the individual service integrations without firing up all the collaborators.

Introducing WireMock

As stated on it's webpage, WireMock is "a web service test double for all occasions". It supports stubbing and mocking of HTTP calls, as well as request verification, record and playback of stubs, fault injection and much more. It actually fires up a small embedded HTTP server, so your code and test interacts with it on the protocol level.

The most convenient way to use WireMock in your test cases is via a JUnit 4.x rule that handles the lifecycle of starting and stopping the mock server before and after each test. There also is a class rule available in case it is sufficient to use the same WireMock instance for the lifetime of the whole test case. The official documentation for the rule can be found here.

The good thing is that you can use the WireMock rule in your Spock specification just like in an ordinary JUnit based test. No magic here. Let's have a look at the following example.
class BookServiceClientSpec extends Specification {

    WireMockRule wireMockRule = new WireMockRule(18080)

    def client = new BookServiceClient("http://localhost:18080")

    def "Find all books using a WireMock stub server"() {
        given: "a stubbed GET request for all books"
        // TODO

        when: "we invoke the REST client to find all books"
        def books = client.findAll()

        then: "we expect two books to be found"
        books.size() == 2

        and: "the mock to be invoked exactly once"
        // TODO
First, the JUnit WireMock rule is created and initialized to listen on port 18080. Next the REST client component under test is created and configured to access the local wire mock server. The test method itself does not do much yet. For it to work we need to stub the response for the findAll() query and we want to check that the mock has been invoked once. Before we continue, let's have a look at the test dependencies required to compile and run the example.
dependencies {
    testCompile 'junit:junit:4.12'
    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'

    testCompile 'com.github.tomakehurst:wiremock:1.57'
    testCompile 'com.github.tomjankes:wiremock-groovy:0.2.0'

Making WireMock Groovy

The last dependency is a small Groovy binding library for WireMock that plays together nicely with Spock. It allows for a more concise stubbing and verification syntax instead of using WireMock's default static imports API. Have a look at the following example to get the idea.
def wireMock = new WireMockGroovy(18080)

def "Find all books using a WireMock stub server"() {
    given: "a stubbed GET request for all books"
    wireMock.stub {
        request {
            method "GET"
            url "/book"
        response {
            status 200
            body """[
                      {"title": "Book 1", "isbn": "4711"},
                      {"title": "Book 2", "isbn": "4712"}
            headers { "Content-Type" "application/json" }

    when: "we invoke the REST client to find all books"
    def books = client.findAll()

    then: "we expect two books to be found"
    books.size() == 2

    and: "the mock to be invoked exactly once"
    1 == wireMock.count {
        method "GET"
        url "/book"
First, we create the WireMock Groovy binding to create stubbed requests and responses. The stub closure takes the definitions of the REST request and response using the WireMock JSON API. As you can see we can even specify the response body as inline JSON multiline GString. Finally, we check that the invocation count for the expected request is correct.

Clearly, specifying the responses inline is not very maintenance friendly especially for large response structures. So a better alternative is to externalize the response body in a separate file. The file needs to be located in a directory named __files within src/test/resources.
The bodyFileName value is relative to the __files directory and contain any content. You could even return binary files like JPEGs using this mechanism.
response {
    status 200
    bodyFileName "books.json"
    headers { "Content-Type" "application/json" }
A further way of specifying the response body is by using plain Java or Groovy objects that get serialized to JSON automatically.
response {
    status 200
    jsonBody new Book(title: "WireSpock Showcase", isbn: "4713")
    headers { "Content-Type" "application/json" }
The stubbing capabilities of WireMock are quite powerful. You can perform different matchings on the URL, request headers, query parameters or the request body to determine the correct response. Have a look at the WireMock stubbing documentation for a complete description of all features.

So there is only one thing left to say: Test long and prosper with Spock!


Dec 17, 2015

Open Source Project Chronix: An efficient and fast time series database based on Apache Solr.
We are pleased to announce the open source project Chronix. Chronix is a fast and efficient time series storage. It is based on Apache Solr, a distributed NoSQL database with impressive search capabilities. Chronix uses the features of Solr and enriches it with specialized concepts for storing time series data. Thereby Chronix allows you to store about 15 GB of raw time series data (csv files) in about 238 MB. An average query for a bunch of time series data needs 21 ms using a single Solr server and one core. On Benchmarks Chronix outperforms related time series databases like OpenTSDB, InfluxDB or Graphite in both storage demand and query times.

Why is Chronix Open Source?
We use Chronix in several applications like the Software EKG or as central time series storage in a research project called "Design for Diagnosability". We believe that the community can benefit from using Chronix in other projects and hope that their experiences flow back into Chronix. Thus download and use Chronix, fork it, improve it, and raise a pull request. :-)

How can I start?
The homepage of Chronix contains a 5 minute quick start guide. The guide uses an example JavaFX application for time series exploration. You can easily perform range queries, do some analyses and examine the results in real time.
Chronix Quick Start Guide - Time Series Exploration

The listing below shows an example integration using Chronix-API, the Chronix-Kassiopeia time series package, and the Chronix-Solr storage. All libraries are available on bintray. A build script for use in all Gradle versions is:
repositories {
    maven {
        url ""
dependencies {
   compile 'de.qaware.chronix:chronix-api:0.1'
   compile 'de.qaware.chronix:chronix-server-client:0.1'
   compile 'de.qaware.chronix:chronix-kassiopeia-simple:0.1'
   compile 'de.qaware.chronix:chronix-kassiopeia-simple-converter:0.1'
Full Source build.gradle

The following snipped first constructs a Chronix client with a connection to Apache Solr, and then streams the maximum from all time series whose metric matches *Load*.
//Connection to Solr
SolrClient solr = new HttpSolrClient("http://host:8983/solr/chronix/");
//Define a group by function for the time series records
Function<MetricTimeSeries, String> groupBy = 
     ts -> ts.getMetric() + "-" + ts.attribute("host");

//Define a reduce function for the grouped time series records
BinaryOperator<MetricTimeSeries> reduce = (ts1, ts2) -> {
      MetricTimeSeries.Builder reduced = new MetricTimeSeries
         .data(concat(ts1.getTimestamps(), ts2.getTimestamps()),
               concat(ts1.getValues(), ts2.getValues()))

//Instantiate a Chronix Client
ChronixClient<MetricTimeSeries,SolrClient,SolrQuery> chronix = 
  new ChronixClient(new KassiopeiaSimpleConverter(),
            new ChronixSolrStorage<>(200,groupBy,reduce));

//We want the maximum of all time series that metric matches *Load*.
SolrQuery query = new SolrQuery("metric:*Load*");

//The result is a Java Stream. We simply collect the result into a list.
List<MetricTimeSeries> maxTS =, query)

//Just print it out."Result for query {} is: {}", query, prettyPrint(maxTS));

But I want to use my own fancy time series implementation! No worries!
In the example above we use the default Chronix time series class but you can use Chronix to store your own time series. You only have to implement two simple methods of the TimeSeriesConverter interface shown below:
//Binary time series (data as blob) into your custom time series
T from(BinaryTimeSeries binaryTimeSeries, long queryStart, long queryEnd);

//Your custom time series into a binary time series
BinaryTimeSeries to(T document);

Afterwards you can use Chronix to store and stream you custom time series. For more details check the Chronix GitHub repository and website.

Can I contribute to Chronix?
You are highly welcome to contribute your improvements to the Chronix project. All you have to do is to fork the public GitHub repository, improve the code and issue a pull request.

What do I have do now?
Fork Chronix on GitHub and follow us on Twitter. :-)

Oct 28, 2015

Java 9 Jigsaw - The long-awaited Java Module System

Java 9 Jigsaw

The long-awaited Java Module System


This year at the JavaOne San Francisco, Mark Reinhold and his Team at Oracle presented the Java 9 Module System with the codename Jigsaw. The work on Jigsaw has a long history and finally arrives with Java 9. Since 1999 we develop acyclic software components with Java per convention and learned how to do this with build tools like Ant, Maven or Gradle. In 2005 at QAware, we developed a tool for static bytecode analysis to detect wrong dependencies between packages since the Java language had no idea of controlling package dependencies or avoiding cyclic dependencies between packages and jars. With Maven, things got better. Maven enforces acyclic dependencies between projects but has also no concept to hide certain packages in one Jar. If a public class is accessible, all other public classes are also accessible. Then came OSGi. It seemed to be the answer for this problem, but the bad tool support in IDEs and the over-engineered dynamic model made more problems than the architectural enforcements benefits for most applications. Even so applications exists where OSGi is a perfect choice. So finally Jigsaw arrived. We now get the long awaited Module System in Java 9. The Java language team had not only the architecture of applications in focus, they also divided the fat legacy runtime jar (rt.jar) into modules. Try to load a Java 8 rt.jar file in a tool like Structure101 and you will see a chaotic mess of cyclic and acyclic dependencies. That problem is solved in Java 9. The runtime itself has now a clean separated module dependency architecture. Another big focus of the Java Language team was compatibility and interoperability. Some odd looking design is useful for compatibility reasons between modular and non modular code. In this blog I will show a test-drive of the Java 9 Jigsaw Module System. The Java 9 preview build can be downloaded from here:

Email Sample Application

We use a simple modular application for sending emails. The application consist of two modules:
  • A mail module with a public interface and a private implementation class. The public part of that module is a package containing a Java interface for sending messages and a factory which returns the implementation to the caller. 
  • A mail client module wich uses the interface of the mail module to send emails. It should be guaranteed that the client can not access the private implementation of the mail module itself. This principle of encapsulation is long known. David Parnas described the principle of information hiding in modules in 1985. 
Our sample application looks like this:
In that situation, the Java 9 Module System can enforce the following:
  • The module MailClient can only use types of exported packages in the module Mail.
  • The module Mail can not have cyclic dependencies to a type located in the module MailClient.
  • Transitive dependencies from Mail to other components can be visible or not for the Module MailClient. This is controlled by declaring a dependency as public in the Mail module.
The Java 9 Module System can NOT enforce: Types of the exported package in module Mail can direct access implementation classes. There is no contract between public and hidden parts of a single module. If you want to write plugin modules which can be replaced, you are not allowed to call any implementation class from the public part of a module. If you want to do this, you have to write a separate API module which looks like this:

Looking at the Code

There is currently no IDE support for the Jigsaw available. Even the latest builds of Netbeans and IntelliJ do not recognize the module descriptor file. Since the Java team decided to support multi-module builds with the javac compiler, they had to invent a complete new lookup mechanism: The module path which is separated from the normal classpath. This concept allows to use normal Jars as modules and vice versa. Jars which are located in the module path are called automatic modules. These are modules which can access all public parts of other modules without explicit declaration. This is done mostly to support smart and slow migration to the new module system without too much trouble for existing applications. At the time of writing, there was no support of Maven, Gradle or Ant. So we have to use shell scripts in this demonstration.

To use the Java 9 compiler multi-module build feature we have to make two subdirectories in the src directory: Mail and MailClient. The Mail directory contains the complete source code of the Mail module inclusive the private implementation which should be hidden from a caller. The MailClient directory contains the main class MailClient which uses methods from the public interface of the Mail module. The image shows the directory layout of our sample:

   ├── Mail 
   │    ├── de 
   │    │   └── qaware
   │    │       └── mail
   │    │            ├──
   │    │            ├──
   │    │            └── impl
   │    │                 └──
   │    └──
   └── MailClient    
       ├── de    
       │   ├── qaware    
       │   └── mail    
       │         └── client                 
       │                └──    

Both modules have the new module descriptor which is per convention in the root package and has the name „“. The module descriptor is not a class, enum or interface. It uses the new keywords „module“, „exports“ and „requires“. The module descriptor for the mail and the client are quite simple:

module MailClient {
 requires Mail;

module Mail {
 exports de.qaware.mail;

In our example the code of client and mail implementation looks like this:

package de.qaware.mail.client;

import de.qaware.mail.MailSender;
import de.qaware.mail.MailSenderFactory;

public class MailClient {

 public static void main(String [] args) {
  MailSender mail = new MailSenderFactory().create();
                              "A message from JavaModule System");


package de.qaware.mail.impl;

import de.qaware.mail.MailSender;

public class MailSenderImpl implements MailSender {
 public void sendMail(String address, String message) {
  System.out.println("Sending mail to: " + address + 
                                          " message: " + message);

The Type MailSender and the MailSenderFactory are from the exported package de.qaware.mail of the module Mail. As you might expect, the MailSender Java interface has only one method: sendMail(). The real implementation is in the impl package which non exported in the Mail module and therefore invisible. If you try to access the MailSenderImpl class direct in the client code you get an compiler error:

../ error: MailSenderImpl is not visible because package de.qaware.mail.impl is not visible
1 error
Thats exactly what we want. Nobody can violate the access rules of our Mail module. Non exported packages are hidden.

Multi module building, packaging and running

The multi module build is a cool feature of the Jigsaw javac compiler. The compiler command for both modules in one step looks like this:

# compile
javac -d build -modulesourcepath src $(find src -name "*.java")

This command compiles all modules in the subpath of ./src and outputs the resulting classfiles in a identical structure in the folder ./build. So the content of the ./build folder can be simply packed in separate jar files: The jar command creates the module jar file from the compiled output. For the MailClient module we also specify the java main class for the next step.

jar --create --file mlib/Mail@1.0.jar --module-version 1.0 -C build/Mail .
jar --create --file mlib/MailClient@1.0.jar --module-version 1.0 --main-class de.qaware.mail.client.MailClient -C build/MailClient .

We can now run our modular application by using the module path where the generated jar files are stored. In our example this is the path ./mlib. In this path the both generated jar files are located. So we can now test our application with the following command:

# run
java -mp mlib -m MailClient


What does linking in Java mean? It means that only the required modules are packed together in a mini application together wich a platform depended starter script and a minified java runtime.

# link
jlink --modulepath $JAVA_HOME/jmods:mlib --addmods MailClient --output mailclient

So the application can be run direct without any knowledge of java or modules. The generated output directory looks like this:

├── bin
│   ├── MailClient
│   ├── java
│   └── keytool
├── conf
│   ├──
│   └── security
│       ├── java.policy
│       └──
├── lib
│   ├── classlist
│   ├── jli
│   │   └── libjli.dylib
│   ├── jspawnhelper
│   ├── jvm.cfg
│   ├── libjava.dylib
│   ├── libjimage.dylib
│   ├── libjsig.diz
│   ├── libjsig.dylib
│   ├── libnet.dylib
│   ├── libnio.dylib
│   ├── libosxsecurity.dylib
│   ├── libverify.dylib
│   ├── libzip.dylib
│   ├── modules
│   │   └── bootmodules.jimage
│   ├── security
│   │   ├── US_export_policy.jar
│   │   ├── blacklist
│   │   ├── blacklisted.certs
│   │   ├── cacerts
│   │   ├── local_policy.jar
│   │   ├── trusted.libraries
│   │   └── unlimited_policy
│   │       ├── README.txt
│   │       ├── US_export_policy.jar
│   │       └── local_policy.jar
│   ├── server
│   │   ├── Xusage.txt
│   │   ├── libjsig.diz
│   │   ├── libjsig.dylib
│   │   ├── libjvm.diz
│   │   └── libjvm.dylib
│   └── tzdb.dat
└── release

The directory tree shows the complete minified runtime for the modular application. In the bin directory there is a generated MailClient starter which can be used to run the application direct without an explicit java command. The log directory contains only the required boot modules and native libraries. After linking you can start the application with the following command:

cd mailclient/bin 
Sending mail to: message: A message from JavaModule System


Gratulation to Mark and his Team. You have done an awesome job. Since this feature is very invasive for tools and existing applications, it will take a long time until we can forget the classpath hell. For me personally it is the proof, that our design principles we used in the last 20 years of modular programming are finally arrived the java world as well. Thanks for that.

Aug 31, 2015

Exploratory Open Source Software Testing with Spock

Exploratory software testing is a technique every agile developer should know about. It's about test driving your application without a predetermined course of action. Although this seems like random free style testing, in the hands of an experienced developer this proves to be a powerful technique to find bugs and undesired behaviour.

But in this article I will talk about exploratory software testing of open source software components, libraries or whole frameworks. So why would you want to do this? Think about this: the amount of hand written code in any modern application is somewhere between 3 to 10 percent of the overall byte code instructions. The rest are usually 3rd party open source libraries and frameworks used by the application such as Apache Commons or the Spring Framework.

But how do you as a software developer or architect decide which open source component to use for a certain required functionality? How do you know that this fancy framework you read about in a programming magazine suites your requirements? How do you evaluate how a library is integrated best into your application?

This is when exploratory testing of open source software comes into play. In summary, the goals of exploratory testing of open source components are:
  • To gain an understanding of how the library or framework works, what its interface looks like, and what functionality it implements: The goal here is to explore the functionality of the open source component in-depth and to find new unexplored functionality.
  • To force the open source software to exhibit its capabilities: This will provide evidence that the software performs the function for which it was designed and that it satisfies your requirements.
  • To find bugs or analyse the performance: Exploring the edges of the open source component and hitting potential soft spots and weaknesses.
  • To act as a safeguard when upgrading the library to a new version: This allows for easier maintenance of your application and its dependencies. The exploratory test detect regressions a new version might introduce.

Scenario based software exploration

If you want to use a new open source component in your project and application you usually already have a rough vision of what you expect and want to gain from its usage. So the idea of scenario based software exploration is: describe your visions and expectations in the form of usage scenarios. These scenarios are your map of the uncharted terrain of the libraries' functionality. And the scenarios will guide you through the test process. In general, a useful scenario will do one or more of the following:
  • Tell a user story and describe the requirements
  • Demonstrate how a certain functionality works
  • Demonstrate an integration scenario
  • Describe cautions and things that could go wrong

Exploratory Testing with Spock

Of course you can write exploratory tests with more traditional xUnit frameworks like JUnit or TestNG. So why use Spock instead? I think the Spock Framework is way better suited to write exploratory tests because it supports the scenario based software exploration by its very nature: 
  • Specification as documentation
  • Reduced, beautiful and highly expressive specification language syntax
  • Support for Stubbing and Mocking
  • Good integration into IDEs and build tools
The following sections will showcase these points in more detail by using Spock to write exploratory tests for the Kryo serialization library.

Specification as documentation

The good thing about Spock is that it allows to use natural language in your specification classes. Have a look at the following example. Currently it does not test anything, it is pure documentation. Even if you do not know Spock at all, I think you can understand what the test is supposed to do just by reading the specification. Awesome.
@Title('Exploratory test for the shallow/deep copy functionality of Kryo')
   Making object copies in Java is an often required functionality.
   Writing explicit copy constructors is fast but also laborious. 
   Instead the Java Serialization is often misused to make copies. 
   Kryo performs fast automatic deep and shallow copying by copying 
   from object to object.
class KryoShallowAndDeepCopySpec extends Specification {

    def kryo = new Kryo()

    def "Making a shallow copy of a semi complex POJO"() {
        given: "a single semi complex POJO instance"

        when: "we make a shallow copy using Kryo"

        then: "the object is a copy, all nested instances are references"

    def "Making a deep copy of a semi complex POJO construct"() {
        given: "a semi complex POJO instance construct"

        when: "we make a deep copy using Kryo"

        then: "the object and all nested instances are copies"

Reduced, beatiful and highly expressive language syntax

The reduced syntax offered by Spock mainly comes from its Groovy nature. Exploratory tests really benefit from this because it helps you to focus on the important bits: the open source component you want to explore. In addition to this Spock brings along its own DSL to make your specification even more expressive. Every feature method in a specification is structured into so-called blocks.
see Spock Primer for more details
These blocks not only allow you to express the different phases of your test. By using these blocks you can demonstrate how an open source component works and how it can be integrated into your codebase. The setup: or given: block sets up the required input using classes from your application domain. The when: and then: blocks will exhibit how a certain functionality works by interacting with the test subject and asserting the desired behaviour. Again, due to the Groovy nature of Spock your assertions only need to evaluate to true or false. And for the last bit of expressiveness you can use your good old Hamcrest matchers.
def "Deserialize a GCustomer object from a temporary data file"() {
    given: "a Kryo input for a temporary data file"
    def input = new Input(new FileInputStream(temporaryFile))

    when: "we deserialize the object"
    def customer = kryo.readObject input, GCustomer

    then: "the customer POJO is initialized correctly"
    expect customer, notNullValue() == 'Mr. Spock'
    expect, equalTo('Mr. Spock')

Support for Stubbing and Mocking

The Spock Framework also brings its own support for Mocks and Stubs to provide the means for interaction based testing. This testing technique focuses on the behaviour of the object under test and helps us to explore how a component interacts with its collaborators, by calling methods on them, and how this influences the component's behavior. Being able to define mocks or stubs for every interface and almost any class also alleviates you from having to manually implement fake objects. Stubs only provide you with the ability to return predefined responses for defined interactions, whereas Mocks additionally provide you with the ability to verify the interactions.
def "Explore writing an object using a Serializer mock"() {
    given: "a Kryo Serializer mock and dummy output"
    def serializer = Mock(Serializer)
    def output = new Output(new byte[1])

    when: "serializing the string Mr. Spock"
    kryo.writeObject(output, 'Mr. Spock', serializer)

    then: "we expect 1 interaction with the serializer"
    1 * serializer.write(kryo, output, 'Mr. Spock')

def "Explore reading an object using a Serializer stub"() {
    given: "a dummy input and a Kryo Serializer stub"
    def input = new Input([1] as byte[])
    def serializr = Stub(Serializer)

    and: "a stubbed Customer response", _, GCustomer) >> new GCustomer(name: 'Mr. Spock')

    when: "deserializing the input"
    def customer = kryo.readObject(input, GCustomer, serializr)

    then: "we get Mr. Spock again" == 'Mr. Spock'

Good Integration into IDEs and Build Tools

A good IDE and build tool integration is an important feature, since we want our exploratory tests to be an integral part of our applications' codebase. Fortunately, the Spock support is already quite good, mainly due to the fact that Spock tests are essentially translated to JUnit tests. To get proper syntax high lighting and code completion you can install dedicated plugins for your favourite IDE. For IntelliJ there is the Spock Framework Enhancements Plugin and for Eclipse there is the Spock Plugin available.

The build tool integration is also pretty straight forward. If you are using Gradle for your build, the integration is only a matter of specifying the correct dependencies and applying the groovy plugin as shown in the follow snippet.
apply plugin: 'groovy'

dependencies {
    // mandatory dependencies for using Spock
    compile 'org.codehaus.groovy:groovy-all:2.4.1'
    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
    testCompile 'junit:junit:4.12'
    testCompile 'org.mockito:mockito-all:1.10.19'

    // optional dependencies for using Spock
    // only necessary if Hamcrest matchers are used
    testCompile 'org.hamcrest:hamcrest-core:1.3' 
    // allows mocking of classes (in addition to interfaces)
    testRuntime 'cglib:cglib-nodep:3.1'
    // allows mocking of classes without default constructor
    testRuntime 'org.objenesis:objenesis:2.1'
For Maven you have to do a little more than just specifying the required Spock dependencies in your POM file. Because Spock tests are written in Groovy, you will also have to include the GMavenPlus Plugin into your build so that your Spock tests get compiled. You may also have to tweak the Surefire Plugin configuration to include **/* as valid tests.
        <!-- Mandatory plugins for using Spock -->
        <!-- Optional plugins for using Spock -->

    <!-- Mandatory dependencies for using Spock -->

    <!-- Optional dependencies for using Spock -->
    <!-- only required if Hamcrest matchers are used -->
    <!-- enables mocking of classes (in addition to interfaces) -->
    <!-- enables mocking of classes without default constructor -->


That's all folks!

Jul 24, 2015

Solr with Spark(s) - Or how to submit a Spark-Task which utilizes a Solr-Cloud from Java-Code

What this tutorial is about

We are going to setup an architecture combining the Big-Data computing framework Apache Spark with a sharded Apache Solr-Cloud. After that we will learn how to start Spark tasks from Java without Spark having any knowledge of the task itself.

For that purpose we will install three virtual machines using VirtualBox.
The following image briefly illustrates our goal.

Technologies used

 For this tutorial we will need to get in touch with the listed technologies.
  • Apache Spark 1.4.0
  • Apache Solr 5.2.1
  • Zookeeper 3.4.6
  • VirtualBox 4.3
  • Ubuntu Server 64bit 15.04 (on each of our VMs)
  • Java 8 (openjdk-8-jdk)
  • Maven

Jun 25, 2015

Profiling Java in Production on Oracle Linux and Mac OS X with DTRACE

Profiling Java in Production

DTRACE on Oracle Linux

Today at the JavaOne Latin America 2015 I heard an interesting talk about Oracle Linux. I is basically another Red Hat Linux Clone. But it has one interesting tool on board which can save your life if you run into problems in a mission critical Java application.

This tool is called DTRACE. It was developed from Sun Microsystems for Solaris 10 and gives you access to thousands of instrumentation hooks called probes. This tool is non invasive for your application and has a very low performance overhead. So you can use this tool in production.

Since you can write custom probes on application level things get interesting. In the hotspot virtual machine there is a DTRACE provider integrated since JDK 6 and ready for use. Since DTRACE works on an OS level it has not idea of the virtual machine code running inside the native JVM wrapper. But the DTRACE provider inside the JVM gives DTRACE information about the Java code like the current Java call stack or method enter/leave information. I found a good slideshow from Gregg Brendan which explains things in more detail. Have a look here:

Oracle has ported DTRACE to Oracle Linux and you can use DTRACE in production of your Java application. What is even better: DTRACE was also ported to Mac OS X and runs on every Mac out of the box. This helps a lot to build your own D scripts for profiling or debugging purposes.

After the session I tried this with NetBeans. And it worked in one minute. The results are shown in the video. I am really impressed.

As you have seen in the video there is only a short script necessary to count each method call and remember the name of the called method. Since method calling scripts could have impact on your performance you have to enable the method call probe by starting the JVM with an special -X parameter.

Why the Java community needs this Tool

There can be problems when your server will be slow and normal tools like stack-traces or heap dumps show nothing abnormal. Then you need any information what your program is really doing. A profiler can help you. But the problem here: In most enterprises you will not get access to the root level of your production servers. But sending a small DTRACE script to the administrator team for execution will work fine since this tool does not need any special system configuration like open ports or the installation of profiler software.
It is much simpler and easy to use than the JRocket based Flightrecorder which is also an option when you try to profile production systems.

Since the JVM has support for DTRACE it would be great to have this functionality available on every Linux Platform.  

May 27, 2015

Accessibility - Detecting high contrast mode to work around a bug in Firefox

Visually impaired users might want to watch your website in high contrast mode. This display mode aims to increase the contrast of displayed content by overriding your styling.

Here is how you enable high contrast mode in Windows:

Unfortunately there is a bug in Firefox, which prevents borders from being rendered in  high contrast mode:

In order to work around that bug my colleague Alexander Krauss came up with 2 step workaorund:
  1. check if the browser is in high contrast mode 
  2. style with border:inset instead of regular borders

Check if the browser is in high contrast mode

The JavaScript snippet checks if the background image of a hidden div has been removed. If so we can assume the browser is operating in high contrast mode. Therefore we add a high-contrast-mode css class to the html element.

Style with border inset instead of regular borders

Now we can add special styles for high contrast mode to our style sheet. In this case we want to make sure the borders are visible. Since the above bug prevents regular borders we use inset borders instead.

Apr 16, 2015

GUI Tests for JavaFX

This article summarizes my experiences when setting up UI tests for a new JavaFX application at QAware.

I had the following requirements for a JavaFX test framework:
  • Free (as in beer) Software  with a permissive license
  • Lean, pure Java framework
  • Integration with  JUnit
Later on, the requirement to do headless tests was added. I will cover headless tests in a follow-up post.
A quick search reveals three contenders:
  • TestFX:
  • JemmyFX:
  • MarvinFX:
Of these three, MarvinFX seems dead. The last commit to MarvinFX was on May the 4th, 2013, almost two years ago, so I discarded that option.

There are several further possibilities such as Automaton and TestComplete. Test Complete is a full-blown GUI testing product and not a lean framework, and the Automaton developers recommend TestFX for testing JavaFX (

Testability restrictions in JavaFX

Before dealing with the test frameworks, let me mention that  JavaFX has two important issues that restrict testability. To fix them, both require modifications to the JavaFX platform.

No lifecylce management on the JavaFX platform

JavaFX allows to start exactly one application exactly once within a JVM. That means:
  • No parallel or sequential execution of UI tests within a JVM.
  • Each test case must run within it's own JVM.
I'm not aware of any way to get around this restriction.

No headless testing of JavaFX applications

The Oracle and OpenJDK desktop builds to not allow testing JavaFX applications without a physical display. I will show a way to get around this restriction in a follow-up post.


TestFX promises "Simple and clean testing for JavaFX". The TestFX Getting Started Guide ( provides a short primer on how to use TestFX.Currently, TestFX 4 is under active development.

TestFX is available in Maven Central:, so integration is just a Maven dependency away.

To create a TestFX GUI test, you have to extend the GuiTest class:
public class SomeGuiTest extends GuiTest {

    protected Parent getRootNode() {
        FXMLLoader fxmlLoader = new FXMLLoader();
        try (InputStream inputStream = getClass()
                .getResourceAsStream(FXML_FILE)) {
            return fxmlLoader.load(inputStream);
        } catch (IOException e) {
            throw new IllegalStateException(e);

You have to override a single method getRootNode(), which will create the node to test. In TestFX 4, currently in alpha state, the idiom is closer to the regular JavaFX application startup. The method to override has the signature void start(Stage stage).
As you can see, TestFX is geared towards testing GUI components of an application. This approach comes with two important caveats:
  1. TestFX does not play nice with customized application startup. Application startup is commonly customized in non-trivial applications, for instance to startup DI containers such as Weld or Spring.
  2. The JavaFX platform does not provide lifecycle management of the JavaFX toolkit. As of JavaFX 8, it is not possible to cleanly startup and shutdown JavaFX within a process. That means while it's nice to be able to easily test each GUI component separately, each of these test cases needs to run in it's own JVM.
There is a workaround for caveat #1:
protected Parent getRootNode() {
    return null;

public void setupStage() throws Throwable {
    new Thread(() -> Application.launch(SomeApplication.class))
Override the setupStage() method and startup your application manually. The result of the getRootNode() method is no longer used.

For the tests themselves, there is a nice and easy fluent API:
public void test() throws Exception {

    // Wait for success with a 10s timeout
    waitUntil("#showLabel", is(visible()), 10);

    String text = ((Label) find("#showLabel")).getText();
    verifyThat(text, containsString("OK"));
I like especially that TestFX exposes the lookup by CSS selector functionality of JavaFX. This makes the creation of page objects superfluous in most cases.

TestFX creates screenshots upon test failures. Unfortunately, it also creates screenshots elsewhen, for example when using the waitUntil()  method. Combine that with the fact that the screenshot location is hard-coded to the current working directory, and the screenshot feature becomes annoying instead of useful.


JemmyFX is another JavaFX UI test library and part of the offical OpenJFX project. Since 2012,  it resides in the test branch of OpenJFX.

JemmyFX is not provided in binary form. To use JemmyFX, you have to checkout and compile the sources yourself:
The Jemmy web site ( is in disrepair which raises doubts about the project's status. However, the test branch is still included in the newest OpenJFX versions and seems to be actively mainained at a low level.
The build produces quite a number of Jars. In total, 7 JARs are required to use JemmyFX:
  • GlassImage.jar
  • GlassRobot.jar
  • JemmyAWTInput.jar
  • JemmyBrowser.jar
  • JemmyCore.jar
  • JemmySupport.jar
  • JemmyFX.jar
You can include these JARs in a Maven or Gradle build by installing them in your local repository or on a custom Nexus.

As soon as all that is done, JemmyFX is quite easy to use. You need to create a setup method that initalizes and launches your application:
public static void setUp() throws Exception {    
    new Thread(() -> Application.launch(LoginApplication.class))

The test cases are somewhat more verbose than with TestFX:
public void test() throws Exception {
 SceneDock scene = new SceneDock();

 // Create docks for the UI elements. Lookup by Id
 TextInputControlDock userNameDock = 
  new TextInputControlDock(
   new ByID<>("userNameTextField"));
 TextInputControlDock passwordDock = 
  new TextInputControlDock(
   new ByID<>("passwordTextField"));
 LabeledDock loginButtonDock = 
  new LabeledDock(
   new ByID<>("loginButton"));



 LabeledDock showDock = new LabeledDock(
  scene.asParent(), new ByID<>("showLabel"));

 // Wait for success with the default 
 // "wait for control" timeout
  .waitValue(true, showDock::isVisible);

JemmyFX requires docks to access each UI element. It offers a intuitive but non-fluent API to operate on these docks.
JemmyFX does not directly expose the JavaFX lookup by CSS selector functionality. You can work around this restriction by creating the docks using the exposed JavaFX UI controls:
TextInputControlDock userNameDock = new TextInputControlDock(
 (TextInputControl) scene.control()


While JavaFX is a powerful and well-engineered GUI framework, it's testability is severely hampered. Once you get around the fundamental restrictions, both TestFX and JemmyFX provide the means to effectively test JavaFX applications.

At the moment, I'd use JemmyFX for a new project because of it's cleaner test case setup and the ability to do headless with Monocle. Also, I dislike abstract base classes used to provide tool-like functionality.
This recommendation will probably change when TestFX 4 is released.

Mar 31, 2015

Secure password storage and authentication

TLDR: If you need to store passwords, use a cryptographic function designed for password hashing like PBKDF2 or bcrypt and use a cryptographic random generator to generate a salt which is at least 64 bit long.
This article describes the common pitfalls when storing passwords and describes best practices for secure password authentication.

We accompany a young developer without the slightest clue of how to store passwords and follow him through his learning process. In each step I will describe the obvious and not-so-obvious problem with his solution. Let’s start!

Just store the passwords as plaintext

So our developer has to store the password of a user of his application. Since he knows how to work with databases, he just stores the password in plaintext.

The obvious problem with this approach is that everyone who has read access to the database can recover the passwords. As they are stored in plaintext and thus can easily be recovered, an attacker could even try to use the user’s username/password for other services, e.g. the email account.

Obfuscate the password

Our developer realizes that storing the passwords as plaintext is a bad idea. So he invents his own algorithm to obfuscate the password.

Problems with this: Cryptography is hard. There are a lot of smart guys who worked together to create secure systems, so please do not roll your own crypto. Even if the output from your self-made crypto may look secure, it probably (and that is a very high probability) is not. Also, there is a principle called security through obscurity which you should avoid.

Encrypting the password

So our developer accepts the fact that creating his own crypto-algorithm is a bad idea. He uses his search engine of choice to find examples of good crypto. And he finds the AES cipher. This cipher is widely used and generally considered as secure. So he utilizes this cipher to encrypt the passwords before storing them in the database. To encrypt the password, he needs to use a key, which is stored somewhere in the application. This key is used to decrypt the password from the database when a user wants to log on to the application.

This approach has a big problem: If the attacker has read access to the database, it is likely that he can also find the application files. And somewhere in this files lies the key to encrypt/decrypt the passwords. The attacker then just needs to read the password from the database, decrypt it with the extracted key, and voila, he gets the plaintext password.

Hashing the password

After discovering the flaw our developer reads about crypto basics and discovers a technique known as hash functions. Some well-known hash functions are MD5, SHA1 and the SHA-2xx family. A hash function is a function which takes data of an arbitrary length and maps it to a fixed-length value. Every same input data maps to the same output value. Hash functions which are used in cryptography have an interesting characteristic: they are non-invertible. That means it is very hard to find the input data for a given output value. “Hey, that’s exactly what I’m looking for”, our developer thinks. “This way I can store the password and nobody can recover the plaintext password!” Okay, you ask, but if you cannot recover the plaintext, how does the system check that the user has entered the correct password on login? Because the same input data maps to the same output value, you can take the password which the user has entered, hash it and compare it to the hash stored in the database. If they are both the same, the user has entered the correct password.

As you may have guessed, this technique also has a flaw: As every same password leads to the same hash, some guys have built big databases which do the inverse mapping, from hash to the plaintext password. As database lookups are usually really fast, the plaintext for a given hash can be looked up quite fast. Test it for yourself: Visit and enter some MD5 hashes. For example:

  • 5ebe2294ecd0e0f08eab7690d2a6ee69 (“secret”)
  • b295ab91a74ae7048c0ba523c03511e3 (“verysecret”)
An attacker can also use Rainbow Tables to get the plaintext to the hash.

Put in some salt

After searching the internet for some time, our developer stumbles across the recommendation to add salt to his hashes. Salt is random data which is added to the plaintext password before hashing. This defeats precomputed hashes or rainbow tables, because the same input password does not produce the same hash anymore. Every password gets its own salt, and the salt is stored in plaintext along with the password hash. When the user wants to log on to the application, the salt and the hash is read from the database. The salt is now added to the password entered from the user and the hash function is applied. Then the hash from the database is compared to the calculated hash. If both hashes match, the user is allowed to login. There are still some mistakes our developer can make:

  • Reuse the salt for every password. The salt should be unique for every password, otherwise the attacker can precalculate the hashes or build a rainbow table
  • Use a salt which is too short: If you add just, say, for example one byte of salt, the precalculated hash table grows at the factor of 256 (2 to the power of 8), but an attacker can compensate for this.
  • Use a non-random salt: If you use the username as salt for example, the salt is not random enough. Use a cryptographic random number generator (for example SecureRandom in Java) to generate the salt. If the salt isn't random, an attacker can build a rainbow table with that salt.

Performance problems the other way round

Even after using a hash function with salt, our developer recognizes, the passwords are not stored secure enough. The main problem is that hash functions are not made for password storage. Admittedly, they have the nice property of being non-invertible, but they have one flaw: they are way too fast. These hash functions have been designed to work with huge amount of data at a fast speed. With the uprising of the ability to calculate on GPUs, billions of hashes can be checked per second (!). A great tool for this is oclHashcat. An NVidia GTX 750 Ti GPU, which costs at the time of writing about 150 Euros, can create roughly 3 billion of MD5 Hashes per second. If you use a more expensive GPU or multiple GPUs, this gets even faster.

The solution, finally

After realizing that using a hash function is not the way to go when storing passwords, our developer reads about cryptographic functions which are more suitable for password storage. One of these functions is PBKDF2, the Password based keyderivation function #2. PBKDF2 applies a salt to the password and uses a hash function (or another pseudorandom function) multiple times. The number of applications of this function is called the iteration count. This technique is known as key stretching and slows down the hashing of passwords. As a result an attacker can calculate considerably fewer hashes per second. The iteration count of PBKDF2 can be adjusted to compensate the growth in computing power. If the computing power grows, just increase the iteration count. Other examples of password hashing functions are Bcrypt and Scrypt. When using such a function you need to decide on how many iterations you need. This depends on your performance requirements and your available performance. ThisStackoverflow article offers some clues.

The QAware solution

We at QAware have developed a library which handles all the unpleasant parts of secure password storage for you. The workflow looks like this:

As you can see in the code, this library has a pretty cool feature: automatic detection of broken algorithms and parameters. If you used, say, PBKDF2 with 1000 iterations, that was secure in the past, but nowadays it is not. The library will detect this and give you the possibility to update the password hash. The library also handles secure salt generation with a sufficient length and has secure defaults for iteration count and the like.


  • Use crypto
  • Do not roll your own crypto
  • Use a salt
  • Apply a random and long enough salt
  • Use a cryptographic function which was designed for password hashing, for example PBKDF2 or bcrypt
  • Adjust the iteration count to fit your needs

Edit 2015-06-30:

We've just made the library open source. You can download (and contribute) here: Enjoy!

Mar 27, 2015

Special Purpose Iterators

This package is for the mathematically inclined only.It contains four types of methods: 

Generating iterators of Integer: There are faculty, fibonacci and hamming. While the first two are well-known, Hamming is rather particular. It is based on an exercise attributed to R. W. Hamming and reported by Dijkstra in "A discipline of programming", p. 129. The problem is this: Let p1, p2, .. pn be n integers, often but not necessarily primes. Create an iterator yielding 1 and all multiples of p1, p2, .. pn in ascending order. So, hamming(2,3,5) returns 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 16, 18, ...

Generating iterators of Double: exp(), cos(), sin() generate the respective series. random(), arithmeticSeries(start, delta), geometricSeries(start, factor) behave as expected. 

Finite and Infinite Polynoms: An Iterator of Double such as (1.0, 1.0, 1.0) can be considered the finite polynom 1 + x + x^2 and hence be evaluated for any Double. The same is true for series such as exp(). The method polynom(Iterator<Double>) returns an UnaryOperator which unwinds the iterator completely or up to a given degree and stores the result in an internal list. It thus consumes the iterator but the polynom itself is reusable.
UnaryOperator<Double> p;
p = polynom(of(1.0, 2.0, 3.0));  // p(x) = 1 + 2x + 3x^2                                                                         
p.apply(0.0);                    // returns 1.0
p.apply(1.0);                    // returns 6.0
p.apply(-1.0);                   // returns 2.0
p.apply(2.0);                    // returns 17.0

p = polynom(exp());              // p(x) = exp(x)
p.apply(0.0);                    // returns 1.0
p.apply(1.0);                    // returns e
p.apply(-1.0);                   // returns 1/e
p.apply(2.0);                    // returns e^2

Operators on Polynoms Two polynoms of finite or infinite length can be added, multiplied, divided and composed, yielding in each case a new polynom. Note that the divisor's first coefficient must not be zero. compose(s, t) returns the coefficients of polynom(t) o polynom(s). It thus holds that

polynom(compose(t, s)) = compose(polynom(t), polynom(s))

Square and inverse of a polynom p are defined by
square(p) = multiply(p, p)
inverse(p) = divide(1, p)

 Everybody knows that sin^2 + cos^2 = 1. With our library, this reads as follows:
Iterator<Double> r;
r = add(square(sin()), square(cos()));;                    // first element is 1.0, all others are 0.0
reduce(limit(r, 10), (a, b) -> abs(a) + abs(b), 0.0);  // returns 0.0
An iterator can also be augmented by a scalar and multiplied with a factor:
Iterator<Double> r;
r = add(random(), 5.0);
r = multiply(random(), 5.0);

Mar 9, 2015

Handling Nulls

Functions, operators and comparators often crash on null operands, e.g. when applying map to a List containing nulls. There are at least two competing approaches to handling nulls: Optional<T> and WeakLogic.
Optional<T>, introduced with Java 8, is a wrapper: variables of type T which might be null are wrapped by an Optional<T>. This works and is clean but highly invasive because types get affected throughout the system. Equal-methods work nicely without Optional due to their different strategy: on null they return false because no non-null object ever equals null. Functions accepting null are called weak. Weak functions are less invasive than Optional because types remain unchanged.
Our class Weaklogic contains some high order functions transforming functions, operators and comparators into their weak counterpart. The idea is this: ignore null operands as long as there is at least one non-null operand, otherwise return null.
So weakEquals compares any two objects using their equals-method if possible and returning true on two nulls. Likewise weakUnaryOperator(UnaryOperator) returns a UnaryOperator identical to the argument but for its gently accepting null.
Weak comparators consider null the smallest element of the universe. weakComparator comes in two flavours: the one without argument returns a weak version of Comparable::compareTo, the other one transforms a Comparator into its weak counterpart.
So weak operators are an appealing alternative if Optional is unsuitable for whatever reason. 

Integer sum;
BinaryOperator<Integer> weakAdd = weakBinaryOperator((Integer a, Integer b) -> a + b);
sum = weakAdd.apply(null, null); // returns null
sum = weakAdd.apply(1, null); // returns 1
sum = weakAdd.apply(null, 2); // returns 2
sum = weakAdd.apply(1, 2); // returns 3
Comparator<Integer> cmp = (x, y) -> x - y;
Comparator<Integer> w = weakComparator(cmp);, 1); // result < 0, 1); // result < 0, null); // result > 0, null); // result == 0
Integer s;
Function<Integer, Integer> weak2 = weakFunction(x -> 2 * x);
s = weak2.apply(5); // returns 10
s = weak2.apply(null); // returns null

Mar 2, 2015

Android Libraries & Tools you should give a try

When we started working on our new internal project, QAwear, we had to choose some libraries to work with. QAwear is an Android app that lets you build your projects via Jenkins and Sonar and review the results - all within the app. Moreover, you can connect your smartwatch with the app. You will get notified about results and you can easily build a project: simply choose one and shake the watch ("shake to build").
We compared different libraries and these are the ones we liked most:

Retrofit is a type-safe REST client library for Android and Java applications, which “turns your REST Api into a Java interface”.  The request method, relative URL, parameters, headers, form encodes and multipart can be added with annotations.

For deserialization, Retrofit uses GSON by default for mapping the JSON responses to POJOs, but XML converters, Protocol Buffers or custom parsers can be configured, as well custom error handlers.

  * Common interface defintion to query Sonar and obtain data via REST. 
public interface SonarApiServiceI { 

      * Get a list of all projects asynchron. 

      * @param cb retrofit callback function 
    @ GET ( "/api/resources" ) 
    void getProjectsCb ( retrofit . Callback < List < SonarProject >> cb ) ; 

      * Get a selected sonar project with passed metrics and violations. 

      * @param project the sonar project's id 
      * @param metrics the mectrics as comma separated String 
      * @param alerts boolean, if true, violations will be returned too 
      * @return a list with one SonarProject object 
    List<SonarProject> getProjectMetrics (@Query("resource") int project, 
    @Query("metrics") String metrics, @Query("includealerts") Boolean alerts); 


//generates an implementation of the SonarApiService interface. 
RestAdapter adapter = new RestAdapter.Builder()
.setRequestInterceptor(new BasicAuthInterceptor(user, token)) //adding basic authentication header 
.setConverter(new GsonConverter(gson)) //setting custom JSON converter 
.setErrorHandler(new NetworkingErrorHandler(this.context)) //using custom Error handler 

service = adapter.create(SonarApiServiceI.class);
Other popular networking libraries are VolleyAndroid Async HTTP and Ion.


Dagger is a dependency injection framework, which checks dependencies already at compile time and generates code to avoid reflection. This approach increases performance significantly for Android applications, because reflection is very slow in the Dalvik VM. View Demo.

Butterknife is a small library, which aims to help you write clearer, easier to understand, and less redundant code. The provided annotations turn time consuming and redundant bindings of views, onClick-Listener and other run-of-the-mill task into a pleasant one-liner. During the compilation of the application, the code for view look-ups gets generated instead of using reflection, so it doesn’t influence your app’s performance.

public class ExampleActivity extends Activity { 

    TextView textMessage; //"injecting" TextView

    List<EditText> credentialFields; //grouping views

    @Override public void onCreate(Bundle savedInstanceState) { 


        //act on all views in the list at once
        ButterKnife.apply(credentialFields, ACTION);

    @OnClick( void submit() { 

EventBus is a tiny and very fast Android optimized EventBus, which simplifies the communication between different components of the app.

//Defining an event
public class MyEvent () {…}

//Register subscriber

//Post an event
EventBus.getDefault().post(new MyEvent());

//Receive an event – based on conventions
public void onEvent (MyEvent event) {…};

The SDK Manager Plugin is a Gradle plugin that downloads and updates missing or out-of-date SDKs and support libraries automatically before building your app. It ensures, that all developer and even your CI system have all necessary SDKs and support libraries to be able to build your application.
You only have to add the dependency for the SDK Manager and apply the plugin before the regular 'android' plugin to your build.gradle file:
buildscript { 

    repositories { 

        classpath '' 
        classpath 'com.jakewharton.sdkmanager:gradle-plugin:0.12.+' 

apply plugin: 'android-sdk-manager' 
apply plugin: ''

Robolectric is a powerful unit testing framework, with the significant benefit of reducing the overhead of deploying apps on a real device or an emulator before every test depending on Android Api components. By replacing all Android classes with so-called shadow objects, behaving similar to the objects of the Android SDK, you can even run your tests inside the regular JVM. You don’t depend on a Dalvik VM and its long test run startup time.
Furthermore, it is possible to provide custom implementations for specific SDK methods for simulating error conditions or hardware features like real-world sensors and behavior, for example. View Demo.

Fabric was announced 2014 at twitter’s first developer conference. By the time of this post it’s still in a closed beta phase, but already one of the best tools for crash reporting and beta distribution (advertising platform not tried).
Fabric gets shipped out of the box, you only have to install the Plugin for your IDE and select the components you want to use in your app. The Plugin will show you, which dependencies have to be added and the positions in your code where changes have to be made and will perform them for you.
The crash reports are very detailed. You can see all information of the affected device, that could help you to retrace the bug’s cause as well as the whole stack trace of the uncaught exceptions over all threads.
Furthermore it provides extensive user statistics and beta distribution, similar to Google Analytics and Google play beta tests.