Distributed Performance Testing With Maven JMeter and Jenkins

In this post, we'll take a look at a practical, simple way to engineer a powerful, distributed performance testing suite, using tools commonly used in the J2EE community

I think that it's commonly accepted at this point that performance testing in software is a useful way of giving us confidence in the robustness and responsiveness of our system under load. The arguments covering these ideas are well covered by better domain experts than myself, so this article will focus on the practicalities of integrating performance testing with your CI environment.

This article assumes familiarity with the following tools

• JMeter - Widely used and highly adaptable performance testing tool. 
• Jenkins - Highly popular continuous integration server. 
• Maven - Build and dependency management tool. 

It's out of scope for this article to introduce, in detail, these tools, but there are a number of online articles that are better suited to providing a better grounding should you be interested. 

Objectives of a well designed performance testing system

 A well designed performance testing system should satisfy some of the following high level properties.

• Have well designed tests. Often, it's understated how important having well written parameterized tests can be, so i'll dedicate a small section of the article to explaining how you can use some of the often overlooked features in JMeter to improve your own tests. 
• Support scalable generation of load. Ideally we don't want to rely on one machine to generate all the load for the performance tests. Doing so will push you into a situation where you might find yourself unneccessarily limited by network congestion and physical resource contention. 
• Provide effective reporting facilities. We need to be able to analyze the results of our tests to determine whether any areas of the system haven't performed the way we expect. 

Luckily, the tools to satisfy these goals already exist, its just a case of putting them together. 

Tips and tricks for your JMeter tests

Ok, lets dive right in. I'm going to try and make this very example centric so you can see whats going on hopefully in the context of your own jmeter tests.

The Thread Group ( Simulating Users )

The thread group is one of the simplest and most powerful ways of simulating a user in your system. Requirements from the business are often expressed in these terms over the classic request per second metric. Thus, it can be a pragmatic approach to model the test in these terms.

Looking at the entries for the Thread Properties gives us a first look at JMeters often overlooked variable dsl, documented here. Lets take the first field as an example :

${__P(users,100)}

So what this statement is trying to do is resolve the value for the users property. If it can't it will use the provided default (100).

Pitfall 1 : Theres some weird truncation logic in play here through the ui ( which i'm assuming 90% of people will be using to edit these fields ), so make sure you check your field names.

Http Request Defaults

So two important factors here about making your test plans generally more maintainable and more robust. 

1. Configurable domain - We set a default here so we can change the environment which we run our tests against. 
2. Timeouts - This is a common pitfall. If you don't set your timeout you could potentially be in a situation where your test hangs forever. 

Pitfall 2 : Always set a timeout on http requests. 

Maven Integration (Lazerycode Plugin)

Now we have some tests we need to start thinking about how to run them outside the JMeter GUI. Thankfully, a decent maven plugin exists provided by Lazerycode. 

So lets assume that you've created a simple maven project in which to put your jmeter test files, properties file and plugin configuration. Lets review what you should probably have in there. 

pom.xml

The JMeter plugin, by default, looks in src/test/jmeter for test plans. If you need to change this you can consult the configuration documentation, but seems like a perfectly sensible place to put test plans.

I've provided some elaboration on the configuration options below:

• ignoreResultFailures - If a single sampler request fails, you dont want to kill the whole test. We want to collect and display this information in the reporting phase and then make assertions at that point. 
• remoteConfig -  Adding the remote config section will tell the plugin to send the test plan to your distributed nodes and collect the results. 
• java.rmi.server.hostname - This will inform the remote nodes where to send results, which can help you avoid a lot of the network configuration problems you might have otherwise when running the test across multiple subnets. 
• propertiesGlobal - Remember all those properties we were making configurable earlier? Heres where the integration comes in. The env.foo syntax is used because thats how variables will be passed from jenkins to the maven plugin. 

Next, we need to specify a couple of additional properties. Another quick look at the documentation will tell you that the plugin also checks the src/test/jmeter dir for a set of properties files. We want to, at this point, create a jmeter.properties file and populate it with the following values : 

remote_hosts=test-server-1.nerdability.com,test-server-2.nerdability.com,test-server-3.nerdability.com

Obviously using your own remote nodes for in place of the defaults i've specified here.

Thats it. Your plugin is now configured to orchestrate execution of your test plans. Onto reporting.

Reporting with Jenkins Performance Plugin

Now go ahead and install the Jenkins performance plugin, which will be responsible for producing pretty graphs from our test reports. After installing, set up a job which runs mvn verify on your project ( this will run the performance tests ). Next add the post build action Publish Performance test result report. It should look like this : 

Finally, remember all those variables we added in the tests earlier? Now we want to add those as parameters to the job so we can configure them when we run the job. 

These will be provided to the build under the env namespace. 

That's pretty much it, tweak and improve the performance tests and configuration as you see fit. The framework should support whatever you need to do. 

Functional Programming with Groovy

I have recently started the Coursera Functional Programming with Scala course (taught by Martin Odersky - the creator of Scala) - which is actually serving as an introduction to both FP and Scala at the same time having done neither before. The course itself is great, however, trying to watch the videos and take in both the new Scala syntax and the FP concepts at the same time can take a bit of effort.

I wanted to work through some of the core FP concepts in a more familiar context, so am going to apply some of the lessons/principles/exercises in Groovy.

Functions:

If you have done any Groovy programming then you will have come across Groovy functions/closures. As Groovy is dynamically typed (compared to Scala's static typing), you can play it fairly fast and loose.

For example, if we take the square root function that is demonstrated in the Scala course, it is defined as follows:



As you can see, Scala expects the values to be typed (aside, you don't actually always need to provide a return type in Scala). But in the Groovy function it is:



A groovy function can be defined and assigned to any variable, thereby allowing it to be passed around as a first class object. If we look at the complete solution for calculating the square root of a number (using Newton's method - To compute the square root of "x" we start with an estimate of the square root, "y" and continue to improve the the guess by taking the mean of x and x/y )


So that's in Scala, if we try Groovy we will see we can achieve pretty much the same thing easily:

Recursion (and tail-recursion):

As FP avoids having mutable state, the most common approach to solve problems is to break the problem down in to simple functions and call them recursively - This avoids having to maintain state whilst iterating through loops, and each function call is given its input and produces an output.

If we again consider an example from the Scala course, with the example of a simple function that calculates the factorial for a given number.

This simple function recursively calculates the factorial, continuing to call itself until all numbers to zero have been considered. As you can see, there is no mutable state - every call to the factorial function simply takes the input and returns an output (the value of n is never changed or re-assigned, n is simply used to calculate output values)

There is a problem here, and that is as soon as you attempt to calculate the factorial of a significantly large enough number you will encounter a StackOverflow exception - this is because in the JVM every time a function is called, a frame is added to the stack, so working recursively its pretty easy to hit upon the limit of the stack and encounter this problem.  The common way to solve this is by using Tail-Call recursion. This trick is simply to have the last code that is evaluated in the function to be the recursive call - normally in FP languages the compiler/interpreter will recognise this pattern and under the hood, it will really just run the code as a loop (e.g. if we know the very last piece of code in the block of code is calling itself, its really not that different to just having the block of code/function inside a loop construct)

In the previous factorial example, it might look like the last code to be executed is the recursive call factorial(n-1) - however, the value of that call is actually returned to the function and THEN multiplied by n - so actually the last piece of code to be evaluated in the function call is actually n * return value of factorial(n-1).
Let's have a look at re-writing the function so it is tail-recursive.


Now, using an accumulator, the last code to be evaluated in the function is our recursive function call. In most FP languages, including Scala, this is enough - however, the JVM doesn't automatically support tail-call recursion, so you actually need to use a rather clunkier approach in Groovy:

The use of the trampoline() method means that the function will now be called using tail-call recursion, so there should never be a StackOverflow exception. It's not as nice as in Scala or other languages, but the support is there so we can continue.

Currying:

This is like function composition - the idea being you take a generic function, and then you curry it with some value to make a more specific application of the function. For example, if we look at a function that given values x and y, it returns z which is the value x percent of y (e.g. given x=10, y=100, it returns the 10 percent of 100,  z=10)

The above simple function is a generic mechanism to get a percentage value of another, but if we consider that we wanted a common application of this function was to always calculate 10% of a given value - rather than write a slightly modified version of the function we can simply curry the function as follows:

Now, if the function tenPercent(x) is called, it uses the original percentage() function, but curries the value 10 as the first argument. (If you need to curry other argument positions you can also use the rcurry() function to curry the right most argument, or ncurry() which also takes an argument position - check the Groovy docs on currying for more info)

Immutability:

Immutability is partially supported in Java normally with use of the final keyword (meaning variables can't be changed after being initially set on object instantiation). Groovy also provides a quick and easy @Immutable annotation that can be added to a class to easily make it immutable.  But really, there is more to avoiding immutable state than just having classes as immutable - As we have functions as first class objects, we can easily assign variables and mutate them within a function - so this is more of a mindset or philosophy that you have to get used to. For example:

The first example is probably more like the Groovy/Java code we are used to writing, but that is mutating the state of the list - where as the second approach leaves the original list unchanged.

Map Reduce:

As a final note, there are some functions in FP that are pretty common techniques - the most famous of which these days (in part thanks to Google) is Map-Reduce, but the trio of functions are actually Map, Reduce(also known as Fold) & Filter - you can read more about the functions here (or just google them!), but these functions actually correlate pretty nicely to core Groovy functions that you probably use a lot of (assuming you are groovy programmers).

Map

map is the easiest to understand of the three. It takes in two inputs - a function, and a list. It then applies this function to every element in the list. You can basically do the same thing with a list comprehension however. 

Sound familiar? This is basically the .collect{} function in Groovy

Reduce/Fold

fold takes in a function and folds it in between the elements of a list. It's a bit hard to understand at first

This one is a bit more complicated to descibe, but is the same as the .inject{} function in groovy

Filter

filter is easy. It takes in a 'test' and a list, and it chucks out any elements of the list which don't satisfy that test.

And another simple one - filtering out a list for desired elements, this is Groovy's .findAll{} function

As I said at the start, I am new to FP and coming from an OO background, but hopefully the above isn't too far from the truth!  As I get further through the Coursera course I will try to post again, maybe with some of the assignments attempted in Groovy to see how it really stands up.

Some useful references:

Groovy docs on FP: http://groovy.codehaus.org/Functional+Programming+with+Groovy

Coursera Scala course: https://www.coursera.org/course/progfun

6 Tips: Getting your Hackathon project into Production

Recently there has been a steady stream of stories about projects that started as a Hackathon idea going into production at some of Silicon Valleys top companies. It's well known that Facebook's 'Like' button started off as a Hackathon project, and Ebay just announced a Hackathon project has led to them using Node.js for the first time in production. So what will give your project a chance of making it out to production for your users to benefit from? The team @ NerdAbility.com are giving you six tips to get things moving when your boss gives you some time to hack away!

1. Share Your Project

So you have made something really cool, maybe on your own or with some other devs. Now is the time to share your Hackathon project and get other people within the company excited! If it's a tool for the dev team why not give a lunch-time talk or send out a note with a quick-start guide. Once people see that you have made something cool and interesting it is surprising how quick it can get added to priority list for more work / release.

2. Don't worry (too much) about the technology

Hackathons are meant to be a chance to try out something new or maybe do something differently, so don't let people tell you your work has to be production ready right away. Often you will do your best work when you are excited and lets face it all of developers love shiny new things! But, once you have the concept nailed and something implemented be prepared to be flexible. Getting a new technology live can often be time consuming, so sometimes you will need to give and take to see your project get the light of day.

3. Be prepared to say 'OK that didn't work'

So you have tried to make something and, well, it didn't quite work out. This is still a valuable project! You will have learnt new things and hopefully take away a few lessons. The most important thing to do is make sure you and your team do another Hackathon! Still share your work with the rest of the team and who knows it may inspire someone else. Next time you can take a fresh idea and see what happens!

4. Upload your project to GitHub

So you have finished your kick ass project and your company decides on a change of direction, your project just isn't needed. Shock horror, this does happen. Why not try to get permission to put your project onto GitHub? This is a great way to share your work and who knows it may be just what another developer / company is looking for. Then instead of your code being in production in one place, it could be used all over the world. Dream big.

5. Work with the business guys

Great things can happen in pairs! Use your technical genius to think up a solution for someone in the business with a problem or idea. If you address a hidden problem or tap into an undiscovered opportunity you will look like hero. The great thing about this is your work will hopefully have business value so you instantly know it will have a better chance getting released. Also working with someone with a different perspective could open your mind to come up with an out of the box solution that will blow the socks off the rest of the dev team!

6. Launch it yourself

If you have built a cool application in your own time, then just launch it yourself! There are plenty of cloud providers offering free basic hosting such as Appfog, AWS and CloudBees amongst many others. Just be careful of any legal complexities that may occur from having users such as data protection and anti-SPAM regulations.

Finally.....

We loved this presentation by Philip Su, Facebook Site Lead London, from back in February this year on Facebook's engineering process and how they use Hackathons as part of their development culture.

Check it out:

Philip Su: Site Lead at Facebook - Engineering process and philosophy at Facebook from HN London on Vimeo.

Essential Resources to Become a Life Long Learner (in tech)

Is one of your New Year Resolutions to re-skill? Thinking about re-training for a new career (or even just a new hobby) in tech? Then you're in luck! 

Today, more than ever, the barrier to entry for starting to learn a new technology or programming language is all but nonexistent, all you really need is a computer (or even a mobile device) and a web connection and you are pretty much good to go - just choose your preferred technology, an IDE and get started.  Almost everything is open source or at least free to use for a single developer just out to learn and there is a wealth of blogs, articles and Q'n'A sites ready to help you with tutorials, walk-through's and helpful advice - many of these backed with ready and running code bases on GitHub free for you to play with and generally work out what is going on.

However, with all these resources it can sometimes be a bit daunting with so much content. Once you have chosen a language how do you know where to start? Here are some of our favourite sites and resources that we have discovered and found useful in learning new skills:

• iTunes U - a lesser known category on iTunes is their academic section, iTunes U(niversity) featuring loads of podcasts and lectures from a range of academic organisations, and some of this stuff is serious! Several large universities have uploaded full lecture series there, and by and large they are free to download (yes, you have to install iTunes, which sucks, we know).  Want to take the full term of Stanford university's iOS course? Its up there. Want to learn AI for chess playing from Cambridge uni? Yep, got that too. And for free.
• MIT OpenWare - MIT have been one of the strongest advocates of open sourced education. A lot of there lecture series are online (can also be found on iTunes, but can be avoided).  Is it just us who thinks its amazing that anyone around the world with a web connection can get educated by the most prestigious academic organisations around?
• Khan Academy - there is a lot of hype around this one, well funded with some pretty big names supporting it (jQuery creator John Resig is a Dean there), a not-for-profit aiming at providing free education for everyone. The academy provides lots of video based courses as well as interactive challenges and detailed stats on how you are doing.
• Udacity - this is another recent, well-funded startup trying to tackle free higher education for all. Founded my three robotocists it is slowly building a very respectable catalogue of uni level courses ranging from CS101 to AI for robotics. As with the Khan academy, the lectures are purely for the web so the videos are clear and designed for remote learning (different from the filmed university lectures which are targeting classroom based learning).  We have recently created and Open Sourced the Spring-Social implementation of the Khan Academy API - so if you are working with the JVM and want to have a play with the Khan Academy API then check it out on GitHub
• CodeAcademy - we have mentioned before we are fans of code academy, code academy is an in-browser development environment that walks you through programming exercises to help you learn with your hands - currently supporting JavaScript, HTML, Ruby and Python
• Free eBooks - there are loads of great free eBooks available online, so many there is no point listing them, instead I will just point you here. Which leads nicely on to the next point..
• StackOverflow - what really needs to be said about SO? It is the definitive q'n'a site for tech. If you are just starting learning head over and sign up, the help from the incredibly active community over there will be invaluable (although be sure to read the posting guides, they can be a little unforgiving at times!).
• Coursera - Another massively popular online learning resource, this one recently generated a lot of interest with its recent Scala course taught by the original creator of the language!  We are currently working on some secret integration with Coursera at NerdAbility, and you will soon be able to integrate your Coursera account and show off which courses you have completed!

Hopefully the above resources help on your path to re-skilling.  In reality, getting your hands dirty with code and trying to solve problems and fix errors is the best way to learn, so don't forget to get stuck in - and maybe when you are more confident try answering questions on StackOverflow!

Of course, with these new found skills you will want to show them off, so we'd recommend heading to NerdAbility and registering (if you haven't as already) and update your skills, add your StackOverflow profile and even add a custom section talking about what you are learning (employers always love to know that candidates are proactive and motivated when it comes to learning new things and keeping up with technology). 

Leave your comments with any other tools and resources you have found useful in your journey of becoming a life long learner.

Tech: Building an RSS Reader Android App

This tutorial will walk through building an RSS reader on the Android platform (focusing on 3.0 + as it will be using Fragments). All the code is available as a complete, working Android app that you can fork/download and fire up straight away on a compatible Android device/emulator. So feel free to go grab that from GitHub before continuing.

It is not an unusual requirement for mobile apps these days to be able to consume an RSS feed from another site (or many) to aggregate the content -  Or maybe you just want to build your own Android app now that Google has announced it will be retiring Reader.

Those of you who have worked with RSS in other JVM languages will know that there are plenty of libraries available that can handle parsing RSS - however, because the android platform doesn't actually contain all the core java classes, almost all of the RSS libraries have not been supported.

Fear not though, as Java's SAX parser is available, so with a bit of code we can get a custom RSS parser up and running in no time!

This walk through will cover off the basics of getting an RSS reader app up and running quickly and will also cover some details of Android's fragment system for tablet optimization as well as some things to watch out for (such as changes in the platform that mean network operations cannot be run in the main thread, which requires some tweaking if you have worked on earlier versions).

All the code for the app is also available on our GitHub so feel free to fork that and try loading it up on your Android device/emulator.

Parsing an RSS Feed:

So to get started we will look at parsing the feed - if you have any experience parsing XML using SAX in Java then you will know how easy this is. All we need to do is to tell the parser which XML nodes we care about, and what to do with them.

If  you have never implemented a SAX parser before, there are three primary methods that we will override: 

• startElement() - This is called by the parser every time a new XML node is found
• endElement() - This is called by the parser every time an XML node is closed (e.g. </.. )
• chars() - this is called when characters are found between nodes

Because we only really care about capturing data from the leaf nodes, our startElement() method is left empty. The chars() element has to be watched, as there is no guarantee when it will be called (e.g. in a node like hello world  this method might be called several times between the start and end) so every time we will just append the contents to a StringBuffer - that way we can be sure that we will have captured all the data in the node.  By the time the endElement() method is called, we know that we have the contents of the node itself, and we just have to store the data.  At this point, we just quickly knocked up a POJO with the attributes that we wanted to capture - the Strings that we match on are the node names from the ATOM RSS feed (that Blogger uses) - if you are using another feed, just have a quick look at the feed and update the node names appropriately.

Using our Feed in an Android App

So, that was easy right? Once that parser has run through (and you could use that code standalone in any java app really) then you will have a list of Java objects that have the core details about the latest blog posts on the feed (title, author, datecreated, content, etc) - So now lets look at using it in an Android app.

We will assume a basic understanding of the Android SDK and the concept of Fragments, so won't go back to basics with that stuff.

What we will do, is create a basic ListFragment and an RSSService class that we will use to populate the list. In our ListFragment we will simply tell our RSS service to populate the list:



Simple right?

Let's take a look at what our helpful RSS service is doing for us.



The first thing to note is that this class is extending Android's AsyncTask- The reason for this is that since Android 3.0, you are no longer able to perform network operations in the main application thread, so being as our class is going to have to fetch some RSS feeds we are going to have to spin off a new thread.

As you can see, the constructor just sets some context info that we will use later on, and then builds a progress dialogue - this is then displayed in the onPreExecute() method - This lets us show a "loading" spinning disk whilst we fetch the data.

Android's AsyncTask's primary method that handles the actual work that you want to do asynchronously is called "doInBackground()" - In our case, this is simple - we just invoke our SAX RSS parser and fetch our feed data:



Finally, we will override the "onPostExecute()" method of the async class to use our newly fetched list to populate our ListFragment.  You note how when we overrode the doInBackground() method earlier we set the return to List of Articles (where Article is my simple POJO containing my RSS blog post info) - well this must correspond to the argument of the "onPostExecute()" method, which looks like this:



Actually, all we really needed to do in this method would be pass our new List or articles to the ListFragment and notify it of the change as below:



However, in our application we have added a bit more sugar on the app - and we have actually backed the app with a simple DB that records the unique IDs of the posts and tracks whether or not they have been read to provide a nicer highlighting of listed blog posts.

So that's it - there's plenty more you can add to your RSS reader app, such as storing posts for reading later, and supporting multiple feeds/feed types - but feel free to fork the code on GitHub, or just download on to your android device to enjoy all our NerdAbility updates!

RSS application running in an Andriod  tablet emulator

Tech: A Rough Guide to Atmosphere with Spring, Tomcat & Jersey: Avoiding the Pitfalls

Welcome to the second  installment of the NerdAbility tech guide series. This post looks at the Atmosphere framework! Enjoy....

Edit: I Previously forgot to include the  project sources and people have rightly asked in the comments about it. I couldn't find the example I wrote this blog post from, but I found a similar example on my github. You can find it here

More and more it's becoming a common requirement to support some sort of bi-directional (asynchronous) channel in modern web applications. Atmosphere is a great library which does much of the heavy lifting if you're running on a Java, servlet based stack. It offers support for WebSockets, Server Side Events (SSE), Long-Polling, HTTP Streaming (Forever frame) and JSONP. There are, however, some pitfalls and, if you've not got a lot of experience in the area, it can be frustrating trying to work out whats going wrong.

Having run through the process once I thought it'd be nice to cover my implementation in some detail. Maybe you'll see something which can help you with your own configuration or implementation problems.

Overview of the stack

This is what I had in place before integrating atmosphere.

• Application server: Tomcat 7.0.32 (Must be greater than 7.0.30 for atmosphere to work correctly).
• Container: Spring 3.2 RC1 ( though anything post 3 is fine )
• Build Enginer/ Dependency Manager: Maven3
• Front-End: Javascript / JQuery

Include the Dependencies

You'll want to add the following entries to your POM :

Properties:

Dependencies:

Just a quick note about the cors-filter dependancy, you'll only need it if you want Cross-Origin-Resource-Sharing support. I thought about covering it in this article, but maybe i'll write it up as a follow up

The web.xml

If you're not using servlet 3 already, time to upgrade. Change your web-app config to look like this:

The next step is to include the standard spring dispatcher etc to your configuration. You will probably you'll already have this anyway.

The Atmosphere Controller 

Using The Atmosphere Servlet

I fully accept that everyone has their own pet way of setting up servlets, this is just one possible mutation. You want to add the following to your web.xml:

Pitfall 1: The broadcasterLifeCyclePolicy tells atmosphere to destroy any connections that been closed by the client, enabling it will prevent OOM (Out of Memory) issues in your application.
Pitfall 2: The recoverFromDestroyedBroadcaster prevents an exception being thrown when you try to reuse a broadcaster that you have recently destroyed ( useful if you want your client to subscribe using its own personal channel ).
So now you're able to use spring MVC alongside your atmosphere code without them stepping on each others' toes. Lets move on.

The Controller Class

Mine was simple enough. It basically passes off the broadcasters that it creates to another service which whatever listeners you have configured ( Mine was RabbitMQ ) will call when they have an appropriate message.

@Configurable: Using @Configurable is a great way to use aop classweaving to inject spring dependencies into non-managed spring classes. There is extensive documentation on how to set this up in spring

Pitfall 3: Make sure you set the suspend period, or you're going to be dealing with a java.net.SocketException: Too many open files in pretty much no time.

The Service Class

Also very simple. Using a map to store the in use channel ids -> broadcasters, the service is simply responsible for looping through the registered broadcasters and broadcasting a generic message.

The Async Class

Probably you'll have some asynchronous process which is waiting for messages, there are many possible implementations that this may take so i'll just show one of the popular ones RabbitMQ.
You'll need to make sure you add the following dependency to your pom:

RabbitMQ has a pretty clean java integration which is nice and quick to set up. The first component is a context configuration file where you bind your queues, connection factory, admin, exchanges and listeners.


Notice the reference to broadcastQueueConsumer we used there ? Lets write that now.

I've omitted the declaration of the Simple Message Converter in the context file but please dont forget about it. Once you're done you've got a pretty clean implementation of the serverside and should be ready to move onto the frontend.

JQuery frontend implementation

Really its quite simple to build a small subscription handler. I've added an example of mine below :
Pitfall 4: Make sure you set enableXDR to true if you want to support cross domain requests.

Pitfall 5: To send custom request params it seems like you need to pass via url (maybe jfrancard has fixed now but not sure)

Thanks for reading, and feel free to ask any questions via a comment below or our twitter @nerd_ability

Tech: How to Fix 'SSLPeerUnverifiedException: peer not authenticated' Exception in Groovy / Java

This is the first in a series of tech posts on the NerdAbility blog. We are aiming to blog any useful tips / gotchas we come across when developing with various technologies. We will still be posting tech recruitment insights and NerdAbility news, so stay tuned!

How to Fix 'SSLPeerUnverifiedException: peer not authenticated' Exception in Java / Groovy

When developing with web services in Java you may come across the need to connect to a HTTPS URL, for example when creating a REST client. In some cases there will be an issue with the type of certificate the web server is using, resulting in a SSLPeerUnverifiedException.

To solve this you could previously export the servers SSL certificate via firefox / chrome and load this directly into the cacerts keystore (jvm's default trusted keystore). In recent versions of firefox / chrome this feature seems to have disappeared.  In this post we will show you how to grab the certificate using command line tools and then load it into the cacerts keystore. Finally we give an example of connecting to a HTTPS URL with Groovy using RESTClient.

Please note this guide is for Linux / Mac users. Windows users may be able to follow along using cygwin, but we have not tested this. If you are using an alternative trusted keystore in your application, use this instead of cacerts in the examples.

Prerequisites: Before loading any key into your cacerts keystore, please verify you are happy with the certificate and its authenticity, and issuer. You can do this by using a tool like this one.

Disclaimer: Follow this guide at your own risk, we can not be held liable / accountable for any damage or issues caused to you or your systems.

Step 1: Download and Store the Certificate

To download and store the certificate run the following command, changing $ADDRESS for the sites address. For example https://www.facebook.com would become facebook.com:

echo -n | openssl s_client -connect $ADDRESS:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$ADDRESS.cert

To check the certificate was grabbed, you can run:

cat /tmp/$ADDRESS.cert

This will output the certificate and you should see something like:

-----BEGIN CERTIFICATE-----
*DATA*
-----END CERTIFICATE-----

Step 2: Load this into the default keystore for the JVM CACERTS

First of all you need to locate the cacerts keystore for the JRE you are using. To find out the version of java run the following command:

java -version

This should give you something similar to:

java version "1.6.0_11" Java(TM) SE Runtime Environment (build 1.6.0_11-b03) Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Next take the Java version number from the previous output, in this case 1.6.0_11 and use locate to find the cacerts keystore for this Java install:

locate cacerts | grep "1.6.0_11"

The output should give you something similar to:

/usr/lib/jvm/jre1.6.0_11/lib/security/cacerts

Now you have enough information to import the key into the keystore. Run the following command, replacing the $ADDRESS with the address variable you used earlier, the $ALIAS with a name for the certificate i.e. facebook. Replace the $PATH variable with the path to the cacert (the output from the locate command we just ran). Also we have added the -storepass argument, passing the default password for the cacerts keystore. You will want to change this, if you have not already, and should be prompted to do so.

sudo keytool -importcert -alias "$ALIAS" -file /tmp/$ADDRESS.cert -keystore $PATH/cacerts -storepass changeit

Once you run this you will be shown the certificate and prompted to confirm you want to import the certificate:

Trust this certificate? [no]:  yes
Certificate was added to keystore

Now you should have the certificate ready for use in your application, providing it is configured to use the default keystore and runs on the JVM we configured the certificate for!

Step 3: Test It!

Here is some example Groovy code using RESTClient: