I think that it's commonly accepted at this point that performance testing in software is a useful way of giving us confidence in the robustness and responsiveness of our system under load. The arguments covering these ideas are well covered by better domain experts than myself, so this article will focus on the practicalities of integrating performance testing with your CI environment.
This article assumes familiarity with the following tools
- JMeter - Widely used and highly adaptable performance testing tool.
- Jenkins - Highly popular continuous integration server.
- Maven - Build and dependency management tool.
It's out of scope for this article to introduce, in detail, these tools, but there are a number of online articles that are better suited to providing a better grounding should you be interested.
Objectives of a well designed performance testing system
A well designed performance testing system should satisfy some of the following high level properties.- Have well designed tests. Often, it's understated how important having well written parameterized tests can be, so i'll dedicate a small section of the article to explaining how you can use some of the often overlooked features in JMeter to improve your own tests.
- Support scalable generation of load. Ideally we don't want to rely on one machine to generate all the load for the performance tests. Doing so will push you into a situation where you might find yourself unneccessarily limited by network congestion and physical resource contention.
- Provide effective reporting facilities. We need to be able to analyze the results of our tests to determine whether any areas of the system haven't performed the way we expect.
Luckily, the tools to satisfy these goals already exist, its just a case of putting them together.
Tips and tricks for your JMeter tests
Ok, lets dive right in. I'm going to try and make this very example centric so you can see whats going on hopefully in the context of your own jmeter tests.
The thread group is one of the simplest and most powerful ways of simulating a user in your system. Requirements from the business are often expressed in these terms over the classic request per second metric. Thus, it can be a pragmatic approach to model the test in these terms.
Looking at the entries for the Thread Properties gives us a first look at JMeters often overlooked variable dsl, documented here. Lets take the first field as an example :
${__P(users,100)}
So what this statement is trying to do is resolve the value for the users property. If it can't it will use the provided default (100).
Pitfall 1 : Theres some weird truncation logic in play here through the ui ( which i'm assuming 90% of people will be using to edit these fields ), so make sure you check your field names.
The Thread Group ( Simulating Users )
The thread group is one of the simplest and most powerful ways of simulating a user in your system. Requirements from the business are often expressed in these terms over the classic request per second metric. Thus, it can be a pragmatic approach to model the test in these terms.
Looking at the entries for the Thread Properties gives us a first look at JMeters often overlooked variable dsl, documented here. Lets take the first field as an example :
${__P(users,100)}
So what this statement is trying to do is resolve the value for the users property. If it can't it will use the provided default (100).
Pitfall 1 : Theres some weird truncation logic in play here through the ui ( which i'm assuming 90% of people will be using to edit these fields ), so make sure you check your field names.
Http Request Defaults
So two important factors here about making your test plans generally more maintainable and more robust.
- Configurable domain - We set a default here so we can change the environment which we run our tests against.
- Timeouts - This is a common pitfall. If you don't set your timeout you could potentially be in a situation where your test hangs forever.
Pitfall 2 : Always set a timeout on http requests.
Maven Integration (Lazerycode Plugin)
Now we have some tests we need to start thinking about how to run them outside the JMeter GUI. Thankfully, a decent maven plugin exists provided by Lazerycode.
So lets assume that you've created a simple maven project in which to put your jmeter test files, properties file and plugin configuration. Lets review what you should probably have in there.
pom.xml
I've provided some elaboration on the configuration options below:
- ignoreResultFailures - If a single sampler request fails, you dont want to kill the whole test. We want to collect and display this information in the reporting phase and then make assertions at that point.
- remoteConfig - Adding the remote config section will tell the plugin to send the test plan to your distributed nodes and collect the results.
- java.rmi.server.hostname - This will inform the remote nodes where to send results, which can help you avoid a lot of the network configuration problems you might have otherwise when running the test across multiple subnets.
- propertiesGlobal - Remember all those properties we were making configurable earlier? Heres where the integration comes in. The env.foo syntax is used because thats how variables will be passed from jenkins to the maven plugin.
Next, we need to specify a couple of additional properties. Another quick look at the documentation will tell you that the plugin also checks the src/test/jmeter dir for a set of properties files. We want to, at this point, create a jmeter.properties file and populate it with the following values :
remote_hosts=test-server-1.nerdability.com,test-server-2.nerdability.com,test-server-3.nerdability.com
Obviously using your own remote nodes for in place of the defaults i've specified here.
Thats it. Your plugin is now configured to orchestrate execution of your test plans. Onto reporting.
Reporting with Jenkins Performance Plugin
Now go ahead and install the Jenkins performance plugin, which will be responsible for producing pretty graphs from our test reports. After installing, set up a job which runs mvn verify on your project ( this will run the performance tests ). Next add the post build action Publish Performance test result report. It should look like this :
Finally, remember all those variables we added in the tests earlier? Now we want to add those as parameters to the job so we can configure them when we run the job.
These will be provided to the build under the env namespace.
That's pretty much it, tweak and improve the performance tests and configuration as you see fit. The framework should support whatever you need to do.
