BlazeMeter provides a plugin for executing automatic load test scripts in Jenkins, an open source continuous integration tool. The plugin is named the "BlazeMeter Plugin" or "BlazeMeter's Jenkins Plugin".
This BlazeMeter-Jenkins plugin allows you to load test your site with BlazeMeter and then to view and analyze the test report in Jenkins. It also allows you to set the final Build status as good, unstable or failed, based on the reported error percentage and response times KPIs.
- What is Jenkins?
- How do I install BlazeMeter’s plugin for Jenkins?
- How do I configure BlazeMeter's plugin for Jenkins?
- How to configure a job using BlazeMeter’s Jenkins plugin
- How do I set up the Build's test?
- How do I run a BlazeMeter test in Jenkins without a Plugin?
- How do I build a job with BlazeMeter's Jenkins plugin?
- Using BlazeMeter with Jenkins pipeline
- Reports
- Adding notes to public report
- Adding JMeter session properties to test session
- Jenkins Report
- Performance Trend Report
- Performance Report
- Comprehensive Reporting Dashboard
- Pass/Fail Critieria
- Logging
- Working with Proxy server
- Plugin Changelog
What is Jenkins?
Jenkins is an open source continuous integration tool written in Java for automating tests. Builds can be started in various ways. For example, they can be triggered by a commit in a version control system, scheduled via a cron-like mechanism, or built when other builds have completed and by the request of a specific build URL.
How do I install BlazeMeter’s plugin for Jenkins?
- Login to Jenkins and go to the dashboard. Select ‘Manage Jenkins’ and then ‘Manage Plugins’. Then, from the top tabs, select ‘Available’. (You can also download BlazeMeter's Plugin to Jenkins).
- Hit 'Ctrl + F' / ‘CMD +F’ in your browser and search for BlazeMeter. Then mark the BlazeMeter check box.
- Scroll down and click ‘Download now and install after restart’.
- After the download starts, click to restart Jenkins.
How do I configure BlazeMeter's plugin for Jenkins?
- Once restarted, click 'Manage Jenkins' and then click Configure System.
- Scroll down to the 'BlazeMeter Cloud' section. You will see the URL 'https://a.blazemeter.com'.
- Click Save.
- Click 'Credentials' --> 'System' --> 'Global Credentials (unrestricted)'.
- Click the 'Add Credentials' button and choose the "BlazeMeter Credentials" button. Add your BlazeMeter API Key Id and API Key Secret.
- Add a nickname for your key in the Description field.
- Leave the 'ID' field empty - it will be generated automatically once you save the credential.
- Click the 'Test BlazeMeter credentials' button.
How do I configure a job using BlazeMeter's Jenkins plugin?
- Click on an existing job or create a new one, then press configure.
- Scroll down to the ‘Build’ steps. Click 'Add Build Step' and choose 'BlazeMeter'.
How do I set up the Build's test?
- If your keys were pre-configured correctly in the global settings, you will see your Workspaces and tests in two drop down menus.
- Click on the "Workspace ID" menu to choose the Workspace you want to see tests from.
- You will now see all your tests for your chosen Workspace in the drop down menu 'BlazeMeter Test'.
- Select a test to run.
- Now, you can either click 'Save' to save your Jenkins project and then click 'Build Now' to run the test. Or, you can use the plugin's additional configurations and only then save and build. There are a few options:
- 'Download JTL report' checkbox - downloads the JTL report after finishing the test.
- 'Download JUNIT report' - downloads the Junit report after finishing the test.
- 'Advanced' - options include the following properties:
- Path to the JTL and/or JUnit reports (if you checked the checkboxes)
- Path to the main and additional test files. This files will be upload in your BlazeMeter test (override if this files already exists) before run test. (Read more here)
- Set Notes and JMeter properties for pushing to the BlazeMeter load test
-Interrupt FreeStyleProject build, if BlazeMeter Build Step was failed.
- Set custom report link name to BlazeMeter report in your Jenkins build
Reports
Plugin supports downloading two kinds of reports: JUnit & JTL.
All reports are saved to ${WORKSPACE}
by default, but this behavior can be overriden by setting paths. Path can contain Jenkins variables, which will be resolved into actual values.
When using plugin with master/agent Jenkins configuration, please keep in mind that permissions Jenkins have in master filesystem by default differ from agent ones. E.g. if you want to save jtl/junit
artifacts outside of $JENKINS_DATA_FOLDER
, you should set permissions to 757 for the folder you want to use at master.
On remote agent Jenkins agent by default has 777 permissions for folder in /home. E.g. if you're using ssh @ for connecting to agent from master, than jenkins agent will have 777 permissions for /home/<username>
.
Adding notes to public report
There is an ability to add notes to public report. This feature is available both in Freestyle project
and in pipeline
mode.
- open job;
- go to BlazeMeter step;
- expand
Advanced
section; - enter note to
Notes
field;
Refer to pipeline usage guidenotes:'a\n\b\n\t\r'
will produce multiline note.
Adding JMeter session properties to test session
There is an ability for adding jmeter properties to test session.
- open job configuration;
- expand
Advanced
section; - enter session properties to
Session properties
field in the following format:key=value,key1=value1,key2=value2
Snippet for using session properties with pipeline
Sending properties for test session is currently not supported for multi tests.
How do I run a BlazeMeter test in Jenkins without a Plugin?
Jenkins, like any other CI tool for that matter, enables executing Shell commands. With shell commands you can execute API calls to start tests, create tests, update them, etc.
How do I build a job with BlazeMeter's Jenkins plugin?
- Once inside the job, click 'Build Now'.
- Click the build that is running.
- Click 'Console Output' to see the build progress. You’ll see the test parameters and an update every few seconds.
- When the test finishes, you'll see a summary of the test results and their impact on the build.
You can now click on ‘BlazeMeter Report’ to see the test report , the same way you would see them on the BlazeMeter site.
You can also edit and compare test results to previous runs.
All logs and data generated during test runs are stored in a ZIP - which you can access directly from your BlazeMeter account, in the 'Logs' tab of a report.
Using BlazeMeter with Jenkins pipeline
See this link for more information:
https://github.com/Blazemeter/blazemeter-jenkins-plugin/blob/master/usingPipeline.md
Jenkins Reports
Jenkins’ Reports provide great out of the box intelligence and visualization. Let's take a look at them:
Performance Trend Report
The Trend Report presents information about the trends and robustness of the successful and failed test results, over time. This report is generated by fetching the JUnit.xml report when a test comes to an end. This capability is inherent to Jenkins and is provided by the common Performance Jenkins plugin.
Performance Report
The Performance Report provides a basic KPI/Transactions report that visualizes a JTL that is an artifact of the test run. This report is built by fetching the JTL file when a test comes to an end.
Comprehensive Reporting Dashboard
A far more comprehensive reporting dashboard is available, during the test and after the test has ended. By pressing the BlazeMeter link, the BlazeMeter reports are visible providing a deep dive into the performance KPIs, allowing one to look and find performance bottlenecks.
PASS/FAIL Criteria
The Plugin is using /ci-status request for setting the build result. Depending on this request, the plugin sets one of the available in Jenkins build result:
SUCCESS - This status is set in next cases:
- test has no thresholds;
test has thresholds and they were not violated;
there were no errors in ci-status response;
UNSTABLE - Test was completed successfully, but there were errors in /ci-status response;
FAILURE - This status is set in next cases:
- test has thresholds and they were violated;
- test has no thresholds, but there were specific errors in /ci-status response:
- "errors":[{"code": 70404,"message": "Session ended without load report data","details": null}]
- "errors":[{"code":0,"Message":"Not enough resources"}]
- jenkins job got error on /test/start request;
- jenkins job got UNSTABLE status due to errors, but their were one of possible reasons, which caused FAILURE. E.g. FAILURE overrides UNSTABLE in all cases.
The default behaviour in case when request to BlazeMeter does not respond or fail is make job status as FAILURE. But you can change this status to UNSTABLE in 'Manage Jenkins' -> 'Configure System' -> 'BlazeMeter Cloud' section using checkbox 'Mark build as UNSTABLE if BlazeMeter does not respond with pass or fail'
The other way to make your build more stable is set number of request attempt that plugin will do to BlazeMeter. By default this value is 3 (plugin will repeat GET request to BlazeMeter if response was fail or BlazeMeter didn’t respond in time). Also you can set other number of request attempt using Java System properties:
Go to 'Manager Jenkins' -> 'Script Console' and set property 'bzm.request.retries.count' using command 'System.setProperty("bzm.request.retries.count", "5");'
Logging
This Plugin has <bzm-log>
file, which can simplify debugging process in case of any issues with the running test.
This file is kept in a workspace. Its location differs depending on which kind of build (Freestyle job, or pipeline) is run.
If you run Freestyle job, then you can use the following URL:http://<jenkins-host>/job/<job-name>/ws/<build-number>/
In case of pipeline, do the following to find log files:
- go to
http://<jenkins-host>/job/<job-name>/<build-number>/flowGraphTable/
- in
Step
table click atAllocate node:Start
- switch to
Workspace
and get intobuild-number
folder.
Working with Proxy Server
Since version 4.2, the Setup Proxy configuration for master, and all agents are in the section,
‘Manage Jenkins’ > ‘Manage Plugins’ > ‘Advanced’
This config will be applied for each of the agents and master node.
In case that you need to use some agent with other Proxy server, or without it, you should add JVM params to this agent in the section,
‘Manage Jenkins’ > ‘Manage Nodes’ > ‘Select Node’ > ‘Configure’ > ‘Launch method’ > ‘Advanced’ >’JVM Options’.
The argument -Dproxy.override=true means that this agent will use other proxy configuration.
In case that this agent wouldn’t use proxy server, simply pass this argument below:
If you want to use other proxy server, after the 'proxy.override' option, also add the following options:
-Dhttp.proxyHost=<IP>
-Dhttp.proxyPort=<PORT>
-Dhttp.proxyUser=<USER>
-Dhttp.proxyPass=<PASSWORD>
The other way for passing these Java properties to the agent without restart, is by using a Script Console, as described below,
System.setProperty("proxy.override", "true");
System.setProperty("-Dhttp.proxyHost", "<IP>");
System.setProperty("-Dhttp.proxyPort", "<PORT>");
System.setProperty("-Dhttp.proxyUser", "<USER>");
System.setProperty("-Dhttp.proxyPass", "<PASSWORD>");
Set Custom report link name
Since version 4.2, you can set custom link name for link to your BlazeMeter report:
By default, the Max report link name length = 35, but you can increase this value using the Java System property, or Jenkins environment variable "bzm.reportLinkName.length" .
Using Job DSL - https://github.com/jenkinsci/blazemeter-plugin/wiki/Using-Job-DSL
Using BlazeMeter Jenkins plugin with Github - https://www.blazemeter.com/blog/how-to-keep-blazemeter-test-data-updated-with-github-and-jenkins
Learn more about running performance testing within Jenkins CI from out free on-demand recording, How to Scale Test Automation with Jenkins & JMeter.
Start performance testing now with BlazeMeter. Request a demo to learn more.
0 Comments