BLOG

Testing Tool Profile: Why Threat Stack Uses ThoughtWorks Gauge

F5 Miniatura
F5
Updated March 16, 2022

Threat Stack is now F5 Distributed Cloud App Infrastructure Protection (AIP). Start using Distributed Cloud AIP with your team today.

Threat Stack has numerous tests running daily, verifying that things are working as expected in our Threat Stack Cloud Security Platform®. To supplement the Software Engineers’ unit and integration tests, our Test Engineering team has created the following as part of our automated regression test suite:

  • Browser-based tests, written in Capybara, verifying that only valid users can log into our Dashboard, navigate our site, view the trail of events created by your AWS fleet, and create and update rules based on these events that generate alerts your security team can triage, all through our web-based user interface: 150+ tests.
  • API Tests, using the Net/HTTP Ruby gem to interact with our Threat Stack API, verifying that you can retrieve the past thirty days of audit logs, list individual and groups of alerts by severity and type, list all virtual environments monitored and investigated by Threat Stack, and view the base compliance rulesets for HIPAA, ISO 27001, MPAA, PCI, and SOC 2. The tests also verify that alerts generated by these pre-configured and custom rules can be viewed, suppressed, or dismissed, all through the Threat Stack API: 130+ tests.

How can we keep track of almost three hundred acceptance tests? Test Engineers here at Threat Stack set up their tests with ThoughtWorks Gauge, a free, lightweight, cross-platform test automation framework. 

There are two main parts to a Gauge test:

  • The test specification, written in plain language, is a step-by-step list of instructions in a bullet-point fashion on how to carry out the test. The specification should read like a manual test plan, clear and concise with enough description that others can follow.
  • The step_implementation folder includes the code executing the test, wrapping each step in a code block that can be reused whenever and wherever it is called in the specification. 

By having Gauge separate out the test plan from the code that executes the test, it makes our tests more readable and maintainable, allowing test steps to be shared between our tests.

In the remainder of this post we focus on ten of the key features that we love about Gauge: 

Snapshot:

  • Website: Gauge.org, Github.com/getgauge
  • Release Date: July 2018
  • Languages Supported: Java, C#, Ruby, Javascript, Go, Python
  • Operating Systems Supported: Linux, macOS, Windows
  • IDE Plugs: Visual Studio, Visual Studio Code, IntelliJ

1. Ease of Setup

Gauge is very easy to install. Go to Gauge’s install page, and by selecting your Operating System, programming language, and IDE, tailored instructions on how to install everything will display. MacOS installs can be done in HomeBrew, Linux installs using APT_GET, and Windows using their Windows Installer, if you don’t want to install from source code. 

After everything is downloaded, the language runner for a programming language such as Ruby can be installed by typing gauge install ruby on the command line.

2. Sample Project Included

With a new test automation framework, it can be difficult determining which files need to be in which folders, and how to set up and kick off tests. 

Gauge bypasses this confusion by including a sample project, including a few tests that new users to the framework can tinker with. Let’s say you have a certain word such as “Gauge,” “Mingle,” or “Snap.” How could you design tests counting the number of vowels in the word, asserting that the expected and the actual count match? 

The example tests are data driven, with input captured in a data table with two columns, ‘Word,’ and the expected ‘Vowel Count.’

The sample project can be run, right after installation, from the command line with gauge run spec .

Sample Output, Command Line: 

3. Separation Between the Test, and How the Test is Implemented

Trying to figure out what is actually being tested within a suite of automated tests can be an exercise in frustration, especially if you need to examine numerous blocks of code before you decipher the purpose of the test, the steps that the test carries out, or the code that executes each test step. 

Gauge bypasses this confusion by separating out the test plan and how the test plan is executed. Gauge organizes steps in a Markdown file, the text formatting language created by Josh Gruber and Aaron Swartz back in 2004. The test steps are saved as bullet points in specification files, which then can be executed by the step_implementations. 

Specifications: Specifications aren’t simply listing the specifications for a feature of a product: They are a way to both organize the test code, and set up the HTML or XML reports. Each element in the *.spec file corresponds to an element in Markdown. Specifications are a “business layer test case which can also act as your feature documentation. They are written in the business language. Typically a spec or specification describes a particular feature of the application under test,” according to the official Gauge documentation. 

Each specification has a header, describing the various test scenarios that will be executed, with related test scenarios described as subheaders. These headers and subheaders are included when viewing the logs, to see what passed and what failed, and in the HTML report. 

Step Definitions: Each step listed in the specification corresponds to either a concept or a reusable step definition pairing the plain language of the spec with the Java, C#, Javascript, Python, or Ruby code that executes. 

By separating out the test from the code that executes the test, it allows other testers, business analysts, and product owners to clearly see how the test was constructed by looking at the spec file, without having to dive into the code.

4. Specifications Are Formatted for Readability

Specifications are set up to be very readable. 

  • Headers are prefixed with a hashtag (“#”) where the author can describe what the scenarios will be doing.
  • Individual test scenarios are listed as subheaders indicated with double hashtags (“##”). 
  • The test steps that carry out the scenario under test are prefixed with an asterisk, as a bullet point (“*”). 

Need an example of a specification file? Here is the specification of the demo project Gauge installs when initializing a new Gauge project:

# Specification Heading
This is an executable specification file. This file follows markdown syntax.Every heading in this file denotes a scenario. Every bulleted point denotes a step.
* Vowels in English language are "aeiou".
## Vowel counts in single word
* The word "gauge" has "3" vowels.
## Vowel counts in multiple word
This is the second scenario in this specification. Here's a step that takes a table.
* Almost all words have vowels
    |Word  |Vowel Count|
    |------|-----------|
    |Gauge |3          |
    |Mingle|2          |
    |Snap  |1     |
    |GoCD  |1     |

Each bullet point in this specification corresponds to a test step listed in the step_implementations folder. 

Need another example? Let’s say you have in the specs folder a test called login.spec that logs into a site:

# Login Test
## Verify valid users can log into site
* LOGIN: Enter Username: “info@threatstack.com”
* LOGIN: Enter Password: “1234”
* LOGIN: Select [SIGN-IN]

Each step placed in the step_implementations folder can automatically be located and executed by the test.

login_spec.rb
step ‘LOGIN: Enter Username: ’ do | username |
 fill_in(EMAIL_TEXTBOX, with: username)
end

Do you find you are executing the same series of steps again and again? You can place these three Login steps in a concept file, under Login.cpt. 

login.cpt
# Login as  and 
* LOGIN: Enter Username: 
* LOGIN: Enter Password: 
* LOGIN: Select [SIGN-IN]

If other tests have a Login component, instead of copying-and-pasting all three steps, you can insert just one line into the test: Login as “info@threatstack.com” and “1234”

5. Good Documentation and Support

Learning a new automation framework can be confusing. Sometimes reading the documentation is not enough. Do you have any questions that can’t be answered through reading Gauge.org’s thorough documentation? The developers are quite responsive on StackOverflow, Google Groups, and Gitter Chat.

6. HTML Reports Created Based on the Test Specification

Test specifications, as we have covered, are written in Markdown. These tests listed in the specification can be leveraged as the true documentation on how the software product under test operates. Because they are listed in Markdown, the specifications can be used to create easy-to-read HTML reports.

Let’s take a look at the HTML report of the demo project, counting the number of vowels in a word:

We can see that the HTML Report contains:

  • The time and date the tests were run and how long they took to execute
  • How many specifications and scenarios were run
  • How many specifications and scenarios passed and failed
  • What steps were run, with passing steps in Green, and failing steps in Red. Error messages are included in the report. 

7. Gauge Supplies Execution Hooks

Any good automation framework includes ways to set up preconditions for a test and a teardown method that runs after the tests have been completed. 

One example would be an automation suite that focuses on browser tests, which would have a setup method that initializes the browser when the test suite is started, and a teardown method which closes the browser when the tests have completed. 

Gauge provides Execution Hooks that can be executed before and after each Suite, Spec, Scenario, or Step in your automation test suite.

By using the execution codes, Gauge removes duplication of code that needs to be executed before every suite, spec, scenario, or step. 

8. Command Line Functionality

Gauge has what they term “first class refactoring support” both in the IDE and on the command line. For example, for those who like continuously tweaking the wording of a test so it becomes more and more clear what the test is actually doing, type the following on the command line: 

$ gauge --refactor "old step name" "new step name"

For more information on command line tools for Gauge, see manpage.gauge.org. 

9. Tests Can Be Data Driven

Tests can be set up to be drafted to be data-driven, so that tests which are simply variations-on-a-theme can be fed a table of test parameters to use. 

Let’s say that you have a series of webservices that need to be tested, hitting an API endpoint. Given the webservice name, the scheme, and the port, you need to check that the HTTP Response will be 200 OK. 

Since each test is similar to the other, the data can be formatted into an easy-to-read table.

webservice_test.spec

Each and every row of information is fed into the test step, and executed by the corresponding step implementation, saving the author from any duplication of work. 

10. Data Can Be Stored On-the-Fly

Sometimes you need to share data between test steps, saving and storing it for later. As in the above API test, once you have the response received from hitting the endpoint, you have test steps to make sure that it is in a valid format with the correct JSON Schema, or if you want to make sure that the proper error messages are listed when running a negative test. 

Gauge uses three types of DataStores: ScenarioStore, SpecStore, and SuiteStore, saving data as key / value pairs for the lifecycle of the Scenario, Spec, or the entire Suite of tests. 

Example: In the step implementation section, you can add element ids like so:

// Adding value
scenario_store = DataStoreFactory.scenario_datastore;
scenario_store.put("element-id", "455678");
// Fetching Value
element_id = scenario_store.get("element-id");

Wrapping Up . . .

As we indicated at the outset, when you’re running as many tests as we do at Threat Stack on a daily basis, you need a cross-platform test automation framework that is at once rugged and agile, and that has the features and capabilities to address your specific testing needs, while providing significant ease-of-use and automation so you can test thoroughly, in a reasonable time frame, with just the right amount of effort and input on the part of the tester.

Stay tuned: In a future post, we’ll be taking a look at some of the other testing tools we use at Threat Stack, including Capybara.

Threat Stack is now F5 Distributed Cloud App Infrastructure Protection (AIP). Start using Distributed Cloud AIP with your team today.