Synthetic Monitoring Tools: 5 Must-Have Features

There are many different classes of web performance tools, from synthetic monitoring to application performance monitoring (APM), to real user monitoring (RUM), and more. These different classes exist because each has its own strengths and weaknesses.

When evaluating open-source or enterprise-grade synthetic monitoring tools, you want to look for capabilities that maximize its strengths. When evaluating solutions, though, it can be tough to distinguish between the nice-to-haves and the must-have features that will enable you to gain the most value out of your synthetic monitoring solution.

So, we’ve put together the lists of features any strong, enterprise-grade synthetic monitoring tool should have. Let’s take a look.

---

---

What is synthetic monitoring?

Synthetic monitoring vendors provide a remote, often global, infrastructure that visits a website periodically and records the performance data for each run. The measured traffic is not of your actual users; it is synthetically generated to collect data on page performance.

The best synthetic monitoring tools enable you to:

• Test at every development stage
• Monitor in a controlled environment
• A/B test third-party performance effects
• Benchmark against competitors
• Monitor 24/7
• Baseline & analyze performance trends across geographies

(Compare synthetic monitoring to RUM, or real user monitoring.)

Synthetic monitoring features for the enterprise

So, which synthetic monitoring tool is the best for your environment? Here is a checklist of must-have features to help you get started:

Deciding on synthetic monitoring tools? This checklist of must-have features will help you select the best synthetic monitoring tool for your environment.

1. Multi-step user flows & business transactions

A key benefit of synthetic monitoring is that you can define the specific actions of a test, allowing you to walk through key flows of your application—a checkout flow or a sign-up flow—to verify its functionality and performance. This is called scripting, and a tool’s scripting capabilities directly determine how valuable it can be.

Here are some of the scripting capabilities that are essential to look for:

• How do you record scripts?
• Is there a browser-based recorder?
• Do you need to manually write code or steps?
• Can you manually edit recorded tests or do you need to completely re-record?
• What is the level of technical knowledge would someone need to be able to record a script?
• Does the tool allow you to import industry-standard formats (like Selenium IDE)?

Of course, since enterprise websites change daily, and scripts can stop working, it is also important to evaluate a tool based on its troubleshooting capabilities, such as:

• Can you test the script?
• Will it tell you what steps it fails on and why?
• Does it capture screenshots or a video of script running, so you can see what the screen looked like when a button or form field couldn’t be found?
• Can you export the script in an industry-standard format so you can troubleshoot it somewhere else if need be?

As an example, here is what the industry-standard Selenium IDE recorder looks like when testing a mission-critical “checkout user flow”.

2. Measuring & comparing performance

A big advantage of synthetic tools is they allow you to experiment with what-if scenarios and see the impact on performance. It is essential to ensure you have flexibility and options that make it easy to get a clear line of sight of the impact of your performance initiatives. Some common examples include:

• Testing your site with and without a content delivery network (CDN) or with different CDNs
• Excluding specific third parties and seeing the impact on performance
• Comparing mobile performance to desktop, or even different mobile devices
• Determining the impact of Single Points of Failure
• Measuring the performance of different A/B test groups
• Testing new features that are not enabled for all users
• Using different global locations to see how geography exacerbates any performance slowdowns

How completely a specific synthetic tool can do these things depends on how much you can control about a test. Here are some of the configuration options to look for to be able to assess the results of common web performance experiments:

• Can you configure the test to exclude requests for certain domains or URLs?
• Can you overload DNS or hostnames to point at different IP addresses?
• Can you pre-load specific cookies for a test?
• Can you test with 4G, 5G, or different networking connections?
• Can you specify custom HTTP headers to add to requests?
• Can you specify the device, viewport, or user-agent used?
• Can you specify the location? How precisely can I define the location? (Am I testing from “Canada”, or am I testing from “Vancouver, in British Columbia, in Canada?”)

Of course, configuring a test with different options is only half the battle. In all these scenarios, you will collect performance data about your sites and applications under different conditions, and then you will need to compare them.

How your synthetic solution allows you to compare data and visualize differences is critical—that determines how easily and quickly you get results. Here are a few ‘must-haves’ to look for:

• Can you compare two specific measurements?
• Can you see both absolute and relative improvement? (For example: Visually Complete improved by 400ms, or 24%.)
• Can you compare the videos or waterfalls side-by-side?
• Can you collect multiple samples of each configuration, and compare those on a graph?

For example, here is what a comparison report looks like in Splunk Synthetic Monitoring:

3. Robust alerting capabilities & integrations

Synthetic monitoring is one of the best ways to detect outages or availability issues since it actively tests your site from the outside. Critical to this is what capabilities does the tool have to define an outage and send the notification.

Here are a few things to look for:

• Can you run it from different locations inside the major geographic regions where most of my visitors are?
• What is the testing frequency?
• Can you verify the presence or absence of text?
• Can you verify with the response code?
• Will connection issues (DNS, TCP, etc.) trigger an outage?
• Can SSL certificate issues trigger an outage?

Just because you had trouble accessing the site once, doesn’t necessarily mean there is an outage, or tell you if that outage is regional or global in scope. False positives can lead to alert fatigue so here are some of the more advanced capabilities to look for as well:

• Does the tool show you a screenshot or the source code of where the error was?
• Will the tool automatically retry a test that fails to verify the outage?
• Can you test from multiple locations?
• Does the tool use results for multiple locations to automatically detect regional vs global outages?

For example, here is a real screenshot (displayed within Splunk Synthetic Monitoring) showing off a page which returned an error:

Once your synthetic monitoring solution has detected an outage, it is critical that it notifies you and your team. How you want a tool to notify you depends on the workflow of your team—email and SMS are minimums. Beyond that, you should focus on notification options that can easily integrate as tightly as possible into your team’s workflow and style. This will optimize how quickly your team can see and react to an outage.

Here are a few options to look for:

• Mobile teams in multiple countries? You may need a tool that sends push notifications.
• Decentralized team? Can the tool send messages to teams or channels inside chat applications (Slack, Microsoft Teams, etc.)?
• Are you already using an operations tool? Look for a tool that comes with turnkey integrations.
• Does the solution have a generic ‘send a webhook’ alert option? This is a great catch-all that allows you to connect the tool to the rest of your process, regardless of how the tools that make up the process evolve.

Here is what a typical custom webhook looks like. Make sure the synthetic tool you choose has similar functionality:

4. Pre-production testing

One of the key strengths of synthetic monitoring solutions is that they can help you assess the performance and user experience (UX) of a site without requiring large volumes of real users driving traffic. This means synthetic monitoring tools can be used in pre-production and lower environments (staging, UAT, QA, etc.) allowing you to understand the performance of your site while it’s still in development.

This is tremendously powerful—it allows you to use performance as a gate and stops performance regressions over time.

To do this, your solution must be able to reach your lower environments and gather performance data. It also must deal with some configuration nuances that are unique to testing environments. Capabilities for accessing pre-production that you should look for:

• Is the testing location outside of your environment? Will you need to whitelist IP addresses? How much work is involved with your security team?
• Can you ignore SSL certificate errors (because it uses an internal Certificate Authority or a self-signed certificate)?
• Can you install something inside of your environment to run tests? Do you need to open ports for it to work properly?
• Is it a physical system or virtual? If virtualized, what is the technology used (VM, Docker, software package)?
• What resources are needed to run the internal testing location?
• How many tests can you run simultaneously through it, and how can you increase that?

As an example, Splunk Synthetic Monitoring provides copy and paste instructions to launch a Docker instance to test pre-production sites:

5. Competitive & industry benchmarking

A major use case of synthetic tools is to measure the performance of an industry to gain clear visibility into how your performance stacks up against others. This application is unique to synthetic monitoring because other tools, like RUM or APM, will require you to place a JavaScript tag on the site or a software agent on the backend infrastructure, something that you obviously cannot do to the websites of other companies.

With a synthetic product, benchmarking a competitor’s site is as easy as testing your own site…you simply provide a URL—and you’re done!

However, there are various web security products that sit in front of websites and can block traffic from synthetic testing tools as a byproduct of trying to block attackers, bots, and other things that can cause fraud. You may often find that the IP addresses of the cloud providers and data centers used by synthetic providers are blocked. So, one thing to consider in your synthetic monitoring solution is:

Can you run tests from locations that your competitors are not blocking?

Another reason security products used by your competitors can block synthetic tests are because of the User Agent. If the User Agent is different than what an actual browser uses, that can cause you to be blocked. Another capability to check for is:

Can you customize the User Agent to remove anything that identifies it as a synthetic testing tool?

Once you can collect performance and user experience data from a competitor, you have everything you need to compare those results to your own site, so it is important to further understand:

• Can you easily compare multiple competitors in a single view?
• Is there an easy-to-understand way to show less technical people internally which site is “Best” and why?
• Does the tool offer a composite score like the Google Lighthouse performance score, which makes it easier to compare one site to another?

Here, Splunk Synthetic Monitoring illustrates a Competitor Benchmark dashboard:

Use cases for synthetic testing

When evaluating a synthetic web performance product, there are ‘nice-to-have’ and ‘must-have’ capabilities that will allow you to extract the most value out of your solution. As you evaluate tools, remember the use cases where synthetic testing excels:

• Measuring the experience of mission-critical business flows
• Running controlled experiments and tests to measure the impact of different features (3PC, CDNs, new features, etc.)
• Detecting and alerting on outages, seamlessly
• Validating performance and UX standards in pre-production and lower environments
• Benchmarking your performance against your competitors and industries

---

---

In order to extract the most value of your synthetic solution, keep these questions in mind:

• How easy is it to configure tests and how flexible are the controls for different experiments?
• How easy is it to integrate into your existing alerting workflow?
• How easy is it to integrate into pre-production testing environments?
• How is it to visualize and compare different tests and trends over time?

Splunk Synthetic Monitoring

Splunk Synthetic Monitoring monitors performance/UX on the client-side and tells you how to improve and make optimizations. You can even integrate this practice into your CI/CD workflows—automate manual performance tasks and operationalize performance across the business.

What is Splunk?

The original version of this blog was published by Billy Hoffman. This posting does not necessarily represent Splunk's position, strategies, or opinion.