3 Reason Why you Should Not be Testing

There are not many reasons to not be testing. As a matter of fact, we struggled to come up with a third reason. The benefits of running even the simplest of tests are so impactful that there need to be very significant and potentially dangerous environments for testing not to be worthwhile. If you find yourself in one of those situations, I strongly recommend reconsidering why they exist and looking deeper into the situation.

Let’s dive into these no-test scenarios:

1. If you, or your team, is not willing to follow through with the results of the test.

In the past, we have ran test for different sized companies. One of these was a company with a mid-6-figure annual revenue. We ran a test that our entire testing team agreed was a relatively insignificant change to the aesthetic and user experience: we changed the background of the website. And so the test ran for two weeks until we received enough results to outline clearly that as a whole, users preferred version A. Resulting in a 2% increase in overall sales. Naturally we all assumed that this outcome would go live, we’d increase the companies profit by thousands in a few short weeks great! We just did not expect that our CMO would veto that option; even after taking a closer look at the statistics involved.

It was not a matter of opinion; it was simply facts at that point. But it didn’t matter, we had wasted resources and time into the test only to end up at the same starting point. Another test or project would have served us better.

If you find yourself in this situation, your company needs to re-evaluate who makes the final decisions regarding the look and feel of yourself. If any one is willing to let their opinion take precedence over facts and analysis, consider why you have placed them in that position.

2. Your content changes too often

If you have a rotator that changes daily, testing the UI or elements within that area will have severe limitations regarding the traffic required. To get good results in any test, we have to have a statically significant group of people seeing each of the content. In statistical surveys, poll takers often carefully select representatives of similar groups and interests to define a clear preference between the various groups. Given that everyone on the internet tends to be anonymous, we have to rely on much larger audiences to ensure that we have significant results.

Also, changing the content of the test midway through often pollutes the result. It’s acceptable if you change all variations at the same time, but we won’t know what’s the main driving force.

If you find yourself in this situation, you have to re-consider your on-page strategy. Are you trying to be busy for the sake of being busy? Are you promotion reaching the right audience? Could the efforts in creating those promotions be better served elsewhere?

3. It’s against your religion

I am not familiar with any religions that many be against statistical testing, but if that’s the case, you shouldn’t do it.

In all seriousness, testing is critical for every website. It’s often a crucial way to gain better conversion, improve customer retention and just gain insights into the behavior of your average user. Even if you were in any of the above situations, you have to be asking yourself why they are prohibitive to your testing, not just ignore them.