A/B testing on a low-traffic website: what still works
3 min read · updated
On a low-traffic site, test fewer and bigger things. Choose changes large enough to move conversion by 30 percent or more, measure a step with a higher rate than final purchase, run each test for whole weeks, and accept that some decisions are better made from customer research than from a test that would take six months. Testing is a tool for settling questions you can afford to settle, not a ritual.
Why do small sites struggle with A/B tests?
Because conversions are rare events and small numbers swing. A site with 300 visitors a day and a 2 percent conversion rate makes about six sales a day. A week of a 50/50 test gives each version around 21 sales, and a difference of five sales either way is as likely to be luck as a real effect.
The usual result is a test that never reaches significance, or worse, one that is stopped early on a lucky streak and shipped as a winner.
What kinds of changes can be tested with little traffic?
Big ones. A new pricing structure, a different offer, a rewritten product page, a removed step in the signup form. Changes like these can move conversion by 30 percent or more, which a small site can detect in a few weeks. Button colours and single word swaps cannot be detected at all at this scale, so there is no point testing them.
- Offer and pricing changes
- Removing a step or field from a form
- A full rewrite of the most visited product or pricing page
- Adding clear delivery and returns information where it was missing
Which metric should a small site test against?
A step closer to the change, with a higher rate. Add to cart happens several times more often than purchase, and click to signup more often than paid conversion. A test on a high rate step reaches a verdict much faster. Check later that the improvement carried through to sales, because a change can raise clicks and lower orders.
How long should a test run?
Until the sample size you calculated in advance is reached, in whole weeks, and no shorter. If the calculation says six months, the test is not worth running in that form. Make the change bigger, pick a higher rate metric, or decide without a test.
Do not keep a test running indefinitely hoping it will turn significant. That is a quiet form of peeking, and the result it eventually produces is not trustworthy.
When should you skip testing and just decide?
When the change is clearly an improvement for customers and easy to reverse: fixing a broken link, adding a missing delivery date, answering a common question on the product page. Testing those wastes weeks. Use customer interviews, chat logs and session analytics to decide, then watch the numbers for anything alarming.
Always on systems help small sites in a different way: they keep a holdout running continuously, so evidence accumulates across many small decisions instead of needing one big test. IntellQ refuses to call a winner until each group has at least 200 visitors and the probability the change is better reaches 95 percent, and its audit tells a store in advance how many days that is likely to take.
What does a realistic plan look like for a small site?
Pick one question per month that matters to revenue, such as whether showing delivery dates on product pages lifts add to cart. Calculate the sample in advance with the calculator, run it for whole weeks, and write the result down even if it is inconclusive. Twelve honest answers a year beat fifty abandoned tests.
Between tests, collect evidence cheaply: five customer calls a month, a short survey after purchase, and a monthly read of chat logs. Those sources tell you which big question to test next.
Keep the bar for shipping honest too. If a change was inconclusive but is clearly better for customers, such as clearer delivery information, ship it and say it was not proven. If it was inconclusive and mainly a matter of taste, keep the original and move on to a bigger question.
Questions
Can you A/B test with 100 visitors a day?
Only very large changes on high rate steps, over many weeks. For most decisions at that scale, customer research is faster and more reliable than a test.
Is a Bayesian test better for low traffic?
It gives a more intuitive answer, such as the probability that B beats A, but it does not create information. Small samples are still uncertain whichever way you read them.
Should I lower the confidence level?
For low risk, reversible changes some teams accept 80 or 90 percent. State it clearly, and do not use the same standard for decisions that are expensive to undo.