We have recently carried out successful split-testing of title tags to improve our search engine optimization (SEO). In this post, we wanted to share some of our additional learnings from our SEO testing. We decided to duplicate the success of our previous experiment by running a series of additional SEO experiments, one of which included changes to:
- Title tags
- Meta descriptions
- H1s
We have found three surprising results:
- Shortening our title tags showed improved performance in terms of visits (and other key metrics).
- Meta descriptions had a statistically significant impact on organic search traffic.
- H1 headings had a statistically significant impact on organic search traffic.
Some notes about our methodology
For this particular test, six unique treatments and two control groups were used. The two control groups remained consistent with each other before and after the experiment.
To derive the estimated exact causal changes made across visits, we use the Causal Impact most used in package Google CausalImpact to standardize the test cubes against the control cubes. In some cases, the method of difference of differences because it was more reliable in estimating effect sizes when working with strong seasonal oscillations.
This experiment was also impacted by strong seasonality and the effects of external events related to holidays, sporting events, and the US elections. Statistical modeling in our final experiment analysis was adjusted for these effects to ensure accurate measurement of the causal effects of test variants.
Learnings
#1: Short stocks continue to win
The results of this experiment aligned with the findings of our previous SEO title tag experiments, where short title tags were shown to lead to more visits.
So far we have validated our hypothesis that short title tags perform better in title tags in multiple independent experiments, including many different variations, and we feel fairly confident that short title tags perform better. We hypothesized that this effect could be happening through a number of different causal mechanisms:
- Low Levenshtein distance and/or higher percentage matching of target search queries is rewarded by Google's search algorithm and therefore improves rankings in Google search results. Wikipedia: “the Levenshtein distance between two words the minimum number of operations required to transform one string of characters into another.
- Short title tags, consisting only of the target search keyword, appear more relevant/attractive to users.
#2: Meta Descriptions Matter
We found that changes to a page's meta description can lead to statistically significant changes in visits. It seemed that long, descriptive meta descriptions performed better and conversely, short, concise meta descriptions performed worse. We hypothesized that longer meta descriptions might perform better through two possible causal mechanisms:
- Long meta descriptions take up more space on a search results page, improving CTRs
- Long meta descriptions give the appearance of more authority or more content, improving CTRs
#3: H1 tags matter
We have found that an H1 change can have a statistically significant impact on organic search traffic. However, changes to the H1 section of a page appear to interact with changes to the title tags in ways that are difficult to predict.
For example, in this experiment, a title tag change in a certain variant increased views. However, when the title tag change was combined with an H1 change, the positive effect of the title tag change was attenuated, although an H1 change by itself in a different variant led to slight increases in visits.
This highlights the importance of SEO testing before deploying even seemingly minor changes to pages.
Accounting for unexpected events
The Donald Trump effect
An example of an event we had to control (among others) was the “Donald Trump effect.” We observed a large spoon tilt on November 9 and 10, 2016 in one of our test groups. Upon investigation, it was found that the bias was due to large increases (+2000% to +5180%) in daily visits to pages related to “Donald Trump” the day after the US presidential election.
Although the spikes in traffic to these pages were short-lived, lasting only several days, they nevertheless had the potential to unduly bias or reduce the statistical significance of the results of our experiment. Therefore, these pages were removed and controlled for when conducting causal impact analyzes for this experiment.
Conclusion
Our SEO experiments illustrate the importance of running SEO tests before making changes to a web page and still delivering amazing results. However, it is important to note that the results of our experiments are true only to us and do not necessarily reflect best practices for sites generally. Therefore, we encourage everyone to discover the strategy that works best for their website through rigorous SEO testing.
