Google’s AI Mode Shows Drastic Inconsistencies in Search Results

Introduction

A recent study has shed light on the performance of Google's new AI Mode, revealing significant inconsistencies in search results. Conducted by SE Ranking, this analysis of 10,000 keywords highlighted that Google's AI Mode returns different URLs for the same query a staggering 91% of the time.

Study Findings

The research emphasizes minimal overlap between the sources cited by AI Mode and those found in traditional organic search results. Specifically, the average overlap of exact URLs across three datasets—AI Mode, Google’s AI Overviews, and the top 10 organic search results—was just 9.2%. This low figure showcases the volatility inherent in the AI Mode.

To determine the consistency of AI Mode, researchers executed the identical set of 10,000 keywords through the feature three times in one day. The results were notably inconsistent, with 21.2% showing no overlapping URLs across the three trials. Additionally, domain-level consistency was calculated to be slightly better at 14.7%, indicating that AI Mode occasionally referenced different pages from the same website.

Impact on Search Results

Only 14% of the URLs returned from AI Mode matched the top 10 traditional organic search results. The comparison at the domain level showed an overlap increase to 21.9%. Alarmingly, in 17.9% of test queries, AI Mode demonstrated zero overlap with organic URLs, suggesting that its selection process operates independently from Google’s traditional ranking algorithms.

On average, each AI Mode response contained 12.6 citations, with the most frequent format being block links at 90.8%, followed by in-text links at 8.9%, and AIM SERP-style links making up a mere 0.3%. Despite the volatility, some domains were consistently acknowledged across all tests, most notably Google properties, which constituted 5.7% of all cited links, predominantly featuring Google Maps business profiles.

AI Mode vs. AI Overviews

In a comparison between AI Mode and Google’s AI Overviews, the study found minimal overlap with an average of just 10.7% for URLs and 16% for domains. This indicates that these two AI-driven systems utilize distinctly different logic and criteria for information sourcing.

Challenges and Opportunities

The volatility of results from AI Mode presents both challenges and new opportunities for content publishers. The variation in responses for identical queries complicates the tracking of visibility. However, this fluidity also opens pathways for greater exposure. Unlike conventional search results that are often dominated by a select number of top-ranking pages, AI Mode appears to refresh its citations frequently. Consequently, publishers offering relevant, high-quality content stand a better chance of featuring in AI Mode responses, even if they aren't listed in the traditional top 10.

Adjustments for SEO and Content Creation

In light of these dynamics, successful adaptation to this evolving landscape is crucial for SEOs and content creators. The study advises embracing new strategies that account for the distinct operational framework of AI Mode and its implications.

Conclusion

In summary, Google’s AI Mode is a groundbreaking yet unpredictable tool that reshapes the landscape of online searches. By diverging from traditional search models, it presents users and content creators with both complex challenges and novel opportunities for engagement.