Rank Tracking Under Pressure: What Google's War on SERP Scraping Means for SEO Tools

How rank tracking works – and why it is so vulnerable

The principle behind classic rank tracking is straightforward. A scraper sends a keyword to Google Search, loads the results page, and reads the source code. From the links and snippets, it determines which URL appears at which position. Repeated across thousands of keywords, locations, and devices, this produces ranking histories, visibility indices, and competitive comparisons.

For this to work at scale, tool providers need three things:

• Access: Requests must look like those of real users. To achieve this, they are routed through so-called residential proxies – networks of private IP addresses.

• Efficiency: The fewer requests required per keyword, the cheaper the data.

• Readability: The source code must contain the target URLs and snippets in a machine-readable format.

Over the past 20 months, Google has targeted all three of these requirements. No official alternative exists: the Custom Search JSON API is already closed to new customers and will, according to Google, be shut down on 1 January 2027. The recommended successor, Vertex AI Search, is an enterprise search product for proprietary content and does not provide public SERP data.

Timeline: How Google has tightened the screws

Viewed individually, each measure looked like a minor technical footnote. Taken together, a clear pattern emerges:

The sequence is telling: first access was made harder, then efficiency, then readability. And finally, the reliability of the data itself.

Google's internal redirects: the goto links

The most obvious method of making life difficult for scrapers is now a reality. Organic search results no longer link directly to the target page, but instead to a Google-internal address following the pattern google.com/goto?url=[hash]. Only when the link is clicked does Google redirect to the actual URL. According to PPC Land, the first sightings occurred as early as June 2026; Google confirmed the rollout on 26 August. The redirects are now also appearing in SERP features.

Earlier redirect variants such as /url?q= posed little problem for tools, because the target URL appeared in plain text within the parameter. With the goto format, only a hash is present. It cannot be decoded locally, and HEAD requests are rejected. Anyone wanting to know which domain sits behind a result must resolve each redirect individually.

The consequences for tools that do not do this are severe:

• All target URLs appear to resolve to google.com.

• A site's own pages are no longer recognised as such.

• Share of voice and competitive comparisons are calculated against incorrect domains.

• Historical time series break off at the cut-off date.

Providers have responded differently. SISTRIX states that it resolves each link with an additional server request; SerpApi announced a solution that restores direct URLs in early September. Nozzle CEO Derek Perkins puts the effort required to fully resolve rankings across five results pages at 500 to 1,000 requests. For context: until September 2025, a single request with num=100 was sufficient.

Google's official justification for the measure is a general reference to protection against misuse. What constitutes misuse is, of course, defined by Google itself.

Data poisoning: when false data is worse than no data

The most consequential escalation to date is also the most recent. According to DemandSphere, Google has been serving manipulated results pages to detected bots since early October, affecting approximately 11 per cent of the SERPs collected daily. Nozzle's own analysis puts the figure at 8 per cent. Both providers describe the same pattern: first came broad IP blocks and more CAPTCHAs, then substituted results. Gaps are preferentially filled with social media and forum pages, while tracked domains disappear from the top 10. Source: PPC Land

Google has not commented on this, and both figures come from providers with a vested interest in the matter. Nevertheless, the mechanism is the critical point:

• A block creates a gap. The provider sees it, can flag it, or remeasure.

• A manipulated SERP looks normal. It produces a plausible but incorrect figure.

In practice, this means: an apparent drop out of the top 10 for an important keyword could be a genuine ranking change. Or it could be a results page that only a bot ever saw.

The timing makes matters worse. Between 24 September and 8 October, the Google Spam Update September 2026 was running concurrently. According to DemandSphere, the share of AI Overviews in brand searches also jumped sharply in September, from around 26 to over 80 per cent of tracked keywords. Genuine updates, new SERP features, and poisoned measurement data are virtually impossible to disentangle during this period.

DemandSphere has responded with a feature called "Observability", which reports error and sampling rates alongside rankings – in other words, data about the quality of the data. Most other providers do not yet offer comparable transparency.

The legal front: Google versus SerpApi

Alongside the technical measures, Google is also attempting to stop scraping through the courts. In December 2025, Google sued the Texas-based provider SerpApi, which sells SERP data via API to SEO tools, AI companies, and other customers. The core allegation: SerpApi circumvents SearchGuard and thereby a technical protection measure within the meaning of US copyright law (DMCA). Google sought $200 to $2,500 in damages per circumvention. With hundreds of millions of requests per day by Google's own estimate, SerpApi noted that the resulting sum would mathematically exceed the entire US economic output.

Google lost the first round. On 20 July 2026, the federal district court in Northern California dismissed the DMCA claim. The reasoning: for search results that contain no copyright-protected content, SearchGuard does not protect access to a protected work. Google cannot relitigate this aspect. SerpApi did not win outright, however. The court found that the circumvention of SearchGuard had been sufficiently demonstrated and gave Google a second opportunity regarding licensed content, such as that found in Knowledge Panels.

Google took that opportunity. The amended complaint of 10 August relies on licensing agreements, including one with Reddit. SerpApi has again moved to dismiss. The hearing scheduled for 13 October was cancelled; the court intends to set a new date or rule on the papers. As of today, the case remains open.

Regardless of the outcome, one thing is clear: no ruling will compel Google to reinstate num=100 or ease its bot defences. The narrower the legal avenue becomes, the more Google is likely to rely on technical measures. Developments since July support this view.

Outlook: What rank tracking is likely to look like in future

Rank tracking will not disappear. But it will become more expensive, slower, and less precise. The following points represent an assessment, not a certainty.

Screenshots instead of source code? Probably not the decisive step

One obvious idea would be to capture results pages as screenshots rather than reading the source code, then extract data using AI or OCR. As long as a snippet displays the domain, the position would theoretically still be determinable. However, this only partially addresses the current problems:

• The browser is already running. Since the JavaScript requirement, scrapers work with genuine, rendered browser sessions. The completed DOM is therefore available in the same way as an image of the page.

• Text is difficult to hide. Masking snippets on the client side – for example via canvas rendering – would break screen readers, browser translation, and in-page search. Accessibility is a hard constraint here, both technically and regulatorily.

• No image helps against manipulated SERPs. A detected bot receives the wrong page whether it parses or photographs it.

The real obstacle is therefore not the reading of data, but access: remaining undetected, solving challenges, using credible IP addresses. This is precisely where Google is focusing its efforts, including action against proxy networks such as NetNut.

What is likely to change in practice

• Higher prices: Each escalation has multiplied the cost per keyword. These costs will sooner or later be passed on to customers.

• Sampling instead of full coverage: Fewer keywords, measured less frequently, with deep queries conducted only on a sample basis. DemandSphere has already confirmed that it is tracking subpages partly via sampling.

• Quality metrics as standard: Error and sampling rates are likely to become a mandatory component of reputable tools.

• Consolidation: Not all providers will be able to sustain the effort indefinitely. Smaller tools without their own infrastructure are likely to find it particularly difficult.

It is worth noting who Google's actual target is likely to be. DemandSphere founder Ray Grieselhuber and Barry Schwartz of Search Engine Roundtable both suspect that the primary focus is on AI companies using Google's results to power their own answers. SEO tools would therefore be more of a case of collateral damage. Google has not confirmed this.

What remains: Search Console as a fallback

As external data becomes less reliable, Google Search Console moves to centre stage. It has one significant advantage: it is based on real users rather than snapshots from a data centre. Manipulated SERPs served to bots do not appear there. In fact, with the removal of num=100, many bot impressions have disappeared from it as well. Many websites saw fewer desktop impressions from mid-September 2025 onwards, alongside an improved average position.

The familiar limitations remain, however:

• Your domain only: Competitors, visibility comparisons, and market analyses cannot be represented. This is the real loss if rank tracking breaks down.

• Averages rather than positions: Average position is weighted across all impressions, including SERP features, locations, and devices. It does not represent a specific position on a specific day.

• Delay: Data arrives with a lag of several days – too slow for rapid responses to updates.

• Gaps: Anonymised queries are absent, and the interface shows only a subset of the data. The API and bulk export to BigQuery help with volume, but not with anonymisation.

• AI Search as a black box: The AI Search report introduced in August shows impressions, according to PPC Land, but neither clicks nor search queries.

Search Console therefore does not replace rank tracking. It is the most reliable source for your own performance and the most important cross-check for assessing the plausibility of external data.

What SEOs should do now

1. Mark the period: Annotate ranking data from mid-September 2026 in reports and dashboards. Evaluate changes within this window only with corroborating evidence.

2. Do not make decisions based on single data points: A sudden drop that reverses within a few days is, in all likelihood, a measurement issue. Only trends sustained over several days should be acted upon.

3. Cross-reference with Search Console: If clicks and impressions in GSC are not moving, a ranking drop shown in a tool should be treated with caution.

4. Ask your tool provider: How are goto links resolved? Are error and sampling rates available? Which days are affected? If you receive no answer, factor that into your assessment of the data.

5. Trim your keyword sets: Tracking fewer, more commercially relevant keywords reduces rising costs and increases the meaningfulness of the data.

6. Build your own data foundation: Store Search Console data permanently via the API or BigQuery export. The interface shows only 16 months; your own storage holds everything.

7. Weight outcome metrics more heavily: Clicks, leads, and revenue from organic search say more about success than a single position – especially since a top position beneath an AI Overview is no longer what it once was.

Conclusion

Google is making SERP scraping progressively more expensive and less reliable. The JavaScript requirement, the removal of num=100, and the goto redirects have multiplied the cost per keyword. With manipulated results pages served to detected bots, a new threshold has been crossed: the problem is no longer just missing data, but false data. Legally, Google lost the first round against SerpApi. But this changes nothing about its technical defences.

Rank tracking will continue to exist, but with higher prices, more sampling, and greater uncertainty. Search Console will become more important as a control mechanism, but cannot replace visibility into the competitive landscape. For SEOs, this means reading data more critically, combining sources, and demanding transparency about data quality from tool providers.

Sources

• PPC Land: Google returns wrong results on 11% of DemandSphere's daily rank checks (13 Oct. 2026)

• PPC Land: Explaining DMCA – Status of Google vs. SerpApi (14 Oct. 2026)

• Search Engine Land: Google loses key DMCA claims against SerpApi (22 Jul. 2026)

• SerpApi: Google tried again. We're moving to dismiss a second time

• PPC Land: Google sues SerpApi over search scraping

• MediaPost: Google Battles SerpApi Over Scraping

• zeo.org: The Impact of &num=100 Parameter Removal on SEO Reporting

• Noble Studios: Google Retires &num=100

• Octoparse: Google official search API – Custom Search JSON API closing