We Built an Indexing Tool and Google Ignored It: What 30 Real Submissions Taught Us About Getting Pages Indexed

The 200 OK that means nothing

I want to start with the result that changed how I think about this whole category, because it took me two months to see it clearly.

SwiftIndex sends URLs to Google's Indexing API. When a push succeeds, Google returns HTTP 200 and a notification timestamp. For a long time I treated that as the job being done. The dashboard said submitted, the log said accepted, and I moved on.

Then I opened the URL Inspection tool in Search Console and checked the pages I had been pushing. Here is what Google said about them, on the same day the pushes were accepted:

Never crawled. Not crawled and rejected, not crawled and held back. Googlebot had not fetched the page at all, weeks after we told Google it existed, over and over.

This matters for anyone shopping for an indexing tool. If a tool reports success the moment a push is accepted, it is reporting delivery and calling it indexing. We did exactly that for a while. We were not lying on purpose, we just never checked.

Why Google ignores a new domain

The obvious question is why. If Google knows the URL exists, why not spend two seconds fetching it?

The answer showed up in the same Search Console property. Our homepage is indexed and has been for months. When I checked it, the last crawl was five days old. Five days between visits to the front page of a site Google already knows.

That is the whole story in one number. Google was giving our entire domain a handful of fetches a week. A new page on that domain is competing for one of those few fetches, and it loses to the homepage every time. The crawl budget was not being spent badly, there simply was not much of one.

New domains get very little crawl attention because Google has no history to justify more. No links pointing in, no track record of pages worth keeping, no pattern of updates that rewarded a return visit. Pushing harder does not create that history. It just adds more notes to the inbox.

The contrast that proved it

Around the same time I submitted a forum thread on a busy message board. It was in Google within about nine hours, and I nearly counted it as a win for our tool.

Then I checked the page we had published to promote it. Google had never crawled that page either. Our signal could not have caused the indexing, because Google never looked at the thing carrying the signal. The forum gets new threads indexed in hours because the forum gets crawled constantly. That thread would have been indexed if we had never existed.

Attribution is the hardest part of this job and almost nobody does it. If you do not record whether a URL was already in Google before you touched it, every page that later appears looks like your win.

What actually got pages indexed

Out of everything we tried, one route has a clean record: pages on a domain that is verified in Google Search Console, pushed through the Indexing API from an account with owner permission on that property.

Those went in. Search Console confirmed them as submitted and indexed, usually within hours. Same API, same code path, same tool. The only difference was ownership.

That difference is not a technicality, it is the entire mechanism. When you own the property, Google has a verified relationship with the site and treats your notification as coming from the person responsible for it. Without ownership, the API answers with a permission error and there is nothing to fall back on that carries the same weight.

The practical version

1. 1Verify the domain in Search Console. Nothing else on this list works properly without it. Use a domain property rather than a URL prefix so subdomains and both http and https are covered by one verification.
2. 2Submit an XML sitemap and keep its lastmod honest. A sitemap whose dates never change is one Google stops re-fetching. Ours was frozen at build time for weeks and we did not notice.
3. 3Link new pages from pages Google already crawls. Our listing page was reachable only from the sitemap. Search Console called it discovered and never fetched it. Adding a link from the indexed homepage is what put it on a path Googlebot already walks.
4. 4Use URL Inspection, then Request Indexing, for pages that matter. It is rate limited and manual, which is exactly why it still carries weight.
5. 5Earn a few real links. This is the slow one and the one that actually raises crawl budget. Nothing in an indexing tool substitutes for it.

There is a fuller walkthrough in our guide on how to get a new website indexed by Google, including how to check each step worked rather than assuming it did.

Two measurement traps that cost us weeks

Both of these made our data wrong in the same direction. They told us pages were not indexed when they were, and told us pages were broken when they were fine.

The site: operator is not a reliable index check

We verified indexing by searching for the full URL with the site: operator. If the page came back, it was indexed. Clean test, or so I thought.

Then a page came back with zero results from site: while ranking first for a normal search of its own title. It was indexed. It was ranking. The operator returned nothing.

The site: operator is a rough diagnostic, not an index lookup, and for deep URLs it drops pages that are plainly in the index. We had been marking those as failures and refunding them. If you are checking your own pages this way, check a few against a plain search of the page title before you trust the answer.

A blocked fetch does not mean a blocked page

Our tool fetched each URL before submitting it, to make sure the page was alive. Sensible, until you see what real sites return to a crawler they do not recognise.

One forum answered our request with a 503 and a proof of work challenge. Same URL, same second, from a browser: a normal page. That site was not down and was not blocking Google. It was blocking us, specifically, because we were an unknown bot. We were treating those as dead links.

Only 404 and 410 are evidence a page cannot be indexed. A 403, 429 or 503 usually means an anti-bot layer decided it did not know you, which tells you nothing about whether Googlebot gets through.

What does not work, and why people keep selling it

We fired every free signal that exists. WebSub notifications, RPC weblog pings, archive snapshot requests, IndexNow. All accepted. None of them produced a crawl of the page carrying the link.

IndexNow is worth keeping because Bing, Yandex and Seznam genuinely use it. Google does not. The RPC ping networks were built for blog directories that mostly no longer exist. Sending more of these does not accumulate into a crawl.

Which brings up the tools promising any link indexed in two minutes. There are only a few ways to produce that claim. One is spam networks pointing at the target, which is against Google's spam policies and puts the receiving domain at risk. One is counting indexing that was going to happen anyway, like my forum thread. One is reporting the push and never checking the outcome.

We worked through each of these in detail in what instant indexing tools can and cannot do, including the questions worth asking a vendor before you pay.

What we changed in the product

Writing this up meant changing things I would rather have left alone.

• Verification no longer relies on a single site: query, because that query was wrong often enough to refund pages that were indexed.
• A URL is only failed on a 404 or 410. Pages behind anti-bot layers are submitted normally.
• We record whether a URL was already in Google before submission, so a page that was going to be indexed anyway is never counted as our result.
• The dashboard used to say indexed by us. It now says indexed, because verification proves a page is in Google, not that we put it there.

The honest summary of where our tool helps: pages on domains you own and have verified in Search Console. That is the route with a clean record, and it is the one worth building on. For links on sites you do not control, we can tell you whether a page is indexed and watch it over time, and that is a real thing, but it is monitoring rather than causation.

If you publish time-sensitive pages, the ownership point is the whole strategy. We wrote up how that plays out for concerts and fixtures in getting event pages indexed fast.

And if Search Console is currently telling you something confusing about your own pages, the guide to discovered versus crawled but not indexed covers what each status means and what to do about it.

Frequently asked questions