Indexation and the Terrible, Horrible, No Good, Very Bad Week - Part 2
but wait...there's more
This is Part 2 of the breakdown of what happened last week and how IMO it is going to impact us moving forward. If you haven’t seen it - here’s the link to Part 1.
What does it mean for us now that Google says that ALL our sites are now being crawled by (mostly) ONLY the mobile crawler agents?
Mobile First Indexing
Google announced they have successfully gone all “Mobile First Indexing” - what does that really mean? They have been working on that for seven (7) years and had announced Dec 2022 that they were taking the due date off of the project and would let us know.
Now we know.
And whatever it means, this is huge because clearly it wasn’t just flipping a switch. How will it impact our work now that ALL our sites are now being crawled by (mostly) ONLY the mobile crawler agents?
[Observation] This is likely a cost saving project — they only have to manage two (2) legacy crawler agents, the previous chrome build and the latest chrome build. (Ask me how I know!)
By only using the mobile crawler to crawl a page instead of BOTH desktop and mobile crawling the same page practically simultaneously - this should reduce the amount of resources ($) to crawl all existing and new content. And they hinted that it might ease up on the crawl budget issue. (hinted - not promised!)
[Observation] Now that they have InspectionTool crawlers it might create an easier path for the system to be more selective in assigning keywords to content. We don’t know if our requesting indexation is going to continue even for our own sites. They’ve already said previously to let Google bot find our content on its own - - - no matter how long it might take. (Also test that from time to time and last time it was almost 2 months)
It also brings search console changes. The Crawl Reporting that is hidden in the search console settings area is going away. I will personally miss that one because in the early days that’s where I found the terms like Discovery and Refresh which explained the types of purpose behind a crawl.
Mobile First Indexing Background For Context
Based on the data from the Indexation Project, between December 2022 and April 2023 the desktop primary crawler designated sites experienced a lower indexation rate of content than the primary smartphone designated sites.
Same type of content, same method of requesting indexation (either search console or Google Indexing API).
To this day people refer to the designations within search console as a “bug” but when you read what google says about the designation, they say that the data contained within that search console for that site was built based on the data from the desktop crawler for desktop sites and mobile crawlers for smartphone sites.
At no time did Google acknowledge that desktop sites received different treatment when it came to servability aka being findable by its keywords. The different treatment was apparent if you knew how to really find your content so you’d know when it wasn’t being served.
[Observation] I think what we have and are experiencing is a result of the shifting of whatever it took for them to make the desktop crawler data process over to mobile not just first, but only.
Previously Google had indicated that “Mobile First” meant something related to ranking.
When we switch a domain to mobile-first indexing, it will see an increase in Googlebot's crawling, while we update our index to your site's mobile version.
My testing in 2022 confirmed this.
I had a domain where the client had separate sets of files for the desktop and mobile versions. I did NOTHING to the desktop set of files and made a high level of optimization on the mobile files content, tested the same pages (desktop against mobile) in Cora SEO Software to confirm that the mobile files contained more of every “good word” on the list - search term, keyword variants, entities and LSI.
Within 48 hours the changes on the mobile files MOVED the desktop version of the site in the SERPS up +11 spots, for related terms up +19 spots.
[Takeaway] Don’t take optimizations out of the mobile version of your site just to make it faster. If you do that, make sure you add them in somewhere on that rendered version of your page.
What About The Desktop Googlebots?
I’d love to share that with you. I have been tracking not just the number of bots, but whether they are desktop or mobile AND their chrome builds because during the lifetime of a chrome crawler agent - what they do changes.
To discover what’s up with the desktop googlebot agents and stay up to date and better yet know if your content is going to get stuck in the indexation system change up - become a subscriber and know what very few actually know!