This is Crawl Or No Crawl. Welcome to its new home where the daily reports will be published and paid subscribers will get up to date, detailed data on what is happening in “Index Land”.
My background as one of the first “official” SEO Testers in 2015 has brought me here. I know how to run html tests.
There are those in our community that say testing is logically unreliable because our test keywords are so “easy” to rank for that it’s not subject to the same algorithm as “DUI Attorney Los Angeles”.
I call BS on that. There is ZERO data that indicates there is more than one (ever-changing) algorithm. Like gravity, the sorting process is the same whether we’re looking for a high value keyword that garners 10K a month plus for agencies or a word that Google doesn’t know. Otherwise - we might get blank results pages back.
This testing of the Indexation System is based on using test keywords that are unique. One keyword is designed to confirm when the content is being served based on the initial crawl (plain html crawl) and a second test keyword is wrapped in javascript to confirm if the rendering portion of the indexation (or 2nd pass) is working.
The Indexation System is comprised of four (4) subsystems - crawling, indexing, ranking and serving. Previous to 2018, indexation was to all extents and purpose - instantaneous. Now it is not which has allowed for daylight to show between the systems and this research project which started in August 2021 is about creating a contemporaneous history of what actually was happening on a daily basis.
Until now, the reports were public and provided at no cost. You can find the Youtube Channel here - https://www.youtube.com/@crawlornocrawl. Moving forward this is the only place that I will be sharing the details and challenges that we are facing now that Google is having issues with the sheer volume of all our content.
What this research has shown me is that indexation, search console, updates and server logs are NOT unrelated to each other. The initial purpose was to provide data to the SEO community to underscore the fact that Google is a machine. There are times when no matter how “good” an SEO you are, the doors are shut.
If having source data, not relying on tools or opinions, matters to you - this is the place for you.
Welcome.