SEO 101: What to do when Google Search Search Console Can’t Crawl Your Site Because of Robots.txt
The first rule of Search Engine Optimization is simple: Make sure that Google – and the other major search engines – can properly access your site! If they can’t “crawl” your site, it doesn’t take a rocket scientist to figure out that your search rankings aren’t going to be very good. In fact, you won’t show up at all. So if Google Search Console is telling you that your robots.txt file is blocking them from crawling the site… you’ve got a problem. The good news is that it’s not a difficult fix, and that’s what we’re going to walk you through here today.
First, some background. What does it mean to “crawl” your site in the first place? Search engines like Google use automated scripts to browse websites across the world wide web. These scripts are referred to as “spiders” and the process of reviewing your website content is known as crawling.
Simply put, crawling is the process that search engines use to gather information about your website. So, if your website won’t allow the search engines to crawl them, your site is basically invisible.
Which is not a good place to be!
If your robots.txt file is blocking Google from crawling the site, it’s usually because you’ve launched the site but forgot to update the robots.txt file beforehand. When a site is in development, it’s standard procedure to block search engines from finding it, because you don’t want website visitors stumbling across your half-built website.
But if you or your developer fails to update the robots.txt file ahead of time, it won’t be accessible to the search engines and you’ll get the following error message in Google Search Console:

So let’s talk about how to fix it.
The first step is to update your robots.txt file. You can generally access this file through a website plugin or the source code itself. If you’re working with a web developer, he or she can do this for you very easily.
But simply fixing the file doesn’t solve the problem. Google Search Console won’t know that you’ve changed your settings unless you take action. Here’s what to do:
Step One: Open up Google Search Console and select “Go to the Old Version” on the left-hand menu.

Step Three: Click “Submit”

Step Two: Choose “Crawl” and then “Robots.txt tester”

Step Four: In the dialogue box, choose “Ask Google to Update”

Step Five: Return to New Google Search Console, Inspect Home Page, and “Request Indexing”

It will take a few moments to process, and then your site will be placed into their indexing queue.
Check back a few moments later and you should be good to go! Once you’ve gone through this process, Google (and other search engines) will be able to access your site and include your website in their search rankings. If you have any further questions, we’d be happy to help. Simply click here to get in touch with us! And if you’d like to learn more about the technical aspects of SEO, click here to subscribe to our YouTube Channel.
See you next time!
Related Articles You May Like
Am I Getting F#*$ed by My Digital Marketing Company?!
Am I Getting F#*$ed by My Digital Marketing Company?! How to Spot the Fakers and Get Real ResultsIn 2020, running a business of any size means working online. Whether you’ve been a business owner for a month or two decades, if you don’t have a presence on the...
Here’s Why You Shouldn’t Advertise on Yelp
Here’s Why You Shouldn’t Advertise on Yelp, HomeAdvisor, or Angie’s List (And What You Should Do Instead)If you’re a small business owner, you have a difficult decision to make. You know that you need to advertise your company locally, but you probably have a limited...
Cloudways Server WordPress Hosting Comparison
Which Cloudways WordPress Server Should You Use for Your Website? Table of Contents What is Cloudways? Quick Disclaimer About CTM and Cloudways 5 Cut Throat Reasons to Use Cloudways Cloudways Speed Test Results Chart Google Cloud Engine Amazon Web Services Vultr...
Get Cut Throat
Your business needs to run ads online to grow. But PPC is a math problem that needs diligence and consistent optimization to run successfully. Get personalized PPC management services from a team with a real track record for success. Call or email us today to find out if our PPC services are right for you.
Location
Upper Loop Way, Columbia SC 29212
(803) 470-6557
Hello@CutThroatMarketing.com