Best Scraping Data From Google Search (Python) article in 2023

Scraping Google Search SERPs (search engine result pages) is as straightforward or as complicated as the tools we use. For this tutorial, we’ll be using Scrapy, a web scraping framework designed for Python. Python and Scrapy combine to create a powerful duo we can use to scrape almost any website.

Scrapy has many useful built-in features that will make scraping Google a walk in the park without compromising any data we would like to scrape.

For example, with Scrapy all it takes is a single command to format our data as CSV or JSON files – a process we would have to code ourselves otherwise.

Before jumping into the code itself, let’s first explore a few reasons a Google scraper can be useful.

Why Scrape Google?

There’s no dispute, that Google is the king of search engines. That means there’s a lot of data available in its search results for a savvy scraper to take advantage of.

Here are a few applications for a Google scraper:

Collecting Customer Feedback Data to Inform Your Marketing

In the modern shopping experience, it is common for consumers to look for product reviews before deciding on a purchase.

With this in mind, a powerful application for a Google SERPs scraper is to collect reviews and customer feedback from your competitor’s products to understand what’s working and what’s not working for them.

It can be to improve your product, find a way to differentiate yourself from the competition or know which features or experiences to highlight in your marketing.

Keep this in mind because we’ll be building our scraper around this issue exactly.

Inform Your SEO and PPC Strategy

According to Oberlo, “Google has 92.18 percent of the market share as of July 2019” and it “has been visited 62.19 billion times this year”. With that many eyes on the SERPs, getting your business to the top of these pages for relevant keywords means a lot of money.

Web scraping is primarily an info-gathering tool. We can use it to know our positions in Google better and benchmark ourselves against the competition.

If we look at our positions and compare ourselves to the top pages, we can generate a strategy to outrank them.

The same goes for PPC campaigns. Because ads appear at the top of every SERP – and sometimes at the bottom – we tell our scraper to bring the name, description, and link to all the ads appearing at the top of the search results for our targeted keywords.

This research will help us find un-targeted keywords, understand our competitor’s strategies, and evaluate the copy of their ads to differentiate ours.

Generate Content Ideas

Google also has many additional features in their SERPs like related searches, “people also ask” boxes, and more. Scraping hundreds of keywords allows you to gather all this information in a couple of hours and organize it in an easy-to-analyze database.

These are just use cases. Depending on the type of data and the ultimate goal you have, you can use a Google scraper for many different reasons.

How to Build a Google Web Scraper Without Getting Blocked

As we stated earlier, for this example we’ll build our Google web scraper to collect competitors’ reviews. So, let’s imagine we’re a new startup building project management software, and we want to understand the state of the industry.

Let’s start from there:

1. Choose Your Target Keywords

Now that we know our main goal, it’s time to pick the keywords we want to scrape to support it.

To pick your target keywords, think of the terms consumers could be searching to find your offering and identify your competitors. In this example, we’ll target four keywords:

• “Asana reviews”
• “ClickUp reviews”
• “best project management software”
• “best project management software for small teams”

We could add many more keywords to this list, but for this scraper tutorial, they’ll be more than enough.

Also, notice that the first two queries are related to direct competitors, while the last two will help us identify other competitors and get an initial knowledge of the state of the industry.

2. Setup Your Development Environment

The next step is to get our machine ready to develop our Google Search scraper. For this, we’ll need a few things:

• Python version 3 or later
• Pip – to install Scrapy and other packages, we might need
• Scraper API

Your machine may have a pre-installed Python version. Enter python -v into your command prompt to see if that’s the case.

If you need to install everything from scratch.

We’ll be using the same setup, so get that done and come back.

Note: something to keep in mind is that the team behind Scrapy recommends installing Scrapy in a virtual environment (VE) instead of globally on your PC or laptop. If you’re unfamiliar, the above Python and Scrapy tutorial shows you how to create the VE and install all dependencies.

In this tutorial, we’re also going to be using Proxy-cheap to avoid any IP bans or repercussions. Google Search doesn’t want us to scrape their SERPs – especially for free. As such, they have implemented advanced anti-scraping techniques that’ll quickly identify any bots trying to extract data automatically.

To get around this, Proxy-cheap is a complex system that utilizes third-party proxies, machine learning, huge browser farms, and years of statistical data to ensure that our scraper won’t get blocked from any site by rotating our IP address for every request, setting wait times between requests and handling CAPTCHAs.

In other words, by just adding a few lines of code, Proxy-cheap will supercharge our scraper, saving us headaches and hours of work.

All we need for this tutorial is to get our API Key from Proxy-cheap. To get it, just create a Proxy-cheap account to redeem API requests.

3. Create Your Project’s Folder

After installing Scrapy in your VE, enter this snippet into your terminal to create the necessary folders:

123	scrapy startproject google_scrapercd google_scraperscrapy genspider google api.Proxy-cheap.io

COPY

Scrapy will first create a new project folder called “google-scraper,” which also happens to be the project’s name. Next, go into this folder and run the “genspider” command to create a web scraper named “google”.

We now have many configuration files, a “spiders” folder containing our scraper, and a Python modules folder containing package files.

4. Import All Necessary Dependencies to Your google.py File

The next step is to build a few components that will make our script as efficient as possible. To do so, we’ll need to make our dependencies available to our scraper by adding them at the top of our file:

importscrapyfromurllib.parse importurlencodefromurllib.parse importurlparseimportjsonfromdatetime importdatetimeAPI_KEY ='YOUR_API_KEY'

COPY

With these dependencies in place, we can use them to build requests and handle JSON files. This last detail is important because we’ll use Proxy-cheap autoparse functionality.

After sending the HTTP request, it will return the data in JSON format, simplifying the process and making it so that we don’t have to write and maintain our parser.

5. Construct the Google Search Query

Google employs a standard and query-able URL structure. You just need to know the URL parameters for the data you need and you can generate a URL to query Google Search with.

That said, the following make up the URL structure for all Google search queries:

http://www.google.com/search

COPY

Several standard parameters make up Google Search queries:

• q represents the search keyword parameter. http://www.google.com/search?q=tshirt, for example, will look for results containing the keyword “tshirt.”
• The offset point is specified by the start parameter. http://www.google.com/search?q=tshirt&start=100 is an example.
• hl is the language parameter. http://www.google.com/search?q=tshirt&hl=en is a good example.
• The as_sitesearch argument allows you to search for a domain (or website). http://www.google.com/search?q=tshirt&as sitesearch=amazon.com is one example.
• The num parameter specifies the number of results per page (maximum is 100). http://www.google.com/search?q=tshirt&num=50 is an example.
• The safe parameter generates only “safe” results. http://www.google.com/search?q=tshirt&safe=active is a good example.

Note: Moz’s comprehensive list of Google search parameters is incredibly useful in building a query-able URL. Bookmark it for more complex scraping projects in the future.

Alright, let’s define a method to construct our Google URL using this information:

defcreate_google_url(query, site=''):    google_dict ={'q': query, 'num': 100, }    ifsite:        web =urlparse(site).netloc        google_dict['as_sitesearch'] =web        return'http://www.google.com/search?'+urlencode(google_dict)    return'http://www.google.com/search?'+urlencode(google_dict)

COPY

In our method, we’re setting ‘q’ as a query because we’ll specify our actual keywords later in the script to make it easier to make changes to our scraper.

6. Define the Scraper API Method

To use Proxy-cheap, all we need to do is send our request through the Proxy-cheap server by appending our query URL to the proxy URL provided by Proxy-cheap using payload and urlencode. The code looks like this:

defget_url(url):payload ={'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}proxy_url ='http://api.Proxy-cheap.io/?'+urlencode(payload)returnproxy_url

COPY

Now that we have defined the logic our scraper will use to construct our target URLs, it’s time to build the main spider.

7. Write the Spider Class

In Scrapy we can create different classes, called spiders, to scrape specific pages or groups of sites. Thanks to this function, we can build different spiders inside the same project, making it much easier to scale and maintain.

classGoogleSpider(scrapy.Spider):    name ='google'    allowed_domains =['api.proxy-cheap.io']    custom_settings ={        'ROBOTSTXT_OBEY': False,        'LOG_LEVEL': 'INFO',        'CONCURRENT_REQUESTS_PER_DOMAIN': 10,        'RETRY_TIMES': 5}

COPY

We need to give our spider a name, as this is how Scrapy will determine which script you want to run. The name you choose should be specific to what you’re trying to scrape, as projects with multiple spiders can get confusing if they aren’t named.

Because our URLs will start with Proxy-cheap domain, we’ll also need to add “api.scraper.com” to allowed_domains. Proxy-cheap will change the IP address and headers between every retry before returning a failed message (which doesn’t count against our total available API credits).

We also want to tell our scraper to ignore the directive in the robots.txt file. This is because by default Scrapy won’t scrape any site that has a contradictory directive inside the said file.

As you can see in the custom_settings code above, we’re telling Proxy-cheap to send 10 concurrent requests and to retry 5 times after any failed response.

8. Send the Initial Request

It’s finally time to send our HTTP request. It is very simple to do this with the start_requests(self) method:

defstart_requests(self):    queries =['asana+reviews', 'clickup+reviews', 'best+project+management+software', 'best+project+management+software+for+small+teams']        url =create_google_url(query)        yieldscrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})

COPY

It will loop through a list of queries that will be passed to the create_google_url function as query URL keywords.

The query URL we created will then be sent to Google Search via the proxy connection we set up in the get_url function, utilizing Scrapy’s yield. The result will then be given to the parse function to be processed (it should be in JSON format). The {‘pos’: 0} key-value pair is also added to the meta parameter, which is used to count the number of pages scraped.

Note: when typing keywords, remember that every word in a keyword is separated by a + sign, rather than a space.

9. Write the Parse Function

Thanks to Proxy-cheap auto-parsing functionality, our scraper should be returning a JSON file as a response to our request. Make sure it is by enabling the parameter ‘auto parse’: ‘true’ in the get_url function.

Next, we’ll load the complete JSON response and cycle through each result, taking the data and combining it into a new item that we can utilize later.

defparse(self, response):    di =json.loads(response.text)    pos =response.meta['pos']    dt =datetime.now().strftime('%Y-%m-%d %H:%M:%S')    forresult indi['organic_results']:        title =result['title']        snippet =result['snippet']        link =result['link']        item ={'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}        pos +=1        yielditem    next_page =di['pagination']['nextPageUrl']    ifnext_page:        yieldscrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})

COPY

This procedure checks to see whether another page of results is available. The request is invoked again if an additional page is present, repeating until there are no additional pages.

10. Run the Spider

Congratulations, we built our first Google scraper! Remember, our code can always be changed to add functionality we discover is missing, but for now we have a functional scraper. If you’ve been following along, your google.py file should look like this by now:

importscrapyfromurllib.parse importurlencodefromurllib.parse importurlparseimportjsonfromdatetime importdatetimeAPI_KEY ='YOUR_API_KEY'defget_url(url):    payload ={'api_key': API_KEY, 'url': url, 'autoparse': 'true', 'country_code': 'us'}    proxy_url ='http://api.proxy-cheap.io/?'+urlencode(payload)    returnproxy_urldefcreate_google_url(query, site=''):    google_dict ={'q': query, 'num': 100, }    ifsite:        web =urlparse(site).netloc        google_dict['as_sitesearch'] =web        return'http://www.google.com/search?'+urlencode(google_dict)    return'http://www.google.com/search?'+urlencode(google_dict)classGoogleSpider(scrapy.Spider):    name ='google'    allowed_domains =['api.Proxy-cheap.io']    custom_settings ={        'ROBOTSTXT_OBEY': False,         'LOG_LEVEL': 'INFO',        'CONCURRENT_REQUESTS_PER_DOMAIN': 10,        'RETRY_TIMES': 5}defstart_requests(self):    queries =['asana+reviews', 'clickup+reviews', 'best+project+management+software', 'best+project+management+software+for+small+teams']    forquery inqueries:        url =create_google_url(query)        yieldscrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})    defparse(self, response):    di =json.loads(response.text)    pos =response.meta['pos']    dt =datetime.now().strftime('%Y-%m-%d %H:%M:%S')    forresult indi['organic_results']:        title =result['title']        snippet =result['snippet']        link =result['link']        item ={'title': title, 'snippet': snippet, 'link': link, 'position': pos, 'date': dt}        pos +=1        yielditem    next_page =di['pagination']['nextPageUrl']    ifnext_page:        yieldscrapy.Request(get_url(next_page), callback=self.parse, meta={'pos': pos})

COPY

Note: If you want to scrape Google Search from different countries (let’s say Italy), all you need to do is change the code inside the country_code parameter in the get_url function. 

To run our scraper, navigate to the project’s folder inside the terminal and use the following command:

scrapy crawl google -o serps.csv

COPY

Now our spider will run and store all scraped data in a new CSV file named “serps.” This feature is a big time saver and one more reason to use Scrapy for web scraping Google Search.

The stored data can then be analyzed and used to provide insight for tools, marketing, and more.

Happy scraping!