Howtouseproxy Selenium cover
How to

How to set up a proxy in Selenium

Welcome to the realm where the strength of proxies combines with Selenium web automation! We’ll look at how proxies can improve your web automation applications in this post. Proxy use unlocks improved security, privacy, and the capacity to mimic actual user behavior.

Prepare to advance your Selenium knowledge and transform your web scraping, testing, and automation projects. Let’s get started and use Selenium’s proxies to their full capacity!

Prerequisites

You must first install Python 3. You may verify if you have it by using the command below to see if most platforms come with it by default:

python --version

It ought to provide a result similar to this:

Python 3.11.2

If it fails or the printed version is Python 2.x, you should download Python 3.x and set it up by following the installation instructions.

Start a fresh Python project after that, and add the Selenium Python binding package:

pip install selenium

Although Selenium can manage a variety of browsers, we’ll choose Google Chrome as it’s the most popular. Ensure that you are using the most recent Chrome installation.

Add WebDriver Manager to your project at this time:

pip install webdriver-manager

That Python module makes driver administration simpler. In more detail, it prevents you from always downloading the newest ChromeDriver version.

It’s time to start using the Python script you installed to command Chrome using Selenium. This code snippet imports the necessary tools, creates a Chrome WebDriver object, and then employs it to access a sample target page.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://example.com/')

Perfect! Just now, you learned how to install Selenium in Python. Let’s examine its proxy usage!

To Set up A Proxy In Selenium

Many unauthenticated free proxy servers eliminate the need for a username and password.
With the help of the subsequent procedures, a Selenium unauthenticated proxy server can be set up:

  • Import Selenium WebDriver from the package
  • Define the proxy server (IP:PORT)
  • Set ChromeOptions()
  • Add the proxy server argument to the options
  • Add the options to the Chrome() instance
from selenium import webdriver
PROXY = "12.402.350.210:8080"
chrome_options = WebDriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
chrome = webdriver.Chrome(chrome_options=chrome_options)
chrome.get("https://www.google.com")

To run tests that use the proxy server, use this instance of Chrome WebDriver. The following line of code, for example, verifies that a search field displays the user’s current city as the default location.

def testUserLocationZurich(self):
self.chrome.get(self.url)
search = self.chrome.find_element_by_id('user-city')
self.assertIn('Paris', search.text)

Create a method that accepts the proxy IP address as an argument so that you can test a website from various places by making this code reusable for various tests.

selenium.webdriver.common.proxy: Proxy offers details about the kind of proxy and the relevant proxy settings.

Testing can be done on a server that is not authenticated. However, they can adhere to the steps below if they want to use an authorized server.

Python Selenium Proxy Authentication: Username & Password

Some proxy servers utilize authentication to only allow users with legitimate credentials access. Typically, premium proxies or commercial solutions are like that.

With Selenium, the syntax to include a username and password in an authenticated proxy URL is as follows:

<PROXY_PROTOCOL>://<USERNAME>:<PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>

It should be noted that because the Chrome driver by default ignores the username and password, entering such a URL in —proxyserver won’t work. A third-party plugin, like Selenium Wire, can help in this situation.

It enhances Selenium by allowing you to see and alter the requests made by the browser. Run the following command to install it:

pip install selenium-wire

Then, handle proxy authentication using Selenium Wire, as seen in the sample below:

from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

# Selenium Wire configuration to use a proxy
proxy_username = 'YourUsername'
proxy_password = 'YourPassword'
seleniumwire_options = {
    'proxy': {
        'http': f'http://{proxy_username}:{proxy_password}@165.145.222.150:7890',
        'verify_ssl': False,
    },
}

driver = webdriver.Chrome(
    seleniumwire_options=seleniumwire_options
)
driver.get('http://httpbin.org/ip')
print(driver.find_element(By.TAG_NAME, 'body').text) # { "origin": "165.145.222.150" }

Note: A 407 Proxy Authentication Required issue could appear when this code is used. When the credentials are incorrect, a proxy server responds with that HTTP status, therefore make sure the proxy URL contains a valid username and password.

To Set up A Rotating Proxy in Selenium

The server may flag your IP as suspicious if your script performs multiple queries in a short period of time. However, a rotating proxy solution, which switches proxies after a predetermined time or number of requests, will prevent it from happening.

The server cannot ban you since your end IP will continually change, making you appear to be a different user each time. That is the strength of rotating proxies!

It’s time to learn how to use Selenium-wire to build a proxy rotator in Selenium.

You must first locate a pool of proxies. We’ll utilize some free proxies in this case.

Organize them as follows in an array:

PROXIES = [
    'http://19.151.94.248:88',
    'http://149.169.197.151:80',
    # ...
    'http://212.78.125.242:97'
]

Then, use random.choice() to extract a random proxy and use it to initialize a new driver instance:

from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
import random

# the list of proxy to rotate on 
PROXIES = [
    'http://19.151.94.248:88',
    'http://149.169.197.151:80',
    # ...
    'http://212.78.125.242:98'
]

# randomly extract a proxy
random_proxy = random.choice(PROXIES)

# set the proxy in Selenium Wire
seleniumwire_options = {
    'proxy': {
        'http': f'{random_proxy}',
        'https': f'{random_proxy}',
        'verify_ssl': False,
    },
}

# create a ChromeDriver instance
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    seleniumwire_options=seleniumwire_options
)

driver.visit('https://example.com/')

# scraping logic...

driver.quit()

# visit other pages...

Every time you wish to go to a new page, follow this logic.

Good work! You created a rotating Selenium proxy. Learn more about rotating proxies in Python in our comprehensive guide.

However, as free proxies are prone to errors, the majority of requests will fail. You should therefore provide retry logic with random timeouts.

But there are other problems as well. A target that employs anti-bot technology might be used to test the IP rotator logic:

driver.visit('https://www.tesla.com/')

The output:

The target server responded with a 403 Unauthorized error after identifying the rotating proxy Selenium request as a bot.

Free proxies actually cause you to get blocked the majority of the time. We utilized them to demonstrate the fundamentals; but, a real-world script should never rely on them.

The answer? A Premium proxy!

Best Practices for Using a Proxy:

Follow these best practices to create a seamless proxy experience:

Test Your Proxy Settings:

  • Before executing complex automation activities, always test your proxy settings. This makes it easier to spot any problems or restrictions early on.

Monitor Proxy Performance:

  • Keep an eye on your proxy’s metrics and response times. If there are any lags or connectivity problems, think about using an alternative proxy server or service.

Maintain Compliance:

  • When utilizing proxies, adhere to all applicable laws and website terms of service. Avoid engaging in illegal acts like aggressive scraping or unauthorized access.

Protect Your Credentials:

  • Store your proxy credentials securely to protect them. When disseminating this knowledge, exercise caution and avoid posting private information in scripts or public repositories.

Stay Updated:

  • Websites and proxy providers frequently update their software. To preserve compatibility and dependability, keep abreast of any changes and modify your scripts accordingly.

Conclusion

Congratulations on discovering Selenium‘s amazing proxy world! You’ve found the secret to improved privacy, security, and the capacity to rep√©licate actual user behavior by utilizing proxies. Your web automation initiatives will achieve new levels of efficiency and effectiveness with proxies as your ally.

Revolutionize your Selenium journey right away by embracing the endless possibilities!

How to use proxy with Netflix

How to use proxy with Facebook

How to use proxy with Wget

How to use a proxy with Telegram

How to use a proxy with WhatsApp

Leave feedback about this

  • Quality
  • Price
  • Service

PROS

+
Add Field

CONS

+
Add Field