How to set up a proxy in Selenium

Testing can be done on a server that is not authenticated. However, they can adhere to the steps below if they want to use an authorized server.

Python Selenium Proxy Authentication: Username & Password

Some proxy servers utilize authentication to only allow users with legitimate credentials access. Typically, premium proxies or commercial solutions are like that.

With Selenium, the syntax to include a username and password in an authenticated proxy URL is as follows:

<PROXY_PROTOCOL>://<USERNAME>:<PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>

It should be noted that because the Chrome driver by default ignores the username and password, entering such a URL in —proxy–server won’t work. A third-party plugin, like Selenium Wire, can help in this situation.

It enhances Selenium by allowing you to see and alter the requests made by the browser. Run the following command to install it:

pip install selenium-wire

Then, handle proxy authentication using Selenium Wire, as seen in the sample below:

from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

# Selenium Wire configuration to use a proxy
proxy_username = 'YourUsername'
proxy_password = 'YourPassword'
seleniumwire_options = {
    'proxy': {
        'http': f'http://{proxy_username}:{proxy_password}@165.145.222.150:7890',
        'verify_ssl': False,
    },
}

driver = webdriver.Chrome(
    seleniumwire_options=seleniumwire_options
)
driver.get('http://httpbin.org/ip')
print(driver.find_element(By.TAG_NAME, 'body').text) # { "origin": "165.145.222.150" }

Note: A 407 Proxy Authentication Required issue could appear when this code is used. When the credentials are incorrect, a proxy server responds with that HTTP status, therefore make sure the proxy URL contains a valid username and password.

To Set up A Rotating Proxy in Selenium

The server may flag your IP as suspicious if your script performs multiple queries in a short period of time. However, a rotating proxy solution, which switches proxies after a predetermined time or number of requests, will prevent it from happening.

The server cannot ban you since your end IP will continually change, making you appear to be a different user each time. That is the strength of rotating proxies!

It’s time to learn how to use Selenium-wire to build a proxy rotator in Selenium.

You must first locate a pool of proxies. We’ll utilize some free proxies in this case.

Organize them as follows in an array:

PROXIES = [
    'http://19.151.94.248:88',
    'http://149.169.197.151:80',
    # ...
    'http://212.78.125.242:97'
]

Then, use random.choice() to extract a random proxy and use it to initialize a new driver instance:

from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
import random

# the list of proxy to rotate on 
PROXIES = [
    'http://19.151.94.248:88',
    'http://149.169.197.151:80',
    # ...
    'http://212.78.125.242:98'
]

# randomly extract a proxy
random_proxy = random.choice(PROXIES)

# set the proxy in Selenium Wire
seleniumwire_options = {
    'proxy': {
        'http': f'{random_proxy}',
        'https': f'{random_proxy}',
        'verify_ssl': False,
    },
}

# create a ChromeDriver instance
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    seleniumwire_options=seleniumwire_options
)

driver.visit('https://example.com/')

# scraping logic...

driver.quit()

# visit other pages...

Every time you wish to go to a new page, follow this logic.

Good work! You created a rotating Selenium proxy. Learn more about rotating proxies in Python in our comprehensive guide.

However, as free proxies are prone to errors, the majority of requests will fail. You should therefore provide retry logic with random timeouts.

But there are other problems as well. A target that employs anti-bot technology might be used to test the IP rotator logic:

driver.visit('https://www.tesla.com/')

The output:

The target server responded with a 403 Unauthorized error after identifying the rotating proxy Selenium request as a bot.

Free proxies actually cause you to get blocked the majority of the time. We utilized them to demonstrate the fundamentals; but, a real-world script should never rely on them.

The answer? A Premium proxy!

Best Practices for Using a Proxy:

Follow these best practices to create a seamless proxy experience:

Test Your Proxy Settings:

• Before executing complex automation activities, always test your proxy settings. This makes it easier to spot any problems or restrictions early on.

Monitor Proxy Performance:

• Keep an eye on your proxy’s metrics and response times. If there are any lags or connectivity problems, think about using an alternative proxy server or service.

Maintain Compliance:

• When utilizing proxies, adhere to all applicable laws and website terms of service. Avoid engaging in illegal acts like aggressive scraping or unauthorized access.

Protect Your Credentials:

• Store your proxy credentials securely to protect them. When disseminating this knowledge, exercise caution and avoid posting private information in scripts or public repositories.

Stay Updated:

• Websites and proxy providers frequently update their software. To preserve compatibility and dependability, keep abreast of any changes and modify your scripts accordingly.

Conclusion

Congratulations on discovering Selenium‘s amazing proxy world! You’ve found the secret to improved privacy, security, and the capacity to repélicate actual user behavior by utilizing proxies. Your web automation initiatives will achieve new levels of efficiency and effectiveness with proxies as your ally.

Revolutionize your Selenium journey right away by embracing the endless possibilities!

Amine Bel

I’m Amine, a 34-year-old mobile enthusiast with a passion for simplifying the world of proxy providers through unbiased reviews and user-friendly guides. My tech journey, spanning from dial-up internet to today’s lightning-fast mobile networks, fuels my dedication to demystifying the proxy world. Whether you prioritize privacy, seek marketing advantages, or are simply curious, my blog is your trusted source.