1. Puppeteer page object’s authenticate() method
There are a few options for using Headless Chrome to authenticate a proxy and using it in Puppeteer.
Puppeteer with a headless browser is found to be a very easy, yet effective tool for developers to automate numerous online tasks such as form filling, data scraping, and web page snapshots.
Puppeteer is nearly unbeatable when combined with a proxy; nevertheless, there may be some challenges when attempting to configure Puppetee appropriately to authorize a proxy that requires a login and password.
Using the Puppeteer page object’s authenticate() method:
Puppeteer has offered a built-in solution for authorizing a proxy with the authenticate() function for over three years. This is the most frequent way to accomplish it with vanilla Puppeteer nowadays.
const puppeteer = require('puppeteer'); const proxy = 'http://proxy.host.address:9000'; const username = 'your_proxy_username'; const password = 'your_proxy_password'; (async () => { // Pass proxy URL into the --proxy-server arg const browser = await puppeteer.launch({
args: [`--proxy-server=${proxy}`], }); const page = await browser.newPage() // Authenticate your proxy with the username and password defined above await page.authenticate({ username, password }); await page.goto('https://www.google.com'); await browser.close(); })();2. Making use of the NPM package proxy-chain:
The proxy-chain NPM package is an open source tool created and maintained by Apify that takes a different route by allowing you to simply “anonymize” an authorized proxy.
This is accomplished by sending your proxy address along with authentication information to the proxyChain.anonymizeProxy function and then getting its value as the —proxy-server parameter when running Puppeteer.
const puppeteer = require('puppeteer'); const proxyChain = require('proxy-chain'); const proxy = 'http://your.proxy.address:9000'; const username = 'your_proxy_username'; const password = 'your_proxy_password'; (async () => { const originalUrl = `http://${username}:${password}@${proxy}`; // Return anonymized version of original URL - looks like http://127.0.0.1:16383 const newUrl = await proxyChain.anonymizeProxy(originalUrl); const browser = await puppeteer.launch({
args: [`--proxy-server=${newProxyUrl}`], }); const page = await browser.newPage(); await page.goto('https://www.google.com'); await browser.close(); // Close any pending connections await proxyChain.closeAnonymizedProxy(newProxyUrl, true); })();When utilizing this approach, it is vital to remember that after exiting the browser, you should use the closeAnonymizedProxy() function to forcefully close any pending connections.
This software handles both simple HTTP proxy forwarding and HTTP CONNECT tunneling to enable protocols like HTTPS and FTP. It also offers many more functionalities, thus it is interesting to explore for other applications.
TL;DR
For developers seeking effective online task automation, using Puppeteer with proxies is an unbeatable option. Our article provides comprehensive guidance on how to configure Puppeteer to authorize a proxy with login and password. A common method is using Puppeteer’s page object’s authenticate() method. Here, your proxy URL is passed into the –proxy-server argument, the browser is launched and your page is authenticated via the username and password.
Another approach involves leveraging the NPM package proxy-chain for anonymizing an authorized proxy. Authentication details are sent to the proxyChain.anonymizeProxy function, which returns a value for use as the –proxy-server parameter when running Puppeteer. Please note the necessity of using the closeAnonymizedProxy() function to forcefully close any pending connections on browser exit. The proxy-chain software additionally supports simple HTTP proxy forwarding and HTTP CONNECT tunneling, thus enabling protocols such as HTTPS and FTP. Explore these functionalities for robust proxy solutions.
