Last active
January 11, 2023 17:31
-
-
Save jancurn/ecf3f04aea3c245498e06a913ce54115 to your computer and use it in GitHub Desktop.
Puppeteer's page.authenticate() does not work for proxy authorization!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
const puppeteer = require('puppeteer'); | |
(async() => { | |
const proxyUrl = 'http://proxy.example.com:8000'; | |
const username = 'bob'; | |
const password = 'password123'; | |
const browser = await puppeteer.launch({ | |
args: [`--proxy-server=${proxyUrl}`], | |
headless: false, | |
}); | |
const page = await browser.newPage(); | |
await page.authenticate({ username, password }); | |
await page.goto('https://www.example.com'); | |
await browser.close(); | |
})(); |
Running Chromium or any headless browser on Lambda is always pain. I have not done it in a long time so can't help unfortunately. It's one of the reasons we created Apify. Because running browsers on existing serverless platforms is a constant struggle.
But Apify platform is also built on aws lambda right?
Not at all :) Apify runs primarily on EC2 and Actors run in Docker.
Not at all :) Apify runs primarily on EC2 and Actors run in Docker.
Thanks a lot for the info :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@mnmkng I am running puppeteer + aws lambda with proxy servers. Check this code snippet. I also tried running crawlee in aws lambda but did not work for me please check this discord discussion on apify discord channel.