How do I connect with authenticated proxy and load extension with launcher manager

[samuel-108]

Rod Version: v0.116.2

The code to demonstrate your question

	extPath, _ := filepath.Abs("assets/extensions/gc")

	l := launcher.New().Set(
		"load-extension", extPath,
	)

	l = l.Set(flags.ProxyServer, i.config.ProxyHost+":"+i.config.ProxyPort)

	// This assumes that the timezone is set in the config
	u, err := l.Env(append(os.Environ(), "TZ="+i.config.TimeZone)...).Headless(i.config.HeadlessMode).Launch()
	if err != nil {
		return err
	}

	i.browser = rod.New().ControlURL(u)

	// https://go-rod.github.io/#/network/README
	//i.browser.MustIgnoreCertErrors(true)

	err = i.browser.Connect()
	if err != nil {
		return err
	}

	i.page, err = stealth.Page(i.browser)
	if err != nil {
		return err
	}

	err = i.setFixedWindowSize()
	if err != nil {
		return err
	}

	// Authenticate the proxy
	go func() {
		defer utils.RecoverFromPanic("proxy authentication")

		err := i.browser.HandleAuth(i.config.ProxyUser, i.config.ProxyPass)()
		if err != nil {
			log.Printf("Error handling proxy authentication: %v", err)
		}
	}()

	// Load a page so that proxy auth can be done before hijacking requests
	err = i.page.Navigate("https://ip.smartproxy.com/json")
	if err != nil {
		return err
	}

	i.page.MustWaitLoad()

	router := i.page.HijackRequests()

	router.MustAdd("*", func(ctx *rod.Hijack) {
		if ctx.Request.Type() == proto.NetworkResourceTypeImage || ctx.Request.Type() == proto.NetworkResourceTypeMedia {
			ctx.Response.Fail(proto.NetworkErrorReasonBlockedByClient)
			return
		}
		ctx.ContinueRequest(&proto.FetchContinueRequest{})
	})

	// Run the router in a goroutine and recover from any panics
	go func() {
		defer utils.RecoverFromPanic("router")

		router.Run()
	}()

	return err

The above code shows how it is being done without using launcher manager.

What you got

No error. However I do not see options on how to set the launcher flags. Specially in regards to proxy, proxy auth and loading extension.

What you expect to see

Sth like:

u := launcher.MustResolveURL("http://localhost:7317")

launcher.ControlURL(u).Set(
		"load-extension", extPath,
).Set(
		flags.ProxyServer, i.config.ProxyHost+":"+i.config.ProxyPort
).Set(....)

What have you tried to solve the question

....

[github-actions]

Please fix the golang code in your markdown:

@@ golang markdown block 1 @@
8:64: missing ',' before newline in argument list
9:7: expected operand, found '...'
10:2: expected ')', found 'EOF'
10:2: expected ';', found 'EOF'
10:2: expected ';', found 'EOF'
10:2: expected '}', found 'EOF'
10:2: missing ',' in argument list

_{generated by check-issue}

[ysmood]

Please check the example file:

rod/examples_test.go

Lines 401 to 403 in 0096e53

	// Adding authentication to the proxy, for the next auth request.
	// We use CLI tool "mitmproxy --proxyauth user:pass" as an example.
	go browser.MustHandleAuth("user", "pass")()

[ysmood]

To load extension with launcher manager, you need to build your own image with the extension file.

[samuel-108]

Thank you @ysmood

But in the above example we can see this:

// Shows how we can further customize the browser with the launcher library.
// Usually you use launcher lib to set the browser's command line flags (switches).
// Doc for flags: https://peter.sh/experiments/chromium-command-line-switches
func Example_customize_browser_launch() {
	url := launcher.New().
		Proxy("127.0.0.1:8080").     // set flag "--proxy-server=127.0.0.1:8080"
		Delete("use-mock-keychain"). // delete flag "--use-mock-keychain"
		MustLaunch()

	browser := rod.New().ControlURL(url).MustConnect()
	defer browser.MustClose()

	// So that we don't have to self issue certs for MITM
	browser.MustIgnoreCertErrors(true)

	// Adding authentication to the proxy, for the next auth request.
	// We use CLI tool "mitmproxy --proxyauth user:pass" as an example.
	go browser.MustHandleAuth("user", "pass")()

	// mitmproxy needs a cert config to support https. We use http here instead,
	// for example
	fmt.Println(browser.MustPage("https://mdn.dev/").MustElement("title").MustText())
}

Here the code shows us how we can add proxy to a local browser instance, but does not show one with a remote instance.

u := launcher.MustResolveURL("http://localhost:7317")

This returns a string. And this is to be used instead of the following to instantiate the browser when we are dealing with remote instances.

url := launcher.New().Proxy("127.0.0.1:8080").Delete("use-mock-keychain").MustLaunch()
browser := rod.New().ControlURL(url).MustConnect()

I want to be able to do (sth similar to) this

u := launcher.MustResolveURL("http://localhost:7317").Proxy("apple.com:11223")
browser := rod.New().ControlURL(url).MustConnect()

[samuel-108]

To load extension with launcher manager, you need to build your own image with the extension file.

When you say this, do you mean that the extension should be present within the docker container, and we give it's path when creating a new browser instance, or some hacky chrome instance with the ext preloaded?

[ysmood]

#1105 (comment)

It's a limit of cdp protocol, we can't do much about it. You have to build a custom image with the proxy deployed to where the chromium actually runs.

#1105 (comment)

Yes. It's a similar problem.

[samuel-108]

@ysmood

It's a limit of cdp protocol, we can't do much about it. You have to build a custom image with the proxy deployed to where the chromium actually runs.

Let's say I have proxy deployed so that it can be accessed remotely as well (i.e. via internet).

Does that mean I am able to connect with proxies on such fully remote systems? How do I do that?

[ysmood]

Yes, as long as the remote chromium can access it.

[samuel-108]

@ysmood

Thank you.

Can you also share an example on how I can launch it?

[samuel-108]

Any update to this?

[samuel-108]

For anyone who comes across this issue in the future:

func runBrowserInstance(instanceNumber int) {
	instanceNumberStr := strconv.Itoa(instanceNumber)
	fmt.Println("Using screen number #" + instanceNumberStr)

	l := launcher.MustNewManaged("http://....:port") // Here you set the ip:port for the docker instance (make sure you can access it remotely)

	l.Set("disable-gpu").Delete("disable-gpu")

	// Launch with headful mode
        // Headful mode is required if you have to install extensions

        // Note: If you want to run multiple instances connected to same manager, then make sure you have each of their instance number different
	l.Headless(false).XVFB("--server-num="+instanceNumberStr, "--server-args=-screen 0 1920x1080x16")

	browser := rod.New().Client(l.MustClient()).MustConnect()

	// You may want to start a server to watch the screenshots of the remote browser.
        // launcher.Open(browser.ServeMonitor("")) => Note: this returns a url from where you can stream screenshot (polling)

        // Your code goes here
        // here is some random ass useless code
 
	err := rod.Try(func() {
		page := browser.MustPage("https://ip.smartproxy.com")

		fmt.Println(
			page.MustEval("() => document.title"),
		)

		page.MustEval("() => window.scrollTo(0, document.body.scrollHeight / 2)")

		time.Sleep(1 * time.Minute)

		fmt.Println("1 minute passed for browser instance #" + instanceNumberStr)

		for i := 0; i < 10; i++ {
			page.MustNavigate("https://www.youtube.com/watch?v=FVwV5BxJ8M4")
			page.MustWaitLoad()
			time.Sleep(30 * time.Second)
			fmt.Println("30 seconds passed for browser instance #"+instanceNumberStr+" with itr: ", i+1)
		}
	})

	if err != nil {
		fmt.Println("Error: ", err)
	}

	browser.MustClose()
}

[samuel-108]

@ysmood

Thank you for your help. You can close the issue now.

[samuel-108]

For loading extensions, proxies and such:

func (i *Instance) Init() error {
	l := launcher.MustNewManaged("http://localhost:7317")
	//extPath, _ := filepath.Abs("/assets/extensions/gc")

	// Add domains to bypass the proxy
	// The domains should be separated by commas
	// Use '<local>' to bypass proxy for localhost and related domains
	bypassList := ProxyBypassList

	// Since this needs to be different for each instance, we will use a random number
	serverNumStr := strconv.Itoa(rand.Intn(1000) + 1)

	l = l.
		Set("load-extension", "/assets/extensions/gc").
		Headless(false).
		XVFB("--server-num="+serverNumStr, "--server-args=-screen 0 1920x1080x16").
		Set(flags.ProxyServer, i.config.ProxyHost+":"+i.config.ProxyPort).
		Set("proxy-bypass-list", bypassList)

[samuel-108]

Note: Make sure that extension path is valid inside the docker container.