DISCUSSION: New parameters to https://www.pandora.com/api/v1/auth/login

[H0r53]

Greetings,
As mentioned in Issue#45 Pandora has somewhat recently partnered with WhiteOps to implement anti-bot / anti-automation technologies. This has effectively broken the REST API documentation. I'm starting this discussion to continue the conversation on how the implemented technologies work and explore potential workarounds. Details of the new API fields and how they are generated have already been explained in Issue#45 by @hacker1024 and @PromyLOPh. It seems that @jasonful, @GodStar88, and @FireController1847 may still be interested in this discussion as well. I've been researching WhiteOps and the Pandora implementation and I've found the login process to be roughly the following:

Login Process
——————————————————

1. GET pandora /account/sign-in
   -) Initial request and page load; store cookies

2. GET s.hoplon.pandora.com /static/pandora/4.80.9/pagespeed.js
   -) loads code to make requests to config.json and main.js

3. GET s.hoplon.pandora.com /sri/config.json
   -) Passes multiple values via query string, notably:
   —) dt: Detection Tag ID (WhiteOps related)
   —) psv: Version of WhiteOps bot detection to use (4.80.9)
   —) ci: Customer ID. Static value indicating the WhiteOps client (Pandora)
   —) spa: Single Page App indicator (WhiteOps config)
   —) pd: Product Key, always acc for Application Integrity
   —) mo: Tag mode. Value is always 2 for Active
   —) di: pandora.com
   —) si: site ID
   —) ui: User ID as defined in client system (optional)
   -) Returns Ozoki fields required in future requests, notably:
   —) OZ_TC: Used as a session identifier and encryption key
   —) OZ_DT: Used as a session token. Base64 encrypted and unmodified during login process

4. POST s.hoplon.pandora.com /2/4.80.9/522268/AHf6m7kADAS5E8S-/postback
   -) Format: /<mo>/<psv>/<ci>/<OZ_TC>/postback
   -) Passes many of the same query fields as seen in Stage 3
   -) The first call to this endpoint sends data: {"loader":{"init":1}}
   -) TODO: Determine significance of the loader data that is sent

5. GET s.hoplon.pandora.com /static/pandora/4.80.9/main.js
   -) Loads main.js (version 4.80.9) into DOM

6. POST s.hoplon.pandora.com /2/4.80.9/522268/AHf6m7kADAS5E8S-/postback
   -) Exact same query parameters as previous call to this endpoint
   -) This call provides data: {"loader":{"load":54}}
   -) TODO: Determine significance of the loader data that is sent

7. GET s.hoplon.pandora.com /2/4.80.9/522268/1.gif
   -) Format: /<mo>/<psv>/<ci>/1.gif
   -) Loads a pixel
   -) NOTE: The parameters and results of this call are static; They are always the same. I'm not certain if the existence of the gif on the page is needed.

8. POST s.hoplon.pandora.com /2/4.80.9/522268/AHf6m7kADAS5E8S-/postback
   -) NOTE: The third request to this endpoint is different than previous ones!
   -) A few additional query strings are added:
   —) oz_sc: The value of session.clientToken. Appears to be a random 24-digit hex string
   —) oz_st: Date.now() at the time of page load
   —) oz_df: The difference between Date.now() - oz_st at the time of this POST (ie: elapsed time)
   —) oz_l: The length of the messages string before it is encrypted (OZ_SG pre encryption)
   -) Most importantly, the body of this request includes the encrypted message data as Content-Type "text/plain"
   —) In other words, this call sends OZ_SG to the hoplon service
   -) NOTE: encrypting messages causes the 4-byte encryption IV to roll! OZ_SG will have a different value when created again. Also important, the IV is always prepended to the encrypted data.

9. GET s.hoplon.pandora.com /sri/config.json
   -) Same as before; Updates local OZ fields. It looks like the first set of OZ fields is used throughout Stages 1-10 but subsequent interactions will use the new OZ fields.

10. POST www.pandora.com /login
    -) params:
    —) existingAuthToken
    —) keepLoggedIn
    —) username
    —) password
    —) OZ_TC
    —) OZ_DT
    —) OZ_SG
    -) NOTE: The unencrypted value of OZ_SG is the same here as in Step 8, but the IV used for encryption rolls on each use, so the actual value of OZ_SG will be different.

This is the process as I currently understand it. However, something else seems to be going on. If everything is as above, then the primary indicator for bot activity is likely the OZ_SG data. If so, then technically you could swap out the event and browser data that becomes OZ_SG before it is encrypted (with a known, accepted, version). However, in certain Chromium/Selenium environments, I'm finding that switching out the pre-encrypted OZ_SG data isn't sufficient. I have found within this data OZ_TC is stored as { "SPA": { "attempt": 0, "firstSession": "<OZ_TC>" } },, but swapping that value with the OZ_TC in context isn't enough when using Chromium/Selenium/Scripting, although doing so does seem to work in a standard browser. Therefore, something else must be going on. I've found that WhiteOps started integrating with Google Analytics, so I'm wondering if the GA communication in the background is an additional factor to consider when recreating the Login workflow.

I am very interested in thoughts, workarounds, and conclusions others may have arrived at.

[H0r53]

Further note - I wrote a script that executes steps 1 - 10 and it was successful up until the POST at stage 10. There is certainly still another piece I must be missing.

[hucario]

but swapping that value with the OZ_TC in context isn't enough when using Chromium/Selenium/Scripting

As a note- when I was still confused about what was going on with a client I help maintain, I looked up the new login parameters and found this article, which says that WhiteOps actively checks for a headless and/or puppetted browser:

...checks for these automation-related properties on window: domAutomation, domAutomationController, _WEBDRIVER_ELEM_CACHE, _phantom, callPhantom, window.chrome._commandLineAPI, window.Debug.debuggerEnabled, __BROWSERTOOLS_CONSOLE, window._FirebugCommandLine, and also if document.documentElement.hasAttribute("webdriver") is true

...tries to check if the browser is headless with isExternalUrlSafeForNavigation, window.opera._browserjsran, window.opera.browserjsran

[H0r53]

That is correct. WhiteOps is fingerprinting the browser environment as a means for bot detection.

[blackjack4494]

Not working on anything pandora related but also have to deal with whiteops/human security and their so called project ozoki. I also try to simulate what they are doing. There is a lot going on actually. The service in my case uses an updated version that is already deployed on other sites as well. pagespeed.js is basically now clear.js while there is still main.js. Most of the internals/scripts are obfuscated but can easily be deobfuscated as the code for it is included. They use rot13 for almost everything or base64.
Do note that you get different scripts (especially main.js) depending on what browser you are using! The webserver will give you different files depending on the user-agent you supply.
So far I know of roughly 3 groups: [Chrome], [IE] - but is uselss, [Edge, FF, Steam Browser, Curl] - so basically a default group if it's not chrome or ie.
When I was debugging in browser and copied all generated results by the internal scripts to my python code to encrypt it, i checked whether the encrypted string is the same as in the browser and it indeed is. My code is similar to this from @perette just that I took it from the original js file and rewrote it to python.
However the hard part is still going thru the internal code that generates all those variables. Some are static (actually a big part) but some are always different.
Furthermore you have to encrypt the almsot same message twice. Once for whiteops to verify so that they approve you and once with the login request itself so they can compare it to the approval.
I am almost certain that you can implement a solution without relying on webdrivers like selenium/puppeteer or so.

So far the postbacks seems to be mandatory - at least the one with the huge encrypted strings. I suppressed those with {"loader":{"init":...}} and there was no negative impact.

If anyone made some more progress yet let me know.