Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: facebook photos page scroll #60

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

fix: facebook photos page scroll #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9ebc18d
fix: facebook photos page scroll
Chickensoupwithrice Sep 13, 2023
55d91ff
fix: use full XPath which doesn't change
Chickensoupwithrice Sep 14, 2023
0616da7
fix: test against the actual substring that can match, rather than
Chickensoupwithrice Sep 14, 2023
e6a7843
fix: maybe pass CI?
Chickensoupwithrice Sep 14, 2023
a9a7a77
fix(ci, facebook): update the CI to point to the new facebook URL
Chickensoupwithrice Sep 14, 2023
425e264
misc: testing CI
Chickensoupwithrice Sep 15, 2023
69a5741
fix: linting
Chickensoupwithrice Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/facebook-photos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ jobs:
SOURCE: ${{ secrets.PROFILE_URL }}
DEST: './profile.tar.gz'

- name: see if profile exists
run: ls

- name: install requirements
run: yarn install

Expand All @@ -41,7 +44,7 @@ jobs:
run: docker pull $BROWSERTRIX_IMAGE

- name: run crawl
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/Capture-Sample-105559135004406/photos --limit 1 --logging behaviors --profile /profile.tar.gz > ./log
run: docker run -v $PWD/dist/behaviors.js:/app/node_modules/browsertrix-behaviors/dist/behaviors.js -v $PWD/profile.tar.gz:/profile.tar.gz $BROWSERTRIX_IMAGE crawl --url https://www.facebook.com/capture.sample/photos/ --limit 1 --logging behaviors --profile /profile.tar.gz > ./log

- name: cat log
run: cat ./log
Expand Down
2 changes: 1 addition & 1 deletion dist/behaviors.js
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 8 additions & 2 deletions src/site/facebook.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ const Q = {
commentMoreComments: "./following-sibling::div/div/div[2][@role='button'][./span/span]",
viewComments: ".//h4/..//div[@role='button']",
photoCommentList: "//ul[../h2]",
firstPhotoThumbnail: "//div[@role='main']//div[3]//div[contains(@style, 'border-radius')]//div[contains(@style, 'max-width') and contains(@style, 'min-width')]//a[@role='link']",
firstVideoThumbnail: "//div[@role='main']//div[contains(@style, 'z-index')]/following-sibling::div/div/div/div[last()]//a[contains(@href, '/videos/') and @aria-hidden!='true']",
firstPhotoThumbnail: "/html/body/div[1]/div/div[1]/div/div[3]/div/div/div/div[1]/div[1]/div/div/div[4]/div/div/div/div/div/div/div/div/div[3]/div[1]/div[1]/div/div/a/img",
firstVideoSimple: "//div[@role='main']//a[contains(@href, '/videos/') and @aria-hidden!='true']",
mainVideo: "//div[@data-pagelet='root']//div[@role='dialog']//div[@role='main']//video",
nextVideo: "following::a[contains(@href, '/videos/') and @aria-hidden!='true']",
Expand Down Expand Up @@ -275,8 +275,12 @@ export class FacebookTimelineBehavior {
firstPhoto.click();
await sleep(waitUnit * 5);
await waitUntil(() => window.location.href !== lastHref, waitUnit * 2);
const fromQuestionMark = window.location.href.indexOf("?");
const firstPhotoHref = window.location.href.substring(fromQuestionMark, window.location.href.length - 7); // Facebook adds &type=3 on click from thumbnail but not on scroll


let nextSlideButton = null;
let currentHref = "";

while ((nextSlideButton = xpathNode(Q.nextSlideQuery))) {
lastHref = window.location.href;
Expand All @@ -290,7 +294,9 @@ export class FacebookTimelineBehavior {
sleep(3000)
]);

if (window.location.href === lastHref) {
// Exit once we've looped
currentHref = window.location.href.substring(window.location.href.indexOf("?"), window.location.href.length);
if (currentHref === firstPhotoHref) {
break;
}

Expand Down
17 changes: 7 additions & 10 deletions test/expected-facebook-photos.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
{"state":{},"msg":"Starting..."}
{"state":{"photos":1,"comments":0},"msg":"Viewing photo https://www.facebook.com/Capture-Sample-105559135004406/photos/105560261670960"}
{"state":{"photos":1,"comments":1},"msg":"Loading comments"}
{"state":{"photos":1,"comments":2},"msg":"Loading comments"}
{"state":{"photos":2,"comments":2},"msg":"Viewing photo https://www.facebook.com/Capture-Sample-105559135004406/photos/105560188337634"}
{"state":{"photos":3,"comments":2},"msg":"Viewing photo https://www.facebook.com/Capture-Sample-105559135004406/photos/105560165004303"}
{"state":{"photos":3,"comments":3},"msg":"Loading comments"}
{"state":{"photos":3,"comments":4},"msg":"Loading comments"}
{"state":{"photos":4,"comments":4},"msg":"Viewing photo https://www.facebook.com/Capture-Sample-105559135004406/photos/105560131670973"}
{"state":{"photos":4,"comments":4},"msg":"done!"}
{"msg":"Starting...","state":{}}
{"msg":"Viewing photo https://www.facebook.com/photo/?fbid=105737024986069&set=pb.100066493156766.-2207520000","state":{"photos":1,"comments":0}}
{"msg":"Viewing photo https://www.facebook.com/photo/?fbid=105736844986087&set=pb.100066493156766.-2207520000","state":{"photos":2,"comments":0}}
{"msg":"Viewing photo https://www.facebook.com/photo/?fbid=105736801652758&set=pb.100066493156766.-2207520000","state":{"photos":3,"comments":0}}
{"msg":"Viewing photo https://www.facebook.com/photo/?fbid=105736774986094&set=pb.100066493156766.-2207520000","state":{"photos":4,"comments":0}}
{"msg":"Viewing photo https://www.facebook.com/photo/?fbid=105736638319441&set=pb.100066493156766.-2207520000","state":{"photos":5,"comments":0}}
{"msg":"done!","state":{"photos":5,"comments":0}}