Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add dailywire #267

Merged
merged 2 commits into from
Jun 19, 2019
Merged

add dailywire #267

merged 2 commits into from
Jun 19, 2019

Conversation

edwardchalstrey1
Copy link
Collaborator

Tested with crawler by reducing the number of load button clicks to 10, getting 155 articles:

2019-06-13 11:43:18     INFO: Processed 155 pages in 0:00:23.592246 => 6.74 Hz
2019-06-13 11:43:18     INFO: Found articles in 155/155 pages => 100.00%
2019-06-13 11:43:18     INFO: ... of these 0/155 had no date => 0.00%
2019-06-13 11:43:18     INFO: ... of these 0/155 had no byline => 0.00%
2019-06-13 11:43:18     INFO: ... of these 0/155 had no title => 0.00%

This is a site from 1.5 list #191

jemrobinson
jemrobinson requested changes Jun 18, 2019
View reviewed changes
misinformation/middlewares/jsloadbuttonmiddleware.py Outdated
@@ -35,6 +35,7 @@ def __init__(self):
'//button[@phx-track-id="load more"]',
'//form[@class="gdpr-form"]/input[@class="btn"]',
'//div[contains(@class, "load-btn")]/a',
'//ul[contains(@class, "pager--infinite-scroll")]/li/a'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this not the <a href="/node?page=1">Show More</a> inside it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, that's better, have changed

@edwardchalstrey1 edwardchalstrey1 merged commit 3828a25 into master Jun 19, 2019
@edwardchalstrey1 edwardchalstrey1 deleted the website/191-dailywire branch June 19, 2019 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jemrobinson jemrobinson jemrobinson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@edwardchalstrey1 @jemrobinson