Sites with no articles in Sept-Dec 2018

[dongpng]

Hi,

We used the following query to find websites in the database with no articles in our time period (Sept 1, 2018 - Dec 31,2018).

DECLARE @MinDate DATE = '20180901',
        @MaxDate DATE = '20181231';

SELECT DISTINCT([site_name]) 
        FROM [dbo].[articles_v5] WHERE [site_name] NOT IN (
                SELECT [site_name] FROM [dbo].[articles_v5] 
                WHERE [publication_datetime] >= @MinDate
                AND [publication_datetime] < @MaxDate);

Out of these sites, we found that most sites were dead or didn't had content in the period. However, there are 3 sites that should have articles:

• aljazeera.com
• huffingtonpost.com
• notallowedto.com

notallowedto.com has hidden pagination. Scraping could be done using https://notallowedto.com/news/page1, https://notallowedto.com/news/page2, https://notallowedto.com/news/page3

[edwardchalstrey1]

• aljazeera.com: James has fixed the issue already
• huffingtonpost.com: I think the problem is that the website index pages only go back as far as page 17, which currently is articles from 2019-05
• notallowedto.com: see Problematic site: notallowedto.com #147