Skip to content

issues Search Results · repo:b4dnewz/robots-parse language:TypeScript

Filter by

6 results
 (67 ms)

6 results

inb4dnewz/robots-parse (press backspace or delete to remove)

b4dnewz/robots-parse
The robot-parse result is unmatched with the actual robots.txt

Hi there, I just curious if this is an issue or not, but the robots-parser result apparently is unmatched from the actual url robots.txt. For example, in tokopedia.com/robots.txt the result would be the ...
  • thoriqadillah
  • Opened 
    on Nov 11, 2021
  • #16

b4dnewz/robots-parse
TypeError: Cannot read property 'allow' of undefined

Steps to Reproduce - npm i request robots-parse - Run the following code const req = require( request ); req( https://optinmonster.com/robots.txt , (err, res, body) = { if (err) { console.log( ...
  • TheUltimateCookie
  • Opened 
    on Apr 17, 2021
  • #11

b4dnewz/robots-parse
Support for crawl delay?

Can this library be used to parse and extract the Crawl Delay for user agents?
enhancement
  • krukru
  • 1
  • Opened 
    on Jan 10, 2020
  • #6

b4dnewz/robots-parse
Issue when Disallow: / does not end with a space

Disallow: / --- works as expected vs Disallow: / --- does not properly parse For a workaround I added: @line 33: if (!line.endsWith( )){ line = line+ }
  • rbedfordpro
  • 1
  • Opened 
    on Sep 13, 2019
  • #4

b4dnewz/robots-parse
Caps sensitive user-agent field causing issues

Currently in robots-parse/lib/parser.js: Line 2: agents: /([Uu]ser-agent:) (.+)/, -- doesn t handle cases where user has manually typed in User-Agent (capital A). Causes error and exit. FIX: Line 2: ...
  • rbedfordpro
  • 1
  • Opened 
    on Sep 13, 2019
  • #3

b4dnewz/robots-parse
AssertionError Protocol mismatch

When I try to execute the example I get a assertion error: const robotsParse = require( robots-parse ); robotsParse( https://sfbay.craigslist.org , (err, res) = { console.log( Result: , res); }); ...
  • Hyllesen
  • 1
  • Opened 
    on Jul 8, 2019
  • #1
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue search results · GitHub