v1.0

bopoda released this 16 Mar 16:21

· 53 commits to master since this release

c41bdaf

class RobotsTxtParser
class RobotsTxtValidator.

all tests passed.

Allowed Directives:

noindex
allow
disallow
host
sitemap
user-agent
crawl-delay
clean-param

Assets 2