Package: robotstxt
Type: Package
Title: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
        Permissions Checker
Version: 0.7.15.9000
Authors@R: c(
        person(
          "Pedro", "Baltazar", role = c("ctb"),
          email = "pedrobtz@gmail.com"
        ),
         person(
          "Jordan", "Bradford", role = c("cre"),
          email = "jrdnbradford@gmail.com"
        ),
        person(
          "Peter", "Meissner", role = c("aut"),
          email = "retep.meissner@gmail.com"
        ),
        person(
            "Kun", "Ren", email = "mail@renkun.me", role = c("aut", "cph"),
            comment = "Author and copyright holder of list_merge.R."
        ),
	    person("Oliver", "Keys", role = "ctb", comment = "original release code review"),
	    person("Rich", "Fitz John", role = "ctb", comment = "original release code review")
	  )
Description: Provides functions to download and parse 'robots.txt'
        files. Ultimately the package makes it easy to check if bots
        (spiders, crawler, scrapers, ...) are allowed to access
        specific resources on a domain.
License: MIT + file LICENSE
BugReports: https://github.com/ropensci/robotstxt/issues
URL: https://docs.ropensci.org/robotstxt/,
        https://github.com/ropensci/robotstxt
Imports: stringr (>= 1.0.0), httr (>= 1.0.0), spiderbar (>= 0.2.0),
        future.apply (>= 1.0.0), magrittr, utils
Suggests: knitr, rmarkdown, dplyr, testthat (>= 3.0.0), covr, curl
Depends: R (>= 3.0.0)
VignetteBuilder: knitr
RoxygenNote: 7.3.2
Encoding: UTF-8
Config/testthat/edition: 3
Config/pak/sysreqs: libicu-dev libssl-dev
Repository: https://ropensci.r-universe.dev
Date/Publication: 2024-11-15 19:32:37 UTC
RemoteUrl: https://github.com/ropensci/robotstxt
RemoteRef: main
RemoteSha: d3d0a4d525e87204010d10162b01a921b05a6149
NeedsCompilation: no
Packaged: 2026-07-01 08:27:13 UTC; root
Author: Pedro Baltazar [ctb],
  Jordan Bradford [cre],
  Peter Meissner [aut],
  Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.),
  Oliver Keys [ctb] (original release code review),
  Rich Fitz John [ctb] (original release code review)
Maintainer: Jordan Bradford <jrdnbradford@gmail.com>