webmockr) async requests via either Async and/or AsyncVaried (thanks @keller-mark) (#180)_PACKAGE pattern for pkg level man file (#177)Via the two issues (#135) and (#159) crul now supports http retries in: Async, AsyncVaried, AsyncQueue, and the request builder class HttpRequest
Async when urls and disk vector lengths do not match (they must match) (#165) thanks @shakrazAsyncQueue docs (#146) thanks @johnbaums !HttpClient docs to state that it's an R6 class, and give some details on what an R6 class is and links to more info (#155)AsyncQueue gains methods: parse, status_code, status, content, and times (#156)$responses() method now returns an S3 class with an associated print method to prevent printing a lot of results to the screen; print method pritns a summary of results, and at most 10 results, just status code and url (#157)Paginator gains support for query parameter combination page/per_page to automatically paginate (#145)Paginator as well as for HttpClient (#151) (#152) (#153) thanks @ateuchercrul would raise an error when this header parsing happens, but now we raise a warning instead (#150)ok() can now accept more than 1 status code so that you can check if the status of a url is within a set of status codes rather than equal to 1 status code (#124)ok() gains a parameter verb to use either head or get requests. in addition added more documentation (#125) to the function on how to get the "right answer" for whether a url is ok/up (#123) (#127)ok() gains parameter ua_random, which if TRUE, will use a random user agent string pulled from a vector of 50 user agent strings generated from charlatan::UserAgentProvider (#138)AsyncQueue for doing async requests with rate limits (#139)curl_verbose() and set_verbose(). curl_verbose() can be set by passing to the initialize step (e.g., HttpClient$new(url, verbose=curl_verbose())), and gets more compact verbose curl output, while also getting request body information (and response body optionally). set_verbose() is sets curl_verbose() globally (#141)HttpClient vs. Async) (#133) (#143)AsyncVaried class (#140)HttpRequest now adds the query (if present) to the printed url in the print method for the class (it was absent before now) (#128)delete-requests and post-requests manual files - mostly redundant with other documentationHttpResponse response object gains new methods for checking response content types, includes: raise_for_ct, raise_for_ct_html, raise_for_ct_json, raise_for_ct_xml. these behave similarly to raise_for_status, and can behave as a warning or raise an error through stop (#119) (#120)as.character.form_file when both httr and crul are loaded (#112)head() verb on HttpClient was no capturing auth when set on initialization (#122)jsonlite package moved to Imports (#112)parse() method in the HttpResponse object now checks whether the response raw bytes can be converted to character, and if not just returns raw bytes (#115) (#116)cainfo curl option, fixes problem arising from change in recent libcurl version (#117)parse() method of an HttpResponse class to the internally called function iconv() to finely control the usage of iconv for cases in which normal encoding conversion doesn't work (#110)httpcode package instead of fauxpas package within ok() function (#108) (#109) thanks @maelle !HttpClient. both request and response hooks supported. not supported in async methods for now (#76) (#107)$parse() behavior (in the HttpResponse object) when using disk or stream. $parse() was throwing a warning when using disk and an error when using stream. and improves behavior when doing async requests (#104)Paginator gains optional progress bar through the new progress parameter. In addition, the cat() calls inside the method were removed, so as not to insert newlines with each page and to not print "OK" when done (#106) thanks @boshekAsync (#101) (#103)AsyncVaried with curl of a certain version, works now (#102) (#103)HttpClient gains a retry method: retries any request verb until successful (HTTP response status < 400) or a condition for giving up is met. (#89) (#95) thanks @hlappHttpClient, HttpRequest, and Async classes gain verb method for doing HTTP requests specifying any of the supported HTTP verbs (#97)HttpClient and Paginator gain a url_fetch method: get the URL that would be sent in an HTTP request without sending the HTTP request. Useful for getting the URL before executing an HTTP request if you need to check something about the URL first. (#92)HttpResponse class as $response_headers_all as an unnamed list, with each element a named list of headers; the last list in the set is the final response headers that match those given in the $response_headers slot (#60) (#99)url_parse: lacked check that input was a string, and that it was length 1 - this PR fixed that (#100) thanks @aaronwolenHttpStubbedResponse was removed from the package - it may have been used at some point, but is not used in the package anymore (#88)Async and AsyncVaried now support simple auth, see ?auth (#70)ok() to ping a URL to see if it's up or not, returns a single boolean (#71) (#73)HttpClient and HttpRequest gain new parameter progress that accepts a function to use to construct a progress bar. For now accepts httr::progress() but will accept other options in the future (#20) (#81)set_auth(), set_headers(), set_opts(), set_proxy(), and crul_settings() (#48) (#85)httpcode::http_code (#80)Async and AsyncVaried to print max of 10 and tell user how many total and remaining not shown (#72)proxy() for socks, e.g. to use with TOR (#79)Async and AsyncVaried requests fail, they don't error but instead we capture the error and pass it back in the result. this way any failure requests don't stop progress of the entire async request suite (#74) (#84)Paginator - it wasn't handling pagination correctly.
In addition, fixed to hopefully handle all scenarios now. added
more tests (#62)urltools::url_encode
to encode strings, but it wasn't encoding correctly in some locales. Using curl::curl_escape fixes the problem. Encoding is done on query values and names (#67) (#68)Paginator to help users automatically paginate through multiple requests. It only supports query parameter based paginating for now. We'll add support later for other types including cursors (e.g., used in Solr servers), and for link headers (e.g., used in the GitHub API). Please get in touch if you find any problems with Paginator. (#56)Async and Asyncvaried gain ability to write to disk and stream data (to disk or elsewhere, e.g. R console or to an R object) (#46) thanks @artemklevtsov for the push to do thisauth to indicate that user and pwd are indeed required - and to further indicate that one can pass in NULL to those parameters (similar to an empty string "" in httr::authenticate) when one e.g. may want to use gssnegotiate method (#43)I() (#55)head requests with HttpClient when passing query parameter - it was failing previously. Added query parameter back. (#52)upload() and examples (#25)crul now drops any options passed in to opts or to ... that
are not in set of allowed curl options, see curl::curl_options() (#49)?cookies for how to set cookies (#44)disk and stream from head method in HttpClient
and HttpRequest as no body returned in a HEAD requestAsyncVaried to return async responses in the order that
they were passed in. This also fixes this exact same behavior in
Async because Async uses AsyncVaried internally. (#41)
thanks @dirkschumacher for reportingwebmockr, which is now on CRAN.auth() to do simple authentication (#33)HttpStubbedResponse for making a stubbed
response object for the webmockr integration (#4)mock() to turn on mocking - it's off by
default. If webmockr is not installed but user attempts
to use mocking we error with message to install
webmockr (#4)gzip-deflate by deafult for each request
to make sure gzip compression is used if the server
can do it (#34)useragent to User-Agent as default user
agent header (#35)url_build and url_parse (#31)AsyncVaried (#30)crul in realistic
scenarios rather than brief examples to demonstrate
individual features (#29)HttpRequest (#28)HttpClient (#26)Async and AsyncVaried. The former being a simpler
interface treating all URLs with same options/HTTP method, and the latter
allowing any type of request through the new R6 class HttpRequest (#8) (#24)HttpRequest to support AsyncVaried - this method
only defines a request, but does not execute it. (#8).... And we
check that user doesn't pass in prohibited options
(curl package takes care of checking that options
are valid) (#5)fauxpas package for dealing with HTTP
conditions. It's a Suggest, so only used if installed (#6)curl::curl_fetch_stream.
stream param defaults to NULL (thus ignored), or pass in a
function to use streaming. Only one of memory, streaming or
disk allowed. (#9)curl::curl_fetch_disk.
disk param defaults to NULL (thus ignored), or pass in a
path to write to disk instead of use memory. Only one of memory,
streaming or disk allowed. (#12)raise_for_status() method on the
HttpResponse class (#10)httpcode but wasn't using it in the package.
Now using the package in HttpResponse