- gains ability to mock (via
webmockr) async requests via eitherAsyncand/orAsyncVaried(thanks @keller-mark) (#180)
- reduce use of single letter variables within crul code to make code easier to understand (#181)
- fix typos (thanks @mdsumner and @espinielli) (#173) (#176)
- change to
_PACKAGEpattern for pkg level man file (#177)
- tweak to test helper base url setup (#178)
Via the two issues (#135) and (#159) crul now supports http retries in: Async, AsyncVaried, AsyncQueue, and the request builder class HttpRequest
- improved error message for
Asyncwhenurlsanddiskvector lengths do not match (they must match) (#165) thanks @shakraz
- fix example in
AsyncQueuedocs (#146) thanks @johnbaums ! - update
HttpClientdocs to state that it's an R6 class, and give some details on what an R6 class is and links to more info (#155)
AsyncQueuegains methods:parse,status_code,status,content, andtimes(#156)$responses()method now returns an S3 class with an associated print method to prevent printing a lot of results to the screen; print method pritns a summary of results, and at most 10 results, just status code and url (#157)
- parsing response headers gains a check for whether encoding is valid, and if not tries to set Latin1 encoding, and if that doesn't work, fails out with message (#163) (#164) thanks @FlukeAndFeather
Paginatorgains support for query parameter combinationpage/per_pageto automatically paginate (#145)
- fix typo (#149) thanks @dpprdan
- Change to how numbers are handled in query parameters. We unfortunately hadn't tested this package with large numbers, which were being converted to scientific notation with a certain number of digits before a decimal. Fixed handling of query parameters to avoid this problem. Fix for
Paginatoras well as forHttpClient(#151) (#152) (#153) thanks @ateucher
- sometimes weird response headers are returned in an HTTP response that can not be easily parsed;
crulwould raise an error when this header parsing happens, but now we raise a warning instead (#150)
ok()can now accept more than 1 status code so that you can check if the status of a url is within a set of status codes rather than equal to 1 status code (#124)ok()gains a parameterverbto use either head or get requests. in addition added more documentation (#125) to the function on how to get the "right answer" for whether a url is ok/up (#123) (#127)ok()gains parameterua_random, which ifTRUE, will use a random user agent string pulled from a vector of 50 user agent strings generated fromcharlatan::UserAgentProvider(#138)
- gains new async class
AsyncQueuefor doing async requests with rate limits (#139) - gains new functions
curl_verbose()andset_verbose().curl_verbose()can be set by passing to the initialize step (e.g.,HttpClient$new(url, verbose=curl_verbose())), and gets more compact verbose curl output, while also getting request body information (and response body optionally).set_verbose()is setscurl_verbose()globally (#141) - gains new vignette "How to choose a client" for choosing which crul class to use (e.g.,
HttpClientvs.Async) (#133) (#143)
- package sticker done, shown in README (#42)
- improve function/class reference page in docs site (#131)
- improvements to the best practices vignette (#132)
- removed unused private variable in the
AsyncVariedclass (#140) - fix inaccuracy in documentation for the RETRY method (#130)
HttpRequestnow adds the query (if present) to the printed url in the print method for the class (it was absent before now) (#128)- use new roxygen2 support for R6 classes (#126)
- removed
delete-requestsandpost-requestsmanual files - mostly redundant with other documentation
HttpResponseresponse object gains new methods for checking response content types, includes:raise_for_ct,raise_for_ct_html,raise_for_ct_json,raise_for_ct_xml. these behave similarly toraise_for_status, and can behave as a warning or raise an error through stop (#119) (#120)
- fix to prep_body internal function to handle various body inputs; now avoids warning about
as.character.form_filewhen both httr and crul are loaded (#112) - finish off "Failing with fauxpas" section of the "API package best practices" vignette (#121)
- the
head()verb onHttpClientwas no capturingauthwhen set on initialization (#122)
jsonlitepackage moved to Imports (#112)- the
parse()method in theHttpResponseobject now checks whether the response raw bytes can be converted to character, and if not just returns raw bytes (#115) (#116) - give vignettes titles (#113) (#114)
- no longer setting
cainfocurl option, fixes problem arising from change in recent libcurl version (#117)
- you can now pass on parameters through the
parse()method of anHttpResponseclass to the internally called functioniconv()to finely control the usage oficonvfor cases in which normal encoding conversion doesn't work (#110)
- use
httpcodepackage instead offauxpaspackage withinok()function (#108) (#109) thanks @maelle ! - fix links to http testing book - ropensci -> ropenscilabs (#111)
- event hooks added to
HttpClient. both request and response hooks supported. not supported in async methods for now (#76) (#107)
- improve
$parse()behavior (in theHttpResponseobject) when using disk or stream.$parse()was throwing a warning when using disk and an error when using stream. and improves behavior when doing async requests (#104) Paginatorgains optional progress bar through the newprogressparameter. In addition, thecat()calls inside the method were removed, so as not to insert newlines with each page and to not print "OK" when done (#106) thanks @boshek
- passing on opts/headers now works with
Async(#101) (#103) - streaming was broken in
AsyncVariedwithcurlof a certain version, works now (#102) (#103)
HttpClientgains aretrymethod: retries any request verb until successful (HTTP response status < 400) or a condition for giving up is met. (#89) (#95) thanks @hlappHttpClient,HttpRequest, andAsyncclasses gainverbmethod for doing HTTP requests specifying any of the supported HTTP verbs (#97)HttpClientandPaginatorgain aurl_fetchmethod: get the URL that would be sent in an HTTP request without sending the HTTP request. Useful for getting the URL before executing an HTTP request if you need to check something about the URL first. (#92)- new vignette for "API package best practices" (#65)
- Package gains manual files for each HTTP verb to facilitate linking to package documentation for information on each HTTP verb (#98)
- Intermediate headers (e.g., those in redirect chains) are now given back in a new slot in the
HttpResponseclass as$response_headers_allas an unnamed list, with each element a named list of headers; the last list in the set is the final response headers that match those given in the$response_headersslot (#60) (#99)
- some dangling file connections were left open - now fixed (#93) (#95)
- fix
url_parse: lacked check that input was a string, and that it was length 1 - this PR fixed that (#100) thanks @aaronwolen
HttpStubbedResponsewas removed from the package - it may have been used at some point, but is not used in the package anymore (#88)
AsyncandAsyncVariednow support simple auth, see?auth(#70)- gains new function
ok()to ping a URL to see if it's up or not, returns a single boolean (#71) (#73) HttpClientandHttpRequestgain new parameterprogressthat accepts a function to use to construct a progress bar. For now acceptshttr::progress()but will accept other options in the future (#20) (#81)- gains a new vignette for curl options (#7)
- can now set curl options globally using new functions
set_auth(),set_headers(),set_opts(),set_proxy(), andcrul_settings()(#48) (#85)
- explicitly import
httpcode::http_code(#80) - fix vignette names to make them more clear and add numbers to order them (#64)
- change print function for
AsyncandAsyncVariedto print max of 10 and tell user how many total and remaining not shown (#72) - added support to
proxy()for socks, e.g. to use with TOR (#79) - now when
AsyncandAsyncVariedrequests fail, they don't error but instead we capture the error and pass it back in the result. this way any failure requests don't stop progress of the entire async request suite (#74) (#84)
- Fixed handling of user agent: you can pass a UA string as a curl option or a header. Previously, we were wrongly overwriting the user input UA if given as a curl option - but were not doing so if given as a header. This is fixed now. (#63) thx to @maelle and @dpprdan
- Fix to
Paginator- it wasn't handling pagination correctly. In addition, fixed to hopefully handle all scenarios now. added more tests (#62) - Fixed handling of query parameters. We were using
urltools::url_encodeto encode strings, but it wasn't encoding correctly in some locales. Usingcurl::curl_escapefixes the problem. Encoding is done on query values and names (#67) (#68)
- Gains a new R6 class
Paginatorto help users automatically paginate through multiple requests. It only supports query parameter based paginating for now. We'll add support later for other types including cursors (e.g., used in Solr servers), and for link headers (e.g., used in the GitHub API). Please get in touch if you find any problems withPaginator. (#56) - Async classes
AsyncandAsyncvariedgain ability to write to disk and stream data (to disk or elsewhere, e.g. R console or to an R object) (#46) thanks @artemklevtsov for the push to do this
- Improved documentation for
authto indicate thatuserandpwdare indeed required - and to further indicate that one can pass inNULLto those parameters (similar to an empty string""inhttr::authenticate) when one e.g. may want to usegssnegotiatemethod (#43) - Fixed query builder so that one can now protect query parameters by wrapping them in
I()(#55)
- Fixed bug in
headrequests withHttpClientwhen passingqueryparameter - it was failing previously. Addedqueryparameter back. (#52)
- file uploads now work, see new function
upload()and examples (#25)
- fixes to reused curl handles - within a connection object only, not across connection objects (#45)
crulnow drops any options passed in tooptsor to...that are not in set of allowed curl options, seecurl::curl_options()(#49)- cookies should now be persisted across requests within
a connection object, see new doc
?cookiesfor how to set cookies (#44) - gather cainfo and use in curl options when applicable (#51)
- remove
diskandstreamfromheadmethod inHttpClientandHttpRequestas no body returned in a HEAD request
- Fixed
AsyncVariedto return async responses in the order that they were passed in. This also fixes this exact same behavior inAsyncbecauseAsyncusesAsyncVariedinternally. (#41) thanks @dirkschumacher for reporting
- Note: This version gains support for integration with
webmockr, which is now on CRAN.
- New function
auth()to do simple authentication (#33) - New function
HttpStubbedResponsefor making a stubbed response object for thewebmockrintegration (#4) - New function
mock()to turn on mocking - it's off by default. Ifwebmockris not installed but user attempts to use mocking we error with message to installwebmockr(#4)
- Use
gzip-deflateby deafult for each request to make sure gzip compression is used if the server can do it (#34) - Change
useragenttoUser-Agentas default user agent header (#35) - Now we make sure that user supplied headers override the default headers if they are of the same name (#36)
- New utility functions
url_buildandurl_parse(#31)
- Now using markdown for documentation (#32)
- Better documentation for
AsyncVaried(#30) - New vignette on how to use
crulin realistic scenarios rather than brief examples to demonstrate individual features (#29) - Better documentation for
HttpRequest(#28) - Included more tests
- Fixed put/patch/delete as weren't passing body
correctly in
HttpClient(#26) - DRY out code for preparing requests - simplify to use helper functions (#27)
- Added support for asynchronous HTTP requests, including two new
R6 classes:
AsyncandAsyncVaried. The former being a simpler interface treating all URLs with same options/HTTP method, and the latter allowing any type of request through the new R6 classHttpRequest(#8) (#24) - New R6 class
HttpRequestto supportAsyncVaried- this method only defines a request, but does not execute it. (#8)
- Added support for proxies (#22)
- Fixed parsing of headers from FTP servers (#21)
- Created new manual files for various tasks to document usage better (#19)
- URL encode paths - should fix any bugs where spaces between words caused errors previously (#17)
- URL encode query parameters - should fix any bugs where spaces between words caused errors previously (#11)
- request headers now passed correctly to response object (#13)
- response headers now parsed to a list for easier access (#14)
- Now supporting multiple query parameters of the same name, wasn't possible in last version (#15)
- Improved options for using curl options. Can manually add
to list of curl options or pass in via
.... And we check that user doesn't pass in prohibited options (curlpackage takes care of checking that options are valid) (#5) - Incorporated
fauxpaspackage for dealing with HTTP conditions. It's a Suggest, so only used if installed (#6) - Added support for streaming via
curl::curl_fetch_stream.streamparam defaults toNULL(thus ignored), or pass in a function to use streaming. Only one of memory, streaming or disk allowed. (#9) - Added support for streaming via
curl::curl_fetch_disk.diskparam defaults toNULL(thus ignored), or pass in a path to write to disk instead of use memory. Only one of memory, streaming or disk allowed. (#12)
- Added missing
raise_for_status()method on theHttpResponseclass (#10)
- Was importing
httpcodebut wasn't using it in the package. Now using the package inHttpResponse
- Released to CRAN.