rnoaa 1.3.7
ncdc* functions - response header content type changed - we had a check for proper content type - that check is now more general so that any json content type will be okay (#390)ghcnd() now accepts more than 1 station identifier (#373) PR from @eliocampersst(): use new v5 version of their service - see ?ersst docs for details (#381) thanks @vonStadarhraun for the tipbuoy() docs to state that a special value of 9999 passsed to the year parameter will give the most up to date data (aka current data) - and an example added using it (#377)ncdc() with add_units = TRUE (#378) (#379) PR from @amcdavidghcnd() handling of unknown/bad/invalid station identifiers: now returns an empty data.frame and gives back empty strings for the two attributes source and file_modified (#374)storm_data(), storm_meta(), storm_shp(), and storm_shp_read(). associated package datasets storm_columns and storm_names removed (#306)lcd() returns a tibble now instead of a tibble with S3 class lcd attached (#369)stormevents_cachesf from Suggests, only used in an examplearc2() fix: when using bbox parameter, it would not have worked as intended, fixed now (#372)argo functions that were not working because of a down API are working again, see ?argo (#358)?rnoaa-defunct manual file. gefs functions are still in the package as those functions may come back at some point. (#359)arc2(): 1) now accepts more than 1 date; 2) gains new parameter box to accept a bounding box to spatially filter results (uses dplyr::filter on the data.frame of spatial data) (#351)Regarding the documentation site at https://docs.ropensci.org/rnoaa
ghncd() (and all functions that build on ghcnd()) can now be altered to use a specific base URL for requests. See the "Base URL" section of the ?ghcnd docs (#353)ghcnd_splitvars() speedup, using data.table instead of dplyr for manipulation (#352) (#355)tibble::as_tibble throughout package instead of dplyr::tbl_df (#354)ghcnd_stations(): internal method get_inventory() was not creating a directory first before trying to download a file into that directory (#349) (#350)rnoaa_options() to toggle package level options; only option for now is cache_messages, a boolean to toggle whether the user gets messages about cached files or not. along with this change, messages about cached files and file sizes and locations are now consistently used across all functions that cache files on disk (#331)?rnoaa_caching - with information on how to access and manage cached files for each of the rnoaa functions that caches files on disk (#346)isd() moved to using hoardr caching - see ?isd_cache for details (#347)path parameter and telling them it's no longer used; been defunct for quite a whileghcnd_stations() now caching data - first time requests should now take just over 1 minute, with subsequent requests (assuming cached data isn't deleted) taking ~ 3 seconds (#164)autoplot method meteo_coverage() fix to visually display gaps in data (#314) (#333) thanks @philipshirkmeteo_nearby_stations() fix: coerce input data.frame to the function to a data.frame before remainder of steps - in case user inputs a tibble (#340)coops_search() fix: when product=predictions, we get no metadata back - so just dont adjust times (#342)lcd() changes: gains lcd_cache for managing cached files; use a new internal function for safely reading each csv file, with more informative error messages; (#344)meteo_pull_monitors() fix: changed internals of meteo_tidy_ghcnd() to set -9999 values to NA slightly differently to avoi failing (#348)lcd() function was unfortunately pulling data from https://www.ncei.noaa.gov/data/global-hourly/access - whereas it should have been pulling data from https://www.ncei.noaa.gov/data/local-climatological-data/access - fixed now; additionaly, lcd_cleanup is defunct because lcd data coming from the appropriate link has all variable names spelled out and data split up (#334) thanks @sayon000 !gefs* functions are now defunct - they are being taken out for now until fixed - see the issues for the details (#335) (#336)gefs function fixes: fixed failing test on CRAN having to do with a date mismatch; gefs now cleans up temporary files (#327) (#328)argo_search, argo_files, argo_qwmo, argo_plan) no longer work, while the functions that use the FTP server still work (argo_buoy_files, argo) (#333)ens and time, that will eventually replace the deprecated parameters ens_idx and time_idx (#321) (#324)isd() now using fetching data using http instead of ftptornadoes(): the URL had changed yet again (#322) (#323) thanks @mbjoseph !sea_ice_tabular() function for fetching tabular .csv sea ice files instead of using the shp files in sea_ice() (#194)seaice() fxn name has changed to sea_ice() (#313)sea_ice() gains option to fetch GeoTIFF format data in addition to shp files (#219) (#313)lcd_cleanup() - takes output of call to lcd(), parsing additional columns that contain comma separated strings (#283)isd() docs to highlight that cached files downloaded with the fxn will be used until deleted by the user! See ?isd docs for details (#205)gefs only mentioned longitude, now both vars discussed (#317) (#318)ncdc() regarding units, and in readme and vignette as well (#265) (#315) from @amoebacpc_prcp(): should have allowed dates back to 1948, but only allowed back to 1979 (#300)buoy() fxn: datasets that did not have lat/lon variables were failing to be parsed by the fxn; now when lat/lon vars missing, we just give. back ncdf4 object for the user to deal with themselves (#303) (#304)gefs(): longitude on the (-180, 180) scale worked but not on the (0,360) scale (#316) (#318) (#319)tornadoes(): the URL for the data had changed (#311) (#312) thanks @mbjosephncdc() parameters startdate/enddate weren't handling dates as input values; now handle date and character inputs (#307)ghcnd_stations(); there was an encoding issue with the data returned from NOAA (#305)This very long US federal government shutdown has allowed time for building in nicer failure behavior and more documentation for government shutdowns. There's a number of related changes:
swdi() function (#298)lcd() function (#295)ncdc*() functions (#293) (#297)swdi(): changed from downloading data with download.file to crul (#298)arc2() tests to not have hard-coded dates (#294)arc2 tests to not be sensitive to the real year that the test is run in, reported in CRAN checks and via email (#294)bsw() for Blended Sea Winds data (#246)ghcnd_read() - to read .dly files directly, e.g., files already downloaded (#223) thanks @shabbychef for the feature requestlcd() for Local Climatological Data (#212)se_data() and se_files() for the Storm Events Database (#282)ghcnd() and ghcnd_search() gain a refresh parameter to refresh data for the query even if it's already cached locally. in addition, these functions now print messages to tell the user what file path the data is locally cached in, and the min and max dates when using ghcnd_search() (#269) thanks @kgmccannncdc() gains add_units parameter (boolean) to toggle adding units to the output data.frame. default is add_units=FALSE. if add_units=TRUE we match dataset id and data type id and return units if we have them. do be in touch if you see a problem with these units! ncdc() now returns tibbles in the data slot (#233) (#266) (#289) (#287)coops docs (#228) thanks @jstaghcnd_search() to arrange data by day instead of by month (#247) thanks @asrivas3 for reportingswdi() to use xml2 and crul packages instead of XML and httr (#275)swdi() tests (#239) thanks @kevin-ht-hoisd_stations_search() no longer renames lat and lon column names (#238) thanks @kevin-ht-hohttr with crul throughout package (#186)vcr now for caching, more to do waiting on vcr being able to handle direct to disk use cases and binary files like pdfs (#284)gefs(). was incorrectly repeating time values within ensembles when it should have repeated time values across ensembles so that each ensemble has a time value for each time period (#230) (#231) thanks for report from @lcsma and fix by @potterzotncdc() - fix to internal function parse_ncdc(), which was failing on strsplit() call if attributes was NULL (#232) thanks for reporting @andypickecpc_prcp(): URLs were changed at some point, fixes for this (#242)cpc_prcp(): to read .gz files correctly with gzfile instead of file (#248)homr() - NOAA server gives back a 200 OK response even if that's not the case - but we can check content type to see if there was likely an error (#250)autoplot.meteo_coverage (#258)buoy_stations(): was getting the wrong station ids - fixed; and use async HTTP requests to get data faster (#261) thanks @johnharley for the bug reportghcnd_stations(): was returning an extra empty row (#267) thanks @joeroe for the bug reportghcnd() was giving a trailing row of NA's - fixed (#270)swdi(): radius parameter doesn't work, update docs to tell users not to use it (#243)ncdc() fix: fix to internal function parse_ncdc(), errored when more than 1 flag returned (#279) thanks @ghaines3 for the bug reportstorm_shp() fix: update to use new URL patterns (#263)swdi() fail better (#274) thanks @OrionDarleymeteo_nearby_stations() to coerce character to numeric lat and lon values (#257) thanks @mondorescue for the bug reportmeteo_nearby_stations() - internally ignored the user supplied column names for lat and lon (#286) thanks @ghaines3 for the bug reporttornadoes() for Windows OS's: utils::untar() was failing on windows, changed to using utils::unzip() (#203)argo_buoy_files(): use fill=TRUE in the read.table call - was erroring on some Windows OS's (#235) thanks @jonmcalder for reportingNote that some NOAA datasets have changed names:
GHCNDMS is now GSOM (Global Summary of the Month)ANNUAL is now GSOY (Global Summary of the Year)isd() gains new parameters additional to toggle whether the
non-mandatory ISD fields (additional + remarks) are parsed and
returned & force to toggle whether download new version or use
cached version. isd_read() gains new parameter additional
(see description above) (#190)cpc_prcp() (#193)arc2() to get data from Africa Rainfall Climatology
version 2 (#201)https - changed internal code
to use https from http for coops, swdi, ersst, and tornadoes
data sources (#187)coops_search() to handle requests better: only certain
date combinations allowed for certain COOPS products (#213) (#214)
thanks @tphilippi !hoardr package to manage caching in some functions. Will
roll out to all functions that cache soon (#191)GHCNDMS is now GSOM and ANNUAL is now GSOY - added to docs
and examples of using GSOM and GSOY (#189)isd() (#168)coops_search() to fix time zone problems (#184) thanks @drf5nghcnd() - fix some column types that were of inappropriate
type before (#211)ghcnd(): we were coercing factors to integers, which caused
nonsense output - first coercing to character now, then integer (#221)ncdc() function. Added metadata to the package to help parse
flags (#199)isd() now using a new package isdparser to parse
NOAA ISD files. We still fetch the file within rnoaa, but the
file parsing is done by isdparser (#176) (#177) (#180) thanks @mrubayet
for the pushghcnd() where internal unexported function
was not found (#179)isd_stations() and isd_stations_search() to work
correctly on Windows (#181) thanks @GuodongZhuncdc) to https from http (#182) thanks @maspottshomr) to https from http (#183)meteo_clear_cache gains parameter force to control force
parameter in unlink()lubridate usage in seaiceurls() function, just using
base R functions.isd_read() to read ISD output from isd() manually
instead of letting isd() read in the data. This is useful when you
use isd() but need to read the file in later when it's already cached.
(#169)rnoaa cache files that are downloaded from
various NOAA web services. File caching is usually done when data comes
from FTP servers. In some of these functions where we cache data, we used
to write to your home directory, but have now changed all these functions
to write to a proper cache directory in a platform independent way.
We determine the cache directory using rappdirs::user_cache_dir().
Note that this may change your workflow if you'd been depending on
cached files to be a in particular place on your file system. In addition,
the path parameter in the changed functions is now defunct, but you
get an informative warning about it (#171)storm_data() now returns a tibble/data.frame not inside of a list. We used
to return a list with a single slot data with a data.frame, but this was
unnecessary.ghcnd_stations() now outputs a data.frame (tbl_df) by itself,
instead of a data.frame nested in a list. This may change how
you access data from this function. (#163)ncdc_*()) (#167)isd() docs that when you get an error similar to
Error: download failed for ftp://ftp.ncdc.noaa.gov/pub/data/noaa/1955/011490-99999-1955.gz,
the file does not exist on NOAA's ftp servers. If your internet is down,
you'll get a different error saying as much (#170)meteo_*, and are meant to find weather monitors near
locations (meteo_nearby_stations), find all monitors within a radius
of a location (meteo_distance), calculate the distances between a
location and all available stations (meteo_process_geographic_data),
calculate the distance between two locations (meteo_spherical_distance),
pull GHCND weather data for multiple weather monitors (meteo_pull_monitors),
create a tidy GHCND dataset from a single monitor (meteo_tidy_ghcnd),
and determine the "coverage" for a station data frame (meteo_coverage()).
In addition, vis_miss() added to visualize missingness in a data.frame. See
the PR diff against master
for all the changes. (#159) Thanks a ton to @geanders et al. (@hrbrmstr,
@maelle, @jdunic, @njtierney, @leighseverson, @RyanGan, @mandilin, @jferreri,
@cpatrizio88, @ryan-hicks, @Ewen2015, @mgutilla, @hakessler, @rodlammers)isd_stations_search() changed internal structure. We replaced
usage of geojsonio and lawn for faster dplyr::filter for
bbox inputs, and meteo_distance() for lat/long/radius inputs
. This speeds up this function significantly. Thanks to @lukas-rokka
(#157)isd_stations_search() and isd_stations() now return
tibble's instead of data.frame'sisd_stations_search() now caches using rappdirs (#161)is() uses to inherits()seaiceeurls() function that's used to generate urls for
the seaice() function - due to change in NOAA urls (#160)ghncd_split_vars() to not fail on dplyr::contains
call (#156) thanks @lawinslow !httr version to call encoding explicitly (#135)gefs functions (#121)isd() function - it's a time consuming task
as we have to parse a nasty string of characters line by line - more speed
ups to come in future versions (#146)dplyr::rbind_all() with dplyr::bind_rows() as the former is
being deprecated (#152)isd() function - was failing on some station names that had
leading zeros. (#136)ncdc_stations() - used to allow more than one station id to
be passed in, but internally only handled one. This is a restriction
due to the NOAA NCDC API. Documentation now shows an example of how
to deal with many station ids (#138)ncdc_*() functions to allow multiple inputs
to those parameters where allowed (#139)ncdc_plot() due to new ggplot2 version (#153)argo() functions: a) with new httr, box input of a vector
no longer works, now manually make a character vector; b) errant file param
being passed into the http request, removed (#155)argo() (#123)
for more, see http://www.argo.ucsd.edu/coops_search()
(#111) for idea from @fmichonneau (#124) for implementing @jsta
also (#126) (#128)rgdal moved to Suggests to make usage easier (#125)ncdc_plot() - made default brakes to just default to what
ggplot2 does, but you can still pass in your own breaks (#131)gefs(), gefs_dimension_values(), gefs_dimensions(), gefs_latitudes(),
gefs_longitudes(), and gefs_variables() (#106) (#119) thanks @potterzot - he's
now an author tooersst() (#96)isd_stations() to get ISD station data.ncdf package for ncdf4 package. Windows binaries weren't
availiable for ncdf4 prior to now. (#117)isd() function to do transformations of certain
variables to give back data that makes more sense (#115)leaflet, geojsonio, and lawn added in Suggests, used in a few
functions.swdi() function man page that the nldn dataset is
available to military users only (#107)buoy() function to accept character class inputs for the
buoyid parameter. the error occurred because matching was not
case-insensitive, now works regardless of case (#118)ggplot2 version (#113)GET request retries for ghncd functions as
some URLs fail unpredictably (#110)utils, methods, and stats (#103)?rnoaa-defunct for more information (#104)radius parameter removed from ncdc_stations() function (#102), was already removed internally within the function in the last version, now not in the function definition, see also (#98) and (#99)plyr and data.table from imports. plyr::rbind.fill() and data.table::rbindlist() replaced with dplyr::bind_rows().httr v1 where empty list not allowed to pass to
the query parameter in GET (#101)ghcnd(), ghcnd_clear_cache(), ghcnd_countries(), ghcnd_search(), ghcnd_splitvars()
ghcnd_states(), ghcnd_stations(), and ghcnd_version() (#85) (#86) (#87) (#88) (#94)isd() functions, including better man file.callopts parameter changed to ... in function swdi().ncdc() requires that users do their own paging - previously this was done internally (#77)RCurl, maptools, stringr, digest.
A few new ones added: dplyr, tidyr.erddap functions now defunct - see the package rerddap,
a general purpose R client for ERDDAP servers. (#51) (#73) (#90) (#95)extent function in noaa_stations() used to accept either a bounding
box or a point defined by lat/long. The lat/long option dropped as it required
two packages, one of which is a pain to install for many users (#98) (#99)ncdc_legacy(). (#54)isd() to get ISD data from NOAA FTP server. (#76)erddap_table(), while gridded datasets are available via erddap_grid(). Helper
function erddap_search() was modified to search for either tabledap or griddap
datasets, and erddap_info() gets and prints summary information differently
for tabledap and griddap datasets. (#63)erddap_data() defunct, now as functions erddap_table() and erddap_grid(), uses new
store parameter which takes a function, either disk(path, overwrite) to store
on disk or memory() to store in R memory.assertthat library removed, replaced with stopifnot()tornadoes(). (#56)storm_*(). (#57)homr_*() (#59)dplyr-like outputs
with a summary of the data.frame, as appropriate.ncdc_* functions changed callopts parameter to .... This parameter
allow you to pass in options to httr::GET to modify curl requests. (#61)check_key() looks for one of two stored keys, as an
environment variable under the name NOAA_KEY, or an option variable under the name
noaakey. Environment variables can be set during session like Sys.setenv(VAR = "..."),
or stored long term in your .Renviron file. Option variables can be set during session
like options(var = "..."), or stored long term in your .Rprofile file.is.* and print.* functions no longer have public man files, but can be seen via
rnoaa::: if needed.sp, rgeos, assertthat, jsonlite, and ncdf4, and new package Suggests: knitr, taxizenoaa*() functions for NCDC data changed to ncdc*(). noaa_buoy() changed to buoy(). noaa_seaice() changed to seaice(). When you call the old versions an error is thrown, with a message pointing you to the new function name. See ?rnoaa-defunct.erddap_info(), erddap_data(), and erddap_search().buoy(), including a number of helper functions.ncdc() now splits apart attributes. Previously, the attributes were returned as a single column, but now there is column for each attribute so data can be easily retrieved. Attribute columns differ for each different datasetid.buoy() function has been removed from the CRAN version of rnoaa. Install the version with buoy() and associated functions via devtools::install_github("ropensci/rnoaa", ref="buoy")noaa_swdi() (function changed to swdi()) gains new parameter filepath to specify path to write a file to if format=kmz or format=shp. Examples added for using format= csv, shp, and kmz.plyr::compact.ncdc() gains new parameter includemetadata. If TRUE, includes metadata, if not, does not, and response should be faster as does not take time to calculate metadata.noaa_stations() gains new parameter radius. If extent is a vector of length 4 (for a bounding box) then radius is ignored, but if you pass in two points to extent, it is interpreted as a point, and then radius is used as the distance upon which to construct a bounding box. radius default is 10 km.datasetid, startdate, and enddate are often required parameters, and changes were made to help users with this.