--- title: rerddap introduction author: Scott Chamberlain date: "2023-06-29" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{rerddap introduction} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- `rerddap` is a general purpose R client for working with ERDDAP servers. ERDDAP is a server built on top of OPenDAP, which serves some NOAA data. You can get gridded data ([griddap](https://upwell.pfeg.noaa.gov/erddap/griddap/documentation.html)), which lets you query from gridded datasets, or table data ([tabledap](https://upwell.pfeg.noaa.gov/erddap/tabledap/documentation.html)) which lets you query from tabular datasets. In terms of how we interface with them, there are similarties, but some differences too. We try to make a similar interface to both data types in `rerddap`. ## NetCDF `rerddap` supports NetCDF format, and is the default when using the `griddap()` function. NetCDF is a binary file format, and will have a much smaller footprint on your disk than csv. The binary file format means it's harder to inspect, but the `ncdf4` package makes it easy to pull data out and write data back into a NetCDF file. Note the the file extension for NetCDF files is `.nc`. Whether you choose NetCDF or csv for small files won't make much of a difference, but will with large files. ## Caching Data files downloaded are cached in a single hidden directory `~/.rerddap` on your machine. It's hidden so that you don't accidentally delete the data, but you can still easily delete the data if you like. When you use `griddap()` or `tabledap()` functions, we construct a MD5 hash from the base URL, and any query parameters - this way each query is separately cached. Once we have the hash, we look in `~/.rerddap` for a matching hash. If there's a match we use that file on disk - if no match, we make a http request for the data to the ERDDAP server you specify. ## ERDDAP servers You can get a data.frame of ERDDAP servers using the function `servers()`. The list of ERDDAP servers is drawn from the *Awesome ERDDAP* page maintained by the Irish Marine Institute . If you know of more ERDDAP servers, follow the instructions on that page to add the server. ## Install Stable version from CRAN ```r install.packages("rerddap") ``` Or, the development version from GitHub ```r remotes::install_github("ropensci/rerddap") ``` ```r library("rerddap") ``` ## Search First, you likely want to search for data, specify either `griddadp` or `tabledap` ```r ed_search(query = 'size', which = "table") #> # A tibble: 36 × 2 #> title dataset_id #> #> 1 CCE Prey Size and Hard Part Size Regressions mmtdPreyS… #> 2 CCE Teleost Prey Size and Hard Part Size Measurements mmtdTeleo… #> 3 CalCOFI Larvae Sizes erdCalCOF… #> 4 CCE Non-Teleost Prey Size and Hard Part Size Measurements mmtdNonTe… #> 5 Channel Islands, Kelp Forest Monitoring, Size and Frequency, Natu… erdCinpKf… #> 6 File Names from the AWS S3 noaa-goes16 Bucket awsS3Noaa… #> 7 File Names from the AWS S3 noaa-goes17 Bucket awsS3Noaa… #> 8 PacIOOS Beach Camera 001: Waikiki, Oahu, Hawaii BEACHCAM-… #> 9 PacIOOS Beach Camera 003: Waimea Bay, Oahu, Hawaii BEACHCAM-… #> 10 PacIOOS Beach Camera 004: Waimea Bay (Offshore), Oahu, Hawaii BEACHCAM-… #> # ℹ 26 more rows ``` ```r ed_search(query = 'size', which = "grid") #> # A tibble: 103 × 2 #> title dataset_id #> #> 1 Audio data from a local source. testGridW… #> 2 Main Hawaiian Islands Multibeam Bathymetry Synthesis: 50-m Bathym… hmrg_bath… #> 3 Main Hawaiian Islands Multibeam Bathymetry Synthesis: 50-m Bathym… hmrg_bath… #> 4 Coastal Upwelling Transport Index (CUTI), Daily erdCUTIda… #> 5 SST smoothed frontal gradients FRD_SSTgr… #> 6 Coastal Upwelling Transport Index (CUTI), Monthly erdCUTImo… #> 7 SST smoothed frontal gradients, Lon0360 FRD_SSTgr… #> 8 Biologically Effective Upwelling Transport Index (BEUTI), Daily erdBEUTId… #> 9 Biologically Effective Upwelling Transport Index (BEUTI), Monthly erdBEUTIm… #> 10 Daily averaged and put on grid 4x daily NCEP reanalysis (psi.2012) noaa_psl_… #> # ℹ 93 more rows ``` There is now a convenience function to search over a list of ERDDAP servers, designed to work with the function `servers()` ```r server_list <- c( emodnet_physics = 'https://erddap.emodnet-physics.eu/erddap/', irish_marine_institute = 'https://erddap.marine.ie/erddap/' ) global_search(query = 'size', server_list, 'griddap') #> title #> 1 EMODnet Physics - Total Suspended Matter - GridSeriesObservation - Concentration of total suspended matter - BALTIC SEA #> 2 EMODnet Physics - Total Suspended Matter - GridSeriesObservation - Concentration of total suspended matter - MEDITERRANEAN SEA #> 3 EMODnet Physics - Total Suspended Matter - GridSeriesObservation - Concentration of total suspended matter - MEDITERRANEAN SEA - LOW RESOLUTION #> 4 EMODnet Physics - TEMPERATURE YEARLY RECORDING DENSITY #> 5 EMODPACE - Monthly sea level derived from CMEMS-DUACS (DT-2018) satellite altimetry (1993-2019) #> 6 EMODnet Physics - Total Suspended Matter - GridSeriesObservation - Concentration of total suspended matter - NORTH SEA #> 7 EMODPACE - Absolute sea level trend (1993 – 2019) - derived from CMEMS-DUACS (DT-2018) satellite altimetry #> 8 EMODPACE - Sea Level monthly mean, EurAsia. This product is based, uses and reprocess the CMEMS product id. SEALEVEL_GLO_PHY_CLIMATE_L4_REP_OBSERVATIONS_008_057 #> 9 COMPASS-NEATL Hindcast 2016-2020 #> dataset_id #> 1 TSM_BALTICSEA #> 2 TSM_MBSEA #> 3 TSM_MBSEA_LOWRESOLUTION #> 4 ERD_EP_TEMP_DNS #> 5 EMODPACE_SLEV_MONTHLY_MEAN_DESEASONALIZED #> 6 TSM_NORTHSEA #> 7 EMODPACE_SLEV_TREND #> 8 EMODPACE_SLEV_MONTHLY_MEAN #> 9 compass_neatl_hindcast_grid #> url #> 1 https://erddap.emodnet-physics.eu/erddap/ #> 2 https://erddap.emodnet-physics.eu/erddap/ #> 3 https://erddap.emodnet-physics.eu/erddap/ #> 4 https://erddap.emodnet-physics.eu/erddap/ #> 5 https://erddap.emodnet-physics.eu/erddap/ #> 6 https://erddap.emodnet-physics.eu/erddap/ #> 7 https://erddap.emodnet-physics.eu/erddap/ #> 8 https://erddap.emodnet-physics.eu/erddap/ #> 9 https://erddap.marine.ie/erddap/ ``` ## Information Then you can get information on a single dataset ```r info('erdCalCOFIlrvsiz') #> erdCalCOFIlrvsiz #> Base URL: https://upwell.pfeg.noaa.gov/erddap #> Dataset Type: tabledap #> Variables: #> calcofi_species_code: #> Range: 19, 946 #> common_name: #> cruise: #> itis_tsn: #> larvae_10m2: ... ``` ## griddap (gridded) data First, get information on a dataset to see time range, lat/long range, and variables. ```r (out <- info('erdMBchla1day')) #> erdMBchla1day #> Base URL: https://upwell.pfeg.noaa.gov/erddap #> Dataset Type: griddap #> Dimensions (range): #> time: (2006-01-01T12:00:00Z, 2023-06-27T12:00:00Z) #> altitude: (0.0, 0.0) #> latitude: (-45.0, 65.0) #> longitude: (120.0, 320.0) #> Variables: #> chlorophyll: #> Units: mg m-3 ``` Then query for gridded data using the `griddap()` function ```r (res <- griddap(out, time = c('2015-01-01','2015-01-03'), latitude = c(14, 15), longitude = c(125, 126) )) #> erdMBchla1day #> Path: [/var/folders/xw/mcmsdzzx4mgbttplylgs7ysh0000gp/T//RtmpoME6FV/R/rerddap/4d844aa48552049c3717ac94ced5f9b8.nc] #> Last updated: [2023-06-29 13:18:53.945082] #> File size: [0.03 mb] #> Dimensions (dims/vars): [4 X 1] #> Dim names: time, altitude, latitude, longitude #> Variable names: Chlorophyll Concentration in Sea Water #> data.frame (rows/columns): [5043 X 5] #> # A tibble: 5,043 × 5 #> longitude latitude altitude time chlorophyll #> #> 1 125 14 0 2015-01-01T12:00:00Z NA #> 2 125. 14 0 2015-01-01T12:00:00Z NA #> 3 125. 14 0 2015-01-01T12:00:00Z NA #> 4 125. 14 0 2015-01-01T12:00:00Z NA #> 5 125. 14 0 2015-01-01T12:00:00Z NA #> 6 125. 14 0 2015-01-01T12:00:00Z NA #> 7 125. 14 0 2015-01-01T12:00:00Z NA #> 8 125. 14 0 2015-01-01T12:00:00Z NA #> 9 125. 14 0 2015-01-01T12:00:00Z NA #> 10 125. 14 0 2015-01-01T12:00:00Z NA #> # ℹ 5,033 more rows ``` The output of `griddap()` is a list that you can explore further. Get the summary ```r res$summary #> $filename #> [1] "/var/folders/xw/mcmsdzzx4mgbttplylgs7ysh0000gp/T//RtmpoME6FV/R/rerddap/4d844aa48552049c3717ac94ced5f9b8.nc" #> #> $writable #> [1] FALSE #> #> $id #> [1] 65536 #> #> $error #> [1] FALSE #> #> $safemode #> [1] FALSE #> ... ``` Get the dimension variables ```r names(res$summary$dim) #> [1] "time" "altitude" "latitude" "longitude" ``` Get the data.frame (beware: you may want to just look at the `head` of the data.frame if large) ```r head(res$data) #> longitude latitude altitude time chlorophyll #> 1 125.000 14 0 2015-01-01T12:00:00Z NA #> 2 125.025 14 0 2015-01-01T12:00:00Z NA #> 3 125.050 14 0 2015-01-01T12:00:00Z NA #> 4 125.075 14 0 2015-01-01T12:00:00Z NA #> 5 125.100 14 0 2015-01-01T12:00:00Z NA #> 6 125.125 14 0 2015-01-01T12:00:00Z NA ``` ## tabledap (tabular) data ```r (out <- info('erdCalCOFIlrvsiz')) #> erdCalCOFIlrvsiz #> Base URL: https://upwell.pfeg.noaa.gov/erddap #> Dataset Type: tabledap #> Variables: #> calcofi_species_code: #> Range: 19, 946 #> common_name: #> cruise: #> itis_tsn: #> larvae_10m2: ... ``` ```r (dat <- tabledap('erdCalCOFIlrvsiz', fields=c('latitude','longitude','larvae_size', 'scientific_name'), 'time>=2011-01-01', 'time<=2011-12-31')) #> erdCalCOFIlrvsiz #> Path: [/var/folders/xw/mcmsdzzx4mgbttplylgs7ysh0000gp/T//RtmpoME6FV/R/rerddap/db7389db5b5b0ed9c426d5c13bc43d18.csv] #> Last updated: [2023-06-29 13:18:57.579066] #> File size: [0.05 mb] #> # A tibble: 1,304 × 4 #> latitude longitude larvae_size scientific_name #> #> 1 32.956665 -117.305 4.5 Engraulis mordax #> 2 32.91 -117.4 5.0 Merluccius productus #> 3 32.511665 -118.21167 2.0 Merluccius productus #> 4 32.511665 -118.21167 3.0 Merluccius productus #> 5 32.511665 -118.21167 5.5 Merluccius productus #> 6 32.511665 -118.21167 6.0 Merluccius productus #> 7 32.511665 -118.21167 2.8 Merluccius productus #> 8 32.511665 -118.21167 3.0 Sardinops sagax #> 9 32.511665 -118.21167 5.0 Sardinops sagax #> 10 32.511665 -118.21167 2.5 Engraulis mordax #> # ℹ 1,294 more rows ``` Since both `griddap()` and `tabledap()` give back data.frame's, it's easy to do downstream manipulation. For example, we can use `dplyr` to filter, summarize, group, and sort: ```r library("dplyr") dat$larvae_size <- as.numeric(dat$larvae_size) dat %>% group_by(scientific_name) %>% summarise(mean_size = mean(larvae_size)) %>% arrange(desc(mean_size)) #> # A tibble: 7 × 2 #> scientific_name mean_size #> #> 1 Anoplopoma fimbria 23.3 #> 2 Engraulis mordax 9.26 #> 3 Sardinops sagax 7.28 #> 4 Merluccius productus 5.48 #> 5 Tactostoma macropus 5 #> 6 Scomber japonicus 3.4 #> 7 Trachurus symmetricus 3.29 ```