NMFS has newly updated web interfaces for accessing landings
https://foss.nmfs.noaa.gov/apexfoss/f?p=215:200::::::
and trade data.
https://foss.nmfs.noaa.gov/apexfoss/f?p=215:2
These interfaces provide a convenient way to access for single use or exploratory data pulls. Web interfaces have a some inherent drawbacks that using the API can helps resolve. APIs provide a means to pull landings and trade data using code within your R script. Some advantages to using an API are:
Querying an API works by sending a Hypertext Transfer Protocol (HTTP) request to NMFS FOSS server.The parameters of the HTTP request are contained in the URL and specify the requested data. The server then returns a response and if the request is properly formed the data that you request.
There are two R packages that can be used to query the API and get data into a usable format (e.g., a dataframe). The package httr is tool for working with URLs and HTTP and is what we’ll use to query the API. As with any package, one can google ‘httr R package’, or something similar, and you can find homepage for the package
https://cran.r-project.org/web/packages/httr/index.html
which has links to a manual and vignettes (i.e., tutorials). Your google search may bring up other kinds of tutorials which could be helpful as well. The second pacakge that we’ll use is jsonlite
https://cran.r-project.org/web/packages/jsonlite/index.html
which is used for working with data in JSON format within R.
The API endpoint for the trade data is
https://www.st.nmfs.noaa.gov/ords/foss/trade_data/
. Entering this URL into a browser will return the first 25 rows of data (by default) from the database in JSON format to your browser. In order to return this data into R rather than your browser we’ll use the function from the GET() function from the httr package.
response <- GET("https://www.st.nmfs.noaa.gov/ords/foss/trade_data")
response## Response [https://www.st.nmfs.noaa.gov/ords/foss/trade_data/]
## Date: 2020-09-23 07:07
## Status: 200
## Content-Type: application/json
## Size: 11.4 kBstr(response)## List of 10
## $ url : chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/"
## $ status_code: int 200
## $ headers :List of 7
## ..$ server : chr "Apache"
## ..$ strict-transport-security : chr "max-age=31536000; includeSubDomains; preload"
## ..$ access-control-allow-origin: chr "floatplan.noaa.gov"
## ..$ content-type : chr "application/json"
## ..$ content-length : chr "11413"
## ..$ date : chr "Wed, 23 Sep 2020 07:07:02 GMT"
## ..$ connection : chr "keep-alive"
## ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ all_headers:List of 2
## ..$ :List of 3
## .. ..$ status : int 301
## .. ..$ version: chr "HTTP/1.1"
## .. ..$ headers:List of 7
## .. .. ..$ server : chr "Apache"
## .. .. ..$ strict-transport-security : chr "max-age=31536000; includeSubDomains; preload"
## .. .. ..$ access-control-allow-origin: chr "floatplan.noaa.gov"
## .. .. ..$ location : chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/"
## .. .. ..$ content-length : chr "0"
## .. .. ..$ date : chr "Wed, 23 Sep 2020 07:07:01 GMT"
## .. .. ..$ connection : chr "keep-alive"
## .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## ..$ :List of 3
## .. ..$ status : int 200
## .. ..$ version: chr "HTTP/1.1"
## .. ..$ headers:List of 7
## .. .. ..$ server : chr "Apache"
## .. .. ..$ strict-transport-security : chr "max-age=31536000; includeSubDomains; preload"
## .. .. ..$ access-control-allow-origin: chr "floatplan.noaa.gov"
## .. .. ..$ content-type : chr "application/json"
## .. .. ..$ content-length : chr "11413"
## .. .. ..$ date : chr "Wed, 23 Sep 2020 07:07:02 GMT"
## .. .. ..$ connection : chr "keep-alive"
## .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
## $ cookies :'data.frame': 0 obs. of 7 variables:
## ..$ domain : logi(0)
## ..$ flag : logi(0)
## ..$ path : logi(0)
## ..$ secure : logi(0)
## ..$ expiration: 'POSIXct' num(0)
## ..$ name : logi(0)
## ..$ value : logi(0)
## $ content : raw [1:11413] 7b 22 69 74 ...
## $ date : POSIXct[1:1], format: "2020-09-23 07:07:02"
## $ times : Named num [1:6] 20.973 0.142 0.166 0.238 21.584 ...
## ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
## $ request :List of 7
## ..$ method : chr "GET"
## ..$ url : chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data"
## ..$ headers : Named chr "application/json, text/xml, application/xml, */*"
## .. ..- attr(*, "names")= chr "Accept"
## ..$ fields : NULL
## ..$ options :List of 2
## .. ..$ useragent: chr "libcurl/7.64.1 r-curl/4.3 httr/1.4.1"
## .. ..$ httpget : logi TRUE
## ..$ auth_token: NULL
## ..$ output : list()
## .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
## ..- attr(*, "class")= chr "request"
## $ handle :Class 'curl_handle' <externalptr>
## - attr(*, "class")= chr "response"Passing ‘response’ to R tells it to print to the console. What gets printed depends on how the author of httr who wrote the print methods. To see a full breakdown of what’s in the object ‘response’ use the str() function.
Take note of the status code. A successful response from the API will have a status code of 200. Any other status code is considered a failure, but different codes can indicate in which way the request failed.
Next we’ll use the content() function to extract the contents. Then the fromJSON() function to convert the data from JSON into an R format.
df <- content(response, as = "text", encoding = "UTF-8")
df <- fromJSON(df, flatten = TRUE) %>% data.frame()
head(df)## items.year items.month items.hts_number
## 1 1989 04 0306130040
## 2 1990 01 1605902000
## 3 1990 08 0305592000
## 4 1990 10 0306130015
## 5 1990 12 0305592000
## 6 1991 01 0306130018
## items.name items.cntry_code
## 1 SHRIMP PEELED FROZEN 2050
## 2 CLAM NSPF BOILED CANNED WHETHER OR NOT MINCED/CHOP > 0.68KG 2050
## 3 SHARK FINS DRIED 2050
## 4 SHRIMP SHELL-ON FROZEN 31/40 2050
## 5 SHARK FINS DRIED 2050
## 6 SHRIMP SHELL-ON FROZEN 41/50 2050
## items.fao items.cntry_name items.district_code items.district_name
## 1 320 GUATEMALA 52 MIAMI, FL
## 2 320 GUATEMALA 27 LOS ANGELES, CA
## 3 320 GUATEMALA 28 SAN FRANCISCO, CA
## 4 320 GUATEMALA 27 LOS ANGELES, CA
## 5 320 GUATEMALA 28 SAN FRANCISCO, CA
## 6 320 GUATEMALA 27 LOS ANGELES, CA
## items.edible_code items.kilos items.val items.source items.association
## 1 E 25719 95760 IMP CBERA, IATTC, ICCAT
## 2 E 789 2400 IMP CBERA, IATTC, ICCAT
## 3 E 117 4000 IMP CBERA, IATTC, ICCAT
## 4 E 1383 5566 IMP CBERA, IATTC, ICCAT
## 5 E 300 6000 IMP CBERA, IATTC, ICCAT
## 6 E 5429 20598 IMP CBERA, IATTC, ICCAT
## items.rfmo items.nmfs_region_code
## 1 IATTC SA
## 2 IATTC PA
## 3 IATTC PA
## 4 IATTC PA
## 5 IATTC PA
## 6 IATTC PA
## items.links
## 1 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAA
## 2 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAB
## 3 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAC
## 4 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAD
## 5 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAE
## 6 self, https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAF
## hasMore limit offset count links.rel
## 1 TRUE 25 0 25 self
## 2 TRUE 25 0 25 edit
## 3 TRUE 25 0 25 describedby
## 4 TRUE 25 0 25 first
## 5 TRUE 25 0 25 next
## 6 TRUE 25 0 25 self
## links.href
## 1 https://www.st.nmfs.noaa.gov/ords/foss/trade_data/
## 2 https://www.st.nmfs.noaa.gov/ords/foss/trade_data/
## 3 https://www.st.nmfs.noaa.gov/ords/foss/metadata-catalog/trade_data/
## 4 https://www.st.nmfs.noaa.gov/ords/foss/trade_data/
## 5 https://www.st.nmfs.noaa.gov/ords/foss/trade_data/?offset=25
## 6 https://www.st.nmfs.noaa.gov/ords/foss/trade_data/str(df)## 'data.frame': 25 obs. of 23 variables:
## $ items.year : chr "1989" "1990" "1990" "1990" ...
## $ items.month : chr "04" "01" "08" "10" ...
## $ items.hts_number : chr "0306130040" "1605902000" "0305592000" "0306130015" ...
## $ items.name : chr "SHRIMP PEELED FROZEN" "CLAM NSPF BOILED CANNED WHETHER OR NOT MINCED/CHOP > 0.68KG" "SHARK FINS DRIED" "SHRIMP SHELL-ON FROZEN 31/40" ...
## $ items.cntry_code : chr "2050" "2050" "2050" "2050" ...
## $ items.fao : chr "320" "320" "320" "320" ...
## $ items.cntry_name : chr "GUATEMALA" "GUATEMALA" "GUATEMALA" "GUATEMALA" ...
## $ items.district_code : chr "52" "27" "28" "27" ...
## $ items.district_name : chr "MIAMI, FL" "LOS ANGELES, CA" "SAN FRANCISCO, CA" "LOS ANGELES, CA" ...
## $ items.edible_code : chr "E" "E" "E" "E" ...
## $ items.kilos : int 25719 789 117 1383 300 5429 258 400 476 3545 ...
## $ items.val : int 95760 2400 4000 5566 6000 20598 3480 8800 3728 23100 ...
## $ items.source : chr "IMP" "IMP" "IMP" "IMP" ...
## $ items.association : chr "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" ...
## $ items.rfmo : chr "IATTC" "IATTC" "IATTC" "IATTC" ...
## $ items.nmfs_region_code: chr "SA" "PA" "PA" "PA" ...
## $ items.links :List of 25
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAA"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAB"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAC"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAD"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAE"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAF"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAG"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAH"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAI"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAJ"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAK"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAL"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAM"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAN"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAO"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAP"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAQ"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAR"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAS"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAT"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAU"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAV"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAW"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAX"
## ..$ :'data.frame': 1 obs. of 2 variables:
## .. ..$ rel : chr "self"
## .. ..$ href: chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/AAEb8FAAHAAAL7DAAY"
## $ hasMore : logi TRUE TRUE TRUE TRUE TRUE TRUE ...
## $ limit : int 25 25 25 25 25 25 25 25 25 25 ...
## $ offset : int 0 0 0 0 0 0 0 0 0 0 ...
## $ count : int 25 25 25 25 25 25 25 25 25 25 ...
## $ links.rel : chr "self" "edit" "describedby" "first" ...
## $ links.href : chr "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/" "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/" "https://www.st.nmfs.noaa.gov/ords/foss/metadata-catalog/trade_data/" "https://www.st.nmfs.noaa.gov/ords/foss/trade_data/" ...After this there is still a little bit of cleanup to do on the data. We’ll drop the following variables but I’ll breifly list them below as they could be useful.
Also, note that the variables year, and month are character and it would be nice for them to be numeric. Finally, I’ll remove “items.” from the beginning of the variable names.
df <- transform(df, items.year=as.numeric(items.year),
items.month = as.numeric(items.month),
links.rel = NULL, links.href = NULL,
items.links = NULL, hasMore = NULL,
limit = NULL, offset = NULL, count = NULL)
str(df)## 'data.frame': 25 obs. of 16 variables:
## $ items.year : num 1989 1990 1990 1990 1990 ...
## $ items.month : num 4 1 8 10 12 1 2 3 11 1 ...
## $ items.hts_number : chr "0306130040" "1605902000" "0305592000" "0306130015" ...
## $ items.name : chr "SHRIMP PEELED FROZEN" "CLAM NSPF BOILED CANNED WHETHER OR NOT MINCED/CHOP > 0.68KG" "SHARK FINS DRIED" "SHRIMP SHELL-ON FROZEN 31/40" ...
## $ items.cntry_code : chr "2050" "2050" "2050" "2050" ...
## $ items.fao : chr "320" "320" "320" "320" ...
## $ items.cntry_name : chr "GUATEMALA" "GUATEMALA" "GUATEMALA" "GUATEMALA" ...
## $ items.district_code : chr "52" "27" "28" "27" ...
## $ items.district_name : chr "MIAMI, FL" "LOS ANGELES, CA" "SAN FRANCISCO, CA" "LOS ANGELES, CA" ...
## $ items.edible_code : chr "E" "E" "E" "E" ...
## $ items.kilos : int 25719 789 117 1383 300 5429 258 400 476 3545 ...
## $ items.val : int 95760 2400 4000 5566 6000 20598 3480 8800 3728 23100 ...
## $ items.source : chr "IMP" "IMP" "IMP" "IMP" ...
## $ items.association : chr "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" "CBERA, IATTC, ICCAT" ...
## $ items.rfmo : chr "IATTC" "IATTC" "IATTC" "IATTC" ...
## $ items.nmfs_region_code: chr "SA" "PA" "PA" "PA" ...head(df)## items.year items.month items.hts_number
## 1 1989 4 0306130040
## 2 1990 1 1605902000
## 3 1990 8 0305592000
## 4 1990 10 0306130015
## 5 1990 12 0305592000
## 6 1991 1 0306130018
## items.name items.cntry_code
## 1 SHRIMP PEELED FROZEN 2050
## 2 CLAM NSPF BOILED CANNED WHETHER OR NOT MINCED/CHOP > 0.68KG 2050
## 3 SHARK FINS DRIED 2050
## 4 SHRIMP SHELL-ON FROZEN 31/40 2050
## 5 SHARK FINS DRIED 2050
## 6 SHRIMP SHELL-ON FROZEN 41/50 2050
## items.fao items.cntry_name items.district_code items.district_name
## 1 320 GUATEMALA 52 MIAMI, FL
## 2 320 GUATEMALA 27 LOS ANGELES, CA
## 3 320 GUATEMALA 28 SAN FRANCISCO, CA
## 4 320 GUATEMALA 27 LOS ANGELES, CA
## 5 320 GUATEMALA 28 SAN FRANCISCO, CA
## 6 320 GUATEMALA 27 LOS ANGELES, CA
## items.edible_code items.kilos items.val items.source items.association
## 1 E 25719 95760 IMP CBERA, IATTC, ICCAT
## 2 E 789 2400 IMP CBERA, IATTC, ICCAT
## 3 E 117 4000 IMP CBERA, IATTC, ICCAT
## 4 E 1383 5566 IMP CBERA, IATTC, ICCAT
## 5 E 300 6000 IMP CBERA, IATTC, ICCAT
## 6 E 5429 20598 IMP CBERA, IATTC, ICCAT
## items.rfmo items.nmfs_region_code
## 1 IATTC SA
## 2 IATTC PA
## 3 IATTC PA
## 4 IATTC PA
## 5 IATTC PA
## 6 IATTC PAnames(df) <- sub(“items.”,"",names(df), fixed= TRUE)
The meta-data catalogue for the trade data can be found here
https://www.st.nmfs.noaa.gov/ords/foss/metadata-catalog/trade_data/
. It’s not that straight forward to read. This shows the variable types one the serve.
Also, the API endpoint for the landings data is
https://www.st.nmfs.noaa.gov/ords/foss/landings/
and the meta-data can be found here
https://www.st.nmfs.noaa.gov/ords/foss/metadata-catalog/landings/
Now that we know we can get data off of the server and get it into a usable format we just need to figure out how to get the data that we want. The way we do this is by carefully adding parameters to the end of the URL that we pass to the API. The set of parameters that we can pass are described in the meta-data catalouge.
For example,
https://www.st.nmfs.noaa.gov/ords/foss/trade_data/?q={"year":"2019"}
returns the first 25 for the year 2019.
Using the GET() function we can retrieve the data in R with
response <- GET('https://www.st.nmfs.noaa.gov/ords/foss/trade_data',
query = list(q = '{"year":"2019"}', limit = 10000))
status_code(response)## [1] 200Modeled somewhat after the web interface I’ve created functions for getting data from the NMFS trade and landings databases through the API.
You can load the functions into R by sourcing them from wherever they are loacted on your computer.
source('~/projects/afsc/NMFSDataAPI/nmfsLandingsData.R')
source('~/projects/afsc/NMFSDataAPI/nmfsTradeData.R')In the web interface the user chooses the trade type (imports/exports) years, product, and country. These functions require you to enter trade type, product type, and years.
The function still fairly rudimentary. It takes as arguments the products listed on the trade data web interface. For example, for groundfish exports from 2019 through all month available in 2020.
plk.trd <- nmfsTradeData(prodType = "GROUNDFISH", tradeType = "EXP",
fromYr = 2019, toYr = 2020)The function essentially performs the operations above.
Similarly, I have a function for extracting landings data. It takes as arguments the species, landings type, and years.
plk.lnd <- nmfsLandingsData(species = "pollock, walleye", landingsType = "Commercial",
fromYr = 2018, toYr = 2019)These functions are yours now. Use them, improve them, share them.
Cameron Spier also has a tutorial he shared on accessing APIs that he shared a little a couple weeks ago. I used some of his material here. Thanks!
Querying APIs in R gives examples of using R to get data from an API.
https://medium.com/@traffordDataLab/querying-apis-in-r-39029b73d5f1
Introduction to APIs is a tutorial on using APIs more generally.
https://www.earthdatascience.org/courses/earth-analytics/get-data-using-apis/intro-to-programmatic-data-access-r/