Collecting all the quirks

Data from JSTOR/DfR is unlike most other data you encounter when doing text analysis. First and foremost, the data about articles and books come from a wide variety of journals and publishers. The level of detail and certain formats vary because of this. jstor tries to deal with this situation with two strategies:

• try to recognise the format and read data accordingly
• if this is not possible, read data as “raw” as possible, i.e. without any conversions

An example for the first case are references. Four different ways how references can be specified are known at this time, and all are imported in specific ways to deal this variation. There might however be other formats, which should lead to an informative error when trying to import them via jst_get_references().

An example for the latter case are page numbers. Most of the time, the entries for page numbers are simply 42, or 61. This is as expected, and could be parsed as integers. Sometimes, there are characters present, like M77. This would pose no problem either, we could simply extract all digits via regex and parse as character. Unfortunately, sometimes the page is specified like this: v75i2p84. Extracting all digits would result in 75284, which is wrong by a long shot. Since there might be other ways of specifying pages, jstor does not attempt to parse the pages to integers when importing. However, it offers a set of convenience functions which deal with a few common cases (see jst_augment() and below).

There are many other problems or peculiarities like this. This vignette tries to list as many as possible, and offer solutions for dealing with them. Unfortunately I have neither the time nor the interest to wade through all the data which you could get from DfR in order to find all possible quirks. The following list is thus inevitably incomplete. If you encounter new quirks/peculiarities, it would be greatly appreciated if you sent me an email, or opened an issue at GitHub. I will then include your findings in future version of this vignette, so this vignette can be a starting point for everybody who conducts text analysis with data from JSTOR/DfR.

Data augmentation

After importing data via jst_get_article(), there are at least two tasks you might typically want to undertake:

• Merge different identifiers for journals into one, so you can filter journals.
• Convert pages from character into integers and calculate the total number of pages per article.

There are four functions which help you to streamline this process:

• jst_clean_page() attempts to turn a character vector with pages into an integer vector.
• jst_add_total_pages() adds a column with the total number of pages per article.
• jst_unify_journal_id() merges different identifiers for journals into one.
• jst_augment() wraps the above functions for convenience.

Known quirks

In the following sections, known issues with data from DfR are described in greater detail.

library(jstor)
library(dplyr)

input <- tibble::tribble(
  ~first_page,   ~last_page,    ~page_range,
  NA_real_,      NA_real_,      NA_character_, 
  1,             10,            "1 - 10",
  1,             10,            NA_character_,
  1,             NA_real_,      NA_character_, 
  1,             NA_real_,      "1-10",
  NA_real_,       NA_real_,      "1, 5-10",
  NA_real_,       NA_real_,      "1-4, 5-10",
  NA_real_,       NA_real_,      "1-4, C5-C10"
)

input %>% 
  mutate(n_pages = jst_get_total_pages(first_page, last_page, page_range))
#> # A tibble: 8 × 4
#>   first_page last_page page_range  n_pages
#>        <dbl>     <dbl> <chr>         <dbl>
#> 1         NA        NA <NA>             NA
#> 2          1        10 1 - 10           10
#> 3          1        10 <NA>             10
#> 4          1        NA <NA>             NA
#> 5          1        NA 1-10             10
#> 6         NA        NA 1, 5-10           7
#> 7         NA        NA 1-4, 5-10        10
#> 8         NA        NA 1-4, C5-C10      10

input %>% 
  jst_add_total_pages()
#> # A tibble: 8 × 4
#>   first_page last_page page_range  n_pages
#>        <dbl>     <dbl> <chr>         <dbl>
#> 1         NA        NA <NA>             NA
#> 2          1        10 1 - 10           10
#> 3          1        10 <NA>             10
#> 4          1        NA <NA>             NA
#> 5          1        NA 1-10             10
#> 6         NA        NA 1, 5-10           7
#> 7         NA        NA 1-4, 5-10        10
#> 8         NA        NA 1-4, C5-C10      10

sample_article <- jst_get_article(jst_example("article_with_references.xml")) 

knitr::kable(sample_article)

sample_article %>% 
  jst_unify_journal_id() %>% 
  left_join(jst_get_journal_overview()) %>% 
  tidyr::gather(variable, value) %>% 
  knitr::kable()
#> Joining with `by = join_by(journal_id)`

sample_article %>% 
  pull(language)
#> [1] "eng"