--- title: "Automating File Import" author: "Thomas Klebel" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Automating File Import} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ```{r} library(jstor) library(purrr) library(stringr) ``` # Intro The `find_*` functions from `jstor` all work on a single file. Data from DfR however contains many single files, from up to 25,000 when using the self-service functions, up to several hundreds of thousands of files when requesting a large dataset via an agreement. Currently `jstor` offers two implementations to import many files: `jst_import_zip()` and `jst_import()`. The first one lets you import data directly from zip archives, the second works for file paths, so you need to unzip the archive first. I will first introduce `jst_import_zip()` and discuss the approach of with `jst_import()` afterwards. # Importing data directly from zip-archives Unpacking and working with many files directly is unpractical for at least three reasons: 1. If you unzip the archive, the single files will occupy a lot more disk space than the single archive. 2. Before you can import the files, you need to list them via `list.files` or `system("find...")` on UNIX. Depending on the size of your data, this can take some time. 3. There might be different types of data in your sample (journal articles, book chapters, etc.). You need to manage matching the paths to the appropriate functions, which is extra work. Importing directly from the zip archive simplifies all those tasks with a single function: `jst_import_zip()`. For the following demonstration, we will use a small sample archive that comes with the package. As a first step, we should take a look at the archive and its content. This is made easy with `jst_preview_zip()`: ```{r} jst_preview_zip(jst_example("pseudo_dfr.zip")) %>% knitr::kable() ``` We can see that we have a simple archive with three metadata files and one ngram file. Before we can use `jst_import_zip()`, we first need to think about, what we actually want to import: all of the data, or just parts? What kind of data do we want to extract from articles, books and pamphlets? We can specify this via `jst_define_import()`: ```{r} import_spec <- jst_define_import( article = c(jst_get_article, jst_get_authors), book = jst_get_book, ngram1 = jst_get_ngram ) ``` In this case, we want to import data on articles (standard metadata plus information on the authors), general data on books and unigrams (ngram1). This specification can then be used with `jst_import_zip()`: ```{r} # set up a temporary folder for output files tmp <- tempdir() # extract the content and write output to disk jst_import_zip(jst_example("pseudo_dfr.zip"), import_spec = import_spec, out_file = "my_test", out_path = tmp) ``` We can take a look at the files within `tmp` with `list.files()`: ```{r} list.files(tmp, pattern = "csv") ``` As you can see, `jst_import_zip()` automatically creates file names based on the string you supplied to `out_file` to delineate the different types of output. If we want to re-import the data for further analysis, we can either use functions like `readr::read_csv()`, or a small helper function from the package which determines and sets the column types correctly: ```{r} jst_re_import( file.path(tmp, "my_test_journal_article_jst_get_article-1.csv") ) %>% knitr::kable() ``` A side note on ngrams: For larger archives, importing all ngrams can take a very long time. It is thus advisable to only import ngrams for articles which you want to analyse, i.e. most likely a subset of the initial request. The function `jst_subset_ngrams()` helps you with this (see also the section on importing bigrams in the [case study](https://docs.ropensci.org/jstor/articles/analysing-n-grams.html#importing-bigrams)). ```{r, echo=FALSE} unlink(tmp) ``` ## Parallel processing Since the above process might take a while for larger archives (files have to be unpacked, read and parsed), there might be a benefit of executing the function in parallel. `jst_import_zip()` and `jst_import()` use `furrr` at their core, therefore adding parallelism is very easy. Just add the following lines at the beginning of your script, and the import will use all available cores: ```{r, eval=FALSE} library(future) plan(multisession) ``` You can find out more about futures by reading the package vignette: ```{r, eval=FALSE} vignette("future-1-overview", package = "future") ``` # Working with file paths The above approach of importing directly from zip archives is very convenient, but in some cases you might want to have more control over how data is imported. For example, if you run into problems because the output from any of the functions provided with the package looks corrupted, you could want to look at the original files. Alongside this, you could unzip the archive and work with the files directly, which I will demonstrate in the following sections. ## Unzip containers For simple purposes it might be sensible to unzip to a temporary directory (with `temp()` and `unzip()`) but for my research I simply extracted files to an external SSD, since I a) lacked disk space, b) needed to read them fast, and c) wanted to be able to look at specific files for debugging. ## List files There are many ways to generate a list of all files: `list.files()` or using `system()` in conjunction with `find` on unix-like systems are common options. For demonstration purposes I use files contained in `jstor` which can be accessed via `system.file`: ```{r, echo=TRUE} example_dir <- system.file("extdata", package = "jstor") ``` ## `list.files` ```{r} file_names_listed <- list.files(path = example_dir, full.names = TRUE, pattern = "*.xml") file_names_listed ``` ### `system` and `find` ```{r, eval=FALSE} file_names <- system(paste0("cd ", example_dir, "; find . -name '*.xml' -type f"), intern = TRUE) ``` ```{r, eval=FALSE} library(stringr) file_names_system <- file_names %>% str_replace("^\\.\\/", "") %>% str_c(example_dir, "/", .) file_names_system #> [1] "/Library/Frameworks/R.framework/Versions/3.4/Resources/library/jstor/extdata/sample_with_footnotes.xml" #> [2] "/Library/Frameworks/R.framework/Versions/3.4/Resources/library/jstor/extdata/sample_book.xml" #> [3] "/Library/Frameworks/R.framework/Versions/3.4/Resources/library/jstor/extdata/sample_with_references.xml" ``` In this case the two approaches give the same result. The main difference seems to be though, that `list.files` sorts the output, whereas `find` does not. For a large amount of files (200,000) this makes `list.files` slower, for smaller datasets the difference shouldn't make an impact. ## Batch import Once the file list is generated, we can apply any of the `jst_get_*`-functions to the list. A good and simple way for small to moderate amounts of files is to use `purrr::map_df()`: ```{r} # only work with journal articles article_paths <- file_names_listed %>% keep(str_detect, "with") article_paths %>% map_df(jst_get_article) %>% knitr::kable() ``` This works well if 1) there are no errors and 2) if there is only a moderate size of files. For larger numbers of files, `jst_import()` can streamline the process for you. This function works very similar to `jst_import_zip()`, the main difference being that it needs file paths as input and can only handle one type of output. ```{r} jst_import(article_paths, out_file = "my_second_test", .f = jst_get_article, out_path = tmp) ``` The result is again written to disk, as can be seen below: ```{r} list.files(tmp, pattern = "csv") ```