--- title: "Control of tests" author: - "Mark Padgham" date: "`r Sys.Date()`" output: html_document: toc: true toc_float: true number_sections: false theme: flatly md_document: variant: markdown_github always_allow_html: true vignette: > %\VignetteIndexEntry{Control of tests} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, warning = TRUE, message = TRUE, width = 120, comment = "#>", fig.retina = 2, fig.path = "README-" ) ``` The [first vignette](https://docs.ropensci.org/autotest/articles/autotest.html) demonstrates the process of applying `autotest` at all stages of package development. This vignette provides additional information for those applying `autotest` to already developed packages, in particular through describing how tests can be selectively applied to a package. By default the [`autotest_package()` function](https://docs.ropensci.org/autotest/reference/autotest_package.html) tests an entire package, but testing can also be restricted to specified functions only. This vignette will demonstrate application to a few functions from the [`stats` package](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html), starting by loading the package. ```{r pkg-load-fakey, eval = FALSE, echo = TRUE} library (autotest) ``` ```{r pkg-lod, eval = TRUE, echo = FALSE, message = FALSE} devtools::load_all (".", export_all = FALSE) ``` ## 1. `.Rd` files, example code, and the autotest workflow To understand what `autotest` does, it is first necessary to understand a bit about the structure of documentation files for R package, which are contained in files called `".Rd"` files. Tests are constructed by parsing individual `.Rd` documentation files to extract the example code, identifying parameters passed to the specified functions, and mutating those parameters. The general procedure can be illustrated by examining a specific function, for which we now choose the [`cor` function](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html), because of its relative simplicity. The following lines extract the documentation for the `cor` function, a `.html` version of which can be seen by clicking on the link above. Note that that web page reveals the name of the `.Rd` file to be "cor" (in the upper left corner), meaning that the name of the `.Rd` file is `"cor.Rd"`. The following lines extract that content, first by loading the entire `.Rd` database for the [`stats` package](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html). ```{r cor-rd} rd <- tools::Rd_db (package = "stats") cor_rd <- rd [[grep ("^cor\\.Rd", names (rd))]] ``` The database itself is a list, with each entry holding the contents of one `.Rd` file in an object of class `r class(cor_rd)`, which is essentially a large nested list of components corresponding to the various `.Rd` tags such as `\arguments`, `\details`, and `\value`. An internal function from the [`tools` package](https://stat.ethz.ch/R-manual/R-devel/library/tools/html/00Index.html) can be used to extract individual components (using the `:::` notation to access internal functions). For example, a single `.Rd` file often describes the functionality of several functions, each of which is identified by specifying the function name as an `"alias"`. The aliases for the `"cor.Rd"` file are: ```{r cor-aliases} tools:::.Rd_get_metadata (cor_rd, "alias") ``` This one file thus contains documentation for those four functions. Example code can be extracted with a dedicated function from the [`tools` package](https://stat.ethz.ch/R-manual/R-devel/library/tools/html/Rd2HTML.html): ```{r Rd2ex-dummy, echo = TRUE, eval = FALSE} tools::Rd2ex (cor_rd) ``` ```{r Rd2ex, echo = FALSE, eval = TRUE} tools::Rd2ex (cor_rd) ``` This is the entire content of the `\examples` portion of `"cor.Rd"`, as can be confirmed by comparing with the [online version](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html). ## 2. Internal structure of the `autotest` workflow For each `.Rd` file in a package, `autotest` tests the code given in the example section according to the following general steps: 1. Extract example lines from the `.Rd` file, as demonstrated above; 2. Identify all function `aliases` described by that file; 3. Identify all points at which those functions are called; 4. Identify all objects passed to those values, including values, classes, attributes, and other properties. 5. Identify any other parameters not explicitly passed in example code, but defined via default value; 6. Mutate the values of all parameters according to the kinds of test described in [`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html). Calling `autotest_package(..., test = FALSE)` implements the first 5 of those 6 steps, and returns data on all possible mutations of each parameter, while setting `test = TRUE` actually passes the mutated parameters to the specified functions, and returns reports on any unexpected behaviour. ## 3. `autotest`-ing the `stats::cov` function The preceding sections describe how `autotest` actually works, while the present section demonstrates how the package is typically used in practice. As demonstrated in the [`README`](https://docs.ropensci.org/autotest/), information on all tests implemented within the package can be obtained by calling the [`autotest_types()` function](https://docs.ropensci.org/autotest/reference/autotest_types.html). The main function for testing package is [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html). The nominated package can be either an installed package, or the full path to a local directory containing a package's source code. By default all `.Rd` files of a package are tested, with restriction to specified functions possible either by nominating functions to exclude from testing (via the `exclude` parameter), or functions to include (via the `functions` parameter). The `functions` parameter is intended to enable testing only of specified functions, while the `exclude` parameter is intended to enable testing of all functions except those specified with this parameter. Specifying values for both of these parameters is not generally recommended. ### 3.1 Listing tests without conducting them The following demonstrates the results of `autotest`-ing the [`cor` function](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html) of the `stats` package, noting that the default call uses `test = FALSE`, and so returns details of all tests without actually implementing them (and for this reason we name the object `xf` for "false"): ```{r at-cor-notest-dummy, echo = TRUE, eval = FALSE} xf <- autotest_package (package = "stats", functions = "cor") print (xf) ``` ```{r print-x, echo = FALSE, collapse = TRUE} xf <- tibble::tibble ( type = rep ("dummy", 15L), test_name = c ( rep ("single_char_case", 2L), "single_par_as_length_2", "return_successful", "return_val_described", "return_desc_include_class", "return_class_matches_desc", rep (c ("par_is_documented", "par_matches_doc"), 4) ), fn_name = rep ("cor", 15L), parameter = c ( rep ("use", 3L), rep ("(return object)", 4L), rep ("x", 2L), rep ("use", 2L), rep ("method", 2L), rep ("y", 2L) ), parameter_type = c ( rep ("single_character", 3L), rep ("(return object)", 4L), rep (NA_character_, 8L) ), operation = c ( "lower-case character parameter", "upper-case character parameter", "Length 2 vector for length 1 parameter", "Check that function successfully returns an object", "Check that description has return value", "Check whether description of return value specifies class", "Compare class of return value with description", rep (c ("Check that parameter is documented", "Check that documentation matches class of input parameter"), 4L) ), content = c ( rep ("(Should yield same result)", 2L), "Should trigger message, warning, or error", rep (NA_character_, 12L) ), test = rep (TRUE, 15L) ) print (xf) ``` The object returned from `autotest_package()` is a simple [`tibble`](https://tibble.tidyverse.org), with each row detailing one test which would be applied to the each of the listed functions and parameters. Because no tests were conducted, all tests will generally have a `type` of `"dummy"`. In this case, however, we see the following: ```{r test-types} table (xf$type) ``` In addition to the `r length (which (xf$type == "dummy"))` dummy tests, the function also returns `r length (which (xf$type == "warning"))` warnings, the corresponding rows of which are: ```{r test-warnings} xf [xf$type != "dummy", c ("fn_name", "parameter", "operation", "content")] ``` Although the `auotest` package is primarily intended to apply mutation tests to all parameters of all functions of a package, doing so requires identifying parameter types and classes through parsing example code. Any parameters of a function which are neither demonstrated within example code, nor given default values can not be tested, because it is not possible to determine their expected types. The above result reveals that neither the `use` parameter of the `var` function, nor the `y` parameter of `cov`, are demonstrated in example code, triggering a warning that these parameter are unable to be tested. ### 3.2 Conducting tests The `r length (which (xf$type == "dummy"))` tests listed above with `type == "dummy"` can then be applied to all nominated functions and parameters by calling the same function with `test = TRUE`. Doing so yields the following results (as an object names `xt` for "true"): ```{r at-cor-test-dummy, echo = TRUE, eval = FALSE} xt <- autotest_package (package = "stats", functions = "cor", test = TRUE) print (xt) ``` ```{r print-xt, echo = FALSE, collapse = TRUE} xt1 <- tibble::tibble ( type = c (rep ("warning", 4L), "diagnostic"), test_name = c (rep ("par_matches_docs", 4L), "single_char_case"), fn_name = rep ("cor", 5L), parameter = c (rep (c ("x", "y"), 2L), "use"), parameter_type = c (rep (NA_character_, 4L), "single character"), operation = c ( rep ("Check that documentation matches class of input parameter", 4L), "upper-case character parameter"), content = c ( rep ("Parameter documentation does not describe class of [integer]", 2L), rep ("Parameter documentation does not describe class of [dist]", 2L), "is case dependent" ), test = rep (TRUE, 5L) ) ``` And the `r length (which (xf$type == "dummy"))` tests yielded `r nrow (xt1)` unexpected responses. The best way to understand these results is to examine the object in detail, typically through `edit(xt)`, or equivalently in RStudio, clicking on the listed object. The different types of tests which produced unexpected responses were: ```{r xt-operations} table (xt1$operation) ``` Two of those reflect the previous results regarding parameters unable to be tested, while the remainder come from only two types of tests. Information on the precise results is contained in the `content` column, although in this case it is fairly straightforward to see that the operation "upper case character parameter" arises because the `use` and `method` parameters of the `cor` and `cov` functions are case-dependent, and are only accepted in lower case form. The other operation is the conversion of vectors to list-column format, as described in the [first vignette](https://docs.ropensci.org/autotest/articles/autotest.html). ### 3.4 Controlling which tests are conducted The `test` parameter of the [`autotest_package()` function](https://docs.ropensci.org/autotest/reference/autotest_package.html) can be used to control whether all tests are conducted or not. Finer-level control over tests can be achieved by specifying the `test_data` parameter. This parameter must be an object of class `autotest_package`, as returned by either the [`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html)] or [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html) functions. The former of these is the function which specifies all unique tests, and so returns a relatively small `tibble` of `r nrow(autotest_types())` rows. The following lines demonstrate how to switch off the list-column test for all functions and parameters: ```{r test_data1-dummy, echo = TRUE, eval = FALSE} types <- autotest_types() types$test [grep ("list_col", types$test_name)] <- FALSE xt2 <- autotest_package (package = "stats", functions = "cor", test = TRUE, test_data = types) print (xt2) ``` ```{r print-xt2, echo = FALSE, eval = TRUE} xt2 <- tibble::tibble ( type = c (rep ("warning", 4L), "diagnostic"), test_name = c (rep ("par_matches_docs", 4L), "single_char_case"), fn_name = rep ("cor", 5L), parameter = c (rep (c ("x", "y"), 2L), "use"), parameter_type = c (rep (NA_character_, 4L), "single character"), operation = c ( rep ("Check that documentation matches class of input parameter", 4L), "upper-case character parameter"), content = c ( rep ("Parameter documentation does not describe class of [integer]", 2L), rep ("Parameter documentation does not describe class of [dist]", 2L), "is case dependent" ), test = rep (TRUE, 5L) ) print (xt2) ``` The result now has four rows with `test == FALSE`, and `type == "no_test"`, indicating that these tests were not actually conducted. That also makes apparent the role of these `test` flags. When initially calling [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html) with default `test = FALSE`, the result contains a `test` column in which all values are `TRUE`. Although potentially perplexing at first, this value must be understood in relation to the `type` column. A `type` of `"dummy"` indicates that a test has not been conducted, in which case `test = TRUE` is a control flag used to determine what would be conducted if these data were submitted as the `test_data` parameter. For all `type` values other than `"dummy"`, the `test` column specifies whether or not each test was actually conducted. The preceding example showed how the results of [`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html) can be used to control which tests are implemented for an entire package. Finer-scale control can be achieved by modifying individual rows of the full table returned by [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html). The following code demonstrates by showing how list-column tests can be switched off only for particular functions, starting again with the `xf` data of dummy tests generated above. ```{r test_data2-fakey, echo = TRUE, eval = FALSE} xf <- autotest_package (package = "stats", functions = "cor") xf$test [grepl ("list_col", xf$test_name) & xf$fn_name == "var"] <- FALSE xt3 <- autotest_package (package = "stats", functions = "cor", test = TRUE, test_data = xf) print (xt3) ``` ```{r print-xt3, echo = FALSE, eval = TRUE} xt3 <- tibble::tibble ( type = c (rep ("warning", 4L), "diagnostic"), test_name = c (rep ("par_matches_docs", 4L), "single_char_case"), fn_name = rep ("cor", 5L), parameter = c (rep (c ("x", "y"), 2L), "use"), parameter_type = c (rep (NA_character_, 4L), "single character"), operation = c ( rep ("Check that documentation matches class of input parameter", 4L), "upper-case character parameter"), content = c ( rep ("Parameter documentation does not describe class of [integer]", 2L), rep ("Parameter documentation does not describe class of [dist]", 2L), "is case dependent" ), test = rep (TRUE, 5L) ) print (xt3) ``` These procedures illustrate the three successively finer levels of control over tests, by switching them off for: 1. Entire packages; 2. Specified functions only; or 3. Specific parameters of particular functions only. ## 4. `autotest`-ing your package `autotest` can be very easily incorporated in your package's `tests/` directory via to simple [`testthat`](https://testthat.r-lib.org) expectations: - `expect_autotest_no_testdata`, which will expect `autotest_package` to work on your package with default values including no additional `test_data` specifying tests which are not to be run; or - `expect_autotest_testdata`, to be used when specific tests are switched off. Using these requires adding `autotest` to the `Suggests` list of your package's `DESCRIPTION` file, along with `testthat`. Note that the use of testing frameworks other than [`testthat`](https://testthat.r-lib.org) is possible through writing custom expectations for the output of [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html), but that is not considered here. To use these expectations, you must first decide which, if any, tests you judge to be not applicable to your package, and switch them off following the procedure described above (that is, at the package level through modifying the `test` flag of the object returned from [`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html), or at finer function- or parameter-levels by modifying equivalent values in the object returned from [`autotest_package(..., test = FALSE)`](https://docs.ropensci.org/autotest/reference/autotest_package.html). These objects must then be passed as the `test_data` parameter to [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html). If you consider all tests to be applicable, then [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html) can be called without specifying this parameter. If you switch tests off via a `test_data` parameter, then the `expect_autotest` expectation requires you to append an additional column to the `test_data` object called `"note"` (case-insensitive), and include a note for each row which has `test = FALSE` explaining why those tests have been switched off. Lines in your test directory should look something like this: ```{r testthat-demo-testdata, collapse = TRUE, message = FALSE, echo = TRUE, eval = FALSE} library (testthat) # as called in your test suite # For example, to switch off vector-to-list-column tests: test_data <- autotest_types (notest = "vector_to_list_col") test_data$note <- "" test_data$note [test_data$test == "vector_to_list_col"] <- "These tests are not applicable because ..." expect_success (expect_autotest_testdata (test_data)) ``` This procedure of requiring an additional `"note"` column ensures that your own test suite will explicitly include all explanations of why you deem particular tests not to be applicable to your package. In contrast, the following expectation should be used when `autotest_package()` passes with all tests are implemented, in which case no parameters need be passed to the expectation, and tests will confirm that no warnings or errors are generated. ```{r testthat-demo-notestdata, collapse = TRUE, echo = TRUE, eval = FALSE} expect_success (expect_autotest_no_testdata ()) ``` ### 4.2 Finer control over testing expectations The two expectations shown above call the [`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html) function internally, and assert that the results follow the expected pattern. There are also three additional [`testthat`](https://testthat.r-lib.org) expectations which can be applied to pre-generated `autotest` objects, to allow for finer control over testing expectations. These are: - `expect_autotest_no_err` to expect no errors in results from `autotest_package()`; - `expect_autotest_no_warn` to expect no warnings; and - `expect_autotest_notes` to expect tests which have been switched off to have an additional `"note"` column explaining why. These tests are demonstrated in one of the testing files used in this package, which the following lines recreate to demonstrate the general process. The first two expectations are that an object be free from both warnings and errors. The tests implemented here are applied to the [`stats::cov()` function](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/cor.html), which actually triggers warnings because two parameters do not have their usage demonstrated in the example code. The tests therefore `expect_failure()`, when they generally should `expect_success()` throughout. ```{r testthat-no-err-warn-fakey, collapse = TRUE, echo = TRUE, eval = FALSE} library (testthat) # as called in your test suite # For example, to switch off vector-to-list-column tests: test_data <- autotest_types (notest = "vector_to_list_col") x <- autotest_package (package = "stats", functions = "cov", test = TRUE, test_data = test_data) expect_success (expect_autotest_no_err (x)) expect_failure (expect_autotest_no_warn (x)) # should expect_success!! ``` ```{r testthat-no-err-warn, collapse = TRUE, echo = FALSE, eval = TRUE} library (testthat) # as called in your test suite # switch off vector-to-list-column tests: test_data <- autotest_types (notest = "vector_to_list_col") x <- tibble::tibble ( type = c (rep ("warning", 2L), rep ("diagnostic", 20L)), test_name = c ( rep ("par_is_demonstrated", 2L), "single_int_for_logical", rep ("single_char_case", 2L), rep ("vector_custom_class", 2L), rep ("single_char_case", 15L) ), fn_name = c ("var", "cov", "var", rep ("cor", 19L)), parameter = rep (c ( "use", "y", "na.rm", "use", "method", "x", "x", "method", "use", "use", "use"), 2L), parameter_type = c ( rep (NA_character_, 2L), "single logical", rep ("single character", 2L), rep ("vector", 2L), rep ("single character", 15L) ), operation = c ( rep ("Check that parameter usage is demonstrated", 2L), "Substitute integer values for logical parameter", rep ("upper-case character parameter", 2L), rep ("Custom class definitions for vector input", 2L), rep ("upper-case character parameter", 4L), "Custom class definitions for vector input", rep ("upper-case character parameter", 6L), "Custom class definitions for vector input", rep ("upper-case character parameter", 2L), "Custom class definitions for vector input" ), content = c ( rep ("Examples do not demonstrate usage of this parameter", 2L), "(Function call should still work unless explicitly prevented)", rep ("is case dependent", 2L), rep ("Function [cor] errors on vector columns with different classes", 2L), rep ("is case dependent", 4L), "Function [cov] errors on vector columns with different classes", rep ("is case dependent", 6L), "Function [cor] errors on vector columns with different classes", rep ("is case dependent", 2L), "Function [cor] errors on vector columns with different classes" ), test = rep (TRUE, 22L) ) expect_success (expect_autotest_no_err (x)) expect_failure (expect_autotest_no_warn (x)) # should expect_success!! ``` The test files then affirms that simply passing the object, `x`, which has tests flagged as `type == "no_test"`, yet without explaining why in an additional `"note"` column, should cause `expect_autotest()` to fail. The following line, removing the logical `testthat` expectation, demonstrates: ```{r testthat-demo-fail-fakey, echo = TRUE, eval = FALSE} expect_autotest_notes (x) ``` As demonstrated above, these `expect_autotest_...` calls should always be wrapped in a direct [`testhat`](https://testthat.r-lib.org) expectation of [`expect_success()`](https://testthat.r-lib.org/reference/expect_success.html). To achieve success in that case, we need to append an additional `"note"` column containing explanations of why each test has been switched off: ```{r testthat-demo-success} x$note <- "" x [grep ("vector_to_list", x$test_name), "note"] <- "these tests have been switched off because ..." expect_success (expect_autotest_notes (x)) ``` In general, using `autotest` in a package's test suite should be as simple as adding `autotest` to `Suggests`, and wrapping either `expect_autotest_no_testdata` or `expect_autotest_testdata` in an `expect_success` call.