abitools

Fingerprint the ABI of installed R packages in order to detect when dependents need to be rebuilt

The purpose is to keep track of ABI imports and exports of any package using S4 or C/C++ such that we can detect (breaking) ABI changes and determine which reverse dependencies should be rebuilt.

Introduction

A binary R package can depend on the binary interface (ABI) of its dependencies in three ways:

abitools computes fingerprints of exactly these parts, to pinpoint which ABIs have changed and therefore which dependents need to be rebuilt. The exports of a package are hashes of the S4 classes, S4 generics, C functions, and C headers that it provides. The imports of a package are the exports of its dependencies that it was built against. Both are read from the installed package: the class definitions from its lazy-load database, the names of the packages and C functions it calls from the strings in its shared library, and the signatures of C functions from its headers, parsed with libclang.

abitools can be used in two ways:

  1. Review a release. Compare the exports of a new version of a package with those of the previous version, to see which classes and C functions changed their ABI or were removed. This tells authors whether a release may require rebuilding dependents.
  2. Find stale dependents. Compare the current exports of a package with the imports that its dependents recorded when they were built. A dependent needs to be rebuilt if it imports a class or C function that has changed or was removed since. This is what package repositories need.

Usage

abi_list() returns the exports and imports of an installed package. Run it right after the package was built, in the library that it was built with.

abi_list("minqa")
#> $exports
#> [1] key  hash
#> <0 rows> (or 0-length row.names)
#>
#> $imports
#>                              key             hash
#> 1 Rcpp::Rcpp_precious_preserve() 33a97ea62b293da6
#> 2   Rcpp::Rcpp_precious_remove() 63d26947ba7e1377
#> 3               Rcpp::demangle() ae88df8ffb66ca8d
#> 4   Rcpp::rcpp_get_stack_trace() c684a9bbe15cdc52
#> 5   Rcpp::rcpp_set_stack_trace() 33a97ea62b293da6

Keys are <package>::<class> for S4 classes, <package>::generic:<name> for S4 generics, <package>::<function>() for C functions, and <package>/include for headers. An import is a copy of the export it was built against, so both are flat data frames with the same columns, which jsonlite::toJSON() turns into lists of entries that are easy to store and query:

{
  "exports": [{"key": "Rsamtools::ApplyPileupsParam", "hash": "ca6334264a0e4b9c"}, ...],
  "imports": [{"key": "S4Vectors::Vector", "hash": "09cab0bd77ae353b"}, ...]
}

abi_check() compares imports with exports, and returns what changed or was removed (no rows means nothing changed).

Review a release

Pass the exports of the previous version as imports:

abi_check(list(imports = old$exports), exports = new$exports)
#>   package                            reason
#>    Matrix                C headers changed
#>    Matrix C function cholmod_nnz() changed

Here, C headers changed only means that some file in the include directory changed; the C functions show whether this affects the ABI.

Find stale dependents

Pass the stored abi_list() of a dependent. The current exports of its dependencies are computed from the installed packages, or can be passed as exports (e.g. from a repository API):

abi_check(x)
#>     package                 reason
#>   S4Vectors S4 class Vector changed

Other functions

abi_deps() shows why a dependency is critical: S4 class, S4 generic, C API, linked library, headers (header-only or static libraries, compiled into the binary, so not critical), or copied code (see limitations). stale_classes() checks the S4 classes of a package in a local library against the current definitions of their superclasses, without stored imports.

See rebuild-procedure.md for using this in a package repository.

What is hashed

The exact formats and rules are specified in SPEC.md, such that other implementations can reproduce the hashes. The specification is a draft and subject to change: hashes may differ between versions of abitools.

S4 classes: the class name, the virtual flag, and the slots and superclasses (including inherited ones) with their packages. A dependent imports the classes that it extends.

S4 generics: the name, the arguments and the arguments to dispatch on. A dependent imports the generics of other packages that it defines methods for.

C functions: every function that a package registers with R_RegisterCCallable(). R keeps no list of these, so abitools tries the identifiers that occur both in the headers and as strings in the shared library. A dependent imports the functions whose names appear as strings in its shared library (the names that it passes to R_GetCCallable()).

The hash describes the signature of the function, as parsed from the headers of the package by libclang (with the preprocessor, and the include directories of R and of the package's own LinkingTo dependencies): the canonical types of the return value and parameters, and for structs defined in these headers their size, alignment and field offsets. Names, qualifiers, formatting, and macros do not matter; a changed array size from a #define or a reordered field does. For example, for S4Vectors::_new_IntAE():

function(UInt64,UInt64,Int32)->*{24,8|0:UInt64;64:UInt64;128:*Int32;}

The signature is taken from the cast at a call site of R_GetCCallable() in the headers if there is one, else from a declaration. This covers the different ways in which packages declare a C API: plain declarations (cli, xts), macros as return type (nloptr), stub prefixes such as _fun (Bioconductor), _pkg_fun (Rcpp interfaces) or M_fun (Matrix), function pointers (libgeos, PreciseSums), and casts at call sites (Rcpp, Matrix, RNifti). Functions that are not found in the headers are not exported.

Headers that include a system library (such as FreeType in systemfonts) only parse with its include directory. Pass the -I flags that the package was built with as flags; install_flags() takes them from the log of R CMD INSTALL. Without them, the functions that do not use types from the library are still found, as clang keeps parsing after an error, and abitools skips only the declarations that were affected. Set options(abitools.verbose = TRUE) to see the errors.

Headers: a hash of all files in the include directory, used for linked library dependencies.

Validation

Installation

abitools requires libclang: apt-get install libclang-dev (Debian, Ubuntu), dnf install clang-devel (Fedora), or brew install llvm (macOS). Set LLVM_CONFIG to choose an LLVM installation. There is no Windows build; the fingerprints of one platform are enough for a repository (see limitations).

Limitations