Package 'drake'

Description

drake is a pipeline toolkit (⁠https://github.com/pditommaso/awesome-pipeline⁠) and a scalable, R-focused solution for reproducibility and high-performance computing.

Author(s)

William Michael Landau [email protected]

References

⁠https://github.com/ropensci/drake⁠

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
library(drake)
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Build everything.
plot(my_plan) # fast call to vis_drake_graph()
make(my_plan) # Nothing is done because everything is already up to date.
reg2 = function(d) { # Change one of your functions.
  d$x3 = d$x^3
  lm(y ~ x3, data = d)
}
make(my_plan) # Only the pieces depending on reg2() get rebuilt.
# Write a flat text log file this time.
make(my_plan, cache_log_file = TRUE)
# Read/load from the cache.
readd(small)
loadd(large)
head(large)
}
# Dynamic branching
# Get the mean mpg for each cyl in the mtcars dataset.
plan <- drake_plan(
  raw = mtcars,
  group_index = raw$cyl,
  munged = target(raw[, c("mpg", "cyl")], dynamic = map(raw)),
  mean_mpg_by_cyl = target(
    data.frame(mpg = mean(munged$mpg), cyl = munged$cyl[1]),
    dynamic = group(munged, .by = group_index)
  )
)
make(plan)
readd(mean_mpg_by_cyl)
})

## End(Not run)

Row-bind together drake plans

Description

Combine drake plans together in a way that correctly fills in missing entries.

Usage

bind_plans(...)

Arguments

See Also

drake_plan(), make()

Examples

# You might need to refresh your data regularly (see ?triggers).
download_plan <- drake_plan(
  data = target(
    command = download_data(),
    trigger = "always"
  )
)
# But if the data don't change, the analyses don't need to change.
analysis_plan <- drake_plan(
  usage = get_usage_metrics(data),
  topline = scrape_topline_table(data)
)
your_plan <- bind_plans(download_plan, analysis_plan)
your_plan

See the time it took to build each target.

Description

Applies to targets in your plan, not imports or files.

Usage

build_times(
  ...,
  path = NULL,
  search = NULL,
  digits = 3,
  cache = drake::drake_cache(path = path),
  targets_only = NULL,
  verbose = NULL,
  jobs = 1,
  type = c("build", "command"),
  list = character(0)
)

Arguments

Details

Times for dynamic targets (⁠https://books.ropensci.org/drake/dynamic.html⁠) only reflect the time it takes to post-process the sub-targets (typically very fast) and exclude the time it takes to build the sub-targets themselves. Sub-targets build times are listed individually.

Value

A data frame of times, each from system.time().

See Also

predict_runtime()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
if (requireNamespace("lubridate")) {
# Show the build times for the mtcars example.
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Build all the targets.
print(build_times()) # Show how long it took to build each target.
}
}
})

## End(Not run)

List targets in the cache.

Description

Tip: read/load a cached item with readd() or loadd().

Usage

cached(
  ...,
  list = character(0),
  no_imported_objects = FALSE,
  path = NULL,
  search = NULL,
  cache = drake::drake_cache(path = path),
  verbose = NULL,
  namespace = NULL,
  jobs = 1,
  targets_only = TRUE
)

Arguments

Value

Either a named logical indicating whether the given targets or cached or a character vector listing all cached items, depending on whether any targets are specified.

See Also

cached_planned(), cached_unplanned(), readd(), loadd(), drake_plan(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
if (requireNamespace("lubridate")) {
load_mtcars_example() # Load drake's canonical example.
make(my_plan) # Run the project, build all the targets.
cached()
cached(targets_only = FALSE)
}
}
})

## End(Not run)

List targets in both the plan and the cache.

Description

Includes dynamic sub-targets as well. See examples for details.

Usage

cached_planned(
  plan,
  path = NULL,
  cache = drake::drake_cache(path = path),
  namespace = NULL,
  jobs = 1
)

Arguments

Value

A character vector of target and sub-target names.

See Also

cached(), cached_unplanned

Examples

## Not run: 
isolate_example("cache_planned() example", {
plan <- drake_plan(w = 1)
make(plan)
cached_planned(plan)
plan <- drake_plan(
  x = seq_len(2),
  y = target(x, dynamic = map(x))
)
cached_planned(plan)
make(plan)
cached_planned(plan)
cached()
})

## End(Not run)

List targets in the cache but not the plan.

Description

Includes dynamic sub-targets as well. See examples for details.

Usage

cached_unplanned(
  plan,
  path = NULL,
  cache = drake::drake_cache(path = path),
  namespace = NULL,
  jobs = 1
)

Arguments

Value

A character vector of target and sub-target names.

See Also

cached(), cached_planned

Examples

## Not run: 
isolate_example("cache_unplanned() example", {
plan <- drake_plan(w = 1)
make(plan)
cached_unplanned(plan)
plan <- drake_plan(
  x = seq_len(2),
  y = target(x, dynamic = map(x))
)
cached_unplanned(plan)
make(plan)
cached_unplanned(plan)
# cached_unplanned() helps clean superfluous targets.
cached()
clean(list = cached_unplanned(plan))
cached()
})

## End(Not run)

Cancel a target mid-build

Description

Cancel a target mid-build. Upon cancellation, drake halts the current target and moves to the next one. The target's previous value and metadata, if they exist, remain in the cache.

Usage

cancel(allow_missing = TRUE)

Arguments

Value

Nothing.

See Also

cancel_if

Examples

## Not run: 
isolate_example("cancel()", {
f <- function(x) {
  cancel()
  Sys.sleep(2) # Does not run.
}
g <- function(x) f(x)
plan <- drake_plan(y = g(1))
make(plan)
# Does not exist.
# readd(y)
})

## End(Not run)

Cancel a target mid-build under some condition

Description

Cancel a target mid-build if some logical condition is met. Upon cancellation, drake halts the current target and moves to the next one. The target's previous value and metadata, if they exist, remain in the cache.

Usage

cancel_if(condition, allow_missing = TRUE)

Arguments

Value

Nothing.

See Also

cancel

Examples

## Not run: 
isolate_example("cancel_if()", {
f <- function(x) {
  cancel_if(x > 1)
  Sys.sleep(2) # Does not run if x > 1.
}
g <- function(x) f(x)
plan <- drake_plan(y = g(2))
make(plan)
# Does not exist.
# readd(y)
})

## End(Not run)

Invalidate and deregister targets.

Description

Force targets to be out of date and remove target names from the data in the cache. Be careful and run which_clean() before clean(). That way, you know beforehand which targets will be compromised.

Usage

clean(
  ...,
  list = character(0),
  destroy = FALSE,
  path = NULL,
  search = NULL,
  cache = drake::drake_cache(path = path),
  verbose = NULL,
  jobs = NULL,
  force = FALSE,
  garbage_collection = FALSE,
  purge = FALSE
)

Arguments

Details

By default, clean() invalidates all targets, so be careful. clean() always:

1. Forces targets to be out of date so the next make() does not skip them.

2. Deregisters targets so loadd(your_target) and readd(your_target) no longer work.

By default, clean() does not actually remove the underlying data. Even old targets from the distant past are still in the cache and recoverable via drake_history() and make(recover = TRUE). To actually remove target data from the cache, as well as any file_out() files from any targets you are currently cleaning, run clean(garbage_collection = TRUE). Garbage collection is slow, but it reduces the storage burden of the cache.

Value

Invisibly return NULL.

See Also

which_clean(), drake_gc()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
# Show all registered targets in the cache.
cached()
# Deregister 'summ_regression1_large' and 'small' in the cache.
clean(summ_regression1_large, small)
# Those objects are no longer registered as targets.
cached()
# Rebuild the invalidated/outdated targets.
make(my_plan)
# Clean everything.
clean()
# But the data objects and files are not actually gone!
file.exists("report.md")
drake_history()
make(my_plan, recover = TRUE)
# You need garbage collection to actually remove the data
# and any file_out() files of any uncleaned targets.
clean(garbage_collection = TRUE)
drake_history()
make(my_plan, recover = TRUE)
}
})

## End(Not run)

Clean the mtcars example from drake_example("mtcars")

Description

This function deletes files. Use at your own risk. Destroys the ⁠.drake/⁠ cache and the report.Rmd file in the current working directory. Your working directory (getcwd()) must be the folder from which you first ran load_mtcars_example() and make(my_plan).

Usage

clean_mtcars_example()

Value

nothing

See Also

load_mtcars_example(), clean()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
# Populate your workspace and write 'report.Rmd'.
load_mtcars_example() # Get the code: drake_example("mtcars")
# Check the dependencies of an imported function.
deps_code(reg1)
# Check the dependencies of commands in the workflow plan.
deps_code(my_plan$command[1])
deps_code(my_plan$command[4])
# Plot the interactive network visualization of the workflow.
outdated(my_plan) # Which targets are out of date?
# Run the workflow to build all the targets in the plan.
make(my_plan)
outdated(my_plan) # Everything should be up to date.
# For the reg2() model on the small dataset,
# the p-value is so small that there may be an association
# between weight and fuel efficiency after all.
readd(coef_regression2_small)
# Clean up the example.
clean_mtcars_example()
}
})

## End(Not run)

Turn a script into a function.

Description

code_to_function() is a quick (and very dirty) way to retrofit drake to an existing script-based project. It parses individual ⁠\*.R/\*.RMD⁠ files into functions so they can be added into the drake workflow.

Usage

code_to_function(path, envir = parent.frame())

Arguments

Details

Most data science workflows consist of imperative scripts. drake, on the other hand, assumes you write functions. code_to_function() allows for pre-existing workflows to incorporate drake as a workflow management tool seamlessly for cases where re-factoring is unfeasible. So drake can monitor dependencies, the targets are passed as arguments of the dependent functions.

Value

A function to be input into the drake plan

See Also

file_in(), file_out(), knitr_in(), ignore(), no_deps(), code_to_plan(), plan_to_code(), plan_to_notebook()

Examples

## Not run: 
isolate_example("contain side effects", {
if (requireNamespace("ggplot2", quietly = TRUE)) {
# The `code_to_function()` function creates a function that makes it
# available for drake to process as part of the workflow.
# The main purpose is to allow pre-existing workflows to incorporate drake
# into the workflow seamlessly for cases where re-factoring is unfeasible.
#

script1 <- tempfile()
script2 <- tempfile()
script3 <- tempfile()
script4 <- tempfile()

writeLines(c(
  "data <- mtcars",
  "data$make <- do.call('c',",
  "lapply(strsplit(rownames(data), split=\" \"), `[`, 1))",
  "saveRDS(data, \"mtcars_alt.RDS\")"
 ),
  script1
)

writeLines(c(
  "data <- readRDS(\"mtcars_alt.RDS\")",
  "mtcars_lm <- lm(mpg~cyl+disp+vs+gear+make,data=data)",
  "saveRDS(mtcars_lm, \"mtcars_lm.RDS\")"
  ),
  script2
)
writeLines(c(
  "mtcars_lm <- readRDS(\"mtcars_lm.RDS\")",
  "lm_summary <- summary(mtcars_lm)",
  "saveRDS(lm_summary, \"mtcars_lm_summary.RDS\")"
  ),
  script3
)
writeLines(c(
  "data<-readRDS(\"mtcars_alt.RDS\")",
  "gg <- ggplot2::ggplot(data)+",
  "ggplot2::geom_point(ggplot2::aes(",
  "x=disp, y=mpg, shape=as.factor(vs), color=make))",
  "ggplot2::ggsave(\"mtcars_plot.png\", gg)"
 ),
  script4
)


do_munge <- code_to_function(script1)
do_analysis <- code_to_function(script2)
do_summarize <- code_to_function(script3)
do_vis <- code_to_function(script4)

plan <- drake_plan(
  munged   = do_munge(),
  analysis = do_analysis(munged),
  summary  = do_summarize(analysis),
  plot     = do_vis(munged)
 )

plan
# drake knows  "script1" is the first script to be evaluated and ran,
# because it has no dependencies on other code and a dependency of
# `analysis`. See for yourself:

make(plan)

# See the connections that the sourced scripts create:
if (requireNamespace("visNetwork", quietly = TRUE)) {
  vis_drake_graph(plan)
}
}
})

## End(Not run)

Turn an R script file or knitr / R Markdown report into a drake plan.

Description

code_to_plan(), plan_to_code(), and plan_to_notebook() together illustrate the relationships between drake plans, R scripts, and R Markdown documents.

Usage

code_to_plan(path)

Arguments

Details

This feature is easy to break, so there are some rules for your code file:

1. Stick to assigning a single expression to a single target at a time. For multi-line commands, please enclose the whole command in curly braces. Conversely, compound assignment is not supported (e.g. target_1 <- target_2 <- target_3 <- get_data()).

2. Once you assign an expression to a variable, do not modify the variable any more. The target/command binding should be permanent.

3. Keep it simple. Please use the assignment operators rather than assign() and similar functions.

See Also

drake_plan(), make(), plan_to_code(), plan_to_notebook()

Examples

plan <- drake_plan(
  raw_data = read_excel(file_in("raw_data.xlsx")),
  data = raw_data,
  hist = create_plot(data),
  fit = lm(Ozone ~ Temp + Wind, data)
)
file <- tempfile()
# Turn the plan into an R script a the given file path.
plan_to_code(plan, file)
# Here is what the script looks like.
cat(readLines(file), sep = "\n")
# Convert back to a drake plan.
code_to_plan(file)

List the dependencies of a function or command

Description

Functions are assumed to be imported, and language/text are assumed to be commands in a plan.

Usage

deps_code(x)

Arguments

Value

A data frame of the dependencies.

See Also

deps_target(), deps_knitr()

Examples

# Your workflow likely depends on functions in your workspace.
f <- function(x, y) {
  out <- x + y + g(x)
  saveRDS(out, "out.rds")
}
# Find the dependencies of f. These could be R objects/functions
# in your workspace or packages. Any file names or target names
# will be ignored.
deps_code(f)
# Define a workflow plan data frame that uses your function f().
my_plan <- drake_plan(
  x = 1 + some_object,
  my_target = x + readRDS(file_in("tracked_input_file.rds")),
  return_value = f(x, y, g(z + w))
)
# Get the dependencies of workflow plan commands.
# Here, the dependencies could be R functions/objects from your workspace
# or packages, imported files, or other targets in the workflow plan.
deps_code(my_plan$command[[1]])
deps_code(my_plan$command[[2]])
deps_code(my_plan$command[[3]])
# You can also supply expressions or text.
deps_code(quote(x + y + 123))
deps_code("x + y + 123")

Find the drake dependencies of a dynamic knitr report target.

Description

Dependencies in knitr reports are marked by loadd() and readd() in active code chunks.

Usage

deps_knitr(path)

Arguments

Value

A data frame of dependencies.

See Also

deps_code(), deps_target()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
load_mtcars_example() # Get the code with drake_example("mtcars").
deps_knitr("report.Rmd")
})

## End(Not run)

Find out why a target is out of date.

Description

The dependency profile can give you a hint as to why a target is out of date. It can tell you if

• the command changed (deps_profile() reports the hash of the command, not the command itself)

• at least one input file changed,

• at least one output file changed,

• or a non-file dependency changed. For this last part, the imports need to be up to date in the cache, which you can do with outdated() or make(skip_targets = TRUE).

• the pseudo-random number generator seed changed. Unfortunately, deps_profile() does not currently get more specific than that.

Usage

deps_profile(target, ..., character_only = FALSE, config = NULL)

Arguments

Value

A data frame of old and new values for each of the main triggers, along with an indication of which values changed since the last make().

See Also

diagnose(), deps_code(), make(), drake_config()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Load drake's canonical example.
make(my_plan) # Run the project, build the targets.
# Get some example dependency profiles of targets.
deps_profile(small, my_plan)
# Change a dependency.
simulate <- function(x) {}
# Update the in-memory imports in the cache
# so deps_profile can detect changes to them.
# Changes to targets are already cached.
make(my_plan, skip_targets = TRUE)
# The dependency hash changed.
deps_profile(small, my_plan)
}
})

## End(Not run)

List the dependencies of a target

Description

Intended for debugging and checking your project. The dependency structure of the components of your analysis decides which targets are built and when.

Usage

deps_target(target, ..., character_only = FALSE, config = NULL)

Arguments

Value

A data frame with the dependencies listed by type (globals, files, etc).

See Also

deps_code(), deps_knitr()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
load_mtcars_example() # Get the code with drake_example("mtcars").
deps_target(regression1_small, my_plan)
})

## End(Not run)

Get diagnostic metadata on a target.

Description

Diagnostics include errors, warnings, messages, runtimes, and other context/metadata from when a target was built or an import was processed. If your target's last build succeeded, then diagnose(your_target) has the most current information from that build. But if your target failed, then only diagnose(your_target)$error, diagnose(your_target)$warnings, and diagnose(your_target)$messages correspond to the failure, and all the other metadata correspond to the last build that completed without an error.

Usage

diagnose(
  target = NULL,
  character_only = FALSE,
  path = NULL,
  search = NULL,
  cache = drake::drake_cache(path = path),
  verbose = 1L
)

Arguments

Value

Either a character vector of target names or an object of class "error".

See Also

drake_failed(), drake_progress(), readd(), drake_plan(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
diagnose() # List all the targets with recorded error logs.
# Define a function doomed to failure.
f <- function() {
  stop("unusual error")
}
# Create a workflow plan doomed to failure.
bad_plan <- drake_plan(my_target = f())
# Running the project should generate an error
# when trying to build 'my_target'.
try(make(bad_plan), silent = FALSE)
drake_failed() # List the failed targets from the last make() (my_target).
# List targets that failed at one point or another
# over the course of the project (my_target).
# drake keeps all the error logs.
diagnose()
# Get the error log, an object of class "error".
error <- diagnose(my_target)$error # See also warnings and messages.
str(error) # See what's inside the error log.
error$calls # View the traceback. (See the rlang::trace_back() function).
})

## End(Not run)

Build/process a single target or import.

Description

Not valid for dynamic branching.

Usage

drake_build(
  target,
  ...,
  meta = NULL,
  character_only = FALSE,
  replace = FALSE,
  config = NULL
)

Arguments

Value

The value of the target right after it is built.

See Also

drake_debug()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
# This example is not really a user-side demonstration.
# It just walks through a dive into the internals.
# Populate your workspace and write 'report.Rmd'.
load_mtcars_example() # Get the code with drake_example("mtcars").
out <- drake_build(small, my_plan)
# Now includes `small`.
cached()
head(readd(small))
# `small` was invisibly returned.
head(out)
}
})

## End(Not run)

Get the cache of a drake project.

Description

make() saves the values of your targets so you rarely need to think about output files. By default, the cache is a hidden folder called ⁠.drake/⁠. You can also supply your own storr cache to the cache argument of make(). The drake_cache() function retrieves this cache.

Usage

drake_cache(path = NULL, verbose = NULL, console_log_file = NULL)

Arguments

Details

drake_cache() actually returns a decorated storr, an object that contains a storr (plus bells and whistles). To get the actual inner storr, use drake_cache()$storr. Most methods are delegated to the inner storr. Some methods and objects are new or overwritten. Here are the ones relevant to users.

• history: drake's history (which powers drake_history()) is a txtq. Access it with drake_cache()$history.

• import(): The import() method is a function that can import targets, function dependencies, etc. from one decorated storr to another. History is not imported. For that, you have to work with the history txtqs themselves, Arguments to import():

  • ... and list: specify targets to import just like with loadd(). Leave these blank to import everything.

  • from: the decorated storr from which to import targets.

  • jobs: number of local processes for parallel computing.

  • gc: TRUE or FALSE, whether to run garbage collection for memory after importing each target. Recommended, but slow.

• export(): Same as import(), except the from argument is replaced by to: the decorated storr where the targets end up.

Value

A drake/storr cache in a folder called ⁠.drake/⁠, if available. NULL otherwise.

See Also

new_cache(), drake_config()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
clean(destroy = TRUE)
# No cache is available.
drake_cache() # NULL
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
x <- drake_cache() # Now, there is a cache.
y <- storr::storr_rds(".drake") # Nearly equivalent.
# List the objects readable from the cache with readd().
x$list()
# drake_cache() actually returns a *decorated* storr.
# The *real* storr is inside.
drake_cache()$storr
}
# You can import and export targets to and from decorated storrs.
plan1 <- drake_plan(w = "w", x = "x")
plan2 <- drake_plan(a = "a", x = "x2")
cache1 <- new_cache("cache1")
cache2 <- new_cache("cache2")
make(plan1, cache = cache1)
make(plan2, cache = cache2)
cache1$import(cache2, a)
cache1$get("a")
cache1$get("x")
cache1$import(cache2)
cache1$get("x")
# With txtq >= 0.1.6.9002, you can import history from one cache into
# another.
# nolint start
# drake_history(cache = cache1)
# cache1$history$import(cache2$history)
# drake_history(cache = cache1)
# nolint end
})

## End(Not run)

Get the state of the cache.

Description

Get the fingerprints of all the targets in a data frame. This functionality is like make(..., cache_log_file = TRUE), but separated and more customizable. Hopefully, this functionality is a step toward better data versioning tools.

Usage

drake_cache_log(
  path = NULL,
  search = NULL,
  cache = drake::drake_cache(path = path),
  verbose = 1L,
  jobs = 1,
  targets_only = FALSE
)

Arguments

Details

A hash is a fingerprint of an object's value. Together, the hash keys of all your targets and imports represent the state of your project. Use drake_cache_log() to generate a data frame with the hash keys of all the targets and imports stored in your cache. This function is particularly useful if you are storing your drake project in a version control repository. The cache has a lot of tiny files, so you should not put it under version control. Instead, save the output of drake_cache_log() as a text file after each make(), and put the text file under version control. That way, you have a changelog of your project's results. See the examples below for details. Depending on your project's history, the targets may be different than the ones in your workflow plan data frame. Also, the keys depend on the hash algorithm of your cache. To define your own hash algorithm, you can create your own storr cache and give it a hash algorithm (e.g. storr_rds(hash_algorithm = "murmur32"))

Value

Data frame of the hash keys of the targets and imports in the cache

See Also

cached(), drake_cache()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
# Load drake's canonical example.
load_mtcars_example() # Get the code with drake_example()
# Run the project, build all the targets.
make(my_plan)
# Get a data frame of all the hash keys.
# If you want a changelog, be sure to do this after every make().
cache_log <- drake_cache_log()
head(cache_log)
# Suppress partial arg match warnings.
suppressWarnings(
  # Save the hash log as a flat text file.
  write.table(
    x = cache_log,
    file = "drake_cache.log",
    quote = FALSE,
    row.names = FALSE
  )
)
# At this point, put drake_cache.log under version control
# (e.g. with 'git add drake_cache.log') alongside your code.
# Now, every time you run your project, your commit history
# of hash_lot.txt is a changelog of the project's results.
# It shows which targets and imports changed on every commit.
# It is extremely difficult to track your results this way
# by putting the raw '.drake/' cache itself under version control.
}
})

## End(Not run)

List cancelled targets.

Description

List the targets that were cancelled in the current or previous call to make() using cancel() or cancel_if().

Usage

drake_cancelled(cache = drake::drake_cache(path = path), path = NULL)

Arguments

Value

A character vector of target names.

See Also

drake_running(), drake_failed(), make()

Examples

## Not run: 
isolate_example("contain side effects", {
plan <- drake_plan(x = 1, y = cancel_if(x > 0))
make(plan)
drake_cancelled()
})

## End(Not run)

Ending of _drake.R for r_make() and friends

Description

Call this function inside the ⁠_drake.R⁠ script for r_make() and friends. All non-deprecated function arguments are the same between make() and drake_config().

Usage

drake_config(
  plan,
  targets = NULL,
  envir = parent.frame(),
  verbose = 1L,
  hook = NULL,
  cache = drake::drake_cache(),
  fetch_cache = NULL,
  parallelism = "loop",
  jobs = 1L,
  jobs_preprocess = 1L,
  packages = rev(.packages()),
  lib_loc = NULL,
  prework = character(0),
  prepend = NULL,
  command = NULL,
  args = NULL,
  recipe_command = NULL,
  timeout = NULL,
  cpu = Inf,
  elapsed = Inf,
  retries = 0,
  force = FALSE,
  log_progress = TRUE,
  graph = NULL,
  trigger = drake::trigger(),
  skip_targets = FALSE,
  skip_imports = FALSE,
  skip_safety_checks = FALSE,
  lazy_load = "eager",
  session_info = NULL,
  cache_log_file = NULL,
  seed = NULL,
  caching = c("main", "master", "worker"),
  keep_going = FALSE,
  session = NULL,
  pruning_strategy = NULL,
  makefile_path = NULL,
  console_log_file = NULL,
  ensure_workers = NULL,
  garbage_collection = FALSE,
  template = list(),
  sleep = function(i) 0.01,
  hasty_build = NULL,
  memory_strategy = "speed",
  spec = NULL,
  layout = NULL,
  lock_envir = NULL,
  history = TRUE,
  recover = FALSE,
  recoverable = TRUE,
  curl_handles = list(),
  max_expand = NULL,
  log_build_times = TRUE,
  format = NULL,
  lock_cache = TRUE,
  log_make = NULL,
  log_worker = FALSE
)

Arguments

Details

In drake, make() has two stages:

1. Configure a workflow to your environment and plan.

2. Build targets. The drake_config() function just does step (1), which is a common requirement for not only make(), but also utility functions like vis_drake_graph() and outdated(). That is why drake_config() is a requirement for the ⁠_drake.R⁠ script, which powers r_make(), r_outdated(), r_vis_drake_graph(), etc.

Value

A configured drake workflow.

Recovery

make(recover = TRUE, recoverable = TRUE) powers automated data recovery. The default of recover is FALSE because targets recovered from the distant past may have been generated with earlier versions of R and earlier package environments that no longer exist.

How it works: if recover is TRUE, drake tries to salvage old target values from the cache instead of running commands from the plan. A target is recoverable if

1. There is an old value somewhere in the cache that shares the command, dependencies, etc. of the target about to be built.

2. The old value was generated with make(recoverable = TRUE).

If both conditions are met, drake will

1. Assign the most recently-generated admissible data to the target, and

2. skip the target's command.

See Also

make(), drake_plan(), vis_drake_graph()

Examples

## Not run: 
isolate_example("quarantine side effects", {
if (requireNamespace("knitr", quietly = TRUE)) {
writeLines(
  c(
    "library(drake)",
    "load_mtcars_example()",
    "drake_config(my_plan, targets = c(\"small\", \"large\"))"
  ),
  "_drake.R" # default value of the `source` argument
)
cat(readLines("_drake.R"), sep = "\n")
r_outdated()
r_make()
r_outdated()
}
})

## End(Not run)

Run a single target's command in debug mode.'

Description

Not valid for dynamic branching. drake_debug() loads a target's dependencies and then runs its command in debug mode (see browser(), debug(), and debugonce()). This function does not store the target's value in the cache (see ⁠https://github.com/ropensci/drake/issues/587⁠).

Usage

drake_debug(
  target = NULL,
  ...,
  character_only = FALSE,
  replace = FALSE,
  verbose = TRUE,
  config = NULL
)

Arguments

Value

The value of the target right after it is built.

See Also

drake_build()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
# This example is not really a user-side demonstration.
# It just walks through a dive into the internals.
# Populate your workspace and write 'report.Rmd'.
load_mtcars_example() # Get the code with drake_example("mtcars").
# out <- drake_debug(small, my_plan)
# `small` was invisibly returned.
# head(out)
}
})

## End(Not run)

List done targets.

Description

List the targets that completed in the current or previous call to make().

Usage

drake_done(cache = drake::drake_cache(path = path), path = NULL)

Arguments

Value

A character vector of target names.

See Also

drake_running(), drake_failed(), drake_cancelled(), drake_progress(), make()

Examples

## Not run: 
isolate_example("contain side effects", {
plan <- drake_plan(x = 1, y = x)
make(plan)
drake_done()
})

## End(Not run)

Get the environment where drake builds targets

Description

Call this function inside the commands in your plan to get the environment where drake builds targets. Advanced users can use it to strategically remove targets from memory while make() is running.

Usage

drake_envir(which = c("targets", "dynamic", "subtargets", "imports"))

Arguments

Details

drake manages in-memory targets in 4 environments: one with sub-targets, one with whole dynamic targets, one with static targets, and one with imported global objects and functions. This last environment is usually the environment from which you call make(). Select the appropriate environment for your use case with the which argument of drake_envir().

Value

The environment where drake builds targets.

Keywords

drake_plan() understands special keyword functions for your commands. With the exception of target(), each one is a proper function with its own help file.

• target(): give the target more than just a command. Using target(), you can apply a transformation (examples: ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠), # nolint supply a trigger (⁠https://books.ropensci.org/drake/triggers.html⁠), # nolint or set any number of custom columns.

• file_in(): declare an input file dependency.

• file_out(): declare an output file to be produced when the target is built.

• knitr_in(): declare a knitr file dependency such as an R Markdown (⁠*.Rmd⁠) or R LaTeX (⁠*.Rnw⁠) file.

• ignore(): force drake to entirely ignore a piece of code: do not track it for changes and do not analyze it for dependencies.

• no_deps(): tell drake to not track the dependencies of a piece of code. drake still tracks the code itself for changes.

• id_chr(): Get the name of the current target.

• drake_envir(): get the environment where drake builds targets. Intended for advanced custom memory management.

See Also

from_plan()

Examples

## Not run: 
isolate_example("contain side effects", {
plan <- drake_plan(
  large_data_1 = sample.int(1e4),
  large_data_2 = sample.int(1e4),
  subset = c(large_data_1[seq_len(10)], large_data_2[seq_len(10)]),
  summary = {
    print(ls(envir = parent.env(drake_envir())))
    # We don't need the large_data_* targets in memory anymore.
    rm(large_data_1, large_data_2, envir = drake_envir("targets"))
    print(ls(envir = drake_envir("targets")))
    mean(subset)
  }
)
make(plan, cache = storr::storr_environment(), session_info = FALSE)
})

## End(Not run)

Download the files of an example drake project.

Description

The drake_example() function downloads a folder from ⁠https://github.com/wlandau/drake-examples⁠. By default, it creates a new folder with the example name in your current working directory. After the files are written, have a look at the enclosed README file. Other instructions are available in the files at ⁠https://github.com/wlandau/drake-examples⁠.

Usage

drake_example(
  example = "main",
  to = getwd(),
  destination = NULL,
  overwrite = FALSE,
  quiet = TRUE
)

Arguments

Value

NULL

See Also

drake_examples(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (requireNamespace("downloader")) {
drake_examples() # List all the drake examples.
# Sets up the same example from load_mtcars_example()
drake_example("mtcars")
# Sets up the SLURM example.
drake_example("slurm")
}
})

## End(Not run)

List the names of all the drake examples.

Description

You can find the code files of the examples at ⁠https://github.com/wlandau/drake-examples⁠. The drake_examples() function downloads the list of examples from ⁠https://wlandau.github.io/drake-examples/examples.md⁠, so you need an internet connection.

Usage

drake_examples(quiet = TRUE)

Arguments

Value

Names of all the drake examples.

See Also

drake_example(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (requireNamespace("downloader")) {
drake_examples() # List all the drake examples.
# Sets up the example from load_mtcars_example()
drake_example("mtcars")
# Sets up the SLURM example.
drake_example("slurm")
}
})

## End(Not run)

List failed targets.

Description

List the targets that quit in error during make().

Usage

drake_failed(cache = drake::drake_cache(path = path), path = NULL)

Arguments

Value

A character vector of target names.

See Also

drake_done(), drake_running(), drake_cancelled(), drake_progress(), make()

Examples

## Not run: 
isolate_example("contain side effects", {
if (suppressWarnings(require("knitr"))) {
# Build a plan doomed to fail:
bad_plan <- drake_plan(x = function_doesnt_exist())
cache <- storr::storr_environment() # optional
try(
  make(bad_plan, cache = cache, history = FALSE),
  silent = TRUE
) # error
drake_failed(cache = cache) # "x"
e <- diagnose(x, cache = cache) # Retrieve the cached error log of x.
names(e)
e$error
names(e$error)
}
})

## End(Not run)

Do garbage collection on the drake cache.

Description

Garbage collection removes obsolete target values from the cache.

Usage

drake_gc(
  path = NULL,
  search = NULL,
  verbose = NULL,
  cache = drake::drake_cache(path = path),
  force = FALSE
)

Arguments

Details

Caution: garbage collection actually removes data so it is no longer recoverable with drake_history() or make(recover = TRUE). You cannot undo this operation. Use at your own risk.

Value

NULL

See Also

clean()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
# At this point, check the size of the '.drake/' cache folder.
# Clean without garbage collection.
clean(garbage_collection = FALSE)
# The '.drake/' cache folder is still about the same size.
drake_gc() # Do garbage collection on the cache.
# The '.drake/' cache folder should have gotten much smaller.
}
})

## End(Not run)

Session info of the last call to make().

Description

By default, session info is saved during make() to ensure reproducibility. Your loaded packages and their versions are recorded, for example.

Usage

drake_get_session_info(
  path = NULL,
  search = NULL,
  cache = drake::drake_cache(path = path),
  verbose = 1L
)

Arguments

Value

sessionInfo() of the last call to make()

See Also

diagnose(), cached(), readd(), drake_plan(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
drake_get_session_info() # Get the cached sessionInfo() of the last make().
}
})

## End(Not run)

Visualize the workflow with ggraph/ggplot2

Description

This function requires packages ggplot2 and ggraph. Install them with install.packages(c("ggplot2", "ggraph")).

Usage

drake_ggraph(
  ...,
  build_times = "build",
  digits = 3,
  targets_only = FALSE,
  main = NULL,
  from = NULL,
  mode = c("out", "in", "all"),
  order = NULL,
  subset = NULL,
  make_imports = TRUE,
  from_scratch = FALSE,
  full_legend = FALSE,
  group = NULL,
  clusters = NULL,
  show_output_files = TRUE,
  label_nodes = FALSE,
  transparency = TRUE,
  config = NULL
)

Arguments

Value

A ggplot2 object, which you can modify with more layers, show with plot(), or save as a file with ggsave().

See Also

vis_drake_graph(), sankey_drake_graph(), render_drake_ggraph(), text_drake_graph()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
load_mtcars_example() # Get the code with drake_example("mtcars").
# Plot the network graph representation of the workflow.
if (requireNamespace("ggraph", quietly = TRUE)) {
  drake_ggraph(my_plan) # Save to a file with `ggplot2::ggsave()`.
}
})

## End(Not run)

Prepare the workflow graph for visualization

Description

With the returned data frames, you can plot your own custom visNetwork graph.

Usage

drake_graph_info(
  ...,
  from = NULL,
  mode = c("out", "in", "all"),
  order = NULL,
  subset = NULL,
  build_times = "build",
  digits = 3,
  targets_only = FALSE,
  font_size = 20,
  from_scratch = FALSE,
  make_imports = TRUE,
  full_legend = FALSE,
  group = NULL,
  clusters = NULL,
  show_output_files = TRUE,
  hover = FALSE,
  on_select_col = NULL,
  config = NULL
)

Arguments

Value

A list of three data frames: one for nodes, one for edges, and one for the legend nodes. The list also contains the default title of the graph.

See Also

vis_drake_graph()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (requireNamespace("visNetwork", quietly = TRUE)) {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
vis_drake_graph(my_plan)
# Get a list of data frames representing the nodes, edges,
# and legend nodes of the visNetwork graph from vis_drake_graph().
raw_graph <- drake_graph_info(my_plan)
# Choose a subset of the graph.
smaller_raw_graph <- drake_graph_info(
  my_plan,
  from = c("small", "reg2"),
  mode = "in"
)
# Inspect the raw graph.
str(raw_graph)
# Use the data frames to plot your own custom visNetwork graph.
# For example, you can omit the legend nodes
# and change the direction of the graph.
library(visNetwork)
graph <- visNetwork(nodes = raw_graph$nodes, edges = raw_graph$edges)
visHierarchicalLayout(graph, direction = 'UD')
}
}
})

## End(Not run)

History and provenance

Description

See the history and provenance of your targets: what you ran, when you ran it, the function arguments you used, and how to get old data back.

Usage

drake_history(cache = NULL, history = NULL, analyze = TRUE, verbose = NULL)

Arguments

Details

drake_history() returns a data frame with the following columns.

• target: the name of the target.

• current: logical, whether the row describes the data actually assigned to the target name in the cache, e.g. what you get with loadd(target) and readd(target). Does NOT tell you if the target is up to date.

• built: when the target's value was stored in the cache. This is the true creation date of the target's value, not the recovery date from make(recover = TRUE).

• exists: logical, whether the target's historical value still exists in the cache. Garbage collection via (clean(garbage_collection = TRUE) and drake_cache()$gc()) remove these historical values, but clean() under the default settings does not.

• hash: fingerprint of the target's historical value in the cache. If the value still exists, you can read it with drake_cache()$get_value(hash).

• command: the drake_plan() command executed to build the target.

• seed: random number generator seed.

• runtime: the time it took to execute the drake_plan() command. Does not include overhead due to drake's processing.

If analyze is TRUE, various other columns are included to show the explicitly-named length-1 arguments to function calls in the commands. See the "Provenance" section for more details.

Value

A data frame of target history.

Provenance

If analyze is TRUE, drake scans your drake_plan() commands for function arguments and mentions them in the history. A function argument shows up if and only if: 1. It has length 1.
2. It is atomic, i.e. a base type: logical, integer, real, complex, character, or raw.
3. It is explicitly named in the function call, For example, x is detected as 1 in fn(list(x = 1)) but not f(list(1)). The exceptions are file_out(), file_in(), and knitr_in(). For example, filename is detected as "my_file.csv" in process_data(filename = file_in("my_file.csv")). NB: in process_data(filename = file_in("a", "b")) filename is not detected because the value must be atomic.

Examples

## Not run: 
isolate_example("contain side-effects", {
if (requireNamespace("knitr", quietly = TRUE)) {
# First, let's iterate on a drake workflow.
load_mtcars_example()
make(my_plan, history = TRUE, verbose = 0L)
# Naturally, we'll make updates to our targets along the way.
reg2 <- function(d) {
  d$x2 <- d$x ^ 3
  lm(y ~ x2, data = d)
}
Sys.sleep(0.01)
make(my_plan, history = TRUE, verbose = 0L)
# The history is a data frame about all the recorded runs of your targets.
out <- drake_history(analyze = TRUE)
print(out)
# Let's use the history to recover the oldest version
# of our regression2_small target.
oldest_reg2_small <- max(which(out$target == "regression2_small"))
hash_oldest_reg2_small <- out[oldest_reg2_small, ]$hash
cache <- drake_cache()
cache$get_value(hash_oldest_reg2_small)
# If you run clean(), drake can still find all the targets.
clean(small)
drake_history()
# But if you run clean() with garbage collection,
# older versions of your targets may be gone.
clean(large, garbage_collection = TRUE)
drake_history()
invisible()
}
})

## End(Not run)

Write a template file for deploying work to a cluster / job scheduler.

Description

See the example files from drake_examples() and drake_example() for example usage.

Usage

drake_hpc_template_file(
  file = drake::drake_hpc_template_files(),
  to = getwd(),
  overwrite = FALSE
)

Arguments

Value

NULL is returned, but a batchtools template file is written.

See Also

drake_hpc_template_files(), drake_examples(), drake_example(), shell_file()

Examples

## Not run: 
plan <- drake_plan(x = rnorm(1e7), y = rnorm(1e7))
# List the available template files.
drake_hpc_template_files()
# Write a SLURM template file.
out <- file.path(tempdir(), "slurm_batchtools.tmpl")
drake_hpc_template_file("slurm_batchtools.tmpl", to = tempdir())
cat(readLines(out), sep = "\n")
# library(future.batchtools) # nolint
# future::plan(batchtools_slurm, template = out) # nolint
# make(plan, parallelism = "future", jobs = 2) # nolint

## End(Not run)

List the available example template files for deploying work to a cluster / job scheduler.

Description

See the example files from drake_examples() and drake_example() for example usage.

Usage

drake_hpc_template_files()

Value

A character vector of example template files that you can write with drake_hpc_template_file().

See Also

drake_hpc_template_file(), drake_examples(), drake_example(), shell_file()

Examples

## Not run: 
plan <- drake_plan(x = rnorm(1e7), y = rnorm(1e7))
# List the available template files.
drake_hpc_template_files()
# Write a SLURM template file.
out <- file.path(tempdir(), "slurm_batchtools.tmpl")
drake_hpc_template_file("slurm_batchtools.tmpl", to = tempdir())
cat(readLines(out), sep = "\n")
# library(future.batchtools) # nolint
# future::plan(batchtools_slurm, template = out) # nolint
# make(plan, parallelism = "future", jobs = 2) # nolint

## End(Not run)

Create a drake plan for the plan argument of make().

Description

A drake plan is a data frame with columns "target" and "command". Each target is an R object produced in your workflow, and each command is the R code to produce it.

Usage

drake_plan(
  ...,
  list = NULL,
  file_targets = NULL,
  strings_in_dots = NULL,
  tidy_evaluation = NULL,
  transform = TRUE,
  trace = FALSE,
  envir = parent.frame(),
  tidy_eval = TRUE,
  max_expand = NULL
)

Arguments

Details

Besides "target" and "command", drake_plan() understands a special set of optional columns. For details, visit ⁠https://books.ropensci.org/drake/plans.html#special-custom-columns-in-your-plan⁠ # nolint

Value

A data frame of targets, commands, and optional custom columns.

Columns

drake_plan() creates a special data frame. At minimum, that data frame must have columns target and command with the target names and the R code chunks to build them, respectively.

You can add custom columns yourself, either with target() (e.g. drake_plan(y = target(f(x), transform = map(c(1, 2)), format = "fst"))) or by appending columns post-hoc (e.g. plan$col <- vals).

Some of these custom columns are special. They are optional, but drake looks for them at various points in the workflow.

• transform: a call to map(), split(), cross(), or combine() to create and manipulate large collections of targets. Details: (⁠https://books.ropensci.org/drake/plans.html#large-plans⁠). # nolint

• format: set a storage format to save big targets more efficiently. See the "Formats" section of this help file for more details.

• trigger: rule to decide whether a target needs to run. It is recommended that you define this one with target(). Details: ⁠https://books.ropensci.org/drake/triggers.html⁠.

• hpc: logical values (TRUE/FALSE/NA) whether to send each target to parallel workers. Visit ⁠https://books.ropensci.org/drake/hpc.html#selectivity⁠ to learn more.

• resources: target-specific lists of resources for a computing cluster. See ⁠https://books.ropensci.org/drake/hpc.html#advanced-options⁠ for details.

• caching: overrides the caching argument of make() for each target individually. Possible values:

  • "main": tell the main process to store the target in the cache.

  • "worker": tell the HPC worker to store the target in the cache.

  • NA: default to the caching argument of make().

• elapsed and cpu: number of seconds to wait for the target to build before timing out (elapsed for elapsed time and cpu for CPU time).

• retries: number of times to retry building a target in the event of an error.

• seed: an optional pseudo-random number generator (RNG) seed for each target. drake usually comes up with its own unique reproducible target-specific seeds using the global seed (the seed argument to make() and drake_config()) and the target names, but you can overwrite these automatic seeds. NA entries default back to drake's automatic seeds.

• max_expand: for dynamic branching only. Same as the max_expand argument of make(), but on a target-by-target basis. Limits the number of sub-targets created for a given target.

Formats

Specialized target formats increase efficiency and flexibility. Some allow you to save specialized objects like keras models, while others increase the speed while conserving storage and memory. You can declare target-specific formats in the plan (e.g. drake_plan(x = target(big_data_frame, format = "fst"))) or supply a global default format for all targets in make(). Either way, most formats have specialized installation requirements (e.g. R packages) that are not installed with drake by default. You will need to install them separately yourself. Available formats:

• "file": Dynamic files. To use this format, simply create local files and directories yourself and then return a character vector of paths as the target's value. Then, drake will watch for changes to those files in subsequent calls to make(). This is a more flexible alternative to file_in() and file_out(), and it is compatible with dynamic branching. See ⁠https://github.com/ropensci/drake/pull/1178⁠ for an example.

• "fst": save big data frames fast. Requires the fst package. Note: this format strips non-data-frame attributes such as the

• "fst_tbl": Like "fst", but for tibble objects. Requires the fst and tibble packages. Strips away non-data-frame non-tibble attributes.

• "fst_dt": Like "fst" format, but for data.table objects. Requires the fst and data.table packages. Strips away non-data-frame non-data-table attributes.

• "diskframe": Stores disk.frame objects, which could potentially be larger than memory. Requires the fst and disk.frame packages. Coerces objects to disk.frames. Note: disk.frame objects get moved to the drake cache (a subfolder of ⁠.drake/⁠ for most workflows). To ensure this data transfer is fast, it is best to save your disk.frame objects to the same physical storage drive as the drake cache, as.disk.frame(your_dataset, outdir = drake_tempfile()).

• "keras": save Keras models as HDF5 files. Requires the keras package.

• "qs": save any R object that can be properly serialized with the qs package. Requires the qs package. Uses qsave() and qread(). Uses the default settings in qs version 0.20.2.

• "rds": save any R object that can be properly serialized. Requires R version >= 3.5.0 due to ALTREP. Note: the "rds" format uses gzip compression, which is slow. "qs" is a superior format.

Keywords

drake_plan() understands special keyword functions for your commands. With the exception of target(), each one is a proper function with its own help file.

• target(): give the target more than just a command. Using target(), you can apply a transformation (examples: ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠), # nolint supply a trigger (⁠https://books.ropensci.org/drake/triggers.html⁠), # nolint or set any number of custom columns.

• file_in(): declare an input file dependency.

• file_out(): declare an output file to be produced when the target is built.

• knitr_in(): declare a knitr file dependency such as an R Markdown (⁠*.Rmd⁠) or R LaTeX (⁠*.Rnw⁠) file.

• ignore(): force drake to entirely ignore a piece of code: do not track it for changes and do not analyze it for dependencies.

• no_deps(): tell drake to not track the dependencies of a piece of code. drake still tracks the code itself for changes.

• id_chr(): Get the name of the current target.

• drake_envir(): get the environment where drake builds targets. Intended for advanced custom memory management.

Transformations

drake has special syntax for generating large plans. Your code will look something like ⁠drake_plan(y = target(f(x), transform = map(x = c(1, 2, 3)))⁠ You can read about this interface at ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠. # nolint

Static branching

In static branching, you define batches of targets based on information you know in advance. Overall usage looks like ⁠drake_plan(<x> = target(<...>, transform = <call>)⁠, where

• ⁠<x>⁠ is the name of the target or group of targets.

• ⁠<...>⁠ is optional arguments to target().

• ⁠<call>⁠ is a call to one of the transformation functions.

Transformation function usage:

• map(..., .data, .names, .id, .tag_in, .tag_out)

• split(..., slices, margin = 1L, drop = FALSE, .names, .tag_in, .tag_out) # nolint

• cross(..., .data, .names, .id, .tag_in, .tag_out)

• combine(..., .by, .names, .id, .tag_in, .tag_out)

Dynamic branching

• map(..., .trace)

• cross(..., .trace)

• group(..., .by, .trace)

map() and cross() create dynamic sub-targets from the variables supplied to the dots. As with static branching, the variables supplied to map() must all have equal length. group(f(data), .by = x) makes new dynamic sub-targets from data. Here, data can be either static or dynamic. If data is dynamic, group() aggregates existing sub-targets. If data is static, group() splits data into multiple subsets based on the groupings from .by.

Differences from static branching:

• ... must contain unnamed symbols with no values supplied, and they must be the names of targets.

• Arguments .id, .tag_in, and .tag_out no longer apply.

See Also

make, drake_config, transform_plan, map, split, cross, combine

Examples

## Not run: 
isolate_example("contain side effects", {
# For more examples, visit
# https://books.ropensci.org/drake/plans.html.

# Create drake plans:
mtcars_plan <- drake_plan(
  write.csv(mtcars[, c("mpg", "cyl")], file_out("mtcars.csv")),
  value = read.csv(file_in("mtcars.csv"))
)
if (requireNamespace("visNetwork", quietly = TRUE)) {
  plot(mtcars_plan) # fast simplified call to vis_drake_graph()
}
mtcars_plan
make(mtcars_plan) # Makes `mtcars.csv` and then `value`
head(readd(value))
# You can use knitr inputs too. See the top command below.

load_mtcars_example()
head(my_plan)
if (requireNamespace("knitr", quietly = TRUE)) {
  plot(my_plan)
}
# The `knitr_in("report.Rmd")` tells `drake` to dive into the active
# code chunks to find dependencies.
# There, `drake` sees that `small`, `large`, and `coef_regression2_small`
# are loaded in with calls to `loadd()` and `readd()`.
deps_code("report.Rmd")

# Formats are great for big data: https://github.com/ropensci/drake/pull/977
# Below, each target is 1.6 GB in memory.
# Run make() on this plan to see how much faster fst is!
n <- 1e8
plan <- drake_plan(
  data_fst = target(
    data.frame(x = runif(n), y = runif(n)),
    format = "fst"
  ),
  data_old = data.frame(x = runif(n), y = runif(n))
)

# Use transformations to generate large plans.
# Read more at
# `https://books.ropensci.org/drake/plans.html#create-large-plans-the-easy-way`. # nolint
drake_plan(
  data = target(
    simulate(nrows),
    transform = map(nrows = c(48, 64)),
    custom_column = 123
  ),
  reg = target(
    reg_fun(data),
   transform = cross(reg_fun = c(reg1, reg2), data)
  ),
  summ = target(
    sum_fun(data, reg),
   transform = cross(sum_fun = c(coef, residuals), reg)
  ),
  winners = target(
    min(summ),
    transform = combine(summ, .by = c(data, sum_fun))
  )
)

# Split data among multiple targets.
drake_plan(
  large_data = get_data(),
  slice_analysis = target(
    analyze(large_data),
    transform = split(large_data, slices = 4)
  ),
  results = target(
    rbind(slice_analysis),
    transform = combine(slice_analysis)
  )
)

# Set trace = TRUE to show what happened during the transformation process.
drake_plan(
  data = target(
    simulate(nrows),
    transform = map(nrows = c(48, 64)),
    custom_column = 123
  ),
  reg = target(
    reg_fun(data),
   transform = cross(reg_fun = c(reg1, reg2), data)
  ),
  summ = target(
    sum_fun(data, reg),
   transform = cross(sum_fun = c(coef, residuals), reg)
  ),
  winners = target(
    min(summ),
    transform = combine(summ, .by = c(data, sum_fun))
  ),
  trace = TRUE
)

# You can create your own custom columns too.
# See ?triggers for more on triggers.
drake_plan(
  website_data = target(
    command = download_data("www.your_url.com"),
    trigger = "always",
    custom_column = 5
  ),
  analysis = analyze(website_data)
)

# Tidy evaluation can help generate super large plans.
sms <- rlang::syms(letters) # To sub in character args, skip this.
drake_plan(x = target(f(char), transform = map(char = !!sms)))

# Dynamic branching
# Get the mean mpg for each cyl in the mtcars dataset.
plan <- drake_plan(
  raw = mtcars,
  group_index = raw$cyl,
  munged = target(raw[, c("mpg", "cyl")], dynamic = map(raw)),
  mean_mpg_by_cyl = target(
    data.frame(mpg = mean(munged$mpg), cyl = munged$cyl[1]),
    dynamic = group(munged, .by = group_index)
  )
)
make(plan)
readd(mean_mpg_by_cyl)
})

## End(Not run)

Show the code required to produce a given drake plan

Description

You supply a plan, and drake_plan_source() supplies code to generate that plan. If you have the prettycode package, installed, you also get nice syntax highlighting in the console when you print it.

Usage

drake_plan_source(plan)

Arguments

Value

a character vector of lines of text. This text is a call to drake_plan() that produces the plan you provide.

See Also

drake_plan()

Examples

plan <- drake::drake_plan(
  small_data = download_data("https://some_website.com"),
  large_data_raw = target(
    command = download_data("https://lots_of_data.com"),
    trigger = trigger(
      change = time_last_modified("https://lots_of_data.com"),
      command = FALSE,
      depend = FALSE
    ),
    timeout = 1e3
  )
)
print(plan)
if (requireNamespace("styler", quietly = TRUE)) {
  source <- drake_plan_source(plan)
  print(source) # Install the prettycode package for syntax highlighting.
  file <- tempfile() # Path to an R script to contain the drake_plan() call.
  writeLines(source, file) # Save the code to an R script.
}

Get the build progress of your targets

Description

Objects that drake imported, built, or attempted to build are listed as "done" or "running". Skipped objects are not listed.

Usage

drake_progress(
  ...,
  list = character(0),
  cache = drake::drake_cache(path = path),
  path = NULL,
  progress = NULL
)

Arguments

Value

The build progress of each target reached by the current make() so far.

See Also

diagnose(), drake_get_session_info(), cached(), readd(), drake_plan(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
# Watch the changing drake_progress() as make() is running.
drake_progress() # List all the targets reached so far.
drake_progress(small, large) # Just see the progress of some targets.
drake_progress(list = c("small", "large")) # Same as above.
}
})

## End(Not run)

List running targets.

Description

List the targets that either

1. Are currently being built during a call to make(), or

2. Were in progress when make() was interrupted.

Usage

drake_running(cache = drake::drake_cache(path = path), path = NULL)

Arguments

Value

A character vector of target names.

See Also

drake_done(), drake_failed(), drake_cancelled(), drake_progress(), make()

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the targets.
drake_running() # Everything should be done.
# nolint start
# Run make() in one R session...
# slow_plan <- drake_plan(x = Sys.sleep(2))
# make(slow_plan)
# and see the progress in another session.
# drake_running()
# nolint end
}
})

## End(Not run)

Description

A ⁠_drake.R⁠ file is required for r_make() and friends. See the r_make() help file for details.

Usage

drake_script(code = NULL)

Arguments

Value

Nothing.

Examples

## Not run: 
isolate_example("contain side-effects", {
drake_script({
  library(drake)
  plan <- drake_plan(x = 1)
  drake_config(plan, lock_cache = FALSE)
})
cat(readLines("_drake.R"), sep = "\n")
r_make()
})

## End(Not run)

Take a strategic subset of a dataset.

Description

drake_slice() is similar to split(). Both functions partition data into disjoint subsets, but whereas split() returns all the subsets, drake_slice() returns just one. In other words, drake_slice(..., index = i) returns split(...)[[i]]. Other features: 1. drake_slice() works on vectors, data frames, matrices, lists, and arbitrary arrays. 2. Like parallel::splitIndices(), drake_slice() tries to distribute the data uniformly across subsets. See the examples to learn why splitting is useful in drake.

Usage

drake_slice(data, slices, index, margin = 1L, drop = FALSE)

Arguments

Value

A subset of data.

Examples

# Simple usage
x <- matrix(seq_len(20), nrow = 5)
x
drake_slice(x, slices = 3, index = 1)
drake_slice(x, slices = 3, index = 2)
drake_slice(x, slices = 3, index = 3)
drake_slice(x, slices = 3, margin = 2, index = 1)
# In drake, you can split a large dataset over multiple targets.
## Not run: 
isolate_example("contain side effects", {
plan <- drake_plan(
  large_data = mtcars,
  data_split = target(
    drake_slice(large_data, slices = 32, index = i),
    transform = map(i = !!seq_len(32))
  )
)
plan
cache <- storr::storr_environment()
make(plan, cache = cache, session_info = FALSE, verbose = FALSE)
readd(data_split_1L, cache = cache)
readd(data_split_2L, cache = cache)
})

## End(Not run)

drake tempfile

Description

Create the path to a temporary file inside drake's cache.

Usage

drake_tempfile(path = NULL, cache = drake::drake_cache(path = path))

Arguments

Details

This function is just like the tempfile() function in base R except that the path points to a special location inside drake's cache. This ensures that if the file needs to be copied to persistent storage in the cache, drake does not need to copy across physical storage media. Example: the "diskframe" format. See the "Formats" and "Columns" sections of the drake_plan() help file. Unless you supply the cache or the path to the cache (see drake_cache()) drake will assume the cache folder is named ⁠.drake/⁠ and it is located either in your working directory or an ancestor of your working directory.

See Also

drake_cache(), new_cache()

Examples

cache <- new_cache(tempfile())
# No need to supply a cache if a .drake/ folder exists.
drake_tempfile(cache = cache)
drake_plan(
  x = target(
    as.disk.frame(large_data, outdir = drake_tempfile()),
    format = "diskframe"
  )
)

Declare input files and directories.

Description

file_in() marks individual files (and whole directories) that your targets depend on.

Usage

file_in(...)

Arguments

Value

A character vector of declared input file or directory paths.

URLs

As of drake 7.4.0, file_in() and file_out() have support for URLs. If the file name begins with "http://", "https://", or "ftp://", make() attempts to check the ETag to see if the data changed from last time. If no ETag can be found, drake simply uses the ETag from last make() and registers the file as unchanged (which prevents your workflow from breaking if you lose internet access). If your file_in() URLs require authentication, see the curl_handles argument of make() and drake_config() to learn how to supply credentials.

Keywords

drake_plan() understands special keyword functions for your commands. With the exception of target(), each one is a proper function with its own help file.

• target(): give the target more than just a command. Using target(), you can apply a transformation (examples: ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠), # nolint supply a trigger (⁠https://books.ropensci.org/drake/triggers.html⁠), # nolint or set any number of custom columns.

• file_in(): declare an input file dependency.

• file_out(): declare an output file to be produced when the target is built.

• knitr_in(): declare a knitr file dependency such as an R Markdown (⁠*.Rmd⁠) or R LaTeX (⁠*.Rnw⁠) file.

• ignore(): force drake to entirely ignore a piece of code: do not track it for changes and do not analyze it for dependencies.

• no_deps(): tell drake to not track the dependencies of a piece of code. drake still tracks the code itself for changes.

• id_chr(): Get the name of the current target.

• drake_envir(): get the environment where drake builds targets. Intended for advanced custom memory management.

See Also

file_out(), knitr_in(), ignore(), no_deps()

Examples

## Not run: 
isolate_example("contain side effects", {
# The `file_out()` and `file_in()` functions
# just takes in strings and returns them.
file_out("summaries.txt")
# Their main purpose is to orchestrate your custom files
# in your workflow plan data frame.
plan <- drake_plan(
  out = write.csv(mtcars, file_out("mtcars.csv")),
  contents = read.csv(file_in("mtcars.csv"))
)
plan
# drake knows "\"mtcars.csv\"" is the first target
# and a dependency of `contents`. See for yourself:

make(plan)
file.exists("mtcars.csv")

# You may use `.id_chr` inside `file_out()` and `file_in()`
# to refer  to the current target. This works inside
# static `map()`, `combine()`, `split()`, and `cross()`.

plan <- drake::drake_plan(
  data = target(
    write.csv(data, file_out(paste0(.id_chr, ".csv"))),
    transform = map(data = c(airquality, mtcars))
  )
)
plan

# You can also work with entire directories this way.
# However, in `file_out("your_directory")`, the directory
# becomes an entire unit. Thus, `file_in("your_directory")`
# is more appropriate for subsequent steps than
# `file_in("your_directory/file_inside.txt")`.
plan <- drake_plan(
  out = {
    dir.create(file_out("dir"))
    write.csv(mtcars, "dir/mtcars.csv")
  },
  contents = read.csv(file.path(file_in("dir"), "mtcars.csv"))
)
plan

make(plan)
file.exists("dir/mtcars.csv")

# See the connections that the file relationships create:
if (requireNamespace("visNetwork", quietly = TRUE)) {
  vis_drake_graph(plan)
}
})

## End(Not run)

Declare output files and directories.

Description

file_out() marks individual files (and whole directories) that your targets create.

Usage

file_out(...)

Arguments

Value

A character vector of declared output file or directory paths.

Keywords

drake_plan() understands special keyword functions for your commands. With the exception of target(), each one is a proper function with its own help file.

• target(): give the target more than just a command. Using target(), you can apply a transformation (examples: ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠), # nolint supply a trigger (⁠https://books.ropensci.org/drake/triggers.html⁠), # nolint or set any number of custom columns.

• file_in(): declare an input file dependency.

• file_out(): declare an output file to be produced when the target is built.

• knitr_in(): declare a knitr file dependency such as an R Markdown (⁠*.Rmd⁠) or R LaTeX (⁠*.Rnw⁠) file.

• ignore(): force drake to entirely ignore a piece of code: do not track it for changes and do not analyze it for dependencies.

• no_deps(): tell drake to not track the dependencies of a piece of code. drake still tracks the code itself for changes.

• id_chr(): Get the name of the current target.

• drake_envir(): get the environment where drake builds targets. Intended for advanced custom memory management.

See Also

file_in(), knitr_in(), ignore(), no_deps()

Examples

## Not run: 
isolate_example("contain side effects", {
# The `file_out()` and `file_in()` functions
# just takes in strings and returns them.
file_out("summaries.txt")
# Their main purpose is to orchestrate your custom files
# in your workflow plan data frame.
plan <- drake_plan(
  out = write.csv(mtcars, file_out("mtcars.csv")),
  contents = read.csv(file_in("mtcars.csv"))
)
plan
# drake knows "\"mtcars.csv\"" is the first target
# and a dependency of `contents`. See for yourself:

make(plan)
file.exists("mtcars.csv")

 # You may use `.id_chr` inside `file_out()` and `file_in()`
 # to refer  to the current target. This works inside `map()`,
 # `combine()`, `split()`, and `cross()`.

plan <- drake::drake_plan(
  data = target(
    write.csv(data, file_out(paste0(.id_chr, ".csv"))),
    transform = map(data = c(airquality, mtcars))
  )
)

plan

# You can also work with entire directories this way.
# However, in `file_out("your_directory")`, the directory
# becomes an entire unit. Thus, `file_in("your_directory")`
# is more appropriate for subsequent steps than
# `file_in("your_directory/file_inside.txt")`.
plan <- drake_plan(
  out = {
    dir.create(file_out("dir"))
    write.csv(mtcars, "dir/mtcars.csv")
  },
  contents = read.csv(file.path(file_in("dir"), "mtcars.csv"))
)
plan

make(plan)
file.exists("dir/mtcars.csv")

# See the connections that the file relationships create:
if (requireNamespace("visNetwork", quietly = TRUE)) {
  vis_drake_graph(plan)
}
})

## End(Not run)

Show a file's encoded representation in the cache

Description

This function simply wraps literal double quotes around the argument x so drake knows it is the name of a file. Use when you are calling functions like deps_code(): for example, deps_code(file_store("report.md")). See the examples for details. Internally, drake wraps the names of file targets/imports inside literal double quotes to avoid confusion between files and generic R objects.

Usage

file_store(x)

Arguments

Value

A single-quoted character string: i.e., a filename understandable by drake.

Examples

# Wraps the string in single quotes.
file_store("my_file.rds") # "'my_file.rds'"
## Not run: 
isolate_example("contain side effects", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the workflow to build the targets
list.files() # Should include input "report.Rmd" and output "report.md".
head(readd(small)) # You can use symbols for ordinary objects.
# But if you want to read cached info on files, use `file_store()`.
readd(file_store("report.md"), character_only = TRUE) # File fingerprint.
deps_code(file_store("report.Rmd"))
config <- drake_config(my_plan)
deps_profile(
  file_store("report.Rmd"),
  plan = my_plan,
  character_only = TRUE
)
}
})

## End(Not run)

Search up the file system for the nearest drake cache.

Description

Only works if the cache is a file system in a hidden folder named ⁠.drake/⁠ (default).

Usage

find_cache(path = getwd(), dir = NULL, directory = NULL)

Arguments

Value

File path of the nearest drake cache or NULL if no cache is found.

See Also

drake_plan(), make(),

Examples

## Not run: 
isolate_example("Quarantine side effects.", {
if (suppressWarnings(require("knitr"))) {
load_mtcars_example() # Get the code with drake_example("mtcars").
make(my_plan) # Run the project, build the target.
# Find the file path of the project's cache.
# Search up through parent directories if necessary.
find_cache()
}
})

## End(Not run)

Name of the current target

Description

id_chr() gives you the name of the current target while make() is running. For static branching in drake_plan(), use the .id_chr symbol instead. See the examples for details.

Usage

id_chr()

Value

The name of the current target.

Keywords

drake_plan() understands special keyword functions for your commands. With the exception of target(), each one is a proper function with its own help file.

• target(): give the target more than just a command. Using target(), you can apply a transformation (examples: ⁠https://books.ropensci.org/drake/plans.html#large-plans⁠), # nolint supply a trigger (⁠https://books.ropensci.org/drake/triggers.html⁠), # nolint or set any number of custom columns.

• file_in(): declare an input file dependency.

• file_out(): declare an output file to be produced when the target is built.

• knitr_in(): declare a knitr file dependency such as an R Markdown (⁠*.Rmd⁠) or R LaTeX (⁠*.Rnw⁠) file.

• ignore(): force drake to entirely ignore a piece of code: do not track it for changes and do not analyze it for dependencies.

• no_deps(): tell drake to not track the dependencies of a piece of code. drake still tracks the code itself for changes.

• id_chr(): Get the name of the current target.

• drake_envir(): get the environment where drake builds targets. Intended for advanced custom memory management.

Examples

try(id_chr()) # Do not use outside the plan.
## Not run: 
isolate_example("id_chr()", {
plan <- drake_plan(x = id_chr())
make(plan)
readd(x)
# Dynamic branching
plan <- drake_plan(
  x = seq_len(4),
  y = target(id_chr(), dynamic = map(x))
)
make(plan)
readd(y, subtargets = 1)
# Static branching
plan <- drake_plan(
  y = target(c(x, .id_chr), transform = map(x = !!seq_len(4)))
)
plan
})

## End(Not run)