The future framework with the future package at the code comprises many packages - use the ones that suits your needs
future()
, resolved()
, and value()
, is designed to unify parallel processing in R at the lowest possible level. It provides a standard for building richer, higher-level parallel frontends without having to worry about and re-implement common, critical tasks such as identifying global variables and packages, parallel RNG, and relaying of output and conditions - cumbersome tasks that are often essential to parallel processing.
Nearly all parallel processing tasks can be implemented using this low-level API. But, most users and package developers use the futures via higher-level map-reduce functions provided by future.apply, furrr, and foreach with doFuture.future.apply - Apply Function to Elements in Parallel using Future
Implementations of apply()
, by()
, eapply()
, lapply()
, Map()
, .mapply()
, mapply()
, replicate()
, sapply()
, tapply()
, and vapply()
that can be resolved using any future-supported backend, e.g. parallel on the local machine or distributed on a compute cluster. These future_*apply()
functions come with the same pros and cons as the corresponding base R *apply()
functions but with the additional feature of being able to be processed via the future framework.
furrr - Apply Mapping Functions in Parallel using Futures
Implementations of the family of map()
functions from purrr that can be resolved using any future-supported backend, e.g. parallel on the local machine or distributed on a compute cluster. (Developed by Davis Vaughan)
foreach with doFuture - A Universal Foreach Parallel Adapter using the Future API of the future Package
Provides a %dopar%
adapter such that any type of futures can be used as backends for the foreach framework and its foreach()
function.
BiocParallel with BiocParallel.FutureParam - Use Futures with BiocParallel
A ‘BiocParallelParam’ class for using futures with the BiocParallel framework, e.g. BiocParallel::register(FutureParam())
. An alternative, is to use BiocParallel::register(DoParam())
in combination of doFuture.
future - the future package has a set of built-in parallel backends that build upon the parallel package, e.g. multicore
, multisession
, and cluster
.
future.batchtools - A Future API for Parallel and Distributed Processing using batchtools
Implementation of the Future API on top of the batchtools package. This allows you to process futures, as defined by the future package, in parallel out of the box, not only on your local machine or ad-hoc cluster of machines, but also via high-performance compute (HPC) job schedulers such as LSF, OpenLava, Slurm, SGE, and TORQUE/PBS, e.g. y <- future.apply::future_lapply(files, FUN = process)
.
future.callr - A Future API for Parallel Processing using callr
Implementation of the Future API on top of the callr package. This allows you to process futures, as defined by the future package, in parallel out of the box, on your local (Linux, macOS, Windows, …) machine. Contrary to backends relying on the parallel package (e.g. future::multisession
), the callr
backend provided here can run more than 125 parallel R processes.
parallelly - Enhancing the parallel Package
Utility functions that enhance the parallel package and support the built-in parallel backends of the future package.
marshal - Framework to Marshal Objects to be Used in Another R Process
PROTOTYPE: Some types of R objects can be used only in the R session they were created. If used as-is in another R process, such objects often result in an immediate error or in obscure and hard-to-troubleshoot outcomes. Because of this, they cannot be saved to file and re-used at a later time. They can also not be exported to a worker in parallel processing. These objects are sometimes referred to as non-exportable or non-serializable objects. One solution to this problem is to use “marshaling” to encode the R object into an exportable representation that then can be used to re-create a copy of that object in another R process.
globals - Identify Global Objects in R Expressions
Identifies global (“unknown” or “free”) objects in R expressions by code inspection using various strategies (ordered, liberal, or conservative). The objective of this package is to make it as simple as possible to identify global objects for the purpose of exporting them in parallel, distributed compute environments.
listenv - Environments Behaving (Almost) as Lists
List environments are environments that have list-like properties. For instance, the elements of a list environment are ordered and can be accessed and iterated over using index subsetting, e.g. x <- listenv(a = 1, b = 2);
for (i in seq_along(x)) x[[i]] <- x[[i]] ^ 2;
y <- as.list(x)
.