Parallel Backends

By default, future-based code runs sequentially, but with a single line of code, we easily switch to run the exact same code in parallel. The most common approach is to parallelize on the local machine, but we have also the option to harness the CPUs of other local or remote machines. For example, to parallelize on the local machine, the end-user can call:

plan(multisession)

After this, all of Futureverse, including future.apply, furrr, and doFuture, and any package that use these, will run the code in parallel.

To switch back to sequential processing, we can call:

plan(sequential)

If you have Secure Shell (SSH) access to other machines on your local network, or remote machines, call:

plan(cluster, workers = c("n1", "n1", "n2", "remote.server.org"))

This will set up four parallel workers, where two run on the local ‘n1’ machine, another on the local ‘n2’ machine, and the fourth on the remote ‘remote.server.org’ machine.

In addition to the above built-in parallel backends, more are provided by other R packages, as show in the table below.

Parallel backend	Description
`sequential`	Run future-based code sequentially (default).
`multisession`	Parallelize on the local machine via persistent background R processes.
`cluster`	Parallelize across local and remote machines via persistent background R processes.
`multicore`	Parallelize on the local machine via transient, forked R processes.
`callr`	Parallelize on the local machine via transient background R processes. Available in the future.callr package.
`mirai_multisession`	Parallelize on the local machine via persistent background R processes. Available in the future.mirai package.
`mirai_cluster`	Parallelize across local and remote machines via persistent background R processes. Available in the future.mirai package.
`batchtools_lsf`	Parallelize via the high-performance-compute (HPC) scheduler Load Sharing Facility (LSF). Available in the future.batchtools package.
`batchtools_openlava`	Parallelize via the high-performance-compute (HPC) scheduler OpenLava. Available in the future.batchtools package.
`batchtools_pbs`	Parallelize via the high-performance-compute (HPC) scheduler TORQUE/PBS. Available in the future.batchtools package.
`batchtools_sge`	Parallelize via the high-performance-compute (HPC) scheduler Son/Sun/Oracle/Univa Grid Engine (SGE). Available in the future.batchtools package.
`batchtools_slurm`	Parallelize via the high-performance-compute (HPC) scheduler Slurm. Available in the future.batchtools package.

It is straightforward to implement new backends that leverage other ways to harness available compute resources. As soon as a new backend has been validated to be compliant with the Future API specifications, which can be done by the future.tests package, then it can be used anywhere future-based code is used.