Resource Specification Project
Some calls need far more memory than the objects you pass to them, and nothing in your script says so. Fitting a cross-validated elastic net with cv.glmnet(x, y) of the glmnet package needs roughly four times the size of x. When that memory is not there, the error arrives long after the work started, or the operating system kills the R process outright:
> fit <- cv.glmnet(x, y)
Error: cannot allocate vector of size 1.7 GbThe goal of the Resource Project is to give memory and other types of resource requirements a home - a resource specification, written in R next to the code that knows it. The same declaration is meant to do two jobs: checked before the work starts, it fails fast instead of hours later, and handed to a parallel framework, it also decides how many tasks can run at the same time.
Ultimately, the outcome of this project is not only reduced amounts of wasted compute resources, but also reduced amounts of wasted human resources from troubleshooting.
The examples below are excerpts from the blog post The Resource Project: Tell R How Much Memory You Need, which explains them in full.
This is a proposal. With one exception - futurize(), which is on CRAN today - none of the features below are implemented.
Declare what a call needs
An expr |> resources(...) pipe leaves the original code and logic as-is, while declaring requirements that R can act on:
fit <- cv.glmnet(x, y) |> resources(memory(4 * object.size(x)))
#> Error: UnmetResourceError: cv.glmnet(x, y) requires memory 7.5 GiB, available 3.1 GiBFail fast for a whole batch
When the declaration is a function of the data, it can be checked for all elements before the first model is fit, e.g. so that ten assay-specific fits do not waste effort on the first three before failing on the fourth:
fits <- lapply(xs, function(assay) cv.glmnet(assay, y)) |>
resources(function(assay) memory(4 * object.size(assay)))
#> Error: UnmetResourceError: cv.glmnet(assay, y) requires memory 7.5 GiB for xs[[4]], available 3.1 GiBLet the function declare its own needs
Nobody should have to investigate what cv.glmnet() needs. The function maintainer, or you yourself for code you do not maintain, can attach the specification to the function:
resources(cv.glmnet) <- function(x, ...) memory(4 * object.size(x))With that in place, the caller declares nothing:
fit <- cv.glmnet(x, y) |> resources()
#> Error: UnmetResourceError: cv.glmnet(x, y) requires memory 7.5 GiB, available 3.1 GiBThe same declaration schedules parallel tasks
Without memory protection, there is a great risk that you run out of memory before you run out of CPU cores. The plan is for futurize to use the very same specification to limit how many memory-hungry tasks run concurrently:
fits <- lapply(xs, function(assay) cv.glmnet(assay, y)) |>
resources(function(assay) memory(4 * object.size(assay))) |>
futurize()On a high-performance compute (HPC) cluster, future.batchtools could translate each calculated requirement into a job-scheduler declaration, e.g. Slurm’s --mem=10G, so that every task lands in an appropriately sized slot.
Why this matters
A long-running script failing due to out-of-memory errors is never fun. Even an experienced developer might end up spending hours troubleshooting the problem, identifying the root cause, and coming up with a fix - and if the retry fails too, even more time is lost. It is better to fail fast, because the resource specification was provided upfront.
Also, compute resources such as CPU, GPU, and memory are scarce and expensive. For example, memory prices have risen ~350% in the last year1 and up to 10x the lowest price2. In other words, the cost of blindly over-requesting memory for an analysis to protect against memory errors increases as well. Requesting twice as much memory as actually needed translates to a real monetary cost, e.g. more expensive hardware investments or cloud compute. Simplified and improved methods for declaring resource needs help mitigate such costs.
Another example is high-performance compute (HPC) clusters where shared resources are wasted due to both under- and over-specified resource needs. For instance, a job that over-specifies how many CPU cores it needs will result in CPU cores sitting idle, instead of being available to other jobs stuck in the queue. Similarly, a job that under-specifies how much memory or run-time it needs will fail to complete, resulting in wasted compute resources. As an example, a utilization study of a UCSF high-performance compute (HPC) cluster shared by thousands of researchers shows that ~27% of all CPU time was consumed by failed jobs running out of run-time or memory.3 Compensating for that loss would cost hundreds of thousands of USD in new hardware investments. Better and easier resource specifications with good feedback can help mitigate these hidden costs.
Much more
Memory is only the beginning. Walltime, scratch space, graphics processing unit (GPU) cores, and GPU memory limit us in the same way.
See the Roadmap for other features related to this project.
Feedback and suggestions are welcome on the Futureverse Discussions forum.
Footnotes
As of August 2026, memory prices have risen 345% since September 2025 to a new high, and a 32 GB DDR5 kit that cost 84-126 EUR in September 2025 cost 400-539 EUR in August 2026 (ComputerBase, August 2026).↩︎
In the US, a 128 GB DDR5 kit is listed at 3,399 USD, up to 10x the lowest price ever tracked (Tom’s Hardware, August 2026).↩︎
Two ordinary weeks of SGE accounting data from UCSF’s Wynton HPC cluster show ~100,000 failed jobs (1.85% of all jobs) that were killed by the scheduler for running out of run-time or memory, consuming 26.9% of all CPU time. That works out to 8.4 CPU-years of compute burned every day, on one university cluster, during two ordinary weeks - results that have been reproduced over the years (see the
read_sge_accounting()reference examples).↩︎