How do I perform parallel processing in R?

      Monday 21 August 2023 |  by Kris Chan

      Category:

* We recommend you have administrative rights and internet access as Windows users will need to install Rtools and grant R networking access to assigning tasks to individual cores (as parallel processing on Windows requires the use of ‘sockets’). Parallel processing can take time to set up, and unless you are doing tasks well suited to parallel processing (e.g. analysing multiple very large files), it is probably better (faster) to run your analyses as normal. Nevertheless, instructions for parallel processing below:

1. 1. (Windows only): Install RTools from https://cran.r-project.org/bin/windows/Rtools/

1. 1. Load the ‘parallel’ package (part of R core’s code after 2.14.0, i.e. no need to install):

library(parallel)

3. 3. Install and load the ‘doParallel’ package.
      • • This requires additional dependencies, i.e. ‘foreach’ and ‘iterators’. If R is connected to the internet, these should be automatically installed – otherwise you will need to manually install them.

install.packages('doParallel')
library(doParallel)

4. 4. Detect the number of cores:

clusters <- detectCores()

5. 5. Specify the number of cores you want to use.
      • • This will depend on just how much of your computer’s processing ability you want to devote to the task.
      • • Though you CAN use all cores, it can be useful to keep one core free to avoid crashes and allow you some processing capability to check whether the process is still running, etc. In the example below we keep one core free for the above reason.

cl <- makeCluster(detectCores()-1)
registerDoParallel(cl)

6. 6. Load variables and packages to each core.
      • • By default, variables and packages which are already loaded into your R environment will not exist or be recognised by individual cores. You will therefore need to parse variables and packages over to each core using the function below

#create an example variable
x <- 99

#parse variable to cluster
clusterExport(cl, "x")

#parse a required R package to the core, e.g. tidyr. Note, if you do not have not tidyr installed, you will first need to call install.packages('tidyr')
clusterEvalQ(cl, library(tidyr))

#Note that you can also parse multiple variables and libraries by replacing the variable with { } and entering multiple inbetween:
dummyArray <- c(42,36,21)
n <- length(dummyArray)

clusterExport(cl, {
  "dummyArray"
  "n"
})

clusterEvalQ(cl, {
  library(tidyr)   #Should be installed (as above)
  library(ggplot2) #if not installed, will require installing
})

7. 7. You can now tell R to process the data. It will automatically assign the next task to the next free core.

#Example parallel processing - e.g. add x to each number in dummyArray. We combine each result to a new column using cbind. We could add them to new rows using rbind.
result <- foreach(a=1:n, .combine = cbind) %dopar% {
  dummyArray[a]+x
}
result

#equivalent simple 'for' loop
result2 <- NULL
for(a in 1:n){
  result2 <- cbind(result2, dummyArray[a]+x)
}
result2

#Sequentially create each column and combine
system.time(m <- foreach(i=1:144, .combine=cbind) %do% { (100/2.3)^3.17*matrix(rnorm(1000*1000)) } )
#my total elapsed time was 4.74

#create each column in parallel processing and combine
system.time(m <- foreach(i=1:144, .combine=cbind) %dopar% { (100/2.3)^3.17*matrix(rnorm(1000*1000))} )
#my total elapsed time was 2.75

stopCluster(cl)

Disadvantages and caveats

Run example code snippet