Parallel Workers in the Cloud

Henrik Bengtsson

Introduction

This vignettes illustrates how to launch parallel workers on cloud services such as Amazon AWS (https://aws.amazon.com/) and Google Compute Engine (https://cloud.google.com/products/compute/).

Examples

Example: Remote worker running on GCE

This example launches a parallel worker on Google Compute Engine (GCE) running a container based VM (with a #cloud-config specification).

library(parallelly)

public_ip <- "1.2.3.4"
user <- "johnny"
ssh_private_key_file <- "~/.ssh/google_compute_engine"
cl <- makeClusterPSOCK(
  ## Public IP number of GCE instance
  public_ip,
  ## User name (== SSH key label (sic!))
  user = user,
  ## Use private SSH key registered with GCE
  rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
  ),
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-parallel",
    "Rscript"
  )
)

Example: Remote worker running on AWS

This example, which is a bit dated, launches a parallel worker on Amazon AWS EC2 running one of the Amazon Machine Images (AMI) provided by Posit (https://www.louisaslett.com/RStudio_AMI/).

library(parallelly)

public_ip <- "1.2.3.4"
ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"

cl <- makeClusterPSOCK(
  ## Public IP number of EC2 instance
  public_ip,
  ## User name (always 'ubuntu')
  user = "ubuntu",
  ## Use private SSH key registered with AWS
  rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
  ),
  ## Set up .libPaths() for the 'ubuntu' user
  ## and then install the future package
  rscript_startup = quote(local({
    p <- Sys.getenv("R_LIBS_USER")
    dir.create(p, recursive = TRUE, showWarnings = FALSE)
    .libPaths(p)
    install.packages("future")
  }))
)