Best Scripting Practices

Below I list a few commandments of R scripting that will guide us through this lesson. Future you and all those around you will sing your praises if you follow these.

We’ll go through all of these in some detail with an example script!

1. Conform to a predictable script structure

1.1. Provide your metadata about the document (not required, but helpful)

It is always a good idea to write at the top of a script who wrote it (give yourself some credit!), when, and why it was written. In the case of an R script (as opposed to an R Markdown document), we might write something like the below at the top of our document.

#' ---
#' title: The best title
#' purpose: blarg
#' author: You
#' date: YYYY-MM-DD
#' ---

Since the above is commented out, it is completely inert. The R does not run or use this information in anyway.

Similarly, R Mardown documents have an element built into the top of their documents called a YAML, which is used in a slightly different way by R, but accomplishes a similar feat. The words you use in a YAML (like ‘title,’ ‘author,’ etc.) are specific to the YAML, so don’t add anything in there that YAMLs don’t use.

---
title: "The best title"
author: "You"
date: "YYYY-MM-DD"
output: html_document
---

1.2. Add your subsections

Next, so you can see exactly what you are building, we are going to open the document outline in R Studio. To open the document outline, either:

• Press Control + Shift+ O

• Click the ‘outline’ button in the most upper right corner of your text editor window.

Here is the outline of an example anaysis.

Outline of an example analysis.

(And here is an example of what the outline for this R Markdown HTML report looks like, which is completely different.)

Outline for this R Markdown HTML Notebook.

Outlines are available for a document types, including R Markdown and R Scripts. You’ll notice in this R Markdown document that different sections headers are listed in the outline for this document. You can click on those to navigate this document.

You can view subsections in your document outline. This structure is up to the user, but, personally, I use the following.

# Directories ---------

# Libraries -------

# Import Data -----

# Functions -------

# Analysis

# *** Correlation ---------

# *** Visualizations ------

# *** Save Outputs -----

In the above example, I use # *** [text] ---- in an attempt to make a level down subsection. Here I consider “Correlation”, “Visualizations”, and “Save Outputs” are all part of doing an analysis. You can use anything to take the space of the asterisk (*) but I think asterisks look best.

These headers work the same as if I wrote them as # Libraries ####.

Below is a short example of a reasonably organized R script that we will build on more. This script is going to correlate some columns and make some plots using one of R’s built in datasets called mpg from the ggplot2 package. To learn more about mpg, type ??mpg in your console, but essentially this dataset describes fuel economy data from 1999 to 2008 for 38 popular models of cars where:

• displ: engine displacement, in litres

• cty: city miles per gallon

• hwy: highway miles per gallon

#' ---
#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

# Directories ---------

# Libraries -------

library(ggplot2)
library(ggpubr)

# Import Data -----

data(mpg)

# Functions -------

# Analysis -----

# *** Correlation ---------

cor_hwy<-cor(mpg$displ, mpg$hwy)

cor_cty<-cor(mpg$displ, mpg$cty)

# *** Visualizations ------

g1<-ggplot(mpg, aes(displ, hwy)) +
  geom_point() + 
  geom_smooth(method = "lm") + 
  ggtitle(paste0("Pearson correlation coefficient = ", round(cor_hwy, digits = 2))) +
  ylim(5, 45)

g2<-ggplot(mpg, aes(displ, cty)) +
  geom_point() + 
  geom_smooth(method = "lm") + 
  ggtitle(paste0("Pearson correlation coefficient = ", round(cor_cty, digits = 2))) +
  ylim(5, 45)

g<-ggarrange(plotlist = list(g1, g2),
               nrow=1, ncol = 2)

# *** Save Outputs -----

ggsave(filename = "graph.png", plot = g)
g

Output from script

This is a great start! Here we are using a predictable structure for our code, we know where to look for each section, and it looks great! But get ready, there are a few more things to do!

2. Annotate

So, what is actually happening in this script? I just wrote it, so I know, in this moment, exactly what is going on. But will “future me” remember? Future me is pretty forgetful and likes quick answers. As to not upset this irratable future me, it might be best to write a few notes in the document right now. We can easily do that with a single (or more, if you like) hash mark (#) before comment text. See below how I integrate it into the script we just wrote.

Admittedly, I don’t usually write this many comments, but for the purposes of this example I want to be extra explicit.

#' ---
#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

# Directories ---------

dir<-getwd() #This is the current directory. This is not currently needed, thought. 

# Libraries ---------

library(ggplot2) # Makes pretty plots
library(ggpubr) # Arranges multiple ggplots on the same page

# Import Data -----

# Download data describing Fuel economy data from 1999 to 2008 for 38 popular models of cars. 
# This data is built into the ggplot2 library. Use "?mpg" to learn more about this dataset. 
data(mpg)
#Columns of interest include: 
## "displ" = engine displacement, in litres
## "cty" = city miles per gallon
## "hwy" = highway miles per gallon

# Functions -------

# There are currently no functions for this script. 

# Analysis -----

# *** Correlation ---------

#Find the Pearson correlation coefficient for:
## engine displacement (litres) ~ highway mpg
cor_hwy<-cor(mpg$displ, mpg$hwy) 

## engine displacement (litres) ~ city mpg
cor_cty<-cor(mpg$displ, mpg$cty)

# *** Visualizations ------

#Create plots displaying data points, a smooth linear model trend line (note 'method = lm'), and a useful title. 

## engine displacement (litres) ~ highway mpg
g1<-ggplot(mpg, aes(displ, hwy)) +
  geom_point() + 
  geom_smooth(method = "lm") + 
  ggtitle(paste0("Pearson correlation coefficient = ", round(cor_hwy, digits = 2))) +
    ylim(5, 45)

## engine displacement (litres) ~ city mpg
g2<-ggplot(mpg, aes(displ, cty)) +
  geom_point() + 
  geom_smooth(method = "lm") + 
  ggtitle(paste0("Pearson correlation coefficient = ", round(cor_cty, digits = 2))) +
    ylim(5, 45)

## arrange these two plots side by side so we can compare differences between the plots. 
g<-ggarrange(plotlist = list(g1, g2),
               nrow=1, ncol = 2)

# *** Save Outputs -----

#Save your plot so you can use and find it later. 
ggsave(filename = "graph.png", plot = g)
g

Output from script

Wow! I will never be in the dark again about past me’s true genius. I would feel comfortable sending this off to any colleague - maybe even colleagues who doesn’t code?! - because I know they will understand what I was trying to do.

3. Never repeat code

I see a lot of redundancy in this code. What if I told you I could remove a third of the content in this script with one simple function? Repeating code to do the same thing over and over again will inevitably lead to mistakes, conflicts, and typos. Further, what if we suddenly decided we wanted all graphs to have a new title or change the colors? In the old script, we would have to change the code going into each plot individually. Now, with a function, we can change that once in just one place and it will be changed everywhere.

Though this is a small example and this version of the code would really be sufficient, I am sure you can see how these issues could become a massive problem in a long script you have been working tirelessly on.

analysis.R

(I’ll save the below file as the above name so you can view it later, too).

#' ---
#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

# Directories ---------

dir<-getwd() #This is the current directory. This is not currently needed, thought. 

# Libraries ---------

library(ggplot2) # Makes pretty plots
library(ggpubr) # Arranges multiple ggplots on the same page

# Import Data -----

# Download data describing Fuel economy data from 1999 to 2008 for 38 popular models of cars. 
# This data is built into the ggplot2 library. Use "?mpg" to learn more about this dataset. 
data(mpg)
#Columns of interest include: 
## "displ" = engine displacement, in litres
## "cty" = city miles per gallon
## "hwy" = highway miles per gallon

# Functions -------

createplot<-function(x, y) {
  #Createplot is a function for calculating the Pearson correlation coefficient and plotting two variables (x and y) against each other 
  
  #Find the Pearson correlation coefficient
  cor_<-cor(x, y)
  
  #Create plots displaying data points, a smooth linear model trend line (note 'method = lm'), and a useful title. 
  g<-ggplot(mpg, aes(x, y)) +
    geom_point() + # plots points
    geom_smooth(method = "lm") + #plots trend line using a linear model
    ggtitle(paste0("Pearson correlation coefficient = ", round(cor_, digits = 2))) + # displays title with Pearson correlation coefficient
    ylim(5, 45) # standardizes the hieght of the y-axis
  
  #return outputs our graph from the function. 
  return(g)
}

# Analysis -----

# *** Visualizations ------

#Use createplot to output our two finished plots. 
g1<-createplot(x = mpg$displ, y = mpg$hwy)

g2<-createplot(x = mpg$displ, y = mpg$cty)

## arrange these two plots side by side so we can compare differences between the plots. 
g<-ggarrange(plotlist = list(g1, g2),
               nrow=1, ncol = 2)

# *** Save Outputs -----

#Save your plot so you can use and find it later. 
ggsave(filename = "graph.png", plot = g)
g

Output from script

4. Name files with predictable and helpful names

“graph.png” is not a terribly helpful name for any file. Good in a pinch, not in the long run. There are a few questions immediately come to mind:

• What analysis does this plot belong to?
• At what part of the analysis was this output created? e.g., order?
• Why do I care about this analysis? e.g., a more descriptive name?
• What run is this analysis from? e.g., date? (something to think about but skipping this for now. We’ll get to it next.)

I usually try to keep all of my files to similar naming structure like this one.

[Order]-[AnalysisName]-[Description].[filetype]

A few rules:

• No spaces. You can use ThisLetterCasing or thisLetterCasing to separate words.
• Use underscores () or dashes (-) to separate ideas so it can be easy to read. Python has issues with underscores (), so you may want to use dashes (-).
• Names should be kept as short as possible. Some applications (including R, though it varies by action and file type) have a letter count limit on file names.

With this in mind, I might change the file name graph.png in ggsave(filename = "graph.png", plot = g) to 1_TestAnalysis_Graph.png by using the below code:

counter<-0
counter<-counter + 1
filename = paste0(counter, "_TestAnalysis_Graph.png")
ggsave(filename = filename, plot = g)

The counter is not really necessary, but is useful if I want to keep the order of these files in my file folder. If you have to make many of these (or any other) plots but don’t want to have to search for them in a sea of files, numbering can help make their location in the folder predictable.

5. Predictable folder structure

Let’s say that you are working on this analysis over the course of several months and you don’t want to overwrite your outputs each time you run your script or get lost in a sea of files. It’s a little complicated to set up, but I suggest the following:

5.1. Set up three new folders

In a given project, I keep all of my files (generally) in three folders. Note that "./" is R shorthand for “current directory,” which would be the directory of your R Project.

• rscripts: where you will save all of your rscripts, including this script we have been working on.
• data: where you will store all of your local data that goes into a script (though not really relevant here since we are using some of R’s in-house data).
• output: where results from your analysis will go.

dir.create(path = "./rscripts")
dir.create(path = "./data")
dir.create(path = "./output")

The root folder’s stucture.

5.2. Create a folder for that analysis run within the output folder

For each day’s run of your analysis, you might like to set up a new file to save everything in. Within that folder, you might also like to keep a copy of what you used to create this file. I recommend automatically creating folders within this new output folder for your rscripts, rawdata, figures, etc. This might seem predicable, but with the:

• figures folder, we save all of our output figures from the analysis.
• rscripts folder, we save all of the rscripts we used to create this analysis with. I know many of you must be thinking that this is redundant in a world with GitHub, but it gives me some peace of mind to have all of these things together. Here I use a simple loop to find all of the files in my ./rscripts/ file in the root directory and copy them to the output’s ./output/[folder for that day's code run]/rscripts/ folder.
• rawdata folder, we store all of the raw data we used to create this analysis.

The output folder’s stucture.

There is nothing stopping you from having other folders here with different outputs, like tables or worddocuments. Just keep the names helpful and predictable.

# Directories -------

outputfolder<-paste0("./output/TestAnalysis_", Sys.Date(), "/") # Define the name of the new output folder. Sys.Date() will add today's name to the file folder's name. 
dir.create(path = outputfolder) # Create folder for today's analysis run

dir.create(path = paste0(outputfolder, "rawdata"))
dir.create(path = paste0(outputfolder, "figures"))
dir.create(path = paste0(outputfolder, "rscripts"))

listfiles<-list.files(path = "./rscripts/") #Find all files in "./rscripts"
for (i in 1:length(listfiles)){ # Save all of those files to the rscripts folder in the output folder
  file.copy(from = paste0("./rscripts/", listfiles[i]), 
            to = paste0("./",outputfolder,"/rscripts/", listfiles[i]), 
            overwrite = T)
}

6. Scripts with specific tasks

In a world where your analyses are a bit more complicated, I find that it is good to split your code into the following files:

• functions.R, this file will be where all of the funcitons used to run your analysis.
• data.R, this file will load all of your data and edit it for your analysis. Never hand edit data from a source. If you can, always manipulate it so you never destroy your original data copy.
• run.R, this file will source the other files you created and run the analysis. We want this to have the bare-bones of what we need for our analysis.

The output rscripts’s stucture.

Such that your new files would look like this and would all be saved in the “rscripts” folder:

functions.R

#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

rm(list=ls())

# Libraries -------
PKG <- c("ggplot2", "ggpubr")
for (p in PKG) {
  if(!require(p,character.only = TRUE)) {  
    install.packages(p)
    require(p,character.only = TRUE)}
}


# Directories -------

outputfolder<-paste0("./output/TestAnalysis_", Sys.Date(), "/") # Define the name of the new output folder. Sys.Date() will add today's name to the file folder's name. 
dir.create(path = outputfolder) # Create folder for today's analysis run

dir.create(path = paste0(outputfolder, "rawdata"))
dir.create(path = paste0(outputfolder, "figures"))
dir.create(path = paste0(outputfolder, "rscripts"))
listfiles<-list.files(path = "./rscripts/") #Find all files in "./rscripts"
for (i in 1:length(listfiles)){ # Save all of those files to the rscripts folder in the output folder
  file.copy(from = paste0("./rscripts/", listfiles[i]), 
            to = paste0("./",outputfolder,"/rscripts/", listfiles[i]), 
            overwrite = T)
}

# Functions -------

createplot<-function(x, y) {
  #Createplot is a function for calculating the Pearson correlation coefficient and plotting two variables (x and y) against each other 
  
  #Find the Pearson correlation coefficient
  cor_<-cor(x, y)
  
  #Create plots displaying data points, a smooth linear model trend line (note 'method = lm'), and a useful title. 
  g<-ggplot(mpg, aes(x, y)) +
    geom_point() + # plots points
    geom_smooth(method = "lm") + #plots trend line using a linear model
    ggtitle(paste0("Pearson correlation coefficient = ", round(cor_, digits = 2))) + # displays title with Pearson correlation coefficient
    ylim(5, 45) # standardizes the hieght of the y-axis
  
  #return outputs our graph from the function. 
  return(g)
}

data.R

#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

# Libraries ---------

library(ggplot2) # Makes pretty plots

# Import Data -----

# Download data describing Fuel economy data from 1999 to 2008 for 38 popular models of cars. 
# This data is built into the ggplot2 library. Use "?mpg" to learn more about this dataset. 

data(mpg)

#Save the rawdata you used for an analysis (if you want to)

write.csv(x = data.frame(mpg), file = paste0(outputfolder, "rawdata/mpg.csv"))

#Columns of interest include: 
## "displ" = engine displacement, in litres
## "cty" = city miles per gallon
## "hwy" = highway miles per gallon

run.R

#' title: Engine displacement in cities and on highways. 
#' purpose: This script is going to correlate the displ, cty, and hwy columns of R's built in dataset called mpg from the ggplot2 package and make some plots of those correlations. 
#' author: You
#' date: YYYY-MM-DD
#' ---

gc() # garbage collection; remove unused variables

# Source Scripts -------

source("./rscripts/functions.R")
source("./rscripts/data.R")

# Analysis -----

#Use createplot to output our two finished plots. 
g1<-createplot(x = mpg$displ, y = mpg$hwy)

g2<-createplot(x = mpg$displ, y = mpg$cty)

## arrange these two plots side by side so we can compare differences between the plots. 
g<-ggarrange(plotlist = list(g1, g2),
               nrow=1, ncol = 2)

# *** Save Outputs -----

#Save your plot so you can use and find it later. 
counter<-0
counter<-counter + 1
filename = paste0(counter, "_TestAnalysis_Graph.png")
ggsave(filename = paste0(outputfolder, "figures/", filename), plot = g)
g

Output from script

See how both the functions and data folders are ‘sourced’ into the run script? It makes the script look so much cleaner and easier to read.

7. Save scripts in an “R Project”

You can find more information about projects here: https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects

R projects are important for:

• organizing your analysis from the nuts and bolts (functions and packages) to analysis output (r scripts).
• sharing your projects to Git Hub
• so you can establish a ‘relative’ directory (as opposed to a local directory that requires a specific address in your computer)
• creating R Shiny apps, packages, etc. (beyond the scope of this example)

To create a project for this work, click File>New Project>Existing Directory

File>New Project>Existing Directory

File>New Project>Existing Directory

We selected “Existing Directory” since we already have one. It is known as our parent directory and what I have been referring to as ./ in the directory name.

Projects manifest themselves, more or less, as fancy folders where R has automatically deposited the following files:

• Creates a project file (with an .Rproj extension) within the project directory. This file contains various project options (discussed below) and can also be used as a shortcut for opening the project directly from the filesystem.
• Creates a hidden directory (named .Rproj.user) where project-specific temporary files (e.g. auto-saved source documents, window-state, etc.) are stored. This directory is also automatically added to .Rbuildignore, .gitignore, etc. if required.

Now, the project is loaded into RStudio and the name is displayed in the Projects toolbar (which is located on the far right side of the main toolbar). Additionally, the project (and all progress on the files) will be easily loadable from File>Recent Projects or File>Open Project.

If you want access to this R Project from my GitHub, for example, you can access it by going to the GitHub link below and downloading it, or with the below code.

install.packages("usethis")
library(usethis) # Automate package and project setup tasks that are otherwise performed manually.
usethis::use_course(url = 'https://github.com/emilyhmarkowitz/BestScriptingPractices/archive/master.zip', 
                    destdir = "your/local/directory/")

8. Other packages to consider using!

Below are some other packages that are worth a closer look as you play around with making beautifully written and organized code!

here

A Simpler Way to Find Your Files

Constructs paths to your project’s files. The ‘here()’ function uses a reasonable heuristics to find your project’s files, based on the current working directory at the time when the package is loaded. Use it as a drop-in replacement for ‘file.path()’, it will always locate the files relative to your project root.

https://github.com/krlmlr/here, http://krlmlr.github.io/here

packrat

A Dependency Management System for Projects and their R Package Dependencies

Manage the R packages your project depends on in an isolated, portable, and reproducible way.

https://github.com/rstudio/packrat/

foreach

Provides Foreach Looping Construct

Support for the foreach looping construct. Foreach is an idiom that allows for iterating over elements in a collection, without the use of an explicit loop counter. This package in particular is intended to be used for its return value, rather than for its side effects. In that sense, it is similar to the standard lapply function, but doesn’t require the evaluation of a function. Using foreach without side effects also facilitates executing the loop in parallel.

https://github.com/RevolutionAnalytics/foreach