{ "metadata": { "kernelspec": { "display_name": "R", "language": "R", "name": "r" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "4.1.0" } }, "nbformat": 4, "nbformat_minor": 5, "cells": [ { "id": "metadata", "cell_type": "markdown", "source": "
\n\n# dplyr & tidyverse for data processing\n\nby [Helena Rasche](https://training.galaxyproject.org/hall-of-fame/hexylena/), [Erasmus+ Programme](https://training.galaxyproject.org/hall-of-fame/erasmusplus/), [Avans Hogeschool](https://training.galaxyproject.org/hall-of-fame/avans-atgm/)\n\nMIT licensed content from the [Galaxy Training Network](https://training.galaxyproject.org/)\n\n**Objectives**\n\n- How can I load tabular data into R?\n- How can I slice and dice the data to ask questions?\n\n**Objectives**\n\n- Read data with the built-in read.csv\n- Read data with dplyr's read_csv\n- Use dplyr and tidyverse functions to cleanup data.\n\n**Time Estimation: 1H**\n
\n", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-0", "source": "

dplyr (Wickham et al. 2021) is a powerful R-package to transform and summarize tabular data with rows and columns. It is part of a group of packages (including ggplot2) called the tidyverse (Wickham et al. 2019), a collection of packages for data processing and visualisation. For further exploration please see the dplyr package vignette: Introduction to dplyr

\n
\n
Comment
\n

This tutorial is significantly based on GenomicsClass/labs.

\n
\n
\n
Agenda
\n

In this tutorial, we will cover:

\n
    \n
  1. Why Is It Useful?
    2. \n
  2. How Does It Compare To Using Base Functions R?
    3. \n
  3. How Do I Get dplyr?
    4. \n
\n
\n

Why Is It Useful?

\n

The package contains a set of functions (or “verbs”) that perform common data manipulation operations such as filtering for rows, selecting specific columns, re-ordering rows, adding new columns and summarizing data.

\n

In addition, dplyr contains a useful function to perform another common task which is the “split-apply-combine” concept. We will discuss that in a little bit.

\n

How Does It Compare To Using Base Functions R?

\n

If you are familiar with R, you are probably familiar with base R functions such as split(), subset(), apply(), sapply(), lapply(), tapply() and aggregate(). Compared to base functions in R, the functions in dplyr are easier to work with, are more consistent in the syntax and are targeted for data analysis around tibbles, instead of just vectors.

\n

How Do I Get dplyr?

\n

To load the required packages:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-1", "source": [ "library(tidyverse)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-2", "source": "
\n
\n

Remember that you can install new packages by running

\n
install.packages(\"tidyverse\")\n
\n

Or by using the Install button on the RStudio Packages interface

\n
\n

Here we’ve imported the entire suite of tidyverse packages. We’ll specifically be using:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PackageUse
readrThis provides the read_csv function which is identical to read.csv except it returns a tibble
dplyrAll of the useful functions we’ll be covering are part of dplyr
magrittrA dependency of dplyr that provides the %>% operator
ggplot2The famous plotting library which we’ll use at the very end to plot our aggregated data.
\n

Data: Mammals Sleep

\n

The msleep (mammals sleep) data set contains the sleep times and weights for a set of mammals. This data set contains 83 rows and 11 variables.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-3", "source": [ "url <- \"https://raw.githubusercontent.com/genomicsclass/dagdata/master/inst/extdata/msleep_ggplot2.csv\"\n", "msleep <- read_csv(url)\n", "head(msleep)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-4", "source": "

The columns (in order) correspond to the following:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
column nameDescription
namecommon name
genustaxonomic rank
vorecarnivore, omnivore or herbivore?
ordertaxonomic rank
conservationthe conservation status of the mammal
sleep_totaltotal amount of sleep, in hours
sleep_remrem sleep, in hours
sleep_cyclelength of sleep cycle, in hours
awakeamount of time spent awake, in hours
brainwtbrain weight in kilograms
bodywtbody weight in kilograms
\n

Compare the above output with the more traditional read.csv that is built into R

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-5", "source": [ "dfmsleep <- read.csv(url)\n", "head(dfmsleep)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-6", "source": "

This is a “data frame” and was the basis of data processing for years in R, and is still quite commonly used! But notice how dplyr has a much prettier and more consice output. This is what is called a tibble (like a table). We can immediately see metadata about the table, the separator that was guessed for us, what datatypes each column was (dbl or chr), how many rows and columns we have, etc. The tibble works basically exactly like a data frame except it has a lot of features to integrate nicely with the dplyr package.

\n

That said, all of the functions below you will learn about work equally well with data frames and tibbles, but tibbles will save you from filling your screen with hundreds of rows by automatically truncating large tables unless you specifically request otherwise.

\n

Important dplyr Verbs To Remember

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
dplyr verbsDescriptionSQL Equivalent Operation
select()select columnsSELECT
filter()filter rowsWHERE
arrange()re-order or arrange rowsORDER BY
mutate()create new columnsSELECT x, x*2 ...
summarise()summarise valuesn/a
group_by()allows for group operations in the “split-apply-combine” conceptGROUP BY
\n

dplyr Verbs In Action

\n

The two most basic functions are select() and filter(), which selects columns and filters rows respectively.

\n

Pipe Operator: %>%

\n

Before we go any further, let’s introduce the pipe operator: %>%. dplyr imports this operator from another package (magrittr).This operator allows you to pipe the output from one function to the input of another function. Instead of nesting functions (reading from the inside to the outside), the idea of piping is to read the functions from left to right. This is a lot more like how you would write a bash data processing pipeline and can be a lot more readable and intuitive than the nested version.

\n

Here’s is the more old fashioned way of writing the equivalent code:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-7", "source": [ "head(select(msleep, name, sleep_total))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-8", "source": "

Now in this case, we will pipe the msleep tibble to the function that will select two columns (name and sleep_total) and then pipe the new tibble to the function head(), which will return the head of the new tibble.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-9", "source": [ "msleep %>% select(name, sleep_total) %>% head(2)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-10", "source": "
\n
Question
\n

How would you rewrite the following code to use the pipe operator?

\n
prcomp(tail(read.csv(\"file.csv\"), 10))\n
\n
👁 View solution\n
\n

Just read from inside to outside, starting with the innermost () and use %>% between each step.

\n
read.csv(\"file.csv\") %>% tail(10) %>% prcomp()\n
\n
\n
\n

Selecting Columns Using select()

\n

Select a set of columns: the name and the sleep_total columns.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-11", "source": [ "msleep %>% select(name, sleep_total)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-12", "source": "

To select all the columns except a specific column, use the “-“ (subtraction) operator (also known as negative indexing):

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-13", "source": [ "msleep %>% select(-name)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-14", "source": "

To select a range of columns by name, use the “:” (colon) operator:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-15", "source": [ "msleep %>% select(name:order)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-16", "source": "

To select all columns that start with the character string “sl”, use the function starts_with():

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-17", "source": [ "msleep %>% select(starts_with(\"sl\"))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-18", "source": "

Some additional options to select columns based on a specific criteria include:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FunctionUsage
ends_with()Select columns that end with a character string
contains()Select columns that contain a character string
matches()Select columns that match a regular expression
one_of()Select column names that are from a group of names
\n

Selecting Rows Using filter()

\n

Filter the rows for mammals that sleep a total of more than 16 hours.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-19", "source": [ "msleep %>% filter(sleep_total >= 16)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-20", "source": "

Filter the rows for mammals that sleep a total of more than 16 hours and have a body weight of greater than 1 kilogram.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-21", "source": [ "msleep %>% filter(sleep_total >= 16, bodywt >= 1)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-22", "source": "

Filter the rows for mammals in the Perissodactyla and Primates taxonomic order

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-23", "source": [ "msleep %>% filter(order %in% c(\"Perissodactyla\", \"Primates\"))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-24", "source": "

You can use the boolean operators (e.g. >, <, >=, <=, !=, %in%) to create the logical tests.

\n

Arrange Or Re-order Rows Using arrange()

\n

To arrange (or re-order) rows by a particular column, such as the taxonomic order, list the name of the column you want to arrange the rows by:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-25", "source": [ "msleep %>% arrange(order) %>% select(order, genus, name)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-26", "source": "

Now we will select three columns from msleep, arrange the rows by the taxonomic order and then arrange the rows by sleep_total. Finally, show the final tibble:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-27", "source": [ "msleep %>%\n", " select(name, order, sleep_total) %>%\n", " arrange(order, sleep_total)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-28", "source": "

Same as above, except here we filter the rows for mammals that sleep for 16 or more hours, instead of showing the whole tibble:

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-29", "source": [ "msleep %>%\n", " select(name, order, sleep_total) %>%\n", " arrange(order, sleep_total) %>%\n", " filter(sleep_total >= 16)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-30", "source": "

Something slightly more complicated: same as above, except arrange the rows in the sleep_total column in a descending order. For this, use the function desc()

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-31", "source": [ "msleep %>%\n", " select(name, order, sleep_total) %>%\n", " arrange(order, desc(sleep_total)) %>%\n", " filter(sleep_total >= 16)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-32", "source": "

Create New Columns Using mutate()

\n

The mutate() function will add new columns to the tibble. Create a new column called rem_proportion, which is the ratio of rem sleep to total amount of sleep.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-33", "source": [ "msleep %>%\n", " mutate(rem_proportion = sleep_rem / sleep_total) %>%\n", " select(starts_with(\"sl\"), rem_proportion)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-34", "source": "

You can many new columns using mutate (separated by commas). Here we add a second column called bodywt_grams which is the bodywt column in grams.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-35", "source": [ "msleep %>%\n", " mutate(rem_proportion = sleep_rem / sleep_total,\n", " bodywt_grams = bodywt * 1000) %>%\n", " select(sleep_total, sleep_rem, rem_proportion, bodywt, bodywt_grams)" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-36", "source": "

Create summaries of the tibble using summarise()

\n

The summarise() function will create summary statistics for a given column in the tibble such as finding the mean. For example, to compute the average number of hours of sleep, apply the mean() function to the column sleep_total and call the summary value avg_sleep.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-37", "source": [ "msleep %>%\n", " summarise(avg_sleep = mean(sleep_total))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-38", "source": "

There are many other summary statistics you could consider such sd(), min(), max(), median(), sum(), n() (returns the length of vector), first() (returns first value in vector), last() (returns last value in vector) and n_distinct() (number of distinct values in vector).

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-39", "source": [ "msleep %>%\n", " summarise(avg_sleep = mean(sleep_total),\n", " min_sleep = min(sleep_total),\n", " max_sleep = max(sleep_total),\n", " total = n())" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-40", "source": "

Group operations using group_by()

\n

The group_by() verb is an important function in dplyr. As we mentioned before it’s related to concept of “split-apply-combine”. We literally want to split the tibble by some variable (e.g. taxonomic order), apply a function to the individual tibbles and then combine the output.

\n

Let’s do that: split the msleep tibble by the taxonomic order, then ask for the same summary statistics as above. We expect a set of summary statistics for each taxonomic order.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-41", "source": [ "msleep %>%\n", " group_by(order) %>%\n", " summarise(avg_sleep = mean(sleep_total),\n", " min_sleep = min(sleep_total),\n", " max_sleep = max(sleep_total),\n", " total = n())" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-42", "source": "

ggplot2

\n

Most people want to slice and dice their data before plotting, so let’s demonstrate that quickly by plotting our last dataset.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-43", "source": [ "library(ggplot2)\n", "msleep %>%\n", " group_by(order) %>%\n", " summarise(avg_sleep = mean(sleep_total),\n", " min_sleep = min(sleep_total),\n", " max_sleep = max(sleep_total),\n", " total = n()) %>%\n", " ggplot() + geom_point(aes(x=min_sleep, y=max_sleep, colour=order))" ], "cell_type": "code", "execution_count": null, "outputs": [ ], "metadata": { "attributes": { "classes": [ "r" ], "id": "" } } }, { "id": "cell-44", "source": "

Notice how we can just keep piping our data together, this makes it incredibly easier to experiment and play around with our data and test out what filtering or summarisation we want and how that will plot in the end. If we wanted, or if the data processing is an especially computationally expensive step, we could save it to an intermediate variable before playing around with plotting options, but in the case of this small dataset that’s probably not necessary.

\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "cell_type": "markdown", "id": "final-ending-cell", "metadata": { "editable": false, "collapsed": false }, "source": [ "# Key Points\n\n", "- Dplyr and tidyverse make it a lot easier to process data\n", "- The functions for selecting data are a lot easier to understand than R's built in alternatives.\n", "\n# Congratulations on successfully completing this tutorial!\n\n", "Please [fill out the feedback on the GTN website](https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-dplyr/tutorial.html#feedback) and check there for further resources!\n" ] } ] }