10  Basic Tidyverse Concepts

The Treachery of Images (Rene Magritte, 1948).

In this chapter we will introduce a few tools from the tidyverse set of R-packages: * the pipe-operator %>% that, like R’s native pipe |>, is used to chain function-calls in an inutitive, readable way; * the tibble class, a variant of the data frame that is especially suitable for large data sets; * data manipulation functions from the dplyr package suitable for use with the pipe operator: * filter() and select() for sub-setting; * mutate() for transforming variables; * group_by() and summarise() for numerical summaries of data.

10.1 The Tidyverse

The tidyverse isn’t a package, exactly—it’s a collection of packages. Go ahead and attach it:

library(tidyverse)

You’ll get an account of the packages that have been attached. We have worked before with ggplot and by the end of CSC 215 we will have worked with all of the others. You need not worry about the fact that filter() and lag() mask functions from the stats package.

10.2 The magrittr Pipe Operator

In Section 6.8.4 you met R’s native pipe operator |>. The tidyverse uses another pipe operator, %>% from the magrittr package.1.

Tip

The keyboard shortcut for %>% is:

  • Ctrl+Shift+M on Windows/Linux, or
  • Cmd+Shift+M on Mac OS.

Like the native R pipe operator, %>% connects two function calls by making the value returned by the first call the first argument of the second call. Here’s an example:

"hello" %>% rep(times = 4)
[1] "hello" "hello" "hello" "hello"

This is the same as the more familiar:

rep("hello", times = 4)
[1] "hello" "hello" "hello" "hello"

Here’s another example:

# same as nrow(bcscr::m111survey)
bcscr::m111survey %>% nrow()
[1] 71

Here’s two pipes:

"hello" %>% rep(times = 4) %>% length()
[1] 4

By default the value of the left-hand call is piped into the right-hand call as the first argument. You can make it some other argument using the dot . as a placeholder, for example:

4 %>% rep("hello", times = .)
[1] "hello" "hello" "hello" "hello"

(Recall that the placeholder for the native R pipe is _. Do not interchange _ with .)

Since sub-setting is actually a function call under the hood, you can use the dot there, too:

# gets the third element of the sequence 1, 4, 9, ..., 97:
seq(1, 100, by = 4) %>% .[3]
[1] 9

The pipe operator isn’t all that useful when you only use it once or twice in succession. Its true value becomes apparent in the chaining together of many manipulations involving data frames.

10.2.1 Practice Exercises

Rewrite the following call with the tidyverse pipe operator, in three different ways:

Use the pipe operator with subset() to find the row of mosaicData::CPS85 containing the worker who made more than 40 dollars per hour. Display only the sex, age and wage of the worker.

10.3 Tibbles

The tibble package gives us tibbles, which are very nearly the same thing as a data frame. Indeed, the name “tibble” is supposed to remind us of a data “table.”

Consider the class of bcscr::m111survey (see Data Table 7.1):

class(bcscr::m111survey)
[1] "data.frame"

Yep, it’s a data frame. But we can convert it to a tibble, as follows:

survey <- as_tibble(bcscr::m111survey)
class(survey)
[1] "tbl_df"     "tbl"        "data.frame"

You can treat tibbles like data frames. For now the primary practical difference is manifest when you print a tibble to the Console:

The output is automatically truncated, and the number of columns printed is determined by the width of your screen. This is a great convenience when one is dealing with larger data sets.

Many larger data tables in packages will come to you as tibbles.

10.4 Subsetting with dplyr

The dplyr function filter() is the rough equivalent of select(): it picks out rows of a data frame (or similar objects such as a tibble). The dplyr function select() subsets for columns.

Thus you can use the two functions together to do perform sub-setting. With the pipe operator, your code can be quite easy to read:

Note that dplyr data-functions like filter() and select() take a data table as their first argument, and return a data table as well. Hence they may be chained together as we saw in the above example.

With select() it’s easy to leave out columns, too:

10.4.1 Practice Exercises

Can you use the pipe to chain dplyr functions along with nrow() to find out how many people in survey believe in love at first sight and drove more than 120 miles per hour?

In survey, find the three largest heights of the males who drove more than 120 miles per hour.

This is one way, using functions you already know:

Use the pipe and filter() to make violin plots of the wages of men and women in CPS85 (see Data Table 7.3), where the outlier-person (whose wage was more than 40 dollars per hour) has been eliminated prior to making the graph.

The babynames package has a data table called babynames (see Data Table 10.2):

babynames
# A tibble: 1,924,665 × 5
    year sex   name          n   prop
   <dbl> <chr> <chr>     <int>  <dbl>
 1  1880 F     Mary       7065 0.0724
 2  1880 F     Anna       2604 0.0267
 3  1880 F     Emma       2003 0.0205
 4  1880 F     Elizabeth  1939 0.0199
 5  1880 F     Minnie     1746 0.0179
 6  1880 F     Margaret   1578 0.0162
 7  1880 F     Ida        1472 0.0151
 8  1880 F     Alice      1414 0.0145
 9  1880 F     Bertha     1320 0.0135
10  1880 F     Sarah      1288 0.0132
# ℹ 1,924,655 more rows

Each row is the set of babies born in U.S. hospitals in the given year who were of the stated sex and had the stated name. In addition, the table gives:

  • n: the number of babies in the group;
  • propr: the proportion of all babies of the stated sex who had the stated name.

Use babynames to investigate the relative popularity of Leslie as a name for boys vs. a name for girls, over the years. You should make a graph like this:

Hint: The line-graphs are made with geom_line().

10.5 Transforming Variables with dplyr

In dplyr you transform variables with the function mutate(). Here is an example:

In mutate() there is always a variable-name on the left-hand side of the = sign. It could be the same as an existing variable in the table if you are content to overwrite that variable. On the right side of the = is a function that can depend on variables in the data table.

You can transform more than one variable in a single call to mutate(), as in the code below. Try it!

10.5.1 Practice Exercises

In mosaicData::CPS85 (see Data Table 7.3) transform the wage variable to units of dollars per day. (Assume an 8-hour working day.)

10.6 Grouping and Summaries

The next two dplyr data-functions are useful for generating numerical summaries of data.

Consider, for example, CPS85 (see Data Table 7.3). We know from graphical studies that the men in the study are paid more than women, but how might we verify this fact numerically? One approach would be to separate the men and the women into two different groups and compute the mean wage for each group. This is accomplished by calling group_by() and summarise() in succession:

It’s possible to create more than one summary variable in a single call to summarise(), for example:

In the previous example, dplyr::n() was used to count the number of cases in each group.

For a more complete account of a numerical variable, one might consider the five-number summary:

  • the minimum value
  • the first quartile (Q1)
  • the median
  • the third quartile (Q3)
  • the maximum value

These quantities are conveniently computed by R’s fivenum() function:

CPS85 %>% 
  .$wage %>% 
  fivenum()
[1]  1.00  5.25  7.78 11.25 44.50

Let’s find the five number summaries for the wages of men and women:

It’s also possible to group by more than one variable at a time. For example, suppose that we wish to compare the wages of men and women in the various sectors of employment. All we need to do is group by both sex and sector:

Note that there were no women in the construction sector, so that group did not appear in the summary.

10.6.1 Note on Binding

Keep in mind that you can always “save” the results of any computation by binding them to a variable name, thus:

Note that the result has data.frame as one of its classes, so you may extract components in any of the ways you have learned. The old ways, for instance, are fine:

sexSector <-
  CPS85 %>% 
  group_by(sector, sex) %>% 
  summarise(
    n = n(),
    min = fivenum(wage)[1],
    Q1 = fivenum(wage)[2],
    median = fivenum(wage)[3],
    Q3 = fivenum(wage)[4],
    max = fivenum(wage)[5]
  )
# minimum wage among male professionals:
with(sexSector, min[sex == "M" & sector == "prof"])
[1] 5

10.6.2 Practice Exercises

These exercises deal with the flights data table from the nycflights13 package.

Description

 On-time data for all flights that departed NYC (i.e. JFK, LGA or EWR)
 in 2013.

Format

 Data frame with columns
 
 year, month, day Date of departure.
 
 dep_time, arr_time Actual departure and arrival times (format HHMM or
           HMM), local tz.
 
 sched_dep_time, sched_arr_time Scheduled departure and arrival times
           (format HHMM or HMM), local tz.
 
 dep_delay, arr_delay Departure and arrival delays, in minutes. Negative
           times represent early departures/arrivals.
 
 carrier Two letter carrier abbreviation. See 'airlines' to get name.
 
 flight Flight number.
 
 tailnum Plane tail number. See 'planes' for additional metadata.
 
 origin, dest Origin and destination. See 'airports' for additional
           metadata.
 
 air_time Amount of time spent in the air, in minutes.
 
 distance Distance between airports, in miles.
 
 hour, minute Time of scheduled departure broken into hour and minutes.
 
 time_hour Scheduled date and hour of the flight as a 'POSIXct' date.
           Along with 'origin', can be used to join flights data to
           'weather' data.

View