Lecture 4: tidyverse for data manipulation

Data: Hollywood movies

Prep work

  • Tidyverse is a popular collection of R packages for working with data in a tidy format: each variable is a column, each observation is a row, and each value is a cell.
library(tidyverse)
library(Lock5Data)
  • We’ll use the HollywoodMovies dataset.
str(HollywoodMovies)
head(HollywoodMovies)
?HollywoodMovies

Review questions

  • How many genres are there? How many movies are in each genre?
  • What are the 10 movies with largest difference (in absolute value) between RottenTomatoes and AudienceScore?
  • Compare the mean AudienceScore by Genre of the movies.
  • What are the five action movies with the largest WorldwideBO?
  • Which movie has the largest WorldwideBO in each genre?

A tidyverse workflow

Tidyverse functions take a data frame as their first input and return a data frame, so functions can be chained with the native R pipe |> or tidyverse pipe %>%.

HollywoodMovies |>
  filter(Genre == "Action") |>
  select(Movie, Genre, WorldwideBO)

Read the pipe as “then”: start with HollywoodMovies, then keep action movies, then select the three named columns.

Subsetting: ‘[’

The square bracket syntax can be used as a function, with the function name ‘[’

HollywoodMovies$Movie[1:5]
'['(HollywoodMovies$Movie, 1:5)  # as a function
HollywoodMovies$Movie |> '['(1:5) # pipe version

HollywoodMovies[5,1]
HollywoodMovies |> '['(5,1)

HollywoodMovies[5,]
HollywoodMovies |> '['(5,)

Subsetting: filter

# Get info of action movies
HollywoodMovies |>
  filter(Genre == "Action")

# Multiple conditions
HollywoodMovies |>
  filter(Genre == "Action", Profitability > 500)

HollywoodMovies |>
  filter(Genre == "Action" | Genre == "Adventure")

(Use ?dplyr::filter to read the help page)

Subsetting: slice and select

# Keep selected rows by position
HollywoodMovies |>
  slice(c(1, 3, 5))

# Select top rows
HollywoodMovies |>
  slice_head(n = 5)

# Keep or drop selected columns
HollywoodMovies |>
  select(Movie, Genre, WorldwideBO)

HollywoodMovies |>
  select(-Distributor, -TheatersOpen)

Sampling: slice_sample()

# Randomly sample 10% of rows, without replacement
HollywoodMovies |>
  slice_sample(prop = 0.1)

# Sample five rows, allowing repeats
HollywoodMovies |>
  slice_sample(n = 5, replace = TRUE)

Sorting: arrange() and slice_max()

# Smallest five worldwide grosses
HollywoodMovies |>
  arrange(WorldwideBO) |>
  slice_head(n = 5)

# Largest five worldwide grosses
HollywoodMovies |>
  slice_max(order_by = WorldwideBO, n = 5, na_rm = TRUE) |>
  select(Movie, Genre, WorldwideBO)

HollywoodMovies |>
  arrange(desc(WorldwideBO)) |>
  slice(1:5)

# Top three values of a derived quantity; ties are retained by default
HollywoodMovies |>
  mutate(score_difference = abs(RottenTomatoes - AudienceScore)) |>
  slice_max(order_by = score_difference, n = 3)

Summary statistics: summarise()

HollywoodMovies |>
  summarise(mean_worldwide = mean(WorldwideBO, na.rm = TRUE), 
            median_worldwide = median(WorldwideBO, na.rm = TRUE), 
            movie_count = n())

Grouping: group_by()

If you are only interested in action movies, say, you might use filter().

HollywoodMovies |>
  filter(Genre == "Action") |>
  summarise(mean_worldwide = mean(WorldwideBO, na.rm = TRUE), 
            median_worldwide = median(WorldwideBO, na.rm = TRUE), 
            movie_count = n())

If you now want to generalize to other genres, rather than manually changing “Action” to other genres, you can group by genre instead.

HollywoodMovies |>
  group_by(Genre) |>
  summarise(mean_worldwide = mean(WorldwideBO, na.rm = TRUE), 
            median_worldwide = median(WorldwideBO, na.rm = TRUE), 
            movie_count = n(),
            .groups = "drop")

The .groups = "drop" argument ensures that the resulting data frame is no longer grouped by genre, so that later operations are not affected.

Can group by multiple variables.

HollywoodMovies |>
  group_by(Genre, Year) |>
  summarise(mean_worldwide = mean(WorldwideBO, na.rm = TRUE), 
            median_worldwide = median(WorldwideBO, na.rm = TRUE), 
            movie_count = n(),
            .groups = "drop")

Example: top 5 most profitable genres

Show code
HollywoodMovies |>
  filter(!is.na(Profitability)) |>
  group_by(Genre) |>
  summarise(
    mean_profit = mean(Profitability),
    movie_count = n(),
    .groups = "drop"
  ) |>
  slice_max(order_by = mean_profit, n = 5)
# A tibble: 5 × 3
  Genre               mean_profit movie_count
  <chr>                     <dbl>       <int>
1 Concert/Performance       1743.           1
2 Horror                     812.          59
3 Black Comedy               359.           6
4 Romantic Comedy            354.           7
5 Drama                      325.          75

Note: nothing is stored; no change is made in the data frame.

Modify and create variables: mutate

# Create a derived column and select the five largest values
HollywoodMovies |>
  mutate(score_gap = abs(RottenTomatoes - AudienceScore)) |>
  slice_max(order_by = score_gap, n = 5) |>
  select(Movie, RottenTomatoes, AudienceScore, score_gap)

# Save the updated data frame using explicit assignment
movies_with_gap <- HollywoodMovies |>
  mutate(score_gap = abs(RottenTomatoes - AudienceScore))

# Keep only selected columns while creating a derived column
HollywoodMovies |>
  transmute(Movie, score_gap = abs(RottenTomatoes - AudienceScore))

# Rename existing columns
HollywoodMovies |>
  rename(open_weekend_gross = OpenBO, theaters_at_open = TheatersOpen)

Tibble

A tibble is a modern version of an R data frame. It prints a compact preview and preserves familiar data-frame behavior while integrating with tidyverse functions.

tbl_movie <- tibble::as_tibble(HollywoodMovies)
class(tbl_movie)

Review question solutions

1. Movie counts by genre

How many genres are there, and how many movies are in each genre?

Show code
# Concise approach
HollywoodMovies |>
  count(Genre, name = "movie_count", sort = TRUE)

# Alternative using group_by(), summarise(), and n()
HollywoodMovies |>
  group_by(Genre) |>
  summarise(movie_count = n(), .groups = "drop") |>
  arrange(desc(movie_count))

2. Largest score differences

What are the 10 movies with the largest absolute difference between RottenTomatoes and AudienceScore?

Show code
# slice_max() approach
HollywoodMovies |>
  mutate(score_gap = abs(RottenTomatoes - AudienceScore)) |>
  slice_max(order_by = score_gap, n = 10) |>
  select(Movie, RottenTomatoes, AudienceScore, score_gap)

# Alternative using arrange() and slice()
HollywoodMovies |>
  mutate(score_gap = abs(RottenTomatoes - AudienceScore)) |>
  arrange(desc(score_gap)) |>
  slice(1:10) |>
  select(Movie, RottenTomatoes, AudienceScore, score_gap)

3. Mean audience score by genre

Compare the mean AudienceScore by genre. This uses the grouped summary method introduced above.

Show code
HollywoodMovies |>
  group_by(Genre) |>
  summarise(
    mean_audience = mean(AudienceScore, na.rm = TRUE),
    .groups = "drop"
  ) |>
  arrange(desc(mean_audience))

4. Highest-grossing action movies

What are the five action movies with the largest WorldwideBO?

Show code
# slice_max() approach
HollywoodMovies |>
  filter(Genre == "Action") |>
  slice_max(order_by = WorldwideBO, n = 5) |>
  select(Movie, Genre, WorldwideBO)

# Alternative using arrange() and slice()
HollywoodMovies |>
  filter(Genre == "Action") |>
  arrange(desc(WorldwideBO)) |>
  slice(1:5) |>
  select(Movie, Genre, WorldwideBO)

5. Highest-grossing movie in each genre

Which movie has the largest WorldwideBO in each genre? slice_max() retains ties.

Show code
HollywoodMovies |>
  group_by(Genre) |>
  slice_max(order_by = WorldwideBO, n = 1) |>
  ungroup() |>
  select(Movie, Genre, WorldwideBO)