Gender Parity and Dialogue in 2016’s Highest Grossing Films

Choosing the Movies

Originally, I planned to look at the transcripts of all the Star Wars movies to see if female representation really has changed over the years, but again, several of those movies were created almost 40 years ago. Is it fair to compare them directly? So, I started thinking about comparing movies that were all created at the same time. That is, movies that were all released in 2016. To limit this project to some sort of semi-reasonable scope, I decided to keep it to the Top 10 Highest Grossing Movies Worldwide from 2016. That left me with the following movies in this order:

• Captain America: Civil War
• Finding Dory
• Zootopia
• The Jungle Book
• The Secret Life of Pets
• Batman V. Superman: Dawn of Justice
• Rogue One: A Star Wars Story
• Deadpool
• Fantastic Beasts and Where to Find Them
• Suicide Squad

That’s 4 superhero/action movies, 2 Sci-Fi/Fantasy, and 4 animated or family movies. Not a perfect distribution of movie genres, but at least there’s a bit of variety.

Deciding on Data Collection

I decided to delve a little bit into similar data exploration projects that have been conducted (like this one or this one) and found that I could look at a few things:

• On-Screen Time
• Words Spoken per Character
• Number of Female Characters
• Number of Female “Lead” Characters

While some really cool technology exists to help measure screen time in movies, it seemed to either be something that I couldn’t use for a side project fueled by my own curiosity or had to be done manually and would thus have taken forever.

Instead, I decided to focus on dialogue. I considered using screenplays as my source for movie dialogue, but so many of them were not publicly available. However, dedicated movie fans tend to transcribe movie dialogue and post the transcripts on the internet. Tracking those down took some work, but once I found them, it was relatively smooth sailing. For a few, I couldn’t find a transcript, but I was able to find closed caption files. For those, I had to re-watch the movie and manually add in which character said each line.

I did mention that I was really curious about this issue, right?

Assigning Gender to Characters

Now that I had a data frame with both Character and Words columns, I had to assign Genders to each Character. This was sometimes quite challenging. Here’s how I did it:

1. When possible, use the pronoun that other characters use. If a character is referred to by others as “he” or “him”, then it’s a male character.

2. If there is no pronoun used throughout the movie but the character is named or credited, use the gender of the actor/actress (using IMDB).

Disclaimer: Gender of actors/actresses assumed based on publicly available information as of January 2017.

3. If there is no pronoun used for a character and the character is not named or credited, look at the closed captions. Sometimes they will name the character that spoke.

4. If all else fails, make an educated guess based on the character’s voice.

I’ll be the first to say that these methods are not perfect. But for a personal project where I just want to explore some data, I feel like it’s pretty good. However, if you caught a mistake on my part, please let me know.

In R, I assigned genders to characters like this:

# Assign Genders to Rogue One Characters
RO_males <- c("Admiral Raddus", "Announcer", "Bail Organa", "Baze Malbus", "Blue Squadron 1", "Blue Squadron 2", "Blue Squadron 3", "Blue Squadron 4", "Bodhi Rook", "C3PO", "Captain Antilles", "Cassian", "Chirrut Imwe", "Corvette 5", "Edrio", "Imperial Soldiers", "Engineers", "Flight Control", "Galen", "Gate Control", "General Dodonna", "General Draven", "General Merrick", "General Ramda", "Gold Leader", "Governor Tarkin", "Imperial Assistant", "Imperial Assistant 2", "Imperial Assistant 3", "Imperial Controller", "Imperial Mission Control", "Imperial Mission Control 2", "Imperial Officer", "Imperial Officer 2", "Imperial Officer 3", "Imperial Pilot", "Imperial Pilot 2", "Inspector", "K2SO", "Krennic", "Lieutenant Sefla", "Dr. Cornelius Evazan", "Pad 12", "Prisoner", "Rebel Announcer", "Rebel Army", "Rebel Fighter", "Rebel Officer", "Rebel Officer 2", "Rebel Pilot", "Rebel Pilot 2", "Rebel Pilot 3", "Rebel Pilot 4", "Rebel Pilot 5", "Rebel Soldier", "Rebel Soldier 1", "Rebel Soldier 10", "Rebel Soldier 2", "Rebel Soldier 3", "Rebel Soldier 4", "Rebel Soldier 5", "Rebel Soldier 6", "Rebel Soldier 7", "Rebel Soldier 8", "Rebel Soldier 9", "Red 5", "Red Leader", "Saw Gerrera", "Senator Jebel", "Senator Vaspar", "Sergeant Melshi", "Ship Computer", "Stormtrooper 1", "Stormtrooper 2", "Stormtrooper 3", "Stormtrooper 4", "Stormtrooper 5", "Stormtrooper 6", "Stormtrooper 7", "Stormtrooper 8", "Stormtrooper 9", "Stormtrooper 10", "Stormtrooper 11", "Stormtrooper 12", "Stormtrooper 13", "Stormtrooper 14", "Stormtrooper 15", "Stormtrooper 16", "Stormtrooper 17", "Stormtrooper 18", "Technician", "Tivik", "Unknown Senator", "Unknown Senator 2", "Vader", "Vader's Assistant")
RO_females <- c("Antenna Computer", "Girl's Mother", "Jyn", "Leia", "Lyra", "Mon Mothma", "Rebel Pilot (Female)", "Rebel Pilot (Female) 2", "Rebel Pilot (Female) 3", "Senator Pamlo")
RO_full <- RO_full %>% 
            mutate(Gender = ifelse(Character %in% RO_females, "female", 
                                ifelse(Character %in% RO_males, "male", "unknown")))

Some questions I’ve already received about this process:

• What if a male character is voiced by a female actress (e.g., Tommy Pickles in Rugrats) or a female character is voiced by a male actor (e.g., Roz in Monster’s Inc.)?
  • Well, in both of those cases, I’d have assigned the gender based on the pronouns used by other characters before getting to the actor/actress’ gender. The same methodology applies here. To my knowledge, I didn’t actually run into that problem with these movies. However, if an unnamed rhino was voiced by a female, I did assume it was a female rhino.
• Can you really know the gender of an animated character?
  • Nope! Again, I am relying on the other characters in the film to know what to call one another. This may be slightly flawed but it’s the best I can do with the information I have.

Counting Words

Now that each of the characters has an assigned gender, it’s time to count how many words each character spoke. In R, I used the dplyr and stringi packages for this.

# Counting words per character
RO_full2 <- RO_full %>% 
                mutate(count = stri_count(Words, regex = "\\S+")) %>% 
                group_by(Character, Gender) %>% 
                summarise(Total_Words = sum(count)) %>% 
                filter(!(Gender == "unknown"))

I should note that I did not include a “minimum number of spoken words limit” to this project. So that means that any character that spoke even a single word is included. And yes, that means every stormtrooper that uttered at least one word will be in here.

In R

In Illustrator

In the end, I decided to make an axis-free bubble chart. I should point out that I am perfectly aware of the arguments against bubble charts (look here), but in this case, I thought the visualization would be effective.

The problem? I had no idea how to make an axis-free bubble chart in R.

Before going down the rabbit-hole of making a new type of graphic in R, I decided to use my trusty Adobe Illustrator to quickly see what these visualizations might look like first. So I pulled my highcharter bubble charts into Illustrator and traced the circles to get an idea of relative size. I ended up with lots of groups of bubbles that looked like this:

But there were a few issues with this. While I liked that you could easily spot the main characters, I knew that there were several characters in Rogue One who had fewer than 10 words, but their bubbles seemed to indicate more. I’d clearly have to alter how I sized the circles. It was also difficult to get an idea of what percentage of the characters or spoken words actually came from female characters.

So, I adjusted the figure. I calculated the proper radius of a circle so that each circle’s area was scaled to the total number of words spoken by each character.

I did this in R.

RO_full2 <- RO_full2 %>% 
            mutate(radius = sqrt(Total_Words),
                   diameter = radius*2)

I could now manually enter the necessary diameter for each circle into Adobe Illustrator.

While a full how-to for Adobe Illustrator design is outside the scope of this post, you can find lots of information on Adobe’s site.

That definitely helped make the circles a more reasonable size. Now to add some extra information, like the percentage of female characters and spoken words. Again, I calculated in R and added the data to Illustrator. I thought that the best way to display that information was just to put a circle around the bubbles, cut the circle, and color it to match the bubbles it was near. Then add a call-out of information. It looked like this:

I was decently happy with it, until I (luckily) asked a friend to look at it. Her first response was asking about which percentage the pink outer line represented: the percentage of female characters or percentage of words spoken by females. It was an excellent question because, although I had intended the outer lines to be mostly decorative, they were so reminiscent of a pie or donut graph that we instinctively read it this way. So either I had to find a way to get the bubbles into “pie” pieces or get rid of the outer lines.

Back to the drawing board.

Adding Interactivity with d3.js

While I was still trying to figure out the best way to show the overall character and word percentages for each movie, I ran into another issue. I loved the bubbles, but I wanted to know who each bubble represented. Where were my favorite characters in this bubble mass? I briefly toyed with the idea of adding an arrow with a character name to each bubble or trying to put names in the bubbles, but with so many characters and such a wide range of bubble sizes, I knew it would get too cluttered.

Luckily for me, I had recently stumbled upon a fantastic data visualization collaboration called data sketch|es. I absolutely loved their unique data visualizations and the way they added interactive elements when necessary. After looking through their write-ups, I knew that interactivity, and more specifically: tooltips, were exactly what my visualization needed and it looked like the best way to do that would be using d3.js.

However, when I started this project, I had never once touched d3.js or Javascript. I used packages like highcharter which wrapped Javascript and let me use it in R. When I first decided to add tooltips to this visualization I was hoping for something similar. I looked for a bridge between the programs I knew (R or Adobe Illustrator) and d3.js. I found that you could import .svg files from Illustrator into d3.js and add interactive tooltips to them, but all of the values had to be manually entered. That seemed like it was asking for typos.

The more I looked, the more I realized I’d need to learn some d3.js to do this the way I want. So, to internet classes and StackOverflow I went. Somewhere along the line, I stumbled upon a fantastic interactive bubble chart from the New York Times and decided that if I was going to make my bubbles interactive, I also wanted to add the ability to separate them by gender. I was able to find a few excellent tutorials on how to make this happen (here and here). So I played and experimented and played some more. When I got stuck I returned to Google searching and reading. At one point I was stuck on a problem for hours and could not find a workable solution, so I turned to the kind people of the internet, asked for help, and was lucky enough to have a very knowledgable user respond right away.

I’m uploading my .js, .css, and .html files to Github, so anyone interested in exploring them, please do. As I learn more Javascript I hope to come back to this project and optimize the code. I know right now that it is far from perfect, but for my first project with d3.js, I’m very happy with how it turned out. To give some brief insight into parts of the d3.js code:

I once again scale the circles by area, with a maximum circle diameter of 40 px.

var radiusScale = d3.scaleSqrt().domain([1, 4692]).range([1, 40])

I define a few forces to move the circles throughout their .svg area (a 300 px by 300 px box). There are two separate X forces (forces that move the circles along the X axis) since I needed two conditions: all the bubbles combined and bubbles separated by gender.

// Define the two forces along the X axis to split by gender
// the drawing force for males is at 30% of the width of the svg box
// the drawing force for females is at 70% of the width of the svg box
var forceXSplit = d3.forceX(d => width * (d.Gender === "male" ? 0.3 : 0.7))
        .strength(0.2);
// Define the force along the X axis to combine all bubbles together 
// (at half the width of the svg box)
var forceXCombine = d3.forceX((width)/2).strength(0.1)

Then, in order to keep the bubbles from overlapping one another, I added a collision force. The radius of this force was scaled based on circle size so that larger circles pushed circles further from their center than smaller circles.

var forceCollide = d3.forceCollide(function(d){
        return radiusScale(d.Total_Words) + 1
      })
      .iterations(10);

With all of my forces defined, I could combine them in a simulation to determine the proper location of each bubble in the group.

var simulation = d3.forceSimulation()
    .force("x", forceXCombine)
    .force("y", d3.forceY((height / 3) + 10).strength(0.15))
    .force("collide", forceCollide);

Then I just needed to include my data file so that each circle could be created automatically and sized according to the total number of words spoken by a character. At the same time, I added the tool tips that triggered when the user’s mouse hovered over a circle. Again, the tool tips were automatically filled with information from the data file.

# Export data from R
write.csv(RO_full2, "RogueOne.csv")

// Import data into Javascript
d3.queue()
    .defer(d3.csv, "RogueOne.csv")
    .await(ready)
function ready (error, datapoints) {
    datapoints.forEach(d => d.Total_Words = +d.Total_Words);
    var mousemove = function() {
                return tooltip.style("top", (d3.event.pageY-10)+"px").style("left",(d3.event.pageX+10)+"px");
            }
  var mouseout = function(){return tooltip.style("visibility", "hidden");} 
    var circles = svg.selectAll(".Character")
        .data(datapoints)
        .enter().append("circle")
        .attr("class", "Character")
        .attr("r", function(d){
        return radiusScale(d.Total_Words)
    })
        .style("fill", d => d.Gender === "male" ? "#3b3f93" : "#ff4a6b")
        .on("mouseover", function(d) {
              tooltip.html(d.Character + "<br><br> Words Spoken: " + d.Total_Words);
              tooltip.style("visibility", "visible");
      })
      .on("mousemove", mousemove)
      .on("mouseout", mouseout);
}         

I added a clickable element within each svg box to toggle between the combined and separated conditions for the bubbles. The style and functionality were inspired by the toggle switch widget from the SaVaGe library. I generated rectangles of sizes corresponding to the percentages of male and female characters and words spoken beneath my bubble grouping to give viewers an overall idea of representation within each movie. This percentage-wise stacked bar graph seemed to make the overarching trends clearest. I used Font Awesome icons alongside each bar graph to differentiate between character and word percentages. And that’s about it! The final design looks like this:

Again, the entire code used to generate the d3.js figure is available on my Github and the interactive version is here

Color Considerations

Once I had my bubble charts working with their corresponding bar graphs, I needed to make a design decision: what color should these be? The simple/stereotypical answer when displaying genders (at least in the US) is “blue for boys and pink for girls”. I really wanted to use a different color scheme. I’m not the biggest fan of the color dichotomy we use in America today and didn’t want to perpetuate it in my data visualization. So I tried a few other color schemes. I found beautiful color palettes on the following websites:

• ColourLovers
• Coolers
• I Want Hue
• Dribbble

But I ran into an issue. I must have tried hundreds of color schemes (many of which looked better with the background gray than the ones that I screenshot here).

But regardless of the color scheme, I couldn’t easily tell which group represented males and which represented females. Yes, that’s what legends are for, but in the end, data visualization should be as easy to interpret as possible. In this case, that meant that some shade of blue and pink/red would be the easiest to understand. So, I found my favorite combination of the two and moved on.