Extracting Tweets

The Twitter API makes it very easy to download tweet history for a user, a the rtweet (Kearney 2018) package has been developed to provide an interface with this to R. You will need to sign up for a developer account to be able to access the API. From my experience, the process was not overly difficult, but there was almost a three week wait in my application being approved. Once you have an account, you will need to authenticate it with R as explained here.

Having setup the package, the tweet history for a user can be extracted using the get_timelines function. This extracts up to 3200 recent tweets from a user and provides lots of metdata for each tweet (date, time, text, links, location etc.). This is shown below:

# scrape the tweets
tweets_pab <- get_timelines(c("pab_james"), n = 3200)

Cleaning the Data

Once the tweet history has been extracted, it must be formatted and cleaned for the plot. Firstly, the column text is collapsed into a single character vector:

# Clean the data
text <- str_c(tweets_pab$text, collapse = "")

We need to clean the text in the string. The str_remove function is used to remove linebreaks, hyperlinks, any hashtags and mentions. We are also not interested in keeping any basic words such as “a”, “the”, “and” etc., so we can use the removeWords and stopwords function from the tm (Feinerer and Hornik 2018) package. In addition, the qdapRegex package (Rinker 2017) is used to strip out the URLs:

# continue cleaning the text
text <- 
  text %>%
  str_remove("\\n") %>%                   # remove linebreaks
  rm_twitter_url() %>%                    # Remove URLS
  rm_url() %>%
  str_remove_all("#\\S+") %>%             # Remove any hashtags
  str_remove_all("@\\S+") %>%             # Remove any @ mentions
  removeWords(stopwords("english")) %>%   # Remove common words (a, the, it etc.)
  removeNumbers() %>%
  stripWhitespace() %>%
  removeWords(c("amp"))                   # Final cleanup of other small changes

Having cleaned the data, we can format the table. The function ‘TermDocumentMatrix’ is used to construct a frequency table of the words from the text string above. This table is sorted by frequency to make it easier to inspect. A quick summary of the most common words is shown in Table 1.

# Convert the data into a summary table
textCorpus <- 
  Corpus(VectorSource(text)) %>%
  TermDocumentMatrix() %>%
  as.matrix()

textCorpus <- sort(rowSums(textCorpus), decreasing=TRUE)
textCorpus <- data.frame(word = names(textCorpus), freq=textCorpus, row.names = NULL)

Building the Wordcloud

Finally, we can build the wordcloud. There are two main options which can be used this: either wordcloud or wordcloud2. For the example, I have used the wordcloud2 package (Lang 2018), as it offered a few more functions for customising the output. Below, we use the frequency table developed above to create the wordlcloud, as shown in Figure 1.

# build wordcloud 
wordcloud <- wordcloud2(data = textCorpus, minRotation = 0, maxRotation = 0, ellipticity = 0.6)
wordcloud

We can play around with this basic setup, and I would recommend checking out the package documentation to see some of the things that can be done. For example, we can provide our own image as a mask to customise the shape of the wordcloud.