10  String Processing

lines <- readLines("./data/clinton.txt")

A “string” is a sequence of characters that are bound together, where a character is a symbol is a written language. To illustrate the concept of strings and string manipulation, we will use Hillary Clinton’s speech from 2016 in which she accepts the Democratic Party nomination for president. This speech is filled with strings that are provided over a number of lines (each separated by a newline character). The R function readLines() simply reads the strings from the source file, line by line; here, the first line is

lines[1]

[1] "Thank you! Thank you all very much! Thank you for that amazing welcome. "

Note that in R, a string is of class character and is bounded by quotes (either single or double). Double quotes are preferable, because then one can use single quotes as apostrophes in strings.

Let’s look again at the first line of Clinton’s speech. How long is it?

length(lines[1])

[1] 1

length() does not tell us how many characters are in a string, but how many strings are contained in the input. (Here, there is one string.) To determine the number of characters, we would use nchar():

nchar(lines[1])

[1] 72

There are 72 characters in the string, including punctuation marks and spaces. To see what some of those characters are (like the first ten), it is tempting to type something like the following:

(lines[1])[1:10]

But that will not work…we need to use substr() (for “substring”) instead:

substr(lines[1], 1, 10)

[1] "Thank you!"

Then, to see the last ten, we would combine substr() with nchar(), taking care to note that the range is inclusive, so we’d specify n-9 as opposed to n-10:

n <- nchar(lines[1])
substr(lines[1], n-9, n)

[1] " welcome. "

The next thing we might wish to do is to split up a string, so to (for instance) get rid of spaces and to build up a collection of actual words:

s <- strsplit(lines[1], split = " ")  # split on a single space
s

[[1]]
 [1] "Thank"    "you!"     "Thank"    "you"      "all"      "very"    
 [7] "much!"    "Thank"    "you"      "for"      "that"     "amazing" 
[13] "welcome."

In the output, we immediately notice that…

• strsplit() returns a list, with each element of the list mapping back to each element of the string vector that was input (here, we input one lines, and got back a list with one elements);

• each list element contains a vector of split-up characters; and

• not every output string is actually a word!

Before we turn to the concept of “regular expressions,” which will, for example, help us achieve more “granular” string splitting, we will mention how we can concatenate strings together. The basic string concatenator is paste():

u <- unlist(s)    # turn s into a vector
paste(u[1], u[2]) # now paste the vector elements together

[1] "Thank you!"

#paste(u[1:2])    # this doesn't work the way you think it would

By default, each component string must be specified as a separate argument to paste(). Also note that by default, strings are concatenated with a space separating them. We can explicitly specify the separator using the sep argument

paste(u[1], u[2], sep = " cha cha cha ")

[1] "Thank cha cha cha you!"

Now, when we want to pass in a vector of strings, we have to specify the collapse argument, which itself specifies the separator:

paste(u, collapse = " ")

[1] "Thank you! Thank you all very much! Thank you for that amazing welcome."

We will note for completeness that, R being R, it is possible to combine both sep and collapse, as in the following example:

paste(c(u[1], u[3]), c(u[2], u[4]), sep = " ")

[1] "Thank you!" "Thank you" 

paste(c(u[1], u[3]), c(u[2], u[4]), sep = " ", collapse = ",")

[1] "Thank you!,Thank you"

10.1 Regular Expressions

Regular expressions, or regexes, are specially constructed strings that allow for flexible pattern matching. (It is important to note that the rules for constructing regexes are independent of R!) Here, we will focus on the use of square brackets and metacharacters to define a regex.

We use square brackets to indicate that we want to match any one of a number of defined characters. For instance:

• “[abcde]” means “look for any string that contains a, b, c, d, or e” (case sensitive!)

• “[a-e]” means the same thing; the dash denotes a range

• “[^a-e]” means “look for any string that contain characters other than a, b, c, d, or e”

• ” [1-4][2-6] ” matches strings that contain the numbers 12-16, 22-26, 32-36, or 42-46

To demonstrate how this works, let’s split the first line from the speech on spaces and exclamation points:

strsplit(lines[1], split = "[ !]")

[[1]]
 [1] "Thank"    "you"      ""         "Thank"    "you"      "all"     
 [7] "very"     "much"     ""         "Thank"    "you"      "for"     
[13] "that"     "amazing"  "welcome."

What we observe is that we have effectively removed the spaces and the exclamation points, but not the period, and that we have introduced empty strings into the output. This occurs when there is space and an exclamation point (or vice-versa) together; when they are split, an empty string is introduced. (For instance, “!” is split into “!”, ““, and” “, with the first and last characters discarded.) Empty strings have to be dealt with, but that is easy; we’ll show how to remove them from a vector of strings below.

Now, it might occur to the reader that, e.g., if we want to split on a large number of possible characters, the coding would be tedious. That’s why, when we can, we would employ metacharacters.Commonly used metacharacters include

• [[:alnum:]], which is the same as [a-zA-Z0-9];

• [[:punct:]], which means “match any string that contains a punctuation mark”; and

• [[:space:]], which means “match any string that contains a space, a tab, or a new line”

strsplit(lines[1], split = "( |[[:punct:]])")

[[1]]
 [1] "Thank"   "you"     ""        "Thank"   "you"     "all"     "very"   
 [8] "much"    ""        "Thank"   "you"     "for"     "that"    "amazing"
[15] "welcome" ""       

We’ll stop here because we’ve (basically) made the point of how to turn text input into words…well, except for one last thing. In R, the following characters are special characters

. $ ^ * + ? \ | { } [ ] ( ) \

To find occurrences of these symbols in strings, we have to use an escape sequence, i.e., we have to place a backslash in front of the symbol (but given that the backslash is a special character, it itself needs to be escaped…hence the double backslash in the example below).

strsplit(lines[1], split = "[ !\\.]")

[[1]]
 [1] "Thank"   "you"     ""        "Thank"   "you"     "all"     "very"   
 [8] "much"    ""        "Thank"   "you"     "for"     "that"    "amazing"
[15] "welcome" ""       

10.2 String Searching, Extraction, and Replacement

If we want to see if a particular word appears in a line, we can use the grep family of functions. For instance, if you want to determine if “And” occurs on a line, use grepl(), which turns TRUE or FALSE.

grepl("And", lines[1:5])

[1] FALSE FALSE FALSE FALSE  TRUE

The output indicates that “And” appears on line 5.Another way to determine this is to use the grep() function itself, which returns the number of the line in which the string is observed:

grep("and", lines[1:5])

[1] 5

If we want to have the contents of the line itself as opposed to the line number, we would pass in the argument value=TRUE.

grep("and", lines[1:5], value = TRUE)

[1] "And Chelsea, thank you. I'm so proud to be your mother and so proud of the woman you've become. Thank you for bringing Marc into our family, and Charlotte and Aidan into the world. "

As we might expect, we can utilize regexes when searching for strings, as in the following.

grepl("[and|And]", lines[1:5])

[1]  TRUE FALSE  TRUE FALSE  TRUE

Here, we see that “and” or “And” occurs one or more times on each of the first five lines, except on line four.

Now, we saw above that we can use substr() to extract a substring. However, we need to specify where the substring starts, and where it ends. A more dynamic extractor involves combining the functions gregexpr() and regmatches(). To demonstrate how to use these,let’s first extract every occurrence of “and” and “And” in the first five lines of Clinton’s speech.

out     <- gregexpr("(a|A)nd",lines[1:5])
matches <- regmatches(lines[1:5],out)
unlist(matches)

[1] "And" "and" "and" "and"

Let’s say that instead of splitting on punctuation, as we (eventually) did above, we want to remove or replace them. One way to do that is to use gsub()

gsub("[[:punct:]]", "-", lines[1])

[1] "Thank you- Thank you all very much- Thank you for that amazing welcome- "

10.3 Word Tables and Stopwords

What are the 20 most common words in Clinton’s speech?

sort(table(unlist(strsplit(lines, split = "[ !\\.]"))), decreasing = TRUE)[1:20]

      the   to  and    a   of   in  you  our   we  And that    I  for    -   is 
 253  199  171  157   99   97   72   70   69   69   59   58   56   53   43   42 
 are   it will with 
  36   36   36   36 

We note that we need to do some additional processing to remove the empty strings and that case sensitivity impacts the count (e.g., “and” and “And” are treated separately). Case sensitivity can be mitigated by applying the tolower() function after unlist():

sort(table(tolower(unlist(strsplit(lines, split = "[ !\\.]")))), decreasing = TRUE)[1:20]

      and  the   to    a   of   we  you   in  our that  for    i    -   is   it 
 253  216  207  172  105   97   97   78   76   75   61   59   56   43   42   42 
  he will  are from 
  39   37   36   36 

We see now that we have 216 instances of “and,” rather that 157 of “and” and another 59 of “And.” But…do we really care about how many times Clinton said the word “and”? One last thing for us to look at here is the removal of uninformative stopwords from a document, like “a,” “and,” and “the.” The easiest way to do that is to use a precompiled list of supposedly uninformative words, as given by, e.g., the stopwords package.

library(stopwords)
head(stopwords("en"),10)

 [1] "i"         "me"        "my"        "myself"    "we"        "our"      
 [7] "ours"      "ourselves" "you"       "your"     

Ultimately, the package provides a list of 175 English stopwords. Below, we show the most common words in Clinton’s speech after simplistic processing and stopword removal.

speech           <- tolower(unlist(strsplit(lines, split = "[ ,!\\.]")))
w                <- which(nchar(speech) == 0)
speech           <- speech[-w]
# is word in speech "in" stopwords? [T/F]
stopword.logical <- speech %in% stopwords("en")
speech.edit      <- speech[stopword.logical == FALSE]
sort(table(speech.edit), decreasing = TRUE)[1:20]

speech.edit
        -    people        us       can      just   country   america       now 
       43        37        30        26        26        24        22        20 
  believe      know      make president     going     trump   working      work 
       19        19        19        19        17        17        17        16 
   donald  together       get       one 
       15        15        14        13 

Well…it appears the emdashes still need to be removed. This illustrates a truism about string processing: it is iterative!