Scraping data from the web is a common tool for data analysis. In fact, it is very creative and ensures a unique data set that no one else has analyzed before. Often times, we can use packages such as rvest, scrapeR, or Rcrawler to get the job done. However, sometimes we want to scrape dynamic web pages that can only be scraped with RSelenium. This RSelenium tutorial will introduce you to how web scraping works with the R package.

RSelenium automates a web browser and lets us scrape content that is dynamically altered by JavaScript for example.

In this RSelenium tutorial, we will be going over two examples of how it can be used.

• For example #1, we want to get some latitude and longitude coordinates for some street addresses we have in our data set. In order to do that, we have to let RSelenium type in our addresses, hit the enter button, and then scrape the latitude and longitude coordinates from the website.
• For example #2, we are doing something similar with postal codes.

Let’s jump into our examples and this RSelenium tutorial!

UPDATE 09/11/2019: 

• After having trouble opening a remote driver because the version did not match with the RSelenium package, I changed the web driver version here.
• I also fixed some typos thanks to Sam’s comment!

UPDATE 16/02/2020: 

• After I had trouble again connecting to my chrome browser, I found the following solution on StackOverflow. I copy-pasted the code from there for windows which you can see below.
• You can find the code for this tutorial on my GitHub.

Example #1

Step 1: Navigate to the URL

For the first example, we are going to visit https://www.latlong.net/.

In the picture above, we can see the text box Place Name , where we are going to let RSelenium type in our street addresses. Afterward, we have to let RSelenium click the Find button and then we have to scrape the results that will appear in the Latitude and Longitude boxes.

Step 2: Let RSelenium Type in the Necessary Fields

################
### Original ###
################
library(RSelenium)
library(tidyverse)
 
driver <- rsDriver(browser=c("chrome"))
remote_driver <- driver[["client"]]
remote_driver$open()
 
remote_driver$navigate("https://www.latlong.net/convert-address-to-lat-long.html")

#########################
### UPDATE 09/11/2019 ###
#########################

driver <- rsDriver(browser = c("chrome"), chromever = "78.0.3904.70")
remote_driver <- driver[["client"]] 
remote_driver$navigate("https://www.latlong.net/convert-address-to-lat-long.html")

#########################
### UPDATE 16/02/2020 ###
#########################

driver <- RSelenium::rsDriver(browser = "chrome",
                              chromever =
                                system2(command = "wmic",
                                        args = 'datafile where name="C:\\\\Program Files (x86)\\\\Google\\\\Chrome\\\\Application\\\\chrome.exe" get Version /value',
                                        stdout = TRUE,
                                        stderr = TRUE) %>%
                                stringr::str_extract(pattern = "(?<=Version=)\\d+\\.\\d+\\.\\d+\\.") %>%
                                magrittr::extract(!is.na(.)) %>%
                                stringr::str_replace_all(pattern = "\\.",
                                                         replacement = "\\\\.") %>%
                                paste0("^",  .) %>%
                                stringr::str_subset(string =
                                                      binman::list_versions(appname = "chromedriver") %>%
                                                      dplyr::last()) %>%
                                as.numeric_version() %>%
                                max() %>%
                                as.character())

remote_driver <- driver[["client"]] 
remote_driver$navigate("https://www.latlong.net/convert-address-to-lat-long.html")

First, we have to load the library. Then we are connecting to the Chrome driver and navigate to the desired URL we want to scrape data from.

Now, we have to have a look at what location the Place Name box is located in the HTML code.

When looking at the HTML code, then we can see that the box is located in this snippet above with the XPath @class = “width70”. So, the code below shows how to navigate to that particular text box.

address_element <- remote_driver$findElement(using = 'class', value = 'width70')

Now, we have to let RSelenium type in the address we want to get coordinates for.

address_element$sendKeysToElement(list("Lombard Street, San Francisco"))

We are almost done. Now we have to press the Find button in order to get the coordinates.

In the code below, we are using the XPath @class = “button”, to locate the button.

button_element <- remote_driver$findElement(using = 'class', value = "button")

After we have located the button, we have to click it.

button_element$clickElement()

Step 3: Scrape the Coordinates From the Website

When we scroll down to then we see the coordinates like this:

They are located here in the HTML code:

Under the XPath @class = “coordinatetxt”.

out <- remote_driver$findElement(using = "class", value="coordinatetxt")
lat_long <- out$getElementText()

When we have a lot of addresses we want to get coordinates for, then this could be accomplished like that:

street_names <- c("Lombard Street, San Francisco", 
                  "Santa Monica Boulevard", 
                  "Bourbon Street, New Orleans", 
                  "Fifth Avenue, New York", 
                  "Richards Street, Vancouver")

get_lat_lon <- function(street_names) {
  remote_driver$navigate("https://www.latlong.net/convert-address-to-lat-long.html")
  final <- c()
  for(i in 1:length(street_names)) {
    
    remote_driver$refresh()
    Sys.sleep(1)
    
    address_element <- remote_driver$findElement(using = 'class', value = 'width70')
    
    address_element$sendKeysToElement(list(street_names[i]))
    button_element <- remote_driver$findElement(using = 'class', value = "button")
    
    button_element$clickElement()
    Sys.sleep(3)
    
    out <- remote_driver$findElement(using = "class", value = "coordinatetxt")
    output <- out$getElementText()
    final <- c(final, output)
    
  }
  
  return(final)
}


vector_out <- get_lat_lon(street_names)

After, we can extract the latitude and longitude values with the code below

data.frame(street_names, purrr::flatten_chr(vector_out)) %>%
  dplyr::mutate(., vector_out = stringr::str_remove_all(vector_out, "\\(|\\)")) %>%
  tidyr::separate(., vector_out, into = c("latitude", "longitude"), sep = ",")

Output:

                   street_names   latitude    longitude
1 Lombard Street, San Francisco  37.799999  -122.434402
2        Santa Monica Boulevard -27.867491   153.353973
3   Bourbon Street, New Orleans  29.964100   -90.060791
4        Fifth Avenue, New York  42.105412   -76.247070
5    Richards Street, Vancouver  49.279030  -123.119431

Let’s jump to the next example of this RSelenium tutorial.

Example #2

Step 1: Navigate to the URL

As previously, we want to go to the website where we want to scrape data from. In our second example, we will be using the https://www.canadapost.ca/cpo/mc/personal/postalcode/fpc.jsf# url.

Again, we can see the box where we have to enter our address and the search button we have to click after we inserted our address.

Step 2: Let RSelenium Type in the Necessary Fields

First, we have to navigate to the desired URL.

url <- "https://www.canadapost.ca/cpo/mc/personal/postalcode/fpc.jsf"
remote_driver$navigate(url)

Then, we have to tell RSelenium to put in the desired address in the box. We do that, by locating where the box lies in the HTML code.

The XPath is underlined in green. The code to put text in the text box looks like this:

address_element <- remote_driver$findElement(using = 'id', value = 'addressComplete')
address_element$sendKeysToElement(list("413 Seymour Street Vancouver"))

Now, we have to locate the Search button in order to get the postal code for the address.

 

The XPath is underlined in green.

To click to the search button, we have to execute the following code:

button_element <- remote_driver$findElement(using = 'id', value = 'searchFpc')
button_element$clickElement()


##############
### Update ###
##############

# website has changed a bit so try code below
button_element <- remote_driver$findElement(class = 'class', value = 'clear')
button_element$clickElement()

After that, we only have to extract the desired information and we are done!

Step 3: Scrape the Postal Code From the Website

In order to get the address we have to do the following:

output <- remote_driver$findElement(using = "id", value="HeaderAddressLabel")
output <- output$getElementText()

Output:

"413 SEYMOUR ST\nVANCOUVER BC   V6B 3H5"

To only get the postal code, we can simply do:

unlist(output) %>%
  stringr::str_sub(., start = -7, end = -1)

Output:
"V6B 3H5"

I hope you have enjoyed this short RSelenium tutorial about web scraping. If you have any questions or suggestions then let me know in the comments below.

Additional Resources

• If you are interested in other web scraping tutorials, then you can check out my post about scraping Indeed Job Postings.
• Another example of web scraping would be my post about building a scraper for a real estate website.