Importing Data 2

Question 1

How do you import beautifulsoup?

Answer 1

from bs4 import BeautifulSoup

Question 2

Full sample script of using BeautifulSoup

Answer 2

1. # Import packages
2. import requests
3. from bs4 import BeautifulSoup
4. # Specify url: url
5. url = ‘https://www.python.org/~guido/’
6. # Package the request, send the request and catch the response: r
7. r = requests.get(url)
8. # Extract the response as html: html_doc
9. html_doc = r.text
10. # Create a BeautifulSoup object from the HTML: soup
11. soup=BeautifulSoup(html_doc)
12. # Get the title of Guido’s webpage: guido_title
13. guido_title=soup.title
14. # Print the title of Guido’s webpage to the shell
15. print(guido_title)
16. # Get Guido’s text: guido_text
17. guido_text=soup.get_text()
18. # Print Guido’s text to the shell
19. print(guido_text)

Question 3

What does the BeautifulSoup method find_all(‘a’) do?

Answer 3

It returns a result set of all hyperlinks on the page. Sample code as to how to extract all hyperlinks:

Find all ‘a’ tags (which define hyperlinks): a_tags
a_tags=soup.find_all(‘a’)

Print the URLs to the shell
for link in a_tags:
print(link.get(‘href’))

Question 4

How do you load a json file?

Answer 4

import json with open(“a_movie.json”) as json_file: json_data = json.load(json_file)

Question 5

Code to load and then read and display json as a dictionary?

Answer 5

Import package
import requests

Assign URL to variable: url
url = ‘http://www.omdbapi.com/?apikey=ff21610b&t=social+network’

Package the request, send the request and catch the response: r
r=requests.get(url)

Decode the JSON data into a dictionary: json_data
json_data=r.json()

Print each key-value pair in json_data
for k in json_data.keys():
print(k + ‘: ‘, json_data[k])