5. Practical session: Working with Beautiful Soup

In this lesson, we are going to explore how we can use the package Beautiful Soup to extract content from XML files. We use the same example file that was used in lesson 2 (download here).

This lesson is divided into the following steps:

  • Load the XML file;

  • Examine the structure of the XML file;

  • Extract the booktitles and descriptions;

  • Extract name and surname of the author;

  • Extract the book identifier;

  • Structure all information;

  • Explore namespaces;

  • Extra: Filter information

Open a new Jupyter Notebook and type all the code examples and code exercises in your Notebook.

Install Beautiful Soup

Beautifull Soup is not a standard Python package, so it needs to be installed first. This can be done directly in the Jupyter Notebook using:

!pip install beautifulsoup4

or through the command line (see lesson 1)

pip install beautifulsoup4

Import Beautiful Soup and load the xml file

Before we can use the package, we have to let Python know we want to use it. We do this by importing the package. Type the following in a code cell:

from bs4 import BeautifulSoup  

Now we can use the package to extract data from the XML.

Examine the structure of the file

Now we want to open the XML file from which we want to extract information. Add a new code cell and type:

with open("data/example.xml") as f:
    root = BeautifulSoup(f, 'xml')

Note

In the code above, alter the ‘data/example’ with the path to the folder and the filename of where you stored the file.

When you want to extract information from an XML file, it is important that you are familiar with the structure of the file. There are two ways to do this.

  1. You can open the file in a program like Notepad++ or open it in your browser

  2. You can show the file in your Jupyter Notebook with the following code:

print(root)
<?xml version="1.0" encoding="utf-8"?>
<catalog>
<book id="bk101">
<author>
<name>Matthew</name>
<surname>Gambardella</surname>
</author>
<title>XML Developer's Guide</title>
<genre>Computer</genre>
<price>44.95</price>
<publish_date>2000-10-01</publish_date>
<description>An in-depth look at creating applications 
      with XML.</description>
</book>
<book id="bk102">
<author>
<name>Kim</name>
<surname>Ralls</surname>
</author>
<title>Midnight Rain</title>
<genre>Fantasy</genre>
<price>5.95</price>
<publish_date>2000-12-16</publish_date>
<description>A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world.</description>
</book>
<book id="bk103">
<author>
<name>Eva</name>
<surname>Corets</surname>
</author>
<title>Maeve Ascendant</title>
<genre>Fantasy</genre>
<price>5.95</price>
<publish_date>2000-11-17</publish_date>
<description>After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society.</description>
</book>
<book id="bk104">
<author>
<name>Eva</name>
<surname>Corets</surname>
</author>
<title>Oberon's Legacy</title>
<genre>Fantasy</genre>
<price>5.95</price>
<publish_date>2001-03-10</publish_date>
<description>In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant.</description>
</book>
<book id="bk105">
<author>
<name>Eva</name>
<surname>Corets</surname>
</author>
<title>The Sundered Grail</title>
<genre>Fantasy</genre>
<price>5.95</price>
<publish_date>2001-09-10</publish_date>
<description>The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy.</description>
</book>
<book id="bk106">
<author>
<name>Cynthia</name>
<surname>Randall</surname>
</author>
<title>Lover Birds</title>
<genre>Romance</genre>
<price>4.95</price>
<publish_date>2000-09-02</publish_date>
<description>When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled.</description>
</book>
<book id="bk107">
<author>
<name>Paula</name>
<surname>Thurman</surname>
</author>
<title>Splish Splash</title>
<genre>Romance</genre>
<price>4.95</price>
<publish_date>2000-11-02</publish_date>
<description>A deep sea diver finds true love twenty 
      thousand leagues beneath the sea.</description>
</book>
<book id="bk108">
<author>
<name>Stefan</name>
<surname>Knorr</surname>
</author>
<title>Creepy Crawlies</title>
<genre>Horror</genre>
<price>4.95</price>
<publish_date>2000-12-06</publish_date>
<description>An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects.</description>
</book>
<book id="bk109">
<author>
<name>Peter</name>
<surname>Kress</surname>
</author>
<title>Paradox Lost</title>
<genre>Science Fiction</genre>
<price>6.95</price>
<publish_date>2000-11-02</publish_date>
<description>After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum.</description>
</book>
<book id="bk110">
<author>
<name>Tim</name>
<surname>O'Brien</surname>
</author>
<title>Microsoft .NET: The Programming Bible</title>
<genre>Computer</genre>
<price>36.95</price>
<publish_date>2000-12-09</publish_date>
<description>Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference.</description>
</book>
<book id="bk111">
<author>
<name>Tim</name>
<surname>O'Brien</surname>
</author>
<title>MSXML3: A Comprehensive Guide</title>
<genre>Computer</genre>
<price>36.95</price>
<publish_date>2000-12-01</publish_date>
<description>The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more.</description>
</book>
<book id="bk112">
<author>
<name>Mike</name>
<surname>Galos</surname>
</author>
<title>Visual Studio 7: A Comprehensive Guide</title>
<genre>Computer</genre>
<price>49.95</price>
<publish_date>2001-04-16</publish_date>
<description>Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment.</description>
</book>
</catalog>

Extract the book titles and descriptions

Exercise

Look at the XML structure. Which elements do we need to extract the title and the description?

First, type the following code in your Jupyter Notebook to get the title from every book:

for book in root.find_all('book'):
    title = book.find('title').text
    print(title)
XML Developer's Guide
Midnight Rain
Maeve Ascendant
Oberon's Legacy
The Sundered Grail
Lover Birds
Splish Splash
Creepy Crawlies
Paradox Lost
Microsoft .NET: The Programming Bible
MSXML3: A Comprehensive Guide
Visual Studio 7: A Comprehensive Guide

Note

Although the basic for loop for ElementTree en Beautiful Soup look identical, please note that there is a small difference: ElementTree uses ‘findall’ and Beautiful Soup ‘find_all’ (with an underscore).

We shall explain what every line of the code does.

First, we iterate through the complete XML file and search for every element with the tag name ‘book’.

for book in root.find_all('book'):

Then, for every book element that exist, we create a temporarly new variable with the name ‘title’. As value for this variable, we use the content of the tag ‘title’ (which is a direct child of the element ‘book’). we add ‘.text’. to let Python know that we are interested in the value between the tags. Without the ‘.text’ addition, Python would simply present us the tag in its location, like ‘<Element ‘title’ at 0x000001995B4718B0>’

title = book.find('title').text

Then, we print the output of the title

print(title)

After this, the loop proceeds to the following book elements, extraxts the title and print the title etc.

We can get the description of each book in the same way.

Exercise

Alter the code above to retreive all the descriptions and print out the descriptions.

This leads to the following output:

An in-depth look at creating applications 
      with XML.
A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world.
After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society.
In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant.
The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy.
When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled.
A deep sea diver finds true love twenty 
      thousand leagues beneath the sea.
An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects.
After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum.
Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference.
The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more.
Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment.

We can use one for loop to extract both the book title and the description from the XML file. Combining multiple items is preferable because it saves unnecessary lines of codes and merges the part of the code which does the same thing. This makes the code more readable and better maintainable.

Combining the two codes above leads to the following code:

for book in root.find_all('book'):
	title = book.find('title').text
	description = book.find('description').text
	print(title, description)
XML Developer's Guide An in-depth look at creating applications 
      with XML.
Midnight Rain A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world.
Maeve Ascendant After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society.
Oberon's Legacy In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant.
The Sundered Grail The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy.
Lover Birds When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled.
Splish Splash A deep sea diver finds true love twenty 
      thousand leagues beneath the sea.
Creepy Crawlies An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects.
Paradox Lost After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum.
Microsoft .NET: The Programming Bible Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference.
MSXML3: A Comprehensive Guide The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more.
Visual Studio 7: A Comprehensive Guide Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment.

Extract name and surname of the author

You can use the same method as described above to extract all the names and surnames from the authors from the example XML. However, if we look at the structure of the XML file, there is a difference between the placement of the elements ‘title’ and ‘description’, and the elements ‘name’ and ‘surname’ in the XML structure.

<catalog>
	<book id="bk101">
		<author>
			<name>Matthew</name>
			<surname>Gambardella</surname>
		</author>
		<title>XML Developer's Guide</title>
		<genre>Computer</genre>
		<price>44.95</price>
		<publish_date>2000-10-01</publish_date>
		<description>An in-depth look at creating applications with XML.</description>
   </book>

Exercise

Look at the XML snippet above. What is the difference between the element ‘title’ and the element ‘name’?

Because of the difference in the place between elements, we need to alter our code a bit. We can use two approaches:

  • Add another for loop inside or first loop;

  • ‘escape’ the element hierarchie.

For the first approach, instead of a single for loop that iterates through all the ‘book’ elements, we also need a second for loop that runs through the ‘author’ element of ‘book’. We can do this with the following code:

for book in root.find_all('book'):
    for author in book.find_all('author'):
        name = author.find('name').text
        print(name) 
Matthew
Kim
Eva
Eva
Eva
Cynthia
Paula
Stefan
Peter
Tim
Tim
Mike

Exercise

The above code extracts only the name of an author. Alter the code, so that it extracts both the name and the surname.

Matthew Gambardella
Kim Ralls
Eva Corets
Eva Corets
Eva Corets
Cynthia Randall
Paula Thurman
Stefan Knorr
Peter Kress
Tim O'Brien
Tim O'Brien
Mike Galos

The second approach is to ‘escape’ the element hiearchy and directly select all subelements, on all levels beneath the current element. This is usefull if you have an XML with a lot of As explained in lesson 3, you can just insert the name of the subchild, as shown in the following code:

for book in root.find_all('author'):
    name = book.find('name').text
    surname = book.find('surname').text
    print(name, surname)  
Matthew Gambardella
Kim Ralls
Eva Corets
Eva Corets
Eva Corets
Cynthia Randall
Paula Thurman
Stefan Knorr
Peter Kress
Tim O'Brien
Tim O'Brien
Mike Galos

Extract the book identifier

As you can see in the XML, each book has its own identifier. As books can have the same name, and authors can have written multiple books, it is good practise to always use the identifier to point to a specific item.

In the previous exercises, we extracted the content that was presented between the tags of an element. For example:

<title>XML Developers Guide</title>

In this example, you see that the title ‘XML Developer’s guide’ is stored between the tags title and /title. We extracted this content by adding ‘.text’.

Exercise

Look at this example of the ‘book’ element with its identifier. What is the difference between the place of the content of the identifier and the place of the content of the title?

	<book id="bk101">
	</book>

To extract content from attributes, we need to use the ‘.get’ method. We still use the for loop to iterate through all the books, but instead of the content of certain elements, we now extract the content of the attribute.

for book in root.find_all('book'):
    identifier = book.get('id')
    print(identifier)
bk101
bk102
bk103
bk104
bk105
bk106
bk107
bk108
bk109
bk110
bk111
bk112

Structure all information

We can combine the different codes we have used above into one cell. To do this we can use the following scheme:

Iterate through all books	
	Get content from id attribute
	Get title content	
	Get description content	
	Iterate through all authors
		Get name
		Get surname
	Print id, title, description, name and surname

Exercise

Create the code that extracts all information we have used so far, from every book. And print this information (see scheme above).

This leads to the following output:

bk101 XML Developer's Guide An in-depth look at creating applications 
      with XML. Matthew Gambardella
bk102 Midnight Rain A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world. Kim Ralls
bk103 Maeve Ascendant After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society. Eva Corets
bk104 Oberon's Legacy In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant. Eva Corets
bk105 The Sundered Grail The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy. Eva Corets
bk106 Lover Birds When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled. Cynthia Randall
bk107 Splish Splash A deep sea diver finds true love twenty 
      thousand leagues beneath the sea. Paula Thurman
bk108 Creepy Crawlies An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects. Stefan Knorr
bk109 Paradox Lost After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum. Peter Kress
bk110 Microsoft .NET: The Programming Bible Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference. Tim O'Brien
bk111 MSXML3: A Comprehensive Guide The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more. Tim O'Brien
bk112 Visual Studio 7: A Comprehensive Guide Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment. Mike Galos

As you can see, it displays all information we wanted, but the output is quite unreadable. For example, it is not clear which part of the content belongs to the title, and which to the description.

To make the output more readable, we can put text before our output variables. In Python, this can be done like this:

print(f"This is the string we type and {this_is_the_variable}")

So in our example, we could add the following:

for book in root.find_all('book'):
    identifier = book.get('id')
    title = book.find('title').text
    description = book.find('description').text
    for author in book.find_all('author'):
        name = author.find('name').text
        surname = author.find('surname').text
	## add text to identify the extracted parts
    print(f"Identifier= {identifier} title= {title} description= {description} name= {name} {surname}")
Identifier= bk101 title= XML Developer's Guide description= An in-depth look at creating applications 
      with XML. name= Matthew Gambardella
Identifier= bk102 title= Midnight Rain description= A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world. name= Kim Ralls
Identifier= bk103 title= Maeve Ascendant description= After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society. name= Eva Corets
Identifier= bk104 title= Oberon's Legacy description= In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant. name= Eva Corets
Identifier= bk105 title= The Sundered Grail description= The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy. name= Eva Corets
Identifier= bk106 title= Lover Birds description= When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled. name= Cynthia Randall
Identifier= bk107 title= Splish Splash description= A deep sea diver finds true love twenty 
      thousand leagues beneath the sea. name= Paula Thurman
Identifier= bk108 title= Creepy Crawlies description= An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects. name= Stefan Knorr
Identifier= bk109 title= Paradox Lost description= After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum. name= Peter Kress
Identifier= bk110 title= Microsoft .NET: The Programming Bible description= Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference. name= Tim O'Brien
Identifier= bk111 title= MSXML3: A Comprehensive Guide description= The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more. name= Tim O'Brien
Identifier= bk112 title= Visual Studio 7: A Comprehensive Guide description= Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment. name= Mike Galos

As you can see, we can now detect the various parts that we extracted. However, it is still not easy to read. To resolve this, we can add linebreaks between each variable and between the different books. W e add a line break by adding ‘\n’ after each variable, leading to the following code:

for book in root.find_all('book'):
    identifier = book.get('id')
    title = book.find('title').text
    description = book.find('description').text
    for author in book.find_all('author'):
        name = author.find('name').text
        surname = author.find('surname').text
	## add linebreaks
    print(f"Identifier= {identifier }\n title= {title}\n description= {description} \n name= {name} {surname}\n")
Identifier= bk101
 title= XML Developer's Guide
 description= An in-depth look at creating applications 
      with XML. 
 name= Matthew Gambardella

Identifier= bk102
 title= Midnight Rain
 description= A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world. 
 name= Kim Ralls

Identifier= bk103
 title= Maeve Ascendant
 description= After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society. 
 name= Eva Corets

Identifier= bk104
 title= Oberon's Legacy
 description= In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant. 
 name= Eva Corets

Identifier= bk105
 title= The Sundered Grail
 description= The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy. 
 name= Eva Corets

Identifier= bk106
 title= Lover Birds
 description= When Carla meets Paul at an ornithology 
      conference, tempers fly as feathers get ruffled. 
 name= Cynthia Randall

Identifier= bk107
 title= Splish Splash
 description= A deep sea diver finds true love twenty 
      thousand leagues beneath the sea. 
 name= Paula Thurman

Identifier= bk108
 title= Creepy Crawlies
 description= An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects. 
 name= Stefan Knorr

Identifier= bk109
 title= Paradox Lost
 description= After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems 
      of being quantum. 
 name= Peter Kress

Identifier= bk110
 title= Microsoft .NET: The Programming Bible
 description= Microsoft's .NET initiative is explored in 
      detail in this deep programmer's reference. 
 name= Tim O'Brien

Identifier= bk111
 title= MSXML3: A Comprehensive Guide
 description= The Microsoft MSXML3 parser is covered in 
      detail, with attention to XML DOM interfaces, XSLT processing, 
      SAX and more. 
 name= Tim O'Brien

Identifier= bk112
 title= Visual Studio 7: A Comprehensive Guide
 description= Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are 
      integrated into a comprehensive development 
      environment. 
 name= Mike Galos

Well, that output looks way better, does it not?

Store the information in a .csv or .txt file.

In a lot of cases, you not only want the extracted content in your Jupyter Notebook, but you also want to store them for future use. We will show you how to store the output in two different ways:

  • as one file with the information of all books in .csv format (which, for example, can be opened in Excel)

  • as one textfile per book.

Store in one file

The easiest way to store and save Python output in one file is through storing it in a Dataframe from the Python package ‘Pandas’ and then saving this Dataframe. You can add data directly from the for loops we created above in a Pandas Dataframe, but we prefer the method in which you first create a list and then transform this list in the output, as Pandas Dataframe execution can become fairly slow with large amounts of data.

To create a list, we first have to declare an empty list. This is done with the following syntax:

booklist = []

Now, we alter our for loop a bit. Instead of printing the output to the screen, as we did above, we store our output in a list. We can use the following code:

booklist = []

for book in root.find_all('book'):
    identifier = book.get('id')
    title = book.find('title').text
    description = book.find('description').text
    for author in book.find_all('author'):
        name = author.find('name').text
        surname = author.find('surname').text
    booklist.append([identifier, title, description, name+" "+surname])

This leads to a list, called ‘booklist’, in which for every book all extracted information is stored. We can then easily transform this list to a pandas DataFrame. To do so, we need to import pandas first with the code

import pandas as pd

Then we type:

books = pd.DataFrame(booklist, columns=["identifier", "title", "description", "name"])

This code works as follows. You declare the variable ‘books’, which will be used to store all the information. Then you let Python know that you want to create a Dataframe. The content of this Dataframe is the list ‘booklist’, which we just created. We then tell Python how we want to name the columns (this should be in the same order as the order of the variables in the list).

You can show the dataframe you just created by typing:

books

This results in the following output:

identifier title description name
0 bk101 XML Developer's Guide An in-depth look at creating applications \n ... Matthew Gambardella
1 bk102 Midnight Rain A former architect battles corporate zombies, ... Kim Ralls
2 bk103 Maeve Ascendant After the collapse of a nanotechnology \n ... Eva Corets
3 bk104 Oberon's Legacy In post-apocalypse England, the mysterious \n ... Eva Corets
4 bk105 The Sundered Grail The two daughters of Maeve, half-sisters, \n ... Eva Corets
5 bk106 Lover Birds When Carla meets Paul at an ornithology \n ... Cynthia Randall
6 bk107 Splish Splash A deep sea diver finds true love twenty \n ... Paula Thurman
7 bk108 Creepy Crawlies An anthology of horror stories about roaches,\... Stefan Knorr
8 bk109 Paradox Lost After an inadvertant trip through a Heisenberg... Peter Kress
9 bk110 Microsoft .NET: The Programming Bible Microsoft's .NET initiative is explored in \n ... Tim O'Brien
10 bk111 MSXML3: A Comprehensive Guide The Microsoft MSXML3 parser is covered in \n ... Tim O'Brien
11 bk112 Visual Studio 7: A Comprehensive Guide Microsoft Visual Studio 7 is explored in depth... Mike Galos

Now we can save this dataframe into a csv file by typing:

books.to_csv('book.csv')

Note

This saves the csv in the root folder of your jupyter installation. If you want it saved in a specific location you need to specify the path before the filename followed by a ‘/, for example books.to_csv('C:/Users/Documents/book.csv') Please remember to use a backward slash (‘/’) between the folders

Create a textfile per book

If you want to create a textfile for every book, you can add the code directly in your for loop.

First, you have to declare a textfile in Python and give it a name. Then, you open the file and write content to it. After this, you close the file. Closing the file is important, else the loop will keep adding data to the file. You can try this with the following code:

with open("test.txt", "w") as f:
    f.write("This is just a test file")

Note

By default, Python stores the text file in the same folder as where you run your Jupyter Notebook. You can alter this by adding a path to your textfile, for example:

 myfile = open('C:/Users/Documents/test.txt', 'w') 

Please remember to use a backward slash (/) between the folders

With a few alterations, we can use this code to save our book information to a seperate file per book. First, we give the text file the name of the book identifier. We can do that by adding the variable into the name of the file like this:

with open(f"{identfier}.txt", "w") as f:

Then, we create the content of the file based on the content we extracted from the book.

f.write(name + " " + surname + "\n" + title + "\n" + description)

If we put these lines into our for loop, Python will save every book with its own name and information. The code looks like this:

for book in root.find_all('book'):
    identifier = book.get('id')
    title = book.find('title').text
    description = book.find('description').text
    for author in book.find_all('author'):
        name = author.find('name').text
        surname = author.find('surname').text
    with open(f"{identfier}.txt", "w") as f:
		f.write(name + " " + surname + "\n" + title + "\n" + description)

Filter information

You can also search for specific elements in your XML. For example, just the title information from the book ‘bk109’. To do so, you can start with the same for loop as we created in this lesson. However, before you print the output, you first check if you have the element you want (in this case: book 109). This can be done with an ‘if’ statement and it looks like this:

for book in root.find_all('book'):
    identifier = book.get('id')
    if identifier == "bk109":
        title = book.find('title').text
        print(title)
Paradox Lost

You can also search the content from XML elements, searching the content for a match. For example, if we want to print all titles that contain the word ‘XML’, we can use the following code:

for book in root.find_all('book'):
    title = book.find('title').text
    if "XML" in title:
        print(title)
XML Developer's Guide
MSXML3: A Comprehensive Guide

Note

Strings in Python are capital senstive! This means that ‘XML’ is not equal to ‘xml’ or ‘Xml’ in Python.

Exercise

Print out the title of all books that have England in their description.

After the collapse of a nanotechnology 
      society in England, the young survivors lay the 
      foundation for a new society.
In post-apocalypse England, the mysterious 
      agent known only as Oberon helps to create a new life 
      for the inhabitants of London. Sequel to Maeve 
      Ascendant.
The two daughters of Maeve, half-sisters, 
      battle one another for control of England. Sequel to 
      Oberon's Legacy.

Namespaces

As we mentioned in lesson 2 during our introduction to XML, some XML files contain namespaces. In lesson 3,we mentioned that Beautiful Soup omits these namespaces in elements, so you don’t have to declare them.

Let’s look at the example with namespaces from lesson 2:

<p:student xmlns:p="http//www.imaginarypythoncourses.com/student">
  <p:id>3235329</p:id>
  <p:name>Jeff Smith</p:name>
  <p:language>Python</p:language>
  <p:rating>9.5</p:rating>
</p:student>

Imagine, we want to extract the name of the student from this XML file.

First, we load the file into our Notebook (the file is called ‘namespaces.xml’ and can be downloaded here

with open("data/namespaces.xml") as f:
    root_ns = BeautifulSoup(f, 'xml')

Then, we create a for loop that iterates through the file and returns the values of all ‘name’ elements.

for student in root_ns.find_all('name'):
    print(student.text)
Jeff Smith

As you can see, Beautiful Soup has no problem with printing the name of the student.

However, in some XML documents, attributes can have a namespace. In such cases, you have to put the namespace identifier in your code.

Let’s imagine the XML looks as follows:

<p:student xmlns:p="http//www.imaginarypythoncourses.com/student">
  <p:name p:id='3235329'>Jeff Smith</p:name>
  <p:language>Python</p:language>
  <p:rating>9.5</p:rating>
</p:student>

Imagine we want to extract the attribute ‘id’. We see that this attribute has a namespace, so we need to declare it in the code. This can only be done by putting the identifier into curly brackets before the attribute name.

The code should looks as follows:

for student in root.find_all('name'):
    identifier = student.get('{http//www.imaginarypythoncourses.com/student}id')

We now have a good basis to try exploring some reallife examples of XML files used in Digital Humanities research. We will introduce some of these formats in the following section.