This assignment serves mostly to apply the content covered in topics 1 through 4. For this assignment we will assume the role of a digital librarian that received a large collection of digital material in a mix of formats. Our goal is to enable researchers to search for certain key phrases/words across the whole collection. The system should then present a list of all chapters the researcher should access.

Collection Records

We already provided a preexisting helpers.py file that contains some useful code that can be reused in your solution. This file defines a CollectionRecord class, that will be used to represent a certain chapter in the large digital collection. It also defines a print_collection_records function that takes a list of these CollectionRecord instances and prints them in a visual table as illustrated bellow:

from helpers import CollectionRecord, print_collection_records

print_collection_records([
        CollectionRecord("Y. Malaise", "ABC", "Intro to Python", "CH1. installing python", "..."),
        CollectionRecord("Y. Malaise", "ABC", "Intro to Python", "CH2. python notebooks", "..."),
        CollectionRecord("S. Budts", "DEF", "The Digital Handbook", "CH1. Data visualisation", "..."),
        CollectionRecord("S. Budts", "DEF", "The Digital Handbook", "CH2. Named Entity Recognition", "..."),
    ])

4 records found:
_____________________________________________________________________________________________________________________________________________
| ARCHIVE  |                       BOOK                       |                     CHAPTER                      |          AUTHOR          |
|----------|--------------------------------------------------|--------------------------------------------------|--------------------------|
|   ABC    |                 Intro to Python                  |              CH1. installing python              |        Y. Malaise        |
|   ABC    |                 Intro to Python                  |              CH2. python notebooks               |        Y. Malaise        |
|   DEF    |               The Digital Handbook               |             CH1. Data visualisation              |         S. Budts         |
|   DEF    |               The Digital Handbook               |          CH2. Named Entity Recognition           |         S. Budts         |
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾

Combining the data across the sources

As mentioned previously our library has recently received collections from four organisations that are closing their doors. Unfortunately all these organisations used different systems to track their data, it is up to us to write some code that will iterate through all of the available source material and convert it into instances of CollectionRecord.

In order to achieve this we will provide a dedicated method for each of the archives.

🥚 LITARC - Literary Archives and Resource Consortium

LITARC is a global organization dedicated to preserving historical literary texts, rare manuscripts, and archived works of significant cultural importance.

Their collection can be found in the collections/litarc directory. LITARC stores all their materials in plain text files following a fixed folder structure:

/litarc
    /author
        /book
            /chapter-number_chapter-title.txt

At the top level there will be one folder for each author in the collection. Inside that folder there will be folder for each book written by that author. Inside each of those there will be a .txt file for each chapter of the book. The name of the txt file will consist of the chapter number followed by the title of the chapter. All spaces in the author, book and chapter names will be replaces with underscores (_).

For example:

/litarc
    /yoshi_malaise
        /Intro_to_Python
            /1_installing_python.txt
            /2_python_notebooks.txt

Complete the following get_litarc_records method and ensure that it correctly loops through all of the files and directories and constructs the list of CollectionRecords along the way.

You will likely need to make use of: - os.scandir (https://www.geeksforgeeks.org/python/how-to-iterate-over-files-in-directory-using-python/) - string replace (https://www.w3schools.com/python/ref_string_replace.asp) - Reading text files (https://www.w3schools.com/python/python_file_open.asp)

import os

def get_litarc_records():
    results = []
    # todo get the records and add them to the results array
    return results

🐣 BOOKS - Bibliotheca of Outstanding and Omniscient Knowledge Sources

BOOKS is a digital archive aimed at providing free access to classic and contemporary literature, promoting knowledge sharing and public education.

Their archive consists of a single large xml file with the following structure:

<archive>
    <book title="title" author="Firstname Lastname">
        <chapter title="chapter title">
            At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.
        </chapter>

        <chapter>...</chapter>
        ...
    </book>

    <book> ... </book>

    ...
</archive>

The root archive element has a child element for each book. These book elements have attributes for the title of the book and the name of the author. Nested inside of the book element are chapter elements with attributes for the chapter title and the actual body of the text as the body of the element.

Complete the get_books_records method to transform the source xml file to a list of CollectionRecord instances.

def get_books_records():
    results = []
    # todo get the records and add them to the results array
    return results

🐣 PALMS - Preservation of Ancient Literary Manuscripts Society

PALMS specializes in the conservation and restoration of ancient and medieval literary works, offering expertise in maintaining physical books and manuscripts.

PALMS stores their whole archive in the form of a json file with the following format:

[
    {
        "title": "book title",
        "author": {
            "firstname": "Yoshi",
            "lastname": "Malaise"
        },
        "chapters": [
            {
                "title": "chapter title",
                "content": "..."
            },
            ...
        ]
    },
    ...
]

The root elementis an array (or list) containing books. Each book has a title, an author (with a firstname and a lastname) and a collection of chapters. Each chapter has a title and its content in plain text.

Complete the get_palms_records method to transform the source json file to a list of CollectionRecord instances.

def get_palms_records():
    results = []
    # todo get the records and add them to the results array
    return results

🐔 SCRIBE - Society for the Conservation of Rare and Important Bibliographic Ephemera

SCRIBE is an organization that preserves both written and printed literary works from all genres, with an emphasis on rare books and ephemera like pamphlets and periodicals.

This dataset is an optional bonus if you finished the rest of the assignment. The scribe dataset contains pdf files of all the chapters and you will need to extract the data from the pdf files directly.

def get_scribe_records():
    results = []
    # todo get the records and add them to the results array
    return results

combining it all

Do not change anything in the following code block. If you implemented the previous methods correctly the get_all_records method should return the combined list of all the records across all of the source collections.

def get_all_records():
    results = []
    results.extend(get_litarc_records())
    results.extend(get_books_records())
    results.extend(get_palms_records())
    results.extend(get_scribe_records())
    return results

print_collection_records(get_all_records())

0 records found:
_____________________________________________________________________________________________________________________________________________
| ARCHIVE  |                       BOOK                       |                     CHAPTER                      |          AUTHOR          |
|----------|--------------------------------------------------|--------------------------------------------------|--------------------------|
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾

Making content searchable

Great, We now have access to all of the sources using one consistent data structure!

This will make it a lot easier to enable users to search for documents regardless of the original source.

Over the next few steps we will work on implementing methods that build on top of each other to create incrementally more powerfull and easier to use filtering capabilities.

However, before doing that we will add a small intermediary step that only executes the data collection step once, saves the result in a specialised format and makes it easy to load in the processed data again so not all the source material needs to be processed every time you load the notebook.

No worries, all of the code for this is given in the next two cells.

# only execute this cell once every time you modified the code for generating the records, this will perform all the conversion that can be computationally expensive
# load all records once, and write the result to a specialised pickle file
import pickle
with open('record_collection.pkl', 'wb') as f:  # open a text file
    pickle.dump(get_all_records(), f) # serialize the list

# load all records directly from the .pkl file generated in the previous step no need to crawl through all the source data again every time
import pickle
all_records = None
with open('record_collection.pkl', 'rb') as f:  # open the previously written file
    all_records = pickle.load(f)

The first type of search we will allow, is allowing to filter based on the author, for this we will implement a method filter_based_on_author that takes two arguments:

collection: the list of records that we want to filter based on the author
author: the name of the author we are looking for

def filter_based_on_author(collection, author):
    matching_records = []
    for record in collection:
        # TODO: modify the next line so it actually checks if the author matches instead of always returning false
        is_match = False
        if is_match:
            matching_records.append(record)
    
    return matching_records

Next we will implement a method filter_based_on_regex that takes two arguments:

collection: the list of records that we want to filter based on the author
regex_pattern: the regular expression we are looking for

this method will work very similarly to the earlier one, except now we will use regular expressions to search for patterns across the entire content of the chapters.

Study the following sources and complete the method.

sources: - https://www.w3schools.com/python/python_regex.asp - https://regexr.com - https://docs.python.org/3/library/re.html

def filter_based_on_regex(collection, regex_pattern):
    matching_records = []
    for record in collection:
        # TODO: modify the body so it actually checks if the body of the document matches the regex
        is_match = False
        if is_match:
            matching_records.append(record)
    
    return matching_records

# here we combine the functionality of both the previous methods in one big method that has optional arguments
# for the author and regex_patterns
# if no author is passed, no filter happens based on author, 
# all regex_patterns need to be true to keep the document  
def filter_complete(collection, author=None, regex_patterns=[]):
    matching_records = collection
    
    if author is not None:
        matching_records = filter_based_on_author(matching_records, author)
    
    for pattern in regex_patterns:
        matching_records = filter_based_on_regex(matching_records, pattern)
    
    return matching_records