# IMPORT RELEVANT LIBRARIES
# -------------------------
# standard functionalities
import json, os, re, itertools
# library to download the lyrics from the Genius website
from lyricsgenius import Genius
# library for dataframe (excel-like) functionalities
import pandas as pd
# library for fast mathematical operations
import numpy as np
# library for linguistic processing and enrichment
import spacy
# library to transform songs into vectors (in the machine learning way)
import sklearn
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
# libraries to plot the results
import seaborn as sns
import plotly.express as px
from IPython.display import HTMLStylometric Analysis of Pommelien Thijs’ Lyrics
Has our analysis of Pommelien Thijs’ lyrics sparked your interest, and do you want to know more about the technology behind it? This tutorial walks you through the full process and provides the code you need to replicate our results or tackle a similar question with your own material.
The tutorial assumes that you have a little bit of Python experience: you should be comfortable running code cells, reading error messages, and changing file paths or variable names when needed. You do not need to be an expert in machine learning.
For a more comprehensive introduction and another excellent case study, have a look at the stylometry chapter of the phenomenal and fully open-source book Humanities Data Analysis: Case Studies with Python by Folgert Karsdorp, Mike Kestemont, and Allen Riddell.
What Is Stylometry?
Stylometry literally means “measuring style”. The basic idea is simple: if people write often enough, they tend to develop small, unconscious habits. Some writers use certain function words more than others. Some prefer short sentences, some repeat particular constructions, and some combine words in ways that are hard to imitate consistently.
In authorship studies, stylometry is often used to ask questions such as: who wrote an anonymous text, which texts in a corpus are stylistically close to each other, or whether a writer’s style changes over time. In this tutorial, we use it for a playful but real research question: do Pommelien Thijs’ lyrics resemble those of artists connected to the same songwriting network, or do they form their own stylistic profile?
A key point is that stylometry usually works best with ordinary words rather than spectacular ones. Very striking expressions are easy to notice and easy to imitate. Small words such as ik, je, en, maar, dat, and niet are less glamorous, but they often say more about writing habits because authors use them without thinking too much about them.
Our workflow has six parts:
- Import the Python libraries we need.
- Collect and clean song lyrics.
- Count the words in each song.
- Use PCA to turn word counts into a visual map of writing style.
- Add album information to interpret Pommelien’s position more carefully.
- Repeat the analysis with TF-IDF to look more at content than style.
Step 1: Preparation
Before we start the actual analysis, we first import the Python libraries we need. Some of these libraries help us collect lyrics, others help us clean and analyse the texts, and the final group helps us visualise the results.
Step 2: Collect the Lyrics
Without data, there is no analysis. We start by using the Genius class from the lyricsgenius package to collect lyrics from the Genius website.
Because Genius asks users of its API to identify themselves, you need a client access token. To get one, follow the instructions in the lyricsgenius setup guide, register on the Genius API Clients page, copy your token, and paste it into the API_KEY placeholder below.
The code below then creates a Genius client that can search the Genius database.
# REGISTER FOR LYRICSGENIUS
# -------------------------
API_KEY = "<your_client_access_token>"
if API_KEY == "<your_client_access_token>":
raise ValueError("Replace <your_client_access_token> with your Genius API token before running this cell.")
genius = Genius(API_KEY)Next, we list the artists we want to include in the analysis. In this case, we use Pommelien Thijs as our main case, and Froukje, Metejoor, Bazart, and Boudewijn de Groot as comparison material.
For every artist, we ask the Genius client to search for that artist and download the available lyrics. The result is saved as a JSON file, a structured text file that Python can read again later.
The lyricsgenius package can be a little finicky. Downloads may stop because of a temporary network problem or because Genius changes how a page is structured. For that reason, the code below prints a clear message when something goes wrong. If a download fails, you can simply run the cell again.
Once you have all JSON files on your computer, store them in a dedicated folder called resources. That makes the next step easier.
# DOWNLOAD LYRICS
# ---------------
artists = ["Pommelien-thijs", "Froukje", "Metejoor", "Bazart", "Boudewijn-de-groot"]
for artist in artists:
try:
artist_record = genius.search_artist(artist)
artist_record.save_lyrics()
except Exception as error:
print(f"Could not finish downloading lyrics for {artist}: {error}")Next, we parse the JSON files with artist records and extract the actual song information. We first list all files in the resources folder and keep only files with the .json extension.
Then we create an empty list called all_songs. For every JSON file, we read the artist record into memory, select the value stored under the songs key, and add those song records to our main list.
Finally, we convert the list into a pandas DataFrame. A DataFrame is similar to a spreadsheet: every row is a song, and every column contains information about that song. The final line shows a random song so we can inspect the result.
# EXTRACT LYRICS FROM JSON FILES
# ------------------------------
# List all JSON files in the resources folder
inputfiles = os.listdir("resources")
inputfiles = [os.path.join("resources", f) for f in inputfiles if f.split(".")[-1] == "json"]
# Extract the song records from every artist file
all_songs = []
for f in inputfiles:
with open(f, encoding="utf-8") as infile:
artist_record = json.load(infile)
all_songs.extend(artist_record["songs"])
# Turn the list of song records into a DataFrame
df_songs = pd.DataFrame.from_records(all_songs)
# Show a random record
pd.set_option('display.max_colwidth', 0)
df_songs.sample(1)| annotation_count | api_path | artist_names | full_title | header_image_thumbnail_url | header_image_url | id | lyrics_owner_id | lyrics_state | path | ... | media | producer_artists | song_relationships | translation_songs | verified_annotations_by | verified_contributors | verified_lyrics_by | writer_artists | artist | lyrics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 173 | 1 | /songs/1761902 | Boudewijn de Groot | Als Jij Niet Van Me Houdt by Boudewijn de Groot | https://images.genius.com/550d821fc69285b89c562da7c9ac0be0.300x300x1.jpg | https://images.genius.com/550d821fc69285b89c562da7c9ac0be0.999x999x1.jpg | 1761902 | 1549345 | complete | /Boudewijn-de-groot-als-jij-niet-van-me-houdt-lyrics | ... | [] | [] | [{'relationship_type': 'samples', 'type': 'samples', 'songs': []}, {'relationship_type': 'sampled_in', 'type': 'sampled_in', 'songs': []}, {'relationship_type': 'interpolates', 'type': 'interpolates', 'songs': []}, {'relationship_type': 'interpolated_by', 'type': 'interpolated_by', 'songs': []}, {'relationship_type': 'cover_of', 'type': 'cover_of', 'songs': []}, {'relationship_type': 'covered_by', 'type': 'covered_by', 'songs': []}, {'relationship_type': 'remix_of', 'type': 'remix_of', 'songs': []}, {'relationship_type': 'remixed_by', 'type': 'remixed_by', 'songs': []}, {'relationship_type': 'live_version_of', 'type': 'live_version_of', 'songs': []}, {'relationship_type': 'performed_live_as', 'type': 'performed_live_as', 'songs': []}, {'relationship_type': 'translation_of', 'type': 'translation_of', 'songs': []}, {'relationship_type': 'translations', 'type': 'translations', 'songs': []}] | [] | [] | [] | [] | [] | Boudewijn de Groot | 1 ContributorAls Jij Niet Van Me Houdt LyricsAls jij niet van me houdt\nMoet ik daarmee leren leven\nMaar de nacht is koud en de morgen ver\nGa niet weg en blijf nog even\n\nAls jij niet van me houdt\nBen ik de man niet van je dromen\nEn als, als je bij me blijft vannacht\nZal hij heus wel een keer komen\n\nAls jij niet van mij houdt\nHou ik wel van jou alleen\nZo mooi als je lacht\nZo warm en zo zacht\n\nAls jij niet van mij houdt\nHou ik wel alleen van jou\nBlijf bij me deze ene nacht\nEn laat me niet alleen\n\nAls jij niet van me houdt\nZul je voor een ander kiezen\nMaar wat niet is kan ook niet verloren gaan\nDus kan ik jou nooit verliezen\n\nSoms denk ik 'was jij er vroeger maar'\nVoordat ik je kende misschien\nSoms denk ik dat jij altijd blijven zal\nZelfs als ik je nooit meer zou zien\n\n\nAls jij niet van mij houdt\nHou ik wel van jou alleen\nZo mooi als je lacht\nZo warm en zo zacht\n\nAls jij niet van mij houdt\nHou ik wel alleen van jou\nBlijf bij me deze ene nacht\nEn laat me niet alleen |
1 rows × 51 columns
After collecting the lyrics, we need to decide which songs are suitable for analysis. Lyrics in another language tell us little about someone’s style in Dutch and should be removed. Covers, live versions, collaborations, and special versions can also distort the results because they may contain lyrics written by someone else or changed for a different performance context.
The code below uses the song metadata to keep Dutch-language lyrics, remove artists with too few songs in the dataset, and filter out live editions, covers from Liefde voor Muziek, and other special versions.
# CLEAN UP THE DATASET
# --------------------
# Keep only lyrics that Genius marks as Dutch
df_songs = df_songs[df_songs["language"] == "nl"].copy()
# Discard collaborations and artists with too few songs in the dataset
allowed_artists = [artist for artist, count in df_songs["primary_artist_names"].value_counts().items() if count > 10]
df_songs = df_songs[df_songs["primary_artist_names"].isin(allowed_artists)].copy()
# Discard live editions, covers from Liefde voor Muziek, and other versions
live = [1 if "live" in x.lower() else 0 for x in df_songs["full_title"].values]
lvm = [1 if "liefde voor muziek" in x.lower() else 0 for x in df_songs["full_title"].values]
versie = [1 if "versie" in x.lower() else 0 for x in df_songs["full_title"].values]
df_songs["live"] = live
df_songs["lvm"] = lvm
df_songs["versie"] = versie
df_songs = df_songs[df_songs["live"] + df_songs["lvm"] + df_songs["versie"] == 0].copy()This part of the code cleans up the lyrics themselves so they are easier for a computer to process.
First, we define regular expressions that can remove text between square brackets and merge multiple blank lines. Genius lyrics often contain section labels such as [Chorus], [Verse 1], or [Bridge]. Those labels are useful for readers, but they are not part of the actual lyrics, so we remove them.
We also remove repeated lines. In pop lyrics, the chorus often appears several times. If we kept every repeated chorus, the analysis would be dominated by repetition rather than by the range of words in the song.
# CLEAN UP THE LYRICS
# -------------------
# Define and compile regular expressions for text between square brackets and multiple newlines
square_brackets_pattern = r"\[[^\]]+\]"
square_brackets_query = re.compile(square_brackets_pattern)
more_newlines_pattern = r"\n+"
more_newlines_query = re.compile(more_newlines_pattern)
def remove_square_brackets(lyrics):
"""Take song lyrics and remove everything between square brackets."""
return re.sub(square_brackets_query, "", lyrics)
def merge_multiple_line_endings(lyrics):
"""Take song lyrics and remove empty lines."""
return re.sub(more_newlines_query, "\n", lyrics)
def remove_repetitions(lyrics):
"""Take song lyrics, split them into lines, and remove repeated lines while preserving their order."""
lines = [line.strip() for line in lyrics.split("\n") if len(line.strip()) > 0]
return list(dict.fromkeys(lines))
# Preserve only the part from the word "Lyrics" onwards
df_songs["lyrics_clean"] = df_songs["lyrics"].apply(lambda x: "Lyrics".join(x.split("Lyrics")[1:]))
# Lowercase everything
df_songs["lyrics_clean"] = df_songs["lyrics_clean"].apply(lambda x: x.lower())
# Remove duplicate songs from the database
df_songs = df_songs.drop_duplicates("lyrics_clean").copy()
# Remove parts between square brackets
df_songs["lyrics_clean"] = df_songs["lyrics_clean"].apply(lambda x: remove_square_brackets(x))
# Remove spurious line endings
df_songs["lyrics_clean"] = df_songs["lyrics_clean"].apply(lambda x: merge_multiple_line_endings(x))
# Remove lines that occur more than once, so repeated choruses are preserved only once
df_songs["lyrics_unique"] = df_songs["lyrics_clean"].apply(lambda x: remove_repetitions(x))
# Remove the "by <ARTIST>" part from the title
df_songs["reduced_title"] = df_songs["full_title"].apply(lambda x: " by".join(x.lower().split(" by")[:-1]))The cleaned lyrics are then enriched with linguistic information. Stylometric analysis is based on how often a given writer uses certain words, so it helps if variants of the same word are treated as the same thing.
Humans immediately recognise that Welkom, welkom, and welkom! are basically the same word. A computer does not automatically know that it can ignore capitalisation and punctuation. To prepare the lyrics, we therefore tokenize the text, remove punctuation, and lemmatize the words.
A token is a small unit of text, usually a word or punctuation mark. A lemma is the dictionary form of a word. For example, plural nouns are often reduced to singular nouns, and conjugated verbs are often reduced to their base form. The sentence Ik heb zand in mijn zakken en wind in mijn haar might therefore become something like ik hebben zand in mijn zak en wind in mijn haar. That looks odd to us, but it gives the computer a more consistent representation of the text.
Most of this linguistic enrichment is done with the Python library spaCy. We first load the small Dutch language model into the variable nlp.
Then we define a function called tokenize. This function:
- takes the cleaned lyrics of one song as input;
- creates an empty list where tokens, parts of speech, and lemmas will be stored;
- loops over all lyric lines;
- tokenizes and lemmatizes every line;
- removes spaces;
- returns the full list of linguistic information.
We apply this function to every song and save the result in a new column called tokenposlemma. We then create another column called lemmas, which contains only the lemmatized words and removes punctuation. This lemmas column will form the basis of our analysis later on.
# TOKENIZE AND LEMMATIZE
# ----------------------
# Load spaCy's small language model for Dutch
nlp = spacy.load("nl_core_news_sm")
# Define a method to tokenize
def tokenize(lyric):
"""Take the lyrics of a song as a list of strings. Tokenize each string and return token, part of speech, and lemma. Remove spaces."""
tokens = []
for line in lyric:
doc = nlp(line)
tokens.extend([(token.text, token.pos_, token.lemma_) for token in doc if not token.pos_ == "SPACE"])
return tokens
# Tokenize all lyrics
df_songs["tokenposlemma"] = df_songs["lyrics_unique"].apply(lambda x: tokenize(x))
# Extract lemmas from the tokenized lyrics and discard punctuation
df_songs["lemmas"] = df_songs["tokenposlemma"].apply(lambda x: [lemma.lower() for (token, pos, lemma) in x if not pos == "PUNCT"])To show you the effect of the preprocessing steps, the code below shows the clean lyrics of Pommelien’s “Erop of eronder” side by side with the lemmatised version.
title = "erop of eronder"
index_song= df_songs[df_songs["reduced_title"]==title].index[0]
song_original = " / ".join(df_songs.loc[index_song,"lyrics_unique"])
song_lemmas = " ".join(df_songs.loc[index_song,"lemmas"])
df_example = pd.DataFrame().from_records([{"original":song_original,"lemmatized":song_lemmas}])
df_exampleStep 3: Count the Words
Now that the dataset has been cleaned and lemmatized, we can start counting words.
In stylometry, a text is often represented as a list of numbers. Each number tells us how often a particular word occurs in that text. If we count the same words for every song, every song can be represented as a comparable numerical profile.
We begin by keeping only the columns we need for the analysis: the song id, artist name, shortened title, cleaned lyrics, unique lyric lines, and lemmas. We also create a separate artist column so that the artist name remains easy to access later.
# REFORMAT THE DATASET
# --------------------
# Restrict the dataset to just the columns we need for the analysis
df = df_songs.loc[:, ["id", "primary_artist_names", "reduced_title", "lyrics_clean", "lyrics_unique", "lemmas"]].copy()
# Set the artist as index and keep a regular artist column for plotting
df = df.set_index("primary_artist_names")
df["artist"] = df.index.values
df.sample(1)| id | reduced_title | lyrics_clean | lyrics_unique | lemmas | artist | |
|---|---|---|---|---|---|---|
| primary_artist_names | ||||||
| Pommelien Thijs | 12380167 | ik moet gaan | \nde stad wordt wakker in het donker\nhet licht gaat uit en ik ga aan (na-na-na-na-na)\nsoms vergeet ik dat ik jong ben\nmaar wat moet ik als zelfs de zon kopje onder gaat? (na-na-na-na-na)\nik wil niet denken, ik moet iets doen\nik kan niet zitten hier en wachten op gedoe (na-na-na-na-na)\nik leer vertrouwen op dit gevoel\nweet wat ik wil, maar nog niet hoe het moet\nik wil lachen en het menen\nik wil dansen in de regen\nik wil roepen, ik wil wenen\nmoet het voelen in m'n benen\nik wil spelen, ik wil winnen\nik wil niet opnieuw beginnen\nhet is vol, maar kom maar binnen\nik moet niks, ik wil jou\nik moet gaan (gaan)\nniet weg van hier, maar met ons allemaal (gaan)\nlaat de muziek hier alles nu bepalen\neen en hetzelfde, eenzelfde taal\ndus wij snappen allemaal\nik moet gaan, ik moet gaan\n(na-na-na-na-na, na-na-na-na)\nik moet gaan\n(na-na-na-na-na, na-na-na-na)\nsoms vergeet ik dat ik echt ben\ndan schiet ik wakker, kom weer terug\n(elke keer kom ik weer terug)\ndan wil ik mee zijn, wil ik een deel zijn\nhet was me bijna echt gelukt\nwant zelfs slechtgaan (slechtgaan)\nis ergens heen gaan (heen gaan)\nik raad het niet aan (ah)\nmaar jij moet niks, ik wil jou\nik moet gaan (gaan)\nniet weg van hier, maar met ons allemaal (gaan)\nlaat de muziek hier alles nu bepalen\neen en hetzelfde, eenzelfde taal\ndus wij snappen allemaal\nik moet gaan, ik moet gaan\n(na-na-na-na-na, na-na-na-na)\nik moet gaan\n(na-na-na-na-na, na-na-na-na)\nik lijk niks te hebben, lijk niks te weten\nvanavond beheers ik de kunst van vergeten\nde lucht is elektrisch, de schoenen versleten\nhet zit in ons dna, dna\nna-na, na-na-na-na-na, na-na-na-na\nna-na, na-na-na-na-na\nik moet niks, ik wil jou\nik moet gaan (gaan)\nniet weg van hier, maar met ons allemaal (gaan)\nlaat de muziek hier alles nu bepalen\neen en hetzelfde, eenzelfde taal\ndus wij snappen allemaal\nik moet gaan, ik moet gaan (gaan)\njij moet gaan, jij moet gaan (na-na-na-na-na, na-na-na-na)\njullie moeten gaan (ik moet gaan)\nmoeten gaan, moeten gaan (na-na-na-na-na, na-na-na-na)\nlaat mij nooit meer gaan (ik, ik moet gaan)\nen jij moet gaan, jij moet gaan (na-na-na-na-na, na-na-na-na)\njullie moeten gaan, ik moet gaan (na-na-na-na-na-na)\nik moet niks, ik wil jou\nik moet gaan | [de stad wordt wakker in het donker, het licht gaat uit en ik ga aan (na-na-na-na-na), soms vergeet ik dat ik jong ben, maar wat moet ik als zelfs de zon kopje onder gaat? (na-na-na-na-na), ik wil niet denken, ik moet iets doen, ik kan niet zitten hier en wachten op gedoe (na-na-na-na-na), ik leer vertrouwen op dit gevoel, weet wat ik wil, maar nog niet hoe het moet, ik wil lachen en het menen, ik wil dansen in de regen, ik wil roepen, ik wil wenen, moet het voelen in m'n benen, ik wil spelen, ik wil winnen, ik wil niet opnieuw beginnen, het is vol, maar kom maar binnen, ik moet niks, ik wil jou, ik moet gaan (gaan), niet weg van hier, maar met ons allemaal (gaan), laat de muziek hier alles nu bepalen, een en hetzelfde, eenzelfde taal, dus wij snappen allemaal, ik moet gaan, ik moet gaan, (na-na-na-na-na, na-na-na-na), ik moet gaan, soms vergeet ik dat ik echt ben, dan schiet ik wakker, kom weer terug, (elke keer kom ik weer terug), dan wil ik mee zijn, wil ik een deel zijn, het was me bijna echt gelukt, want zelfs slechtgaan (slechtgaan), is ergens heen gaan (heen gaan), ik raad het niet aan (ah), maar jij moet niks, ik wil jou, ik lijk niks te hebben, lijk niks te weten, vanavond beheers ik de kunst van vergeten, de lucht is elektrisch, de schoenen versleten, het zit in ons dna, dna, na-na, na-na-na-na-na, na-na-na-na, na-na, na-na-na-na-na, ik moet gaan, ik moet gaan (gaan), jij moet gaan, jij moet gaan (na-na-na-na-na, na-na-na-na), jullie moeten gaan (ik moet gaan), moeten gaan, moeten gaan (na-na-na-na-na, na-na-na-na), laat mij nooit meer gaan (ik, ik moet gaan), en jij moet gaan, jij moet gaan (na-na-na-na-na, na-na-na-na), jullie moeten gaan, ik moet gaan (na-na-na-na-na-na)] | [de, stad, worden, wakker, in, het, donker, het, licht, gaan, uit, en, ik, gaan, aan, na-na-na-na-na, soms, vergeeten, ik, dat, ik, jong, zijn, maar, wat, moeten, ik, als, zelfs, de, zon, kop, onder, gaan, na-na-na-na-na, ik, willen, niet, denken, ik, moeten, iets, doen, ik, kunnen, niet, zitten, hier, en, wachten, op, gedoe, na-na-na-na-na, ik, leren, vertrouwen, op, dit, gevoel, weten, wat, ik, willen, maar, nog, niet, hoe, het, moeten, ik, willen, lachen, en, het, menen, ik, willen, dansen, in, de, regen, ik, willen, roepen, ik, willen, wenen, moeten, het, voelen, in, mijn, been, ik, willen, spelen, ik, willen, winnen, ik, ...] | Pommelien Thijs |
The actual counting is done with vectorizers from sklearn. A vectorizer turns texts into vectors: rows of numbers that machine-learning algorithms can work with.
The function below can work in two modes. In count mode, it simply counts how often each word occurs. In tfidf mode, which we use later, it gives more weight to words that are distinctive for particular songs.
For the stylometric analysis, we use CountVectorizer and keep the 100 most frequent words. These frequent words include many ordinary function words, which are useful for studying style.
# VECTORIZE LYRICS
# ----------------
# Turn lyrics into vectors
def vectorize_lyrics(lyrics, mode, max_features):
"""Take a collection of song lyrics and turn them into vectors based on the words they contain."""
if mode == "tfidf":
vectorizer = TfidfVectorizer(max_features=max_features)
else:
vectorizer = CountVectorizer(max_features=max_features)
X = vectorizer.fit_transform(lyrics).toarray()
return XStep 4: Analyse Writing Style with PCA
A table with 100 word counts for every song is useful for the computer, but not very readable for humans. We need one more step that turns all those word counts into a visual overview.
For that, we use Principal Component Analysis, usually shortened to PCA. PCA looks for the strongest patterns of difference in the word counts and turns them into coordinates. These coordinates let us plot every song as a point on a two-dimensional graph.
Songs that use the frequent words in similar proportions should appear close together. If stylometric habits are strong enough, we expect songs by the same artist to cluster near each other.
The code below first defines a helper function to compute PCA coordinates. It then defines a colour palette and a plotting function so we can reuse the same visual style for different analyses.
One practical detail: the plotting function returns the figure as HTML instead of using fig.show(). This avoids a common Plotly/Jupyter error about nbformat and usually makes the notebook easier to run across different environments.
# ANALYSIS: PCA AND PLOTTING
# --------------------------
# Perform PCA on the vectors to compute coordinates
def compute_coordinates(matrix):
pca = sklearn.decomposition.PCA(n_components=2)
documents_proj = pca.fit_transform(matrix)
c1, c2 = documents_proj[:, 0], documents_proj[:, 1]
return c1, c2
# Define the colour scheme for plotting
colorscheme = ["#3366CC", "#ecc73f", "#DC3912", "#990099", "#109618"]
palette = sns.color_palette(colorscheme, len(colorscheme))
artists = sorted(df["artist"].unique())
artist_to_color = {artist: color for (artist, color) in zip(artists, colorscheme)}
# Plot the lyrics based on the coordinates
def plot_lyrics(df, hover_data, title, width=500, height=400, palette=artist_to_color, color="artist", alpha=0.6):
fig = px.scatter(
df,
x="x",
y="y",
color=color,
hover_data=hover_data,
title=title,
opacity=alpha,
color_discrete_map=palette,
width=width,
height=height,
)
fig.update_traces(marker=dict(size=10, line=dict(width=1, color="white")), selector=dict(mode="markers"))
fig.update_traces(hovertemplate="%{customdata[0]}")
fig.update_layout(
hoverlabel=dict(
font_size=9,
font_family="Helvetica",
),
hoverlabel_align="auto",
)
fig.update_layout(yaxis_title=None, xaxis_title=None)
fig.update_xaxes(showticklabels=False)
fig.update_yaxes(showticklabels=False)
return HTML(fig.to_html(full_html=False, include_plotlyjs="cdn"))Now we can run the stylometric analysis itself.
First, we collect the artist names, song titles, lemmatized lyrics, and song ids. Then we join the lemmas of every song into a single string, because the vectorizer expects each text to be represented as one string.
Next, we count the 100 most frequent words, compute PCA coordinates, and store everything in a new DataFrame called song_pca. Finally, we plot the result.
# COUNT VECTORIZER; ONE DOT PER SONG
# -----------------------------------
artists = df["artist"]
titles = df["reduced_title"]
lyrics = [" ".join(lyrics) for lyrics in df["lemmas"]]
ids = [i for i in range(len(titles))]
# Vectorize lyrics and compute coordinates
vectorized_lyrics = vectorize_lyrics(lyrics, "count", 100)
x, y = compute_coordinates(vectorized_lyrics)
# Add coordinates to a DataFrame and plot
song_pca = pd.DataFrame([ids, artists, titles, x, y], index=["id", "artist", "title", "x", "y"]).transpose()
song_pca = song_pca.sort_values(by="artist")
plot_lyrics(song_pca, ["title"], "Stylometric analysis based on the 100 most frequent words", width=1000, height=500)We also save the stylometric coordinates in the main DataFrame. This will make it easy to export all results at the end of the tutorial.
df["stylometry_x"] = x
df["stylometry_y"] = yThe graph visualises our first stylometric analysis. Every dot is one song. The colour shows the artist, and the position shows how the song relates stylistically to the others.
If the analysis works, we should see at least some clustering by artist. That does not mean that every song by the same artist will be in exactly the same place. Songs can differ in tone, theme, genre, and co-writing context. But if the dots of one artist often appear near each other, that suggests that the method is capturing something meaningful about writing style.
In this case, the reference artists form recognisable regions in the graph. Bazart, Metejoor, Froukje, and Boudewijn de Groot do not all occupy the same space. That is encouraging: the method is picking up stylistic differences in the corpus.
The result becomes especially interesting when we look at Pommelien. Her songs are less tightly clustered than those of some of the comparison artists. Some Pommelien songs appear close to Metejoor, others move more towards Froukje, and a few sit closer to the region associated with Boudewijn de Groot.
That spread is precisely what makes the case interesting. It suggests that Pommelien’s lyrics may not have one single stable position in this reference corpus. To interpret that better, we need one extra layer of information: the album each song belongs to.
Step 5: Compare Pommelien’s Albums
Pommelien had released two albums at the time of this analysis: Per ongeluk and Gedoe. If the two albums were written under different creative conditions, we might see that in the stylometric map.
The code below creates a dictionary that links Pommelien song titles to album titles. Songs by other artists are labelled as other. We then plot the same PCA coordinates again, but this time the colour indicates album rather than artist.
# ADD ALBUM INFORMATION
# ---------------------
# Indicate which of Pommelien's songs belong to which album
title_to_album = {
"atlas": "gedoe",
"ben je klaar?": "gedoe",
"het beste moet nog komen": "gedoe",
"tegenwoordige tijd": "gedoe",
"erop of eronder": "per ongeluk",
"niemand is bijzonder": "gedoe",
"bel me als je thuis bent": "gedoe",
"roekeloos": "gedoe",
"cassandra": "gedoe",
"wie we morgen zijn": "gedoe",
"medeplichtig": "per ongeluk",
"authentiek": "gedoe",
"ik moet gaan": "gedoe",
"zilver": "per ongeluk",
"ongewoon": "per ongeluk",
"meisjes van honing": "per ongeluk",
"koning minimaliseren": "gedoe",
"wanneer": "per ongeluk",
"dunne lijn": "per ongeluk",
"mensen zoals jij": "gedoe",
"oog voor een oog": "per ongeluk",
"eclips": "per ongeluk",
"entertainment": "per ongeluk",
"droom het donker weg": "per ongeluk",
"oké": "per ongeluk",
"hypothetisch": "per ongeluk",
"wat een idee!?": "per ongeluk",
"doen of zijn": "gedoe",
}
# Work out the album for every song. If it is not by Pommelien, assign it to "other".
song_pca["album"] = song_pca["title"].map(title_to_album).fillna("other")
df["album"] = df["reduced_title"].map(title_to_album).fillna("other")
# Select colours for the albums
album_to_color = {"per ongeluk": "#E45756", "gedoe": "#ecc73f", "other": "#e1ecf7"}
# Sort the DataFrame by artist and plot
song_pca = song_pca.sort_values("artist")
plot_lyrics(song_pca, ["title", "artist"], "'Per ongeluk' vs. 'Gedoe'", width=1000, height=500, color="album", palette=album_to_color, alpha=0.5)When we add album information, the picture becomes clearer. The Pommelien songs closest to Metejoor mainly come from Per ongeluk. Because Metejoor also worked closely with The Companions, this makes it plausible that the first Pommelien album shares more of that collaborative writing environment.
The songs from Gedoe move further away from that Metejoor-like cluster. This does not prove who wrote which line, but it does suggest that the second album has a different stylistic profile and may reflect Pommelien’s own writing voice more strongly.
The album split also helps interpret the few Pommelien songs that appear closer to Boudewijn de Groot. They are not all from the same album, so their position is probably not simply an album effect. They may be different because of theme, register, or lyrical ambition rather than because of a general shift in Pommelien’s writing style.
Step 6: Compare Content Instead of Style
So far, we have focused on style by counting frequent words. But we can also ask a different question: which songs are similar in content?
To do that, we make one small but important change. Instead of CountVectorizer, we use TfidfVectorizer. TF-IDF gives more weight to words that are distinctive for a particular song and less weight to words that occur almost everywhere. In other words, it shifts the analysis away from function-word habits and towards topic and vocabulary.
The code is almost the same as before. We vectorize the lyrics, compute PCA coordinates, and plot the songs again.
# CONTENT ANALYSIS
# ----------------
# TF-IDF vectorizer; one dot per song
artists = df["artist"]
titles = df["reduced_title"]
lyrics = [" ".join(lyrics) for lyrics in df["lemmas"]]
ids = [i for i in range(len(titles))]
# Vectorize lyrics and compute coordinates
vectorized_lyrics = vectorize_lyrics(lyrics, "tfidf", max_features=3000)
x, y = compute_coordinates(vectorized_lyrics)
# Add coordinates to DataFrame and plot
song_pca = pd.DataFrame([ids, artists, titles, x, y], index=["id", "artist", "title", "x", "y"]).transpose()
song_pca = song_pca.sort_index()
plot_lyrics(song_pca, ["title"], title="Content analysis", width=1000, height=500)The content plot should be interpreted differently from the stylometric plot. Here, nearby songs are not necessarily written in the same style. They are songs that use similar content words. This makes the plot useful for exploring themes, topics, and recurring images.
In the blog post, this second analysis helped us see that Pommelien’s lyrics cover a relatively wide range of themes. That thematic range is part of why the comparison with Boudewijn de Groot becomes interesting: both artists move across a broader lyrical landscape than a simple pop-song stereotype might suggest.
Finally, we save the most important results. The output file contains the song id, title, artist, stylometric coordinates, album label, and content-analysis coordinates. This file can be reused in a visualisation tool such as Flourish, Tableau, Observable, or another Python notebook.
# SAVE RESULTS
# ------------
# Add content-analysis coordinates to the DataFrame
df["content_x"] = x
df["content_y"] = y
# Save the relevant columns
df_to_save = df[["id", "reduced_title", "artist", "stylometry_x", "stylometry_y", "album", "content_x", "content_y"]]
df_to_save.to_csv("resources/songs_with_coordinates.csv", sep=",", index=False)What Next?
This workflow is deliberately compact, but it gives you a reusable template for stylometric exploration. You can adapt it by adding more artists, changing the number of frequent words, comparing albums or periods, or replacing song lyrics with another type of short text.
A few cautions are worth keeping in mind:
- Stylometry is comparative: the results depend on the texts you include as reference material.
- Small corpora can produce unstable patterns, especially when texts are very short.
- Visual clusters are clues, not final proof. They should be interpreted together with close reading, metadata, and knowledge of the cultural context.
That combination of computational overview and human interpretation is exactly what makes digital text analysis useful for humanities research.