# load pretty jupyter's magics
-%load_ext pretty_jupyter
-from plotly.offline import plot
-from IPython.display import IFrame
-import pandas as pd
-from IPython.display import display, HTML
-import plotly.express as px
-import plotly.graph_objects as go
-import plotly.io
-plotly.io.renderers.default = 'notebook_connected'
-Introduction¶
"The Hogwarts Bestiary" will, in a way, classify beasts that come from the wizarding world through the Harry Potter series and, -according to the various intrinsic qualities, descriptions, and the symbolic meaning or essence each individual creature holds, -sort them into their own respective Hogwarts house. The datasets central to our analysis were meticulously created by myself, -leveraging web scraping techniques to gather detailed information from reputable sources and conducting thorough research to ensure -accuracy and completeness. This unique compilation serves as the foundation for our exploratory data analysis, offering insights into -the magical realm.
- - - - - - - -Hogwarts Houses¶
-
A Deep Dive into Symbolism, Traits, and More
-Gryffindor¶
Symbolic Essence: Fire; Lion
-Colors: Scarlet and Gold
-Notable Characteristics: This house values bravery, courage, and determination. Its founder, Godric Gryffindor, hailed from the moors and was renowned as the greatest duelist of his of his time, embodying the house's spirit of valor. However, Gryffindor's emphasis on bravery can sometimes lead to recklessness and an overestimation of one's own abilities, resulting in actions without thought for consequences.
-Hufflepuff¶
Symbolic Essence: Earth; Badger
-Colors: Yellow and Black
-Notable Characteristics: It is celebrated for its values of loyalty, patience, and hard work. Helga Hufflepuff, from the valleys of Wales, was known for her compassionate nature and excellence in charms, magical manufacturing, and promoting an inclusive ethos. Despite these virtues, Hufflepuff is sometimes unfairly perceived as less competitive and overly accommodating, potentially overshadowing its members' achievements.
-Ravenclaw¶
Symbolic Essence: Air; Eagle
-Colors: Blue and Bronze
-Notable Characteristics: This house values intelligence, creativity, and wisdom. Rowena Ravenclaw, from the Scottish glens, was the most brilliant witch of her time, with profound talents in divination, invention, and magical architecture, embodying the house's pursuit of knowledge. However, Ravenclaw's quest for wisdom can lead to a sense of elitism, and an overemphasis on intellect can sometimes detach its members from practical realities.
-Slytherin¶
Symbolic Essence: Water; Serpent
-Colors: Silver and Green
-Notable Characteristics: Known for its ambition, cunning, and resourcefulness, its founder Salazar Slytherin, from the fen-lands, possessed the rare gift of Parseltongue and was skilled in legilimency and the dark arts, reflecting the house's complex and multifaceted nature. However, Slytherin's drive for success can veer into ruthlessness, and its valorization of purity of blood has marred its legacy, contributing to perceptions of exclusivity and manipulation.
- - - - - - - -Methodology: Crafting the Datasets¶
The journey to constructing our datasets began with identifying authoritative and comprehensive sources that catalog magical creatures and details about Hogwarts houses. Utilizing Python alongside web scraping libraries such as BeautifulSoup and Scrapy, I developed scripts to extract the necessary data. The process was iterative, with continuous refinement to handle the nuances of web content and ensure the integrity of the data collected.
-Following the initial data collection, I embarked on a meticulous cleaning and preprocessing phase, employing pandas to normalize, cleanse, and prepare the data for analysis. This phase was critical in transforming raw data into structured datasets that accurately represent the magical entities and their affiliations.
-Web Scraping Overview¶
Creatures Data¶
The creatures' dataset was compiled using Python's BeautifulSoup library, targeting specific fan sites and wikis dedicated to the Harry Potter series. Challenges such as navigating nested HTML structures and handling pages with inconsistent formatting were overcome through a combination of CSS selectors and XPath queries. Key functions like scrape_creature_urls and scrape_creature_info played pivotal roles in efficiently extracting detailed information about each creature's characteristics, habitat, and magical properties.
Houses Data¶
Similarly, the dataset detailing the attributes of Hogwarts houses was scraped from primary sources using Python's requests library and BeautifulSoup for parsing. The scrape_hogwarts_infobox function was crucial in extracting structured data from infoboxes within each house's dedicated page, including house symbols, founders, and key traits.
Conclusion: Datasets¶
The datasets driving this analysis were born out of a keen interest in the magical world and a dedication to unveiling its secrets through data. The creation of these datasets from scratch represents a significant undertaking that involved not just technical skill in web scraping and data processing, but also a deep engagement with the subject matter. As we look to the future, there are ample opportunities to expand this work, adding new dimensions to the datasets and exploring further questions about the magical realm.
-- - - - - - - -
Data Loading and Overview¶
The analysis begins by loading the creatures_data.xlsx file, which contains detailed information about various magical creatures, and the houses_master_dataset.csvfile which contains detailed information about the Hogwarts Houses. The first step is to load the necessary libraries then examine the dataset to understand its structure and content.
Importing Libraries¶
!pip install squarify
-!pip install pandas
-!pip install wordcloud
-!pip install pretty-jupyter
-
-#Importing necessary libraries for data manipulation and visualization
-import pandas as pd
-import matplotlib.pyplot as plt
-from wordcloud import WordCloud
-from plotly.subplots import make_subplots
-import plotly.express as px
-import seaborn as sns
-import re
-from collections import Counter
-import numpy as np
-import squarify
-import ast
-from IPython.display import Image
-import networkx as nx
-from ast import literal_eval
-Importing Datasets¶
In this section, we load two crucial datasets for our analysis:
--
-
creatures_master_dataset.csv: Contains comprehensive details on various magical creatures, including their names, characteristics, and associations with the wizarding world.
-houses_master_dataset.csv: Provides information on the four Hogwarts houses, focusing on their values, symbolic elements, and historical significance.
-
Understanding these datasets is key to exploring the affinity between magical creatures and Hogwarts houses.
-# Load the dataset with detailed information about Hogwarts houses
-houses_df = pd.read_csv('houses_master_dataset.csv')
-# Load the dataset containing information about magical creatures
-# Specifying encoding due to special characters in the dataset
-creatures_df = pd.read_csv('creatures_master_dataset.csv', encoding='latin1')
-creatures_df.head()
-
Magic Functions for Improved Notebook Performance¶
To enhance our notebook's performance and visualization capabilities, we'll employ IPython magic commands. The %matplotlib inline magic function, for example, allows us to display plots directly within the Jupyter notebook.
# Enable inline plotting
-%matplotlib inline
-- - - - - - - -
Data Cleaning and Preprocessing¶
The raw datasets underwent several cleaning steps to ensure their accuracy and usability for analysis.
-Creatures Cleaning¶
Functions such as correct_classifications and correct_origin_names were developed to standardize taxonomy and geographical names. This preprocessing was vital for consistent analysis, particularly for grouping creatures by origin or classification.
Houses Cleaning¶
While the houses dataset required minimal cleaning, due diligence was performed to validate the integrity of the data, ensuring no missing values or inconsistencies that could skew the analysis.
-Preprocessing¶
Specific functions are designed to address inconsistencies, missing values, and to prepare our data for analysis. Preprocessing the text data involves:
--
-
clean_text: Removing or altering unwanted characters, spaces, or symbols.
-tokenize_text: Split text into individual elements or tokens for future scoring.
-remove_stop_words: Filters out common words (such as "the", "is", etc.) that do not contribute significant meaning.
-
The following section outlines the steps taken to clean our datasets. Specific functions are designed to address inconsistencies, missing values, and to prepare our data for analysis.
-# Example of a cleaning function
-def clean_text(column):
- creatures_df[column] = creatures_df[column].str.replace('[^a-zA-Z]', ' ').str.lower()
-clean_text('Name')
-# Example function to filter out stop words from a list of words
-def remove_stop_words(word_list):
- return [word for word in word_list if word not in stop_words]
-# Apply the function to remove stop words from the creatures' cleaned traits
-creatures_df['Cleaned_Traits'] = creatures_df['Traits'].apply(remove_stop_words)
--
Exploratory Data Analysis (EDA)¶
The purpose of this analysis is to examine the affinities between magical creatures and Hogwarts houses - and which factors are -most important in determining these categorizations. With our data cleaned, we're now ready to explore and analyze it to uncover -patterns and insights.
-Initial Categorization¶
#Function to match creatures to houses from dictionary of mapped house keywords
-def match_creatures_to_houses(creature_keywords):
- matches = {house: 0 for house in house_to_keyword_matches}
-# Apply the matching function to each creature's traits
-creatures_df['Initial House'] = creatures_df['Traits'].apply(match_creatures_to_houses)
--
We explore the initial assignment of magical creatures to the four Hogwarts houses based on their characteristic traits. This assignment leverages the foundational traits associated with each house, as famously known from the Harry Potter series:
--
-
- Gryffindor: Brave, Courage, Daring, Determination, Bold -
- Hufflepuff: Loyal, Hard work, Fair, Patience, Dedication -
- Ravenclaw: Wise, Intelligent, Creative, Knowledge, Wisdom -
- Slytherin: Ambitious, Cunning, Resourceful, Leadership, Ambition -
-
-
The bar chart above depicts the number of creatures assigned to each Hogwarts house, as well as combinations for creatures that exhibit traits from multiple houses.
-Key Observations:
--
-
Ravenclaw Dominance: There is a notable dominance of creatures assigned to Ravenclaw. This is likely due to the high occurrence of the keyword "knowledge" in the creatures' descriptions, which aligns strongly with Ravenclaw's emphasis on intelligence and wisdom.
-
-Cross-House Traits: Several creatures display traits that span across multiple houses, as seen in the assignment to 'Hufflepuff, Ravenclaw' and 'Ravenclaw, Slytherin'. This reflects the complexity of the creatures' characteristics and their multifaceted nature.
-
-Rare Traits: Gryffindor and Slytherin have the fewest assignments, suggesting that traits like bravery and ambition are less commonly observed or less emphasized in the descriptions of these creatures.
-
-
The distribution reflects an initial attempt at categorizing magical creatures based on their described traits. However, it raises questions about the method of assignment and the possible over-representation of certain traits. This provides an opportunity to refine the criteria and consider a more nuanced approach to trait analysis.
-Next Steps:
--
-
- Refinement of Keyword Analysis: Refine the methodology to ensure a balanced representation of house traits. -
- Trait Weighting: Consider the relative importance or prominence of each trait within the creatures' descriptions. -
- Additional Data Sources: Include more data sources or descriptions for a comprehensive analysis. -
- - - - - - - -
Key Trait Analysis of Magical Creatures¶
In our quest to understand the intricate details that characterize the magical creatures of the wizarding world, we have employed a rule-based matching process. This technique has revealed recurring key phrases in the trait descriptions, painting a vivid picture of each creature's distinctive features and abilities.
-Methodology Overview¶
The process involves a meticulous examination of descriptive phrases, pinpointing specific behavioral traits, magical abilities, and specialized knowledge. By defining precise patterns and employing regular expression techniques, we have extracted and tallied the most notable traits associated with these mystical beings.
-#Extracts and creates a dictionary of creatures whose descriptions include the regex pattern phrase
-pattern = re.compile(r"most \w+ and \w+", re.IGNORECASE)
-matched_creatures = []
-for index, row in creatures_df.iterrows():
- matches = pattern.findall(row['Traits'].lower())
- matched_creatures.extend([(row['Name'], match) for match in matches])
--
Regex Pattern Phrases:
--
-
- "most and " -
- "knowledge of and " -
- "can produce ___ " -
Visual Exploration of Keywords¶
To deepen our understanding of magical creatures and their corresponding traits, we have distilled the essence of various words into word clouds. These visual representations emphasize the frequencies of keywords that best characterize the creatures within the realms of their knowledge, most notable traits, and any specific abilities.
- - - - - - - - - - -import pandas as pd
-import matplotlib.pyplot as plt
-
-# Load the data
-df = pd.read_csv('final_phrases.csv')
-
-# Filter columns ending in 'phrase'
-phrase_columns = [col for col in df.columns if col.endswith('Phrase')]
-
-# Specify the colors to be used in the word clouds
-colors = ['#013220', '#7F0909', '#FFDB00', '#0E1A40']
-
-# Define stopwords to exclude
-custom_stopwords = set(STOPWORDS).union({'unsorted', 'knowledge', 'produce', 'loud', 'powerful'})
-
-# Setup the figure and axes for the subplots
-# Adjust the figsize to fit your screen and the number of subplots appropriately
-fig, axes = plt.subplots(1, len(phrase_columns), figsize=(len(phrase_columns) * 6, 6))
-
-for i, col in enumerate(phrase_columns):
- # Prepare text data from the column
- text_data = ' '.join(df[col].dropna().values)
-
- # Create a word cloud object with custom stopwords
- wordcloud = WordCloud(background_color='white',
- colormap='tab10',
- stopwords=custom_stopwords,
- color_func=lambda *args, **kwargs: colors[i % len(colors)]).generate(text_data)
-
- # Display the generated word cloud on the respective subplot
- ax = axes[i] if len(phrase_columns) > 1 else axes # This handles both single and multiple plots
- ax.imshow(wordcloud, interpolation='bilinear')
- ax.axis('off')
-
- ax.set_title(f'Word Cloud for {col}')
-
-plt.tight_layout()
-plt.show()
-
-
-
-
-
-
-
--
-
- The word cloud for Knowledge Phrases bursts with terms like "potions," "herbs", "astrology," and "divination" that reflect an intrinsic connection to speciliazed knowledge of the magical world. -
- The Most Phrase word cloud highlights words like "elusive," "intelligent," "friendly," and "cunning", indicating creatures' behaviors. -
- In the Produce Phrase word cloud, we see a range of words from "fire" and "venom" to "squeak" and "healing," suggesting the diverse and magical outputs that creatures are capable of producing. -
These word clouds provide us with a vivid linguistic map, tracing the contours of our magical creatures' most prominent attributes. The size and prominence of each word guide us in recognizing the traits most celebrated and feared in the magical community.
-Comparative Analysis of Behavioral Traits, Specialized Knowledge, and Magical Abilities from Extracted Phrases¶
Our exploration into the magical creatures' affinities with Hogwarts houses reaches a pivotal point with the analysis of key phrases extracted from their descriptions. The donut charts presented below illustrate the final house assignments derived from these phrases, emphasizing the diverse attributes that creatures bring to each house.
-#Function to score each keyword and determine Keyword House by the max_score.
-def score_and_assign_houses(row, house_keywords):
- scores = {house: 0 for house in house_keywords}
- # Handle tie or no house found
- assigned_houses = [house for house, score in scores.items() if score == max_score]
- # If no keywords matched, set to "Unsorted"
- if max_score == 0:
- return "Unsorted"
- else:
- return ', '.join(assigned_houses)
-Behavioral Traits: Which Factors Are More Important?¶
The donut charts present the distribution of creatures among Hogwarts houses based on the frequency of certain keywords in their descriptions, providing a unique perspective on the attributes most valued by each house.
- - - - - - - - - - -import pandas as pd
-import matplotlib.pyplot as donut_plt
-
-# Load the data, replacing NaN values with 'Unsorted'
-df = pd.read_csv('final_phrases.csv').fillna('Unsorted')
-
-# Define the 'house' columns and the colors
-house_columns = ['Knowledge House', 'Produce House', 'Most House']
-house_colors_updated = {
- 'Ravenclaw': '#0E1A40',
- 'Hufflepuff': '#FFDB00',
- 'Slytherin': '#013220',
- 'Gryffindor': '#7F0909',
- 'Hufflepuff, Ravenclaw, Slytherin': '#627C5B', # Combination of Hufflepuff, Ravenclaw, and Slytherin
- 'Ravenclaw, Slytherin': '#02282B', # Combination of Ravenclaw and Slytherin
- 'Gryffindor, Slytherin': '#405214', # Combination of Gryffindor and Slytherin
- 'Hufflepuff, Slytherin': '#80993B', # Combination of Hufflepuff and Slytherin
- 'Gryffindor, Ravenclaw, Slytherin': '#423D49', # Combination of Gryffindor, Ravenclaw, and Slytherin
- 'Unsorted': '#808080',
- 'Gryffindor, Ravenclaw': '#453E4A', # Combination of Gryffindor and Ravenclaw
- 'Hufflepuff, Ravenclaw': '#8E782F', # Combination of Hufflepuff and Ravenclaw
- 'Gryffindor, Hufflepuff, Slytherin': '#857A33', # Combination of Gryffindor, Hufflepuff, and Slytherin
- 'Gryffindor, Hufflepuff': '#B78655', # Combination of Gryffindor and Hufflepuff
- 'Gryffindor, Hufflepuff, Ravenclaw': '#967348' # Combination of Gryffindor, Hufflepuff, and Ravenclaw
-}
-
-
-# Create subplots for the donut charts
-fig, axes = donut_plt.subplots(1, len(house_columns), figsize=(22, 14))
-
-# Iterate over each house column to create a donut chart
-for i, column in enumerate(house_columns):
- # Get the count of each category
- data = df[column].value_counts()
-
- # Assign colors to each category
- colors = [house_colors_updated[house] if house in house_colors_updated else '#808080' for house in data.index]
-
- # If there's only one subplot, wrap 'axes' in a list
- ax = axes[i] if len(house_columns) > 1 else [axes]
-
- # Create the pie chart
- ax.pie(data, labels=data.index, autopct='%1.1f%%', startangle=140, colors=colors, wedgeprops=dict(width=0.3))
- ax.set_title(f' {column}')
-
-# Adjust the layout and show the plots
-donut_plt.tight_layout()
-donut_plt.show()
-
-
-
-
-
-
-
-Knowledge House:
--
-
A dominant unsorted category, suggests that many creatures possess complex areas of knowledge that do not align clearly with a single house's known attributes.
-
-Hufflepuff has a moderate presence, reflecting creatures' affinities for diversity and connection to nature, while Gryffindor and Slytherin have fewer creatures, which may point to more selective or less common knowledge-related associations among the creatures studied.
-
-
Produce House:
--
-
- The unsorted segment is notably larger, emphasizing the difficulty in associating creatures with houses based on their magical capabilities alone. -
- Hufflepuff and Ravenclaw's minimal presence could be attributed to fewer creatures having overt displays of 'productive' prowess associated with the house. -
- Gryffindor's slight increase suggests a small variety of creatures that produce heat and combustion, aligning with the house's core element - fire. -
Most House:
--
-
- There is a more even distribution among the houses, with a smaller unsorted proportion which might indicate that creatures' most prominent behaviors are easier to match with the core values of each house. -
- Gryffindor and Hufflepuff show a notable presence, aligning with the strong and often loyal attributes represented by the keywords, while Ravenclaw and especially Slytherin reflect more specific traits like wisdom and cunning. -
These charts underscore the challenge in categorizing creatures within the established Hogwarts house framework. They also highlight the potential for deeper analysis into the creatures' behavioral traits to better understand their house alignments. The substantial number of unsorted creatures (which include creatures with multiple house alignments) in each category calls for a more refined approach to explore the complexity of the magical creature taxonomy.
-Final House Alignments Based on Keyword Frequencies¶
The interactive charts below provide a clear visual representation and breakdown of the final house alignments and the magical creatures assigned to each. These alignments are based on the frequency of specific keywords found within the creatures' descriptions that resonate with the values and traits of each Hogwarts house.
- - - - - - - - - - -final_house_df = pd.read_csv('final_house.csv')
-
-import pandas as pd
-import plotly.express as px
-
-house_counts = final_house_df['Final House'].value_counts().sort_index()
-colors = {
- 'Gryffindor': '#7F0909',
- 'Hufflepuff': '#FFDB00',
- 'Slytherin': '#013220',
- 'Ravenclaw': '#0E1A40',
- 'Unsorted': '#808080',
- 'Multiple Houses': '#666666'
-}
-
-house_counts_df = house_counts.reset_index()
-house_counts_df.columns = ['House', 'Count']
-house_counts_df['House'] = house_counts_df['House'].apply(lambda x: 'Multiple Houses' if ', ' in x else x)
-house_counts_df = house_counts_df.groupby('House')['Count'].sum().reset_index()
-house_counts_df['Color'] = house_counts_df['House'].map(colors)
-filtered_houses = ['Gryffindor', 'Hufflepuff', 'Slytherin', 'Ravenclaw', 'Unsorted', 'Multiple Houses']
-house_counts_df = house_counts_df[house_counts_df['House'].isin(filtered_houses)]
-colors_filtered = {house: colors[house] for house in filtered_houses}
-
-barchart_fig = px.bar(
- house_counts_df,
- x='House',
- y='Count',
- text='Count',
- color='House',
- color_discrete_map=colors_filtered
-)
-
-barchart_fig.update_layout(
- title='Creature Trait Distribution Across Hogwarts Houses',
- xaxis_title='House',
- yaxis_title='Number of Creatures',
- legend_title='Hogwarts Houses',
- legend=dict(yanchor="top", y=0.99, xanchor="left", x=0.01),
- paper_bgcolor='rgba(0,0,0,0)', # Set the background of the figure to transparent
- plot_bgcolor='rgba(0,0,0,0)', # Set the background of the plotting area to transparent
- # For a white background, replace 'rgba(0,0,0,0)' with 'white' in both lines above
-)
-
-barchart_fig.update_traces(
- hoverinfo='name+y+text',
- textposition='outside'
-)
-
-barchart_fig.show(renderer="notebook_connected")
-from plotly.subplots import make_subplots
-import plotly.express as px
-
-
-# Assuming final_house_df is your DataFrame and it's already loaded
-# Define the color mapping including "Multiple Houses"
-colors = {
- 'Gryffindor': '#7F0909',
- 'Hufflepuff': '#FFDB00',
- 'Slytherin': '#013220',
- 'Ravenclaw': '#0E1A40',
- 'Unsorted': '#808080', # Neutral gray for 'Unsorted'
- 'Multiple Houses': '#666666' # Dark gray for 'Multiple Houses'
-}
-
-
-# Update 'Final House' to group multiple houses into "Multiple Houses"
-final_house_df['Final House Grouped'] = final_house_df['Final House'].apply(lambda x: 'Multiple Houses' if ', ' in x else x)
-
-# Filter the DataFrame to exclude 'Unsorted' and 'Review Needed'
-filtered_df = final_house_df[~final_house_df['Final House Grouped'].isin(['Unsorted', 'Multiple Houses'])]
-
-# Treemap visualization
-treemap_fig = px.treemap(filtered_df,
- path=['Final House Grouped', 'Name'],
- color='Final House Grouped',
- color_discrete_map=colors)
-
-# Update layout for a better look
-treemap_fig.update_layout(margin=dict(t=50, l=25, r=25, b=25))
-
-treemap_fig.show(renderer="notebook_connected")
-Key Observations:¶
-
-
- Hufflepuff emerges with a notably high count, potentially indicating a wide range of nurturing and earth-associated traits among its creatures. -
- Gryffindor and Ravenclaw have similar counts, reflecting their shared appreciation for magical history and intellect, respectively. -
- Slytherin's creatures are fewer in number, which may highlight a selective nature or specific trait preferences within the house. -
- A significant number of creatures remain Unsorted or belong to Multiple Houses, suggesting complex trait combinations that do not fit neatly into one house. -
Distribution Highlights:¶
-
-
- Ravenclaw’s section features creatures like the Augurey and Wampus Cat, symbolizing the house's penchant for wisdom and wit. -
- Hufflepuff is represented by the nurturing, yet mischievous Niffler and industrious Bowtruckle, creatures that echo the house's dedication to diligence and loyalty. -
- In Slytherin, the mystical and cunning nature of the house is mirrored in creatures such as the Basilisk and the Siren. -
- Gryffindor boasts the brave and bold Griffin and the fearless Fire Crab, underscoring the house's valor and strength. -
The assortment of creatures within each house color zone reveals the intrinsic values and traits each house celebrates. From the nurturing care in Hufflepuff to the cunning of Slytherin, each section of the treemap tells a story of the magical creatures’ compatibility with their respective houses.
-- - - - - - - -
Physical Feature Analysis of Magical Creatures¶
In this part of the Exploratory Data Analysis (EDA), we delve into the fascinating world of magical creatures, assessing their alignments with the Hogwarts houses through the lens of their distinct physical features.
-Methodology Overview¶
The investigation categorizes creatures by their most prominent physical attributes — color and species. We seek to discover if there is a significant correlation between these characteristics and their house alignments.
-house_to_keyword_matches = {
- "Gryffindor": ["red", "scarlet", "gold"],
- "Hufflepuff": ["yellow", "black"],
- "Ravenclaw": ["blue", "bronze", "copper", "brown"],
- "Slytherin": ["green", "silver", "grey"]
-}
-
-#Applies scoring function to 'Colors'
-creatures_df['Color House'] = creatures_df['Colors'].apply(determine_house_scored)
-Comparative Analysis of Creatures' Colors and Species¶
Individual House Color Associations¶
The first network graph showcases the creatures associated with each Hogwarts house, color-coded to emphasize their individual house alignments. It visually represents how each house attracts certain creatures, possibly reflecting the extrinsic characteristics or values of the house.
-
-
-
-
-
This visualization uncovers any color patterns specific to a house, perhaps indicating a preference or tendency for certain types of creatures within that house's lore.
-Overlapping House Color Associations¶
The second network graph expands our understanding by revealing creatures with color affiliations to multiple houses. It provides insights into the complexity of creature-house relationships, suggesting a more intricate interconnection between the creatures and the houses they are aligned with.
- - - - - - - - - - -# Read the dataframe
-df = pd.read_csv('physical_features.csv')
-
-
-
-import pandas as pd
-import networkx as nx
-import plotly.graph_objs as go
-
-# Define a function to parse strings that represent lists
-def parse_list_string(list_string):
- return list_string.strip("[]").replace("'", "").split(", ")
-
-# Define house colors
-house_colors = {
- 'Gryffindor': '#7F0909',
- 'Hufflepuff': '#FFDB00',
- 'Slytherin': '#013220',
- 'Ravenclaw': '#0E1A40',
-}
-
- # Initialize graph
-G = nx.Graph()
-
-# Create nodes and edges
-for index, row in df.iterrows():
- creature = row['Name']
- # Only consider main houses by filtering the 'Final House' column
- houses = [house for house in parse_list_string(row['Final House']) if house in house_colors]
- for house in houses:
- # Add 'label_color' attribute for house nodes
- G.add_node(house, size=100, color=house_colors[house], label_color='white')
- G.add_node(creature, size=10, color='grey', label_color='black') # Default color for creatures
- G.add_edge(creature, house)
-
-
-# Position nodes using the Fruchterman-Reingold force-directed algorithm
-pos = nx.spring_layout(G)
-
-# Create edge traces
-edge_x = []
-edge_y = []
-for edge in G.edges():
- x0, y0 = pos[edge[0]]
- x1, y1 = pos[edge[1]]
- edge_x.extend([x0, x1, None])
- edge_y.extend([y0, y1, None])
-
- # Create node traces
-node_x = []
-node_y = []
-node_color = []
-node_size = []
-text_color = [] # Initialize an empty list for text colors
-for node in G.nodes():
- node_x.append(pos[node][0])
- node_y.append(pos[node][1])
- node_color.append(G.nodes[node]['color'])
- node_size.append(G.nodes[node]['size'])
- # Assign 'white' for house nodes and 'black' for creature nodes
- text_color.append(G.nodes[node]['label_color'] if 'label_color' in G.nodes[node] else 'black')
-
-
-
-# Create the Plotly trace for the graph
-edge_trace = go.Scatter(
- x=edge_x, y=edge_y,
- line=dict(width=0.5, color='#888'),
- hoverinfo='none',
- mode='lines')
-
-node_trace = go.Scatter(
- x=node_x, y=node_y,
- mode='markers+text',
- text=list(G.nodes()),
- hoverinfo='text',
- marker=dict(
- showscale=False,
- color=node_color,
- size=node_size,
- line_width=2),
- textfont=dict( # Set text color based on the node type
- color=text_color
-)
-)
-
-# Define the layout of the figure
-network_fig = go.Figure(data=[edge_trace, node_trace], layout=go.Layout(
- title='Creature Color to House Alignments',
- showlegend=False,
- hovermode='closest',
- margin=dict(b=20, l=5, r=5, t=40),
- xaxis=dict(showgrid=False, zeroline=False, showticklabels=False),
- yaxis=dict(showgrid=False, zeroline=False, showticklabels=False),
- plot_bgcolor='rgba(0,0,0,0)',
- paper_bgcolor='#ffffff',
- width=1000,
- height=1000
- ))
-network_fig.show(renderer="notebook_connected")
-By examining the overlaps, we can speculate on the multifaceted nature of these creatures and how their attributes resonate with more than one house, blending the distinct qualities each house embodies.
-Insights from Network Graphs¶
From the network graphs, we can infer the following:
--
-
- Certain houses, such as Gryffindor and Slytherin, may have creatures with more pronounced or bold color features, resonating with the houses' strong identities. -
- Hufflepuff and Ravenclaw may demonstrate a more diverse palette, suggesting an inclusivity for a wider range of creatures. -
- Overlaps in color associations highlight that some creatures cannot be confined to the qualities of just one house. They embody a blend of traits that resonate with multiple houses, such as bravery and intelligence, or ambition and loyalty. -
Species Diversity in Hogwarts Houses¶
In our quest to decode the magical fauna of the wizarding world, we've embarked on a detailed analysis of the species diversity within each Hogwarts house.
-Methodological Approach for Species Analysis¶
The approach taken involved categorizing the myriad of species into a manageable form. We enumerated each species' occurrences across the houses, highlighting the most prevalent and aggregating the rarer ones under an "Other" category for a more streamlined analysis.
-# Python code for species categorization
-species_counts = filtered_species_data['Species'].value_counts()
-top_species = species_counts.head(10).index.tolist()
-filtered_species_data['Species Aggregated'] = filtered_species_data['Species'].apply(
- lambda x: x if x in top_species else 'Other')
-aggregated_counts = filtered_species_data.groupby(
- ['Species House', 'Species Aggregated']).size().unstack(fill_value=0)
-
-if 'Other' not in aggregated_counts.columns:
- aggregated_counts['Other'] = 0
--
With a total of 85 unique species identified, we found that creatures such as Serpents and Snakes, various Plants, and Insects were among the top observed species.
-Visualizing Species Distribution¶
Our heatmap serves as a testament to the diversity of magical creatures in Hogwarts houses.
-
It illustrates the count and distribution of prominent species across Gryffindor, Hufflepuff, Ravenclaw, and Slytherin. The color gradients on the heatmap indicate the frequency of each species within the houses, shedding light on unique species-house correlations and the prevalence of certain creatures in particular houses.
-Additionally, we provide a bar chart showcasing the concentration of symbolic species in their respective houses, underscoring the representational alignment with house mascots.
-
Through this bar chart, we unravel some captivating narratives:
--
-
- Gryffindor resonates with the valor of its lion emblem, housing 5 species that share attributes with its symbolic animal. -
- Hufflepuff, known for its nurturing nature, is associated with 3 species representing its badger emblem and other related natural entities. -
- Ravenclaw's erudition is mirrored in its 10 species that correlate with its symbolic eagle. -
- Slytherin's 11 serpent-like species emphasize the house's infamous connection with cunning and ambition. -
Conclusive Insights¶
The creatures’ association with house symbols reflects the deeper, often mystical attributes they share with their respective houses. This alignment goes beyond the surface, hinting at a more profound, sometimes arcane, connection that these creatures have with the houses of Hogwarts. By scrutinizing the species-house alignments, we gain an enriched perspective of the traits and values each house.
- - - - - - - -Color and Species House Alignment Analysis¶
Exploring the Concordance Between Color and Species Assignments¶
Our study aims to uncover whether the Hogwarts house assignments based on a creature's color are in agreement with those based on species type.
-# Calculate the alignment between color-based and species-based house assignments
-alignment = data.apply(lambda x: x['Color House'] == x['Species House'], axis=1).mean()
-alignment_percentage = alignment * 100 # Convert to percentage
--
To facilitate this comparison, a new binary column flags whether each creature's house assignment due to its color aligns with that due to its species. This computation is succinctly performed by the above Python code snippet, resulting in a percentage value indicating overall alignment.
-
-Only a mere 18.3% of the creatures show concordance in their house assignments based on color and species. This low alignment rate underscores the complexity and diversity in the defining traits of the creatures, suggesting that color and species may often signal distinct qualities valued by different houses. The divergence in house assignment based on these characteristics accentuates the multifaceted nature of the creatures and the nuanced considerations involved in their categorization within the magical ecosystem of Hogwarts.
Comparison of Mapped Houses (Color vs. Species)¶
In our exploration of Hogwarts house assignments, we compare the influence of physical color traits to species-related characteristics.
-Visual Comparison¶
-Insights from House Mappings¶
-
-
Color-Based Assignments: The color traits of creatures reveal a balanced distribution of house alignments, with Ravenclaw and Slytherin having a larger share, hinting at the rich variety in the visual attributes that these houses may resonate with. Interestingly, Gryffindor and the "Unsorted" categories are less represented, suggesting that color alone may not be the definitive factor for these assignments.
-
-Species-Based Assignments: A stark contrast is observed in the species-based assignments, where a whopping 75% fall into the "Unsorted" category. This imbalance implies that species traits alone may not be sufficiently distinctive to align with the specific values or characteristics of the Hogwarts houses. The other houses, including Slytherin, Ravenclaw, Gryffindor, and Hufflepuff, share the remaining quarter, indicating fewer direct correlations between species types and house traits.
-
-
Concluding Observations¶
The side-by-side comparison suggests that while both color and species play roles in determining house assignments, they often do so independently and without a clear pattern of alignment. The notable volume of "Unsorted" in species-based mapping emphasizes the vast diversity within the magical creature community, where many creatures defy straightforward classification. This diversity underscores the need for a deeper, more nuanced understanding of how the varied attributes of magical creatures reflect the ethos and qualities of the Hogwarts houses.
-Final House Distribution based on Physical Features¶
An intriguing aspect of our analysis is the final house distribution based on physical attributes of the creatures. It gives us a bird's-eye view of how species and color traits are reflected in house assignments.
-Distribution Insights¶
Breakdown of House Assignments¶
-
-
- Slytherin stands out as the most frequently assigned house, accounting for 21.7% of the creatures. This predominance may reflect the house's distinct and pronounced characteristics as perceived in the magical creature community. -
- Ravenclaw is not far behind, with 19.2% of assignments, potentially indicative of this house's diverse and inclusive nature. -
- Hufflepuff and the "Unsorted" category are quite balanced, each around the 12% mark, suggesting that the creatures in this category might exhibit traits that resonate with Hufflepuff's values or are too varied to fit into a single house. -
- Gryffindor appears less commonly as a final assignment, only about 5%, suggesting that Gryffindor-associated traits may be less prominent or less easily identified among the creatures studied. -
Color and Species Distribution and Alignments Insights¶
-
-
- The low alignment rate between color and species-based mappings (~18.33%) underscores the multifaceted nature of these features in determining house associations. -
- The diversity in final house assignments highlights not only the prevalence of certain houses like Slytherin and Ravenclaw but also the substantial "Unsorted" group, pointing to the complexity of neatly classifying magical creatures. -
- Color-based mappings distribute more uniformly, suggesting a wider range of interpretations when it comes to color traits, possibly reflecting the diverse values each house upholds. -
- The skew towards "Unsorted" in species-based mappings reveals that a single physical trait like species may be insufficient to determine house placement, emphasizing the nuanced and complex nature of such classifications. -
Concluding Thoughts¶
Our comprehensive analysis illuminates the layered approach required to classify magical creatures into Hogwarts houses. It underscores that asingle attribute may not holistically encapsulate a creature's affinity to a house. Instead, it is the combination and interplay of various features that paint a more accurate picture. This rich tapestry of house alignments showcases the varied and intricate connections between the creatures' physical characteristics and the enduring legacy of Hogwarts houses.
-- - - - - - - -
Analysis of All Traits and Features¶
Trait Distribution Analysis¶
By scoring all trait houses found during our Exploratory Data Analysis (EDA), we determine a final house for each creature and summarize the distribution of creatures across the Hogwarts houses. This provides insights into the characteristic features that influence house assignments. -exploratory data analysis (EDA).
-# Function to tally scores for each house and determine the final house
-def tally_and_determine_final_house(row):
- # Initialize a dictionary to keep track of scores
- house_scores = {"Gryffindor": 0, "Hufflepuff": 0, "Ravenclaw": 0, "Slytherin": 0,}
-# List of house columns to tally scores from
- house_columns = ['Knowledge House', 'Produce House', 'Most House', 'Species House', 'Color House']
- for column in house_columns:
- # Ensure the cell is treated as a string
- cell_value = str(row[column])
- if cell_value != 'Unsorted' and cell_value != 'nan':
- houses = cell_value.split(', ')
- for house in houses:
- if house in house_scores:
- house_scores[house] += 1
-
-
- # Determine the house(s) with the maximum score
- max_score = max(house_scores.values())
- final_houses = [house for house, score in house_scores.items() if score == max_score]
- # Handling ties, unsorted cases, and determining the final house
- if max_score == 0:
- return "Unsorted"
- elif len(final_houses) > 1:
- return ', '.join(final_houses)
- else:
- return final_houses[0]
-
The bar chart above depicts the distribution of creatures across the Hogwarts houses. Notable observations are:
--
-
- A significant number of creatures exhibit traits that align with Ravenclaw, suggesting this house's attributes are highly represented. -
- There's a considerable overlap in traits leading to multiple house assignments, reflecting the complex nature of magical creatures. -
- The presence of creatures classified as 'Unsorted' indicates that their traits do not fit neatly into the existing house categories, highlighting the diversity and uniqueness among them. -
The visualization aids in discerning how creatures are assorted across the houses and where traits intersect, resulting in multiple house affiliations.
-Exploring Associations¶
Investigating the relationships between magical creatures and Hogwarts houses entails analyzing the distribution of species, Ministry of Magic (M.O.M.) classifications, and origins across houses. This exercise seeks to identify if certain species are more aligned with particular houses or if the origin of creatures influences their house sorting.
-Determining Influential Factors¶
Our goal is to discern the key determinants—be it species, traits, or origins—in categorizing creatures into their final houses. To that end, we shall examine the species, M.O.M. classifications, and origins relative to the final house distributions to unearth any discernible patterns or correlations.
--
-
- M.O.M. Class Distribution: The M.O.M. classifications, indicative of a creature's potential threat or significance, vary across houses. More perilous classes tend to align with single houses. -
- Origin Distribution: The geographic origin of creatures also bears on house assignments. Creatures from specific regions often find congruence with certain houses. -
- Species Distribution: We note a propensity for certain species to associate with particular houses. Dragons and serpents, for example, have a strong presence in houses like Gryffindor and Slytherin. -
Ministry of Magic Danger Classification Distribution Across Final Houses¶
# Python code snippet for calculating M.O.M. classification distribution
-mom_class_distribution = data.groupby('Final House')['M.O.M. Class'].apply(lambda x: x.value_counts().index[0]).reset_index()
--
The M.O.M. classification distribution intimates a relationship between a creature's danger level and its house assignment. Creatures deemed most dangerous are more frequently sorted into single houses or certain house combinations, suggesting that the danger level influences house alignment.
-Origin Distribution Across Final Houses¶
# Python code snippet for calculating origin distribution
-origin_distribution = data.groupby('Final House')['Origin'].apply(lambda x: x.apply(eval).apply(lambda y: y[0] if y else None).value_counts().index[0] if not x.apply(eval).apply(lambda y: y[0] if y else None).empty else None).reset_index()
-- - - - - - - - - - -
import pandas as pd
-import plotly.graph_objects as go
-import ast # For parsing the stringified lists in the CSV
-
- # Load the dataset
-data = pd.read_csv('all_houses.csv')
-origin_house_df=data[['Origin', 'Final House']]
-
-
-
-# Assuming 'Origin' and 'Final House' are stringified lists, we convert them back to lists
-data['Origin'] = data['Origin'].apply(ast.literal_eval)
-data['Final House'] = data['Final House'].apply(lambda x: x.split(', '))
-
-# Prepare the data for visualization
-# We'll create a new DataFrame where each row represents an origin, and columns represent counts per house
-from collections import Counter
-
-# Initialize a dictionary to hold our aggregated data
-origin_counts_per_house = {house: Counter() for house in ['Hufflepuff', 'Ravenclaw', 'Slytherin', 'Gryffindor']}
-
-# Aggregate the data
-for _, row in data.iterrows():
- for house in row['Final House']:
- if house in origin_counts_per_house:
- origin_counts_per_house[house].update(row['Origin'])
-
-# Convert the aggregated data into a DataFrame
-origin_house_df = pd.DataFrame.from_dict(origin_counts_per_house, orient='index').fillna(0).T
-
-# Define the colors for each house
-house_colors = {
- 'Ravenclaw': '#0E1A40',
- 'Hufflepuff': '#FFDB00',
- 'Slytherin': '#013220',
- 'Gryffindor': '#7F0909',
-}
-
-# Create a stacked bar chart
-origin_fig = go.Figure()
-
-for house in origin_house_df.columns:
- origin_fig.add_trace(go.Bar(
- name=house,
- x=origin_house_df.index,
- y=origin_house_df[house],
- marker_color=house_colors[house] # Set the color for each house
- ))
-
-origin_fig.update_layout(
- barmode='stack',
- title='Distribution of Origins by Final House',
- xaxis_title='Origin',
- yaxis_title='Count',
- xaxis={'categoryorder': 'total descending'},
- legend_title='Final House',
- paper_bgcolor='rgba(0,0,0,0)', # Set the background of the figure to transparent
- plot_bgcolor='rgba(0,0,0,0)',
- height=600,
- width=1000
-)
-
-# Rotate the x-axis labels for better readability
-origin_fig.update_xaxes(tickangle=-45)
-
-
-origin_fig.show(renderer="notebook_connected")
-The geographical origin distribution implies that the provenance of creatures plays a role in house sorting, with those hailing from certain regions being more likely to correspond with specific houses.
-Species Distribution Across Final Houses¶
- - - - - - - - - -import pandas as pd
-import plotly.express as px
-import ast
-
-
-
-# Load the dataset
-data = pd.read_csv('all_houses.csv')
-
-# Assuming 'Species' is a stringified list, we convert it back to a list
-data['Species'] = data['Species'].apply(ast.literal_eval)
-data['Final House'] = data['Final House'].apply(lambda x: x.split(', '))
-
-# Prepare the data for visualization by expanding both the Species and Final House columns
-rows = []
-for _, row in data.iterrows():
- for species in row['Species']:
- for house in row['Final House']:
- rows.append({'Species': species, 'House': house})
-
-# Convert the list of dictionaries into a DataFrame
-expanded_data = pd.DataFrame(rows)
-
-# Count the occurrences of each species within each house
-species_distribution = expanded_data.groupby(['House', 'Species']).size().reset_index(name='Count')
-
-
-# Assuming your DataFrame `species_distribution` is ready
-
-# Custom color sequence based on house colors and combinations
-# This sequence is arbitrary and serves as an example. Adjust based on your preferences or logic.
-custom_color_scale = [
- '#ffffff',
- '#808080',
- '#02282B',
- '#013220', # Slytherin
- '#0E1A40', # Ravenclaw
- '#7F0909', # Gryffindor
- '#FFDB00', # Hufflepuff
-]
-
-
-# Create a treemap with the custom color scale
-species_treemap_fig = px.treemap(species_distribution, path=['House', 'Species'], values='Count',
- title='Distribution of Species Across Houses',
- color='Count', hover_data=['Species'],
- color_continuous_scale=custom_color_scale)
-
-species_treemap_fig.update_layout(
- plot_bgcolor='rgba(0,0,0,0)',
- margin=dict(t=0, l=0, r=0, b=0),# Light grey background for the plot area
- title='Distribution of Species Across Houses',
- title_x=0.5, # Center the title
- font=dict(
- family="Verdana", # Optional: Specify font family
- size=12, # Optional: Specify font size
- color="black" # Optional: Specify font color
- )
-)
-
-species_treemap_fig.show(renderer="notebook_connected")
--
-
- Certain species are more frequently associated with specific houses. For instance, dragons are commonly found in Slytherin and Gryffindor. Snakes and serpents are notably associated with Slytherin and houses that include Slytherin in their mix (e.g., Ravenclaw, Slytherin). -
- The diversity of species in mixed houses (e.g., Gryffindor, Hufflepuff, Ravenclaw) includes hippogriffs, birds, lions, indicating a broad range of traits valued across these houses. -
-
EDA Insights and Conclusions¶
Cleaning and Preprocessing¶
The data's transformation through clean_text, tokenize_text, and remove_stop_words functions ensured we had a refined canvas for our exploratory endeavors. The steps included:
-
-
- Text Normalization: Unifying text by stripping special characters and standardizing case sensitivity paved the way for more consistent analysis. -
- Tokenization: This granular breakdown of text into tokens laid the groundwork for an in-depth word-based examination. -
- Stop Words Removal: Discarding common words sharpened our focus on the pivotal elements of our textual data, eliminating the noise. -
The result was a cleaner, more analysis-ready dataset that eased the subsequent stages of our exploration.
-Exploratory Data Analysis (EDA)¶
In our EDA, we dove into the "Hogwarts Bestiary" dataset to unearth the nuanced connections between magical creatures and the iconic Hogwarts houses. Highlights of our EDA are:
--
-
- Creature-House Dynamics: We observed a rich tapestry of magical creatures, each with distinctive house alignments that mirror the diverse philosophies of the houses. -
- Identifying House Traits: With
match_creatures_to_houses, we spotlighted traits resonating with house principles—valor for Gryffindor and cunning for Slytherin, among others.
- - Structured House Assignments: Utilizing functions like
score_and_assign_housesenabled us to quantify these alignments systematically, adding rigor to our house assignment process.
-
Conclusions¶
Our project's groundwork has given us a kaleidoscopic view of the interplay between magical fauna and house attributes. With a robust dataset, our analysis has started to demystify complex creature-house affiliations.
-As we proceed, we are poised to apply advanced analytics, aiming to deepen our understanding of this mystical symbiosis. Our endgame is a thorough comprehension of the magical bestiary's integration into the wizarding realm, an endeavor that promises to enrich the lore of the houses and their beastly companions.
-The journey thus far hints at a treasure trove of insights awaiting discovery, with the promise to expand the annals of magical zoology in the wizarding world.
--
Final Thoughts¶
Our investigative journey through data has charted the alignments of magical creatures with Hogwarts house ethos. The use of Python in this Jupyter notebook has exemplified its versatility and strength in tackling complex data science challenges.
-Further investigations might include deep dives into the traits of unsorted creatures or even a dynamic analysis should additional data surface over time, ensuring our bestiary remains as living and breathing as the magical creatures it comprises.
- - - - - - - -Acknowledgments¶
While this project was enriched by a variety of sources and tools that facilitated data collection and analysis, it's important to note that the datasets underpinning this exploration were personally compiled by me. This endeavor required not only technical web scraping but also critical assessment and validation of the information to ensure its reliability and relevance to our analysis goals.
- - - - - - - - - - -!jupyter nbconvert --to html --template pj --embed-images 'Hogwarts Bestiary.ipynb'
-[NbConvertApp] Converting notebook Hogwarts Bestiary.ipynb to html -[NbConvertApp] Writing 7584060 bytes to Hogwarts Bestiary.html -- - - - - - - - -
-