Skip to content

Instantly share code, notes, and snippets.

@ThiagoFPMR

ThiagoFPMR/discord-analysis.ipynb

Created November 7, 2020 05:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ThiagoFPMR/a8403f3ae1ecc8557b136d25e90d475f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ThiagoFPMR/a8403f3ae1ecc8557b136d25e90d475f to your computer and use it in GitHub Desktop.
Download ZIP
Discord-Analysis
Display the source blob
Display the rendered blob
Raw
discord-analysis.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Discord-Analysis",
"provenance": [],
"collapsed_sections": [
"0HR9y4huHU9z"
],
"authorship_tag": "ABX9TyNij8h+PD0eZP1aXYIw7HoU",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/ThiagoFPMR/a8403f3ae1ecc8557b136d25e90d475f/discord-analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EOLnL0gQ4_Vx"
},
"source": [
"# Importing Modules & Reading Data\n",
"Getting the imports out of the way first."
]
},
{
"cell_type": "code",
"metadata": {
"id": "JYCyjv_eufKw",
"outputId": "5c3087e5-4569-4839-a23c-9b296f8c8f8c",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"!pip install emoji"
],
"execution_count": 92,
"outputs": [
{
"output_type": "stream",
"text": [
"Requirement already satisfied: emoji in /usr/local/lib/python3.6/dist-packages (0.6.0)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "CYMPg039z0ca"
},
"source": [
"import re\n",
"import emoji\n",
"import numpy as np\n",
"import pandas as pd\n",
"import plotly.express as px"
],
"execution_count": 95,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "JHubCidC5S-C"
},
"source": [
"## Reading the data"
]
},
{
"cell_type": "code",
"metadata": {
"id": "GcKcgqkJ0-jl"
},
"source": [
"data = pd.read_csv(\"msg_hist.csv\", usecols=[1, 2, 3])"
],
"execution_count": 110,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Tj98-JZbHcuR"
},
"source": [
"# Preparing The Data\n",
"Turning the raw data we extracted into something better to work with."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PF1Z43zn6Wh0"
},
"source": [
"## Anonymizing The Data\n",
"It's not ethical (and probably not even legal) to make the account names of the people in a dataset public without their permission, so we'll anonimize them first."
]
},
{
"cell_type": "code",
"metadata": {
"id": "0nQ6jVei7uV8",
"outputId": "7f2f5590-df82-45a1-b909-7fcbc39aff87",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"data.info()"
],
"execution_count": 111,
"outputs": [
{
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 100000 entries, 0 to 99999\n",
"Data columns (total 3 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 content 97923 non-null object\n",
" 1 time 100000 non-null object\n",
" 2 author 100000 non-null object\n",
"dtypes: object(3)\n",
"memory usage: 2.3+ MB\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rAVojxOE1XVK"
},
"source": [
"def anom_dict (names):\n",
" anonymized = {}\n",
" for index, name in enumerate(names.unique()):\n",
" anonymized[name] = f\"A{index + 1}\"\n",
" return anonymized"
],
"execution_count": 112,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "D3mhcAH42oyh"
},
"source": [
"data.author = data.author.map(anom_dict(data.author))"
],
"execution_count": 113,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "0HR9y4huHU9z"
},
"source": [
"## Basic Cleaning\n",
"The dataset came with missing values resulting from the bot's failed read of a few messages and also has the time column as a string, which we want as a datetime object."
]
},
{
"cell_type": "code",
"metadata": {
"id": "DmXfTEMQ4kaO",
"outputId": "ddf3f062-56e0-4a8d-b7d0-322e7affffe0",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"data.info()"
],
"execution_count": 114,
"outputs": [
{
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 100000 entries, 0 to 99999\n",
"Data columns (total 3 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 content 97923 non-null object\n",
" 1 time 100000 non-null object\n",
" 2 author 100000 non-null object\n",
"dtypes: object(3)\n",
"memory usage: 2.3+ MB\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sJ8SYNJ04MB7"
},
"source": [
"Our time is stored as an object, which drastically limits what we can do with it. To fix that, we'll be turning it into a datetime object."
]
},
{
"cell_type": "code",
"metadata": {
"id": "ZSfkxWBoGkb0"
},
"source": [
"data.time = pd.to_datetime(data.time)"
],
"execution_count": 115,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "PuRjQjJR6RmN",
"outputId": "492debaf-9dd6-4819-cd35-c061a82dd157",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"data.isnull().sum()"
],
"execution_count": 116,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"content 2077\n",
"time 0\n",
"author 0\n",
"dtype: int64"
]
},
"metadata": {
"tags": []
},
"execution_count": 116
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AiKxXdm-AOy-"
},
"source": [
"The missing values consist of messages the bot was unable to read, such as embeds and images. They're all unecessary for our purposes so we can just get rid of them."
]
},
{
"cell_type": "code",
"metadata": {
"id": "oHYWkc_m6_yK",
"outputId": "c20b1197-a700-4831-b18e-fe37d2175813",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"data.dropna(inplace=True)\n",
"data.info()"
],
"execution_count": 117,
"outputs": [
{
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"Int64Index: 97923 entries, 0 to 99999\n",
"Data columns (total 3 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 content 97923 non-null object \n",
" 1 time 97923 non-null datetime64[ns]\n",
" 2 author 97923 non-null object \n",
"dtypes: datetime64[ns](1), object(2)\n",
"memory usage: 3.0+ MB\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4CY3GXFjHmcz"
},
"source": [
"## Preparing The Data For Analysis\n",
"We have dealt with missing values and incorrect value types but we might still want to do a few changes to the dataframe to make getting info out of it easier."
]
},
{
"cell_type": "code",
"metadata": {
"id": "9e2qdQGrHsal",
"outputId": "6004ee67-50b4-4b60-e929-55d62c9683f8",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
}
},
"source": [
"data.head()"
],
"execution_count": 118,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>content</th>\n",
" <th>time</th>\n",
" <th>author</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>D:&lt; i tried something different this year</td>\n",
" <td>2020-10-31 23:44:09.770</td>\n",
" <td>A1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>I like how u had to think about what would go ...</td>\n",
" <td>2020-10-31 23:29:08.644</td>\n",
" <td>A2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Thx</td>\n",
" <td>2020-10-31 23:20:24.956</td>\n",
" <td>A3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Nice costumes</td>\n",
" <td>2020-10-31 23:08:03.268</td>\n",
" <td>A1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ooo</td>\n",
" <td>2020-10-31 23:07:49.062</td>\n",
" <td>A1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" content ... author\n",
"0 D:< i tried something different this year ... A1\n",
"1 I like how u had to think about what would go ... ... A2\n",
"2 Thx ... A3\n",
"3 Nice costumes ... A1\n",
"4 Ooo ... A1\n",
"\n",
"[5 rows x 3 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 118
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Dp0wScFAy-JL"
},
"source": [
"### Emojis\n",
"We'll be also keeping track of the emojis sent in each message via an array."
]
},
{
"cell_type": "code",
"metadata": {
"id": "hMHwnDLnu07h"
},
"source": [
"def emoji_list (msg):\n",
" emote = np.array([])\n",
" pattern = \"[\"u\"\\U0001F600-\\U0001F64F\"u\"\\U0001F300-\\U0001F5FF\"u\"\\U0001F680-\\U0001F6FF\"u\"\\U0001F1E0-\\U0001F1FF\"\"]+\" \n",
" if re.search(pattern, msg):\n",
" for term in msg.split():\n",
" if term in emoji.UNICODE_EMOJI:\n",
" emote = np.append(emote, term)\n",
" return emote"
],
"execution_count": 119,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "35b3B38erIeN"
},
"source": [
"emoji_list = data.content.apply(emoji_list)\n",
"data.insert(3, \"emoji\", emoji_list)"
],
"execution_count": 120,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "D_eV_1Di-Kjm"
},
"source": [
"### Discord Emotes\n",
"Aside from standard emojis, discord also has it's own exclusive guild emotes, which we might want to keep track of."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OI-tNkj38dCC"
},
"source": [
"Filtering out and creating a separate column for discord emojis."
]
},
{
"cell_type": "code",
"metadata": {
"id": "RbMpmjOeSGWK"
},
"source": [
"def emote_list (msg):\n",
" emotes = np.array([])\n",
" pattern = \"<:(.*):[0-9]{18}>\" \n",
" if re.search(pattern, msg):\n",
" for term in msg.split():\n",
" if term[:2] == \"<:\": # We only want to store the name of the emote\n",
" emotes = np.append(emotes, term.split(\":\")[1]) \n",
" return emotes"
],
"execution_count": 121,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "omz2j4gW2zf7"
},
"source": [
"discord_emotes = data.content.apply(emote_list)\n",
"data.insert(4, \"discord_emotes\", discord_emotes)"
],
"execution_count": 122,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fOC8VhcwJdLV"
},
"source": [
"Filtering out and creating a separate column for animated discord emojis."
]
},
{
"cell_type": "code",
"metadata": {
"id": "jXHrHk7L4lSz"
},
"source": [
"def animated_emote_list (msg):\n",
" emotes = np.array([])\n",
" pattern = \"<a:(.*):[0-9]{18}>\" \n",
" if re.search(pattern, msg):\n",
" for term in msg.split():\n",
" if term[:3] == \"<a:\": # We only want to store the name of the emote\n",
" emotes = np.append(emotes, term.split(\":\")[1])\n",
" return emotes"
],
"execution_count": 123,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "hvnMLPaf2Fhj"
},
"source": [
"animated_discord_emotes = data.content.apply(animated_emote_list)\n",
"data.insert(5, \"a_discord_emotes\", animated_discord_emotes)"
],
"execution_count": 124,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "_QNCp3vWzDSs"
},
"source": [
"### Adding a Word Count Column\n",
"We can see the average length of a user's message by adding a column that keeps track of their word count per message."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V3PLMmAXznOP"
},
"source": [
"Adding a new column that contains the word count for each message."
]
},
{
"cell_type": "code",
"metadata": {
"id": "GGuTg3orwghb"
},
"source": [
"def word_count (msg):\n",
" return len(msg.split())"
],
"execution_count": 125,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GBgqlSKpw2en"
},
"source": [
"words = data.content.apply(word_count)\n",
"data.insert(6, \"word_count\", words)"
],
"execution_count": 126,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ioSfzNFz5lLB"
},
"source": [
"# Playing With The Data\n",
"After being done with that entire process, we can finally do some analysis work."
]
},
{
"cell_type": "code",
"metadata": {
"id": "I6CCsDkn5os6",
"outputId": "6137c0f0-00fa-4877-f53a-d88ce652c3ac",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
}
},
"source": [
"data.head()"
],
"execution_count": 127,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>content</th>\n",
" <th>time</th>\n",
" <th>author</th>\n",
" <th>emoji</th>\n",
" <th>discord_emotes</th>\n",
" <th>a_discord_emotes</th>\n",
" <th>word_count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>D:&lt; i tried something different this year</td>\n",
" <td>2020-10-31 23:44:09.770</td>\n",
" <td>A1</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>7</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>I like how u had to think about what would go ...</td>\n",
" <td>2020-10-31 23:29:08.644</td>\n",
" <td>A2</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>22</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Thx</td>\n",
" <td>2020-10-31 23:20:24.956</td>\n",
" <td>A3</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Nice costumes</td>\n",
" <td>2020-10-31 23:08:03.268</td>\n",
" <td>A1</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Ooo</td>\n",
" <td>2020-10-31 23:07:49.062</td>\n",
" <td>A1</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>[]</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" content ... word_count\n",
"0 D:< i tried something different this year ... 7\n",
"1 I like how u had to think about what would go ... ... 22\n",
"2 Thx ... 1\n",
"3 Nice costumes ... 2\n",
"4 Ooo ... 1\n",
"\n",
"[5 rows x 7 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 127
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "WVYB2jwr5q7j",
"outputId": "7dd50c9a-9d6a-4e14-a2f9-4da517dc06b8",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"source": [
"data.info()"
],
"execution_count": 129,
"outputs": [
{
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"Int64Index: 97923 entries, 0 to 99999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 content 97923 non-null object \n",
" 1 time 97923 non-null datetime64[ns]\n",
" 2 author 97923 non-null object \n",
" 3 emoji 97923 non-null object \n",
" 4 discord_emotes 97923 non-null object \n",
" 5 a_discord_emotes 97923 non-null object \n",
" 6 word_count 97923 non-null int64 \n",
"dtypes: datetime64[ns](1), int64(1), object(5)\n",
"memory usage: 6.0+ MB\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ES6n_ub-kax"
},
"source": [
"## User Activity Over Time\n",
"Plotting the messages sent by the server members over the entire period covered by the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dq60h2LyFMmC"
},
"source": [
"When we try to plot the user activity in the server, the plot gets visually crowded from the amount of authors there is. Most of which actually contribute very little to the data."
]
},
{
"cell_type": "code",
"metadata": {
"id": "5Rsne4Ni58E2",
"outputId": "084b58f2-dcc9-42e0-ba59-f90b40648238",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 542
}
},
"source": [
"fig = px.histogram(data, x='time', color='author', opacity=0.5)\n",
"fig.show()"
],
"execution_count": 130,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"<html>\n",
"<head><meta charset=\"utf-8\" /></head>\n",
"<body>\n",
" <div> <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script> <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
@caio-vinicius

caio-vinicius commented May 19, 2021

Copy link
Copy Markdown

Isso é bem útil. Muito obrigado :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment