MMT_Project_with_watson.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MMT_Project with watson",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "1vIxnwi6VbqHWLHLDBpnK4SbevgaK2L9d",
      "authorship_tag": "ABX9TyMmCdABZy0EiHxuFCIg1Erq",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/raghavmittal101/music_mind_tech_project/blob/master/MMT_Project_with_watson.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dvwkm6KqJqk4",
        "colab_type": "text"
      },
      "source": [
        "# MMT Project\n",
        "My aim is to find associations between a person's score in Big-5 personality assessment and the emotions depicted by lyrics of their favourite songs.\n",
        "\n",
        "For this purpose, I have collected data from more than 50 participants. This data consists of\n",
        "* demographic data\n",
        "* big-5 personality assessment data\n",
        "* name and artist of 3 to 10 favourite songs\n",
        "\n",
        "## Process\n",
        "1. Gather lyrics of songs\n",
        "2. Analyze the emotions depected in lyrics\n",
        "3. Analyze associations between the Big-5 personality scores and emotions depected in lyrics\n",
        "\n",
        "\n",
        "1. I will automate the lyrics gathering process by using `lyrics-extractor` package, which searches the song on Google and scrapes the lyrics shown on the results page. I also tried other packages for searching lyrics, but the problem with them is that they are not tolerant to spelling mistakes and incomplete words. The data was collected by circulating a Google form, there are high chances of misspelled and incomplete song names.  \n",
        "After extracting the lyrics, we need to remove any labelling brackets like *\\[chorus\\]* or *\\[bridge\\]* as they may contain meaningful words which may affect the analysis of emotions. For this purpose I will use regular expressions.\n",
        "\n",
        "2. To analyze the emotions depected in lyrics, I will use Tone Analyzer service by IBM Watson. This service takes in the lyrics and return the scorings of lyrics in 13 different components which consist of Big-5 components O,C,E,A,N under social category and anger, disgust, fear, joy, and sadness under emotional category and analytical, confident, and tentative under language category.\n",
        "\n",
        "3. Once we have all the scores for each song, we need to calculate cummulative scores for each participant. To calculate cummulative scores, we will do average of each emotion type accross all the favourite songs. This will give us a score vector against each participant.  \n",
        "Where ever the score is less than 0.5, we will make it zero. It's done because according to the IBM watson documents, it's difficult to percieve the emotions which score less than 0.5.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3OIpGBRXg-dG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "thpLMWE1kSWJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "Load dataset which contains `id`, `songnames`, `songartist`. Always load latest \n",
        "version of dataset which may contain lyrics and scores fetched previously. This \n",
        "will save time and any quota imposed on third-party services which we will use.\n",
        "\"\"\"\n",
        "songs = pd.read_csv(\"/content/drive/My Drive/acads/MMT/MMT project/collected data/songs_dataset.csv\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX5sEQOo3AWY",
        "colab_type": "text"
      },
      "source": [
        "## Fetch and process the lyrics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ML1dcQ4GfPsn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "'''\n",
        "Load external package to fetch lyrics for each song in dataset.\n",
        "\n",
        "Extracting lyrics of songs by using `lyrics_extractor` API\n",
        "(https://pypi.org/project/lyrics-extractor/#description)\n",
        "\n",
        "We will use `lyrics_extractor` API for this purpose. It is tolerant to spelling \n",
        "mistakes and incomplete words because it does a google search for the given data,\n",
        "google autocorrects and autocompletes misspelled or incomplete words. Which is\n",
        "very useful for us because it takes away the tiring task of manually checking and \n",
        "correcting spelling of each entry in dataset. \n",
        "'''\n",
        "!pip install lyrics-extractor\n",
        "\n",
        "# Example\n",
        "# song_title, song_lyrics = extract_lyrics.get_lyrics(\"Its you\tAli Gatie\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fM8gvoBrPDom",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "'''\n",
        "Extract and store lyrics of songs.\n",
        "\n",
        "Go through each row in songs dataset and add lyric to each row. \n",
        "To save time and Google search requests quota, skip rows which already have lyrics.\n",
        "'''\n",
        "from lyrics_extractor import Song_Lyrics\n",
        "extract_lyrics = Song_Lyrics(\"\",\"\") # insert the Google api keys here\n",
        "\n",
        "for i in songs.index:\n",
        "  if len(str(songs['lyrics'][i])) < 15: # is true if the corresponding lyrics coloumn is empty in dataset\n",
        "    try:  # fetch lyrics of song from the Internet\n",
        "      title, lyrics = extract_lyrics.get_lyrics(str(songs['songnames'][i])+\" \"+str(songs['songartists'][i]))\n",
        "      if lyrics == '':  # if `extract_lyrics` is unable to fetch/find lyrics of song, print song details\n",
        "        print(i, str(songs['songnames'][i])+\" \"+str(songs['songartists'][i]), \" lyrics not found!!!\")\n",
        "      songs['lyrics'][i] = lyrics\n",
        "    except: # print song details with keyword `ERROR!!` if try fails\n",
        "      print(i, songs['email'][i], songs['songnames'][i], songs['songartists'][i], \"ERROR!!\")\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bH1_IsI-hMOT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "Just to play safe, save the current dataframe to CSV. This way we are free to\n",
        "play with current dataframe without any worry of destroying the df by mistake. \n",
        "\"\"\"\n",
        "\n",
        "songs.to_csv('songs_dataset.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "1yULgY8yoE9W",
        "colab": {}
      },
      "source": [
        "'''\n",
        "Now we have lyrics in place, let's do some necessary preprocessing.\n",
        "\n",
        "We noticed some labels here and there in the fetched lyrics.\n",
        "Let's remove labelling brackets with labels from lyrics because labels may \n",
        "contain words which will unncessarily influence the sentiment of the lyrics.\n",
        "\n",
        "examples of labels: [chorus], [bridge]\n",
        "'''\n",
        "import re\n",
        "regex1 = '\\[[a-zA-Z 0-9]*\\]'  # select words like [chorus], [bridge]\n",
        "\n",
        "for i in songs.index:\n",
        "  try:\n",
        "    lyrics = str(songs['lyrics'][i])\n",
        "    lyrics = re.sub(regex1, ' ', lyrics)\n",
        "    songs['lyrics'][i] = lyrics\n",
        "  except:\n",
        "    print(\"something went wrong!!!\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TywBRokZYRyj",
        "colab_type": "code",
        "outputId": "843d7159-0044-4b71-9dd5-d769c84f0123",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        }
      },
      "source": [
        "songs.head()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>id</th>\n",
              "      <th>songnames</th>\n",
              "      <th>songartists</th>\n",
              "      <th>lyrics</th>\n",
              "      <th>scoringDone</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>Habit</td>\n",
              "      <td>Still Woozy</td>\n",
              "      <td>I could let you have it\\nYou could be my habit...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.484500</td>\n",
              "      <td>0.249156</td>\n",
              "      <td>0.013715</td>\n",
              "      <td>0.414249</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.972431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>Someday</td>\n",
              "      <td>Flypsyde</td>\n",
              "      <td>Come on\\nShalalala, shalalala\\nSomeday we gonn...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.633487</td>\n",
              "      <td>0.190181</td>\n",
              "      <td>0.032559</td>\n",
              "      <td>0.368816</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.168827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>Venus</td>\n",
              "      <td>Sleeping at Last</td>\n",
              "      <td>The night sky once ruled my imagination.\\nNow ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.109705</td>\n",
              "      <td>0.647529</td>\n",
              "      <td>0.506882</td>\n",
              "      <td>0.228974</td>\n",
              "      <td>0.555642</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.196453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>Will Do</td>\n",
              "      <td>TV on the Radio</td>\n",
              "      <td>It might be impractical to seek out a new roma...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.226913</td>\n",
              "      <td>0.045250</td>\n",
              "      <td>0.516772</td>\n",
              "      <td>0.217690</td>\n",
              "      <td>0.569231</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.927953</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>Every other Freckle</td>\n",
              "      <td>Alt-J</td>\n",
              "      <td>Aah\\nI want to share your mouthful\\nI want to ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.128174</td>\n",
              "      <td>0.268708</td>\n",
              "      <td>0.409157</td>\n",
              "      <td>0.121439</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.812988</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0  id            songnames  ... analytical confident  tentative\n",
              "0           0   2                Habit  ...   0.000000  0.000000   0.972431\n",
              "1           1   2              Someday  ...   0.000000  0.000000   0.168827\n",
              "2           2   2                Venus  ...   0.555642  0.000000   0.196453\n",
              "3           3   2              Will Do  ...   0.569231  0.000000   0.927953\n",
              "4           4   2  Every other Freckle  ...   0.000000  0.812988   0.000000\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CzAWkdLv0PsK",
        "colab_type": "text"
      },
      "source": [
        "## Analyze tone of lyrics\n",
        "We will use **IBM Watson Tone Analyzer** tool for this purpose."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lwI_6PQ10Z9B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "IBM Watson Tone Analyzer API is available as a package in pip.\n",
        "In order to use it, you need to signup to Watson, create a service in \n",
        "ToneAnalyzer and get the credentials.\n",
        "\"\"\"\n",
        "\n",
        "!pip install --upgrade \"ibm-watson>=4.3.0\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f6oDREbH3Rdh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from ibm_watson import ToneAnalyzerV3\n",
        "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator\n",
        "import json\n",
        "authenticator = IAMAuthenticator('') # insert Watson API key here\n",
        "'''\n",
        "The service with version '2017-09-21' can return results for the following tone IDs:\n",
        "anger, fear, joy, and sadness (emotional tones); analytical,\n",
        "confident, and tentative (language tones). The service returns\n",
        "results only for tones whose scores meet a minimum threshold of 0.5.\n",
        "\n",
        "The service with version '2016-05-19' can return results for the following tone IDs of the different categories: \n",
        "for the emotion category: anger, disgust, fear, joy, and sadness; \n",
        "for the language category: analytical, confident, and tentative; \n",
        "for the social category: openness_big5, conscientiousness_big5,\n",
        "extraversion_big5, agreeableness_big5, and emotional_range_big5. \n",
        "The service returns scores for all tones of a category, regardless of their values.\n",
        "'''\n",
        "version='2017-09-21'\n",
        "tone_analyzer = ToneAnalyzerV3(\n",
        "    version=version,\n",
        "    authenticator=authenticator\n",
        ")\n",
        "\n",
        "tone_analyzer.set_service_url('https://api.eu-gb.tone-analyzer.watson.cloud.ibm.com')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VSimKBwhG1DX",
        "colab": {}
      },
      "source": [
        "'''\n",
        "Function to get sentiment scores of lyrics. Based on version of the tone analyzer, \n",
        "it will return the factors.\n",
        "parameter: `lyrics`\n",
        "return: JSON object returned by watson API.\n",
        "'''\n",
        "def analyzeLyrics(lyrics):\n",
        "  try:\n",
        "    tone_analysis = tone_analyzer.tone(\n",
        "        {'text': lyrics},\n",
        "        content_type='application/json',\n",
        "        sentences=False # sentencewise analysis not required\n",
        "    ).get_result()\n",
        "  except:\n",
        "    print(i, songs['id'][i], songs['songnames'][i], 'unable to fetch scores!!!')\n",
        "  return json.dumps(tone_analysis, indent=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y8NZi1ddQSIB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "outputId": "58bca545-5a49-42da-e9c4-1f1cf321842c"
      },
      "source": [
        "# example input\n",
        "# Let's check if our function is working fine or not.\n",
        "i = 436\n",
        "results = analyzeLyrics(songs['lyrics'][i])\n",
        "print(songs['id'][i], songs[\"songnames\"][i])\n",
        "r_dict = json.loads(results)\n",
        "\n",
        "\"\"\" use this snippet when you with version '2017-09-21' of Tone Analyzer \"\"\"\n",
        "for tones in r_dict['document_tone']['tones']:\n",
        "  print(tones['tone_id'], ':', tones['score'])\n",
        "\n",
        "# \"\"\" use this snippet with version '2016-05-19' of Tone Analyzer \"\"\"\n",
        "# for category in r_dict['document_tone']['tone_categories']:\n",
        "#   for scores in category['tones']:\n",
        "#     print(scores['tone_id'], ':', scores['score'])"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "74 Rossetta Stoned\n",
            "fear : 0.611615\n",
            "joy : 0.62383\n",
            "sadness : 0.695139\n",
            "anger : 0.714518\n",
            "tentative : 0.661148\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "afc5gbjB8iXa",
        "colab_type": "code",
        "outputId": "8f7ca68d-153a-46ad-aa6e-1ccfc2214da5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        }
      },
      "source": [
        "# add new columns to DF 'songs'\n",
        "songs_with_scores = pd.concat(\\\n",
        "                              [songs, \\\n",
        "                               pd.DataFrame(\\\n",
        "                                columns = [ 'scoringDone','anger','fear',\\\n",
        "                                           'joy', 'sadness', 'analytical', 'confident',\\\n",
        "                                           'tentative'])])\n",
        "# songs_with_scores = songs\n",
        "songs_with_scores.head()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>id</th>\n",
              "      <th>songnames</th>\n",
              "      <th>songartists</th>\n",
              "      <th>lyrics</th>\n",
              "      <th>scoringDone</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Habit</td>\n",
              "      <td>Still Woozy</td>\n",
              "      <td>I could let you have it\\nYou could be my habit...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.484500</td>\n",
              "      <td>0.249156</td>\n",
              "      <td>0.013715</td>\n",
              "      <td>0.414249</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.972431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Someday</td>\n",
              "      <td>Flypsyde</td>\n",
              "      <td>Come on\\nShalalala, shalalala\\nSomeday we gonn...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.633487</td>\n",
              "      <td>0.190181</td>\n",
              "      <td>0.032559</td>\n",
              "      <td>0.368816</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.168827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Venus</td>\n",
              "      <td>Sleeping at Last</td>\n",
              "      <td>The night sky once ruled my imagination.\\nNow ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.109705</td>\n",
              "      <td>0.647529</td>\n",
              "      <td>0.506882</td>\n",
              "      <td>0.228974</td>\n",
              "      <td>0.555642</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.196453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Will Do</td>\n",
              "      <td>TV on the Radio</td>\n",
              "      <td>It might be impractical to seek out a new roma...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.226913</td>\n",
              "      <td>0.045250</td>\n",
              "      <td>0.516772</td>\n",
              "      <td>0.217690</td>\n",
              "      <td>0.569231</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.927953</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Every other Freckle</td>\n",
              "      <td>Alt-J</td>\n",
              "      <td>Aah\\nI want to share your mouthful\\nI want to ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.128174</td>\n",
              "      <td>0.268708</td>\n",
              "      <td>0.409157</td>\n",
              "      <td>0.121439</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.812988</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0   id            songnames  ... analytical confident  tentative\n",
              "0         0.0  2.0                Habit  ...   0.000000  0.000000   0.972431\n",
              "1         1.0  2.0              Someday  ...   0.000000  0.000000   0.168827\n",
              "2         2.0  2.0                Venus  ...   0.555642  0.000000   0.196453\n",
              "3         3.0  2.0              Will Do  ...   0.569231  0.000000   0.927953\n",
              "4         4.0  2.0  Every other Freckle  ...   0.000000  0.812988   0.000000\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AV6O4BL-tf3-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# iterate through each row in DF songs and calculate the scores for lyrics.\n",
        "# add the scores back to DF in corresponding song row\n",
        "\n",
        "for i in songs_with_scores.index:\n",
        "  results_dict=[]\n",
        "  try:\n",
        "    lyrics = songs_with_scores['lyrics'][i]\n",
        "    if(len(lyrics)>15 and pd.isna(songs_with_scores['scoringDone'][i])):\n",
        "      results = analyzeLyrics(lyrics)\n",
        "      results_dict = json.loads(results)\n",
        "      for category in results_dict['document_tone']['tone_categories']:\n",
        "        for scores in category['tones']:\n",
        "          songs_with_scores.at[i, scores['tone_id']] = scores['score']\n",
        "      songs_with_scores.at[i, 'scoringDone'] = 1\n",
        "  except:\n",
        "    print(i, songs['id'][i], songs['songnames'][i], \"unable to put scores in df!!!\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bn2XTCZEEgp7",
        "colab_type": "code",
        "outputId": "1fd93620-45c5-49b0-a92b-a7398ca384e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        }
      },
      "source": [
        "songs_with_scores.head()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>id</th>\n",
              "      <th>songnames</th>\n",
              "      <th>songartists</th>\n",
              "      <th>lyrics</th>\n",
              "      <th>scoringDone</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Habit</td>\n",
              "      <td>Still Woozy</td>\n",
              "      <td>I could let you have it\\nYou could be my habit...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.484500</td>\n",
              "      <td>0.249156</td>\n",
              "      <td>0.013715</td>\n",
              "      <td>0.414249</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.972431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Someday</td>\n",
              "      <td>Flypsyde</td>\n",
              "      <td>Come on\\nShalalala, shalalala\\nSomeday we gonn...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.633487</td>\n",
              "      <td>0.190181</td>\n",
              "      <td>0.032559</td>\n",
              "      <td>0.368816</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.168827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Venus</td>\n",
              "      <td>Sleeping at Last</td>\n",
              "      <td>The night sky once ruled my imagination.\\nNow ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.109705</td>\n",
              "      <td>0.647529</td>\n",
              "      <td>0.506882</td>\n",
              "      <td>0.228974</td>\n",
              "      <td>0.555642</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.196453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Will Do</td>\n",
              "      <td>TV on the Radio</td>\n",
              "      <td>It might be impractical to seek out a new roma...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.226913</td>\n",
              "      <td>0.045250</td>\n",
              "      <td>0.516772</td>\n",
              "      <td>0.217690</td>\n",
              "      <td>0.569231</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.927953</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>Every other Freckle</td>\n",
              "      <td>Alt-J</td>\n",
              "      <td>Aah\\nI want to share your mouthful\\nI want to ...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.128174</td>\n",
              "      <td>0.268708</td>\n",
              "      <td>0.409157</td>\n",
              "      <td>0.121439</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.812988</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0   id            songnames  ... analytical confident  tentative\n",
              "0         0.0  2.0                Habit  ...   0.000000  0.000000   0.972431\n",
              "1         1.0  2.0              Someday  ...   0.000000  0.000000   0.168827\n",
              "2         2.0  2.0                Venus  ...   0.555642  0.000000   0.196453\n",
              "3         3.0  2.0              Will Do  ...   0.569231  0.000000   0.927953\n",
              "4         4.0  2.0  Every other Freckle  ...   0.000000  0.812988   0.000000\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQdQNnFs1-_s",
        "colab_type": "text"
      },
      "source": [
        "## Prepare the data for correlation analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zz3JJGRDD7ss",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Drop columns which are not required in Data analysis\n",
        "id_lyrics_scores = songs_with_scores.drop(columns=[\"Unnamed: 0\",\"scoringDone\",\"songnames\", \\\n",
        "                                                    \"songartists\", \"lyrics\"],\\\n",
        "                                           errors='ignore')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PRAo-1G204lj",
        "colab_type": "code",
        "outputId": "b1e77530-7ca4-42d6-efc9-7009fd134639",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        }
      },
      "source": [
        "# # group lyrics scores into positive emotions and negative emotions by adding columns\n",
        "\n",
        "# using apply function to create a new column \n",
        "# id_lyrics_scores['negativity'] = id_lyrics_scores.apply(lambda row: (row.anger+row.disgust+row.fear+row.sadness)/4, axis = 1)\n",
        "id_lyrics_scores.head()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.484500</td>\n",
              "      <td>0.249156</td>\n",
              "      <td>0.013715</td>\n",
              "      <td>0.414249</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.972431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.633487</td>\n",
              "      <td>0.190181</td>\n",
              "      <td>0.032559</td>\n",
              "      <td>0.368816</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.168827</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.109705</td>\n",
              "      <td>0.647529</td>\n",
              "      <td>0.506882</td>\n",
              "      <td>0.228974</td>\n",
              "      <td>0.555642</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.196453</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.226913</td>\n",
              "      <td>0.045250</td>\n",
              "      <td>0.516772</td>\n",
              "      <td>0.217690</td>\n",
              "      <td>0.569231</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.927953</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.128174</td>\n",
              "      <td>0.268708</td>\n",
              "      <td>0.409157</td>\n",
              "      <td>0.121439</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.812988</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    id     anger      fear  ...  analytical  confident  tentative\n",
              "0  2.0  0.484500  0.249156  ...    0.000000   0.000000   0.972431\n",
              "1  2.0  0.633487  0.190181  ...    0.000000   0.000000   0.168827\n",
              "2  2.0  0.109705  0.647529  ...    0.555642   0.000000   0.196453\n",
              "3  2.0  0.226913  0.045250  ...    0.569231   0.000000   0.927953\n",
              "4  2.0  0.128174  0.268708  ...    0.000000   0.812988   0.000000\n",
              "\n",
              "[5 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-cMHPyI4jMZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# watson documentation states that any score below 0.5 is unpercievable by humans.\n",
        "# such scores should be floored to 0.\n",
        "for i in id_lyrics_scores.columns:\n",
        "  if(i not in [\"id\"]):\n",
        "    id_lyrics_scores.loc[id_lyrics_scores[i] < 0.5, i] = 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A5xVTWojFP8q",
        "colab_type": "code",
        "outputId": "e6796ab4-2fc3-44b1-9796-cb126b5a123b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        }
      },
      "source": [
        "id_lyrics_scores.head()"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.972431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.633487</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.647529</td>\n",
              "      <td>0.506882</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.555642</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.516772</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.569231</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.927953</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.812988</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    id     anger      fear       joy  sadness  analytical  confident  tentative\n",
              "0  2.0  0.000000  0.000000  0.000000      0.0    0.000000   0.000000   0.972431\n",
              "1  2.0  0.633487  0.000000  0.000000      0.0    0.000000   0.000000   0.000000\n",
              "2  2.0  0.000000  0.647529  0.506882      0.0    0.555642   0.000000   0.000000\n",
              "3  2.0  0.000000  0.000000  0.516772      0.0    0.569231   0.000000   0.927953\n",
              "4  2.0  0.000000  0.000000  0.000000      0.0    0.000000   0.812988   0.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nnOMJ_QCCT5X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 227
        },
        "outputId": "d9ec3714-5a6e-4cc0-aa9e-13c4b7144556"
      },
      "source": [
        "\"\"\"\n",
        "Here we find mean lyrics scores for each participant.\n",
        "\"\"\"\n",
        "\n",
        "# Find mean and sum of lyrics scores over each participant\n",
        "lyrics_scores_mean = id_lyrics_scores.groupby(['id']).mean()\n",
        "lyrics_scores_mean.head()"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>id</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2.0</th>\n",
              "      <td>0.079186</td>\n",
              "      <td>0.152190</td>\n",
              "      <td>0.211465</td>\n",
              "      <td>0.221433</td>\n",
              "      <td>0.216763</td>\n",
              "      <td>0.101624</td>\n",
              "      <td>0.521714</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3.0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.072347</td>\n",
              "      <td>0.226194</td>\n",
              "      <td>0.286345</td>\n",
              "      <td>0.188826</td>\n",
              "      <td>0.059999</td>\n",
              "      <td>0.286870</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4.0</th>\n",
              "      <td>0.079258</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.422219</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.265145</td>\n",
              "      <td>0.210948</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6.0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.057586</td>\n",
              "      <td>0.058399</td>\n",
              "      <td>0.269365</td>\n",
              "      <td>0.145375</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.276035</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7.0</th>\n",
              "      <td>0.054502</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.197023</td>\n",
              "      <td>0.061493</td>\n",
              "      <td>0.226869</td>\n",
              "      <td>0.306120</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        anger      fear       joy   sadness  analytical  confident  tentative\n",
              "id                                                                           \n",
              "2.0  0.079186  0.152190  0.211465  0.221433    0.216763   0.101624   0.521714\n",
              "3.0  0.000000  0.072347  0.226194  0.286345    0.188826   0.059999   0.286870\n",
              "4.0  0.079258  0.000000  0.422219  0.000000    0.000000   0.265145   0.210948\n",
              "6.0  0.000000  0.057586  0.058399  0.269365    0.145375   0.000000   0.276035\n",
              "7.0  0.054502  0.000000  0.197023  0.061493    0.226869   0.306120   0.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BS4EJohrGQlB",
        "colab_type": "code",
        "outputId": "c73e55d9-dfaa-4d07-b9a5-ea8c44bcfaf1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 334
        }
      },
      "source": [
        "# let's get big-5 scores and demographic data here. Later we use these scores and lyrics scores to analyze associations between them\n",
        "big5_scores = pd.read_csv(\"/content/drive/My Drive/acads/MMT/MMT project/collected data/big5_continuous_scores.csv\")\n",
        "demographic_data = pd.read_csv(\"/content/drive/My Drive/acads/MMT/MMT project/collected data/demographics.csv\")\n",
        "\n",
        "# and merge `lyrics_scores_mean` and above DFs by 'id'\n",
        "df1 = pd.merge(left=demographic_data, right=big5_scores, left_on='id', right_on='id')\n",
        "big5_continuous_with_lyrics_mean = pd.merge(left=df1, right=lyrics_scores_mean, left_on='id', right_on='id')\n",
        "\n",
        "big5_continuous_with_lyrics_mean.head()"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>age group</th>\n",
              "      <th>sex</th>\n",
              "      <th>instumental or lyrical music</th>\n",
              "      <th>listen to english lyrics</th>\n",
              "      <th>extraversion</th>\n",
              "      <th>agreeableness</th>\n",
              "      <th>conscientiousness</th>\n",
              "      <th>neuroticism</th>\n",
              "      <th>openness</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>24-26 years</td>\n",
              "      <td>Female</td>\n",
              "      <td>Music with lyrics</td>\n",
              "      <td>Always</td>\n",
              "      <td>21</td>\n",
              "      <td>34</td>\n",
              "      <td>26</td>\n",
              "      <td>26</td>\n",
              "      <td>47</td>\n",
              "      <td>0.079186</td>\n",
              "      <td>0.152190</td>\n",
              "      <td>0.211465</td>\n",
              "      <td>0.221433</td>\n",
              "      <td>0.216763</td>\n",
              "      <td>0.101624</td>\n",
              "      <td>0.521714</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>24-26 years</td>\n",
              "      <td>Male</td>\n",
              "      <td>Music with lyrics</td>\n",
              "      <td>Always</td>\n",
              "      <td>24</td>\n",
              "      <td>41</td>\n",
              "      <td>35</td>\n",
              "      <td>23</td>\n",
              "      <td>43</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.072347</td>\n",
              "      <td>0.226194</td>\n",
              "      <td>0.286345</td>\n",
              "      <td>0.188826</td>\n",
              "      <td>0.059999</td>\n",
              "      <td>0.286870</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4</td>\n",
              "      <td>24-26 years</td>\n",
              "      <td>Female</td>\n",
              "      <td>Music with lyrics</td>\n",
              "      <td>Sometimes</td>\n",
              "      <td>27</td>\n",
              "      <td>34</td>\n",
              "      <td>24</td>\n",
              "      <td>28</td>\n",
              "      <td>33</td>\n",
              "      <td>0.079258</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.422219</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.265145</td>\n",
              "      <td>0.210948</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>24-26 years</td>\n",
              "      <td>Male</td>\n",
              "      <td>Music with lyrics</td>\n",
              "      <td>Often</td>\n",
              "      <td>30</td>\n",
              "      <td>41</td>\n",
              "      <td>38</td>\n",
              "      <td>18</td>\n",
              "      <td>36</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.057586</td>\n",
              "      <td>0.058399</td>\n",
              "      <td>0.269365</td>\n",
              "      <td>0.145375</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.276035</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>27-30 years</td>\n",
              "      <td>Male</td>\n",
              "      <td>Music with lyrics</td>\n",
              "      <td>Often</td>\n",
              "      <td>27</td>\n",
              "      <td>33</td>\n",
              "      <td>33</td>\n",
              "      <td>16</td>\n",
              "      <td>42</td>\n",
              "      <td>0.054502</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.197023</td>\n",
              "      <td>0.061493</td>\n",
              "      <td>0.226869</td>\n",
              "      <td>0.306120</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id    age group     sex  ... analytical confident  tentative\n",
              "0   2  24-26 years  Female  ...   0.216763  0.101624   0.521714\n",
              "1   3  24-26 years    Male  ...   0.188826  0.059999   0.286870\n",
              "2   4  24-26 years  Female  ...   0.000000  0.265145   0.210948\n",
              "3   6  24-26 years    Male  ...   0.145375  0.000000   0.276035\n",
              "4   7  27-30 years    Male  ...   0.226869  0.306120   0.000000\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U0jwFtSW0YeA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "Let's save the curent DF to save time in future.\n",
        "\"\"\"\n",
        "big5_continuous_with_lyrics_mean.to_csv(\"big5_continuous_with_lyrics_mean.csv\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpxiO4A1FJp9",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6xdeBQ24HCyP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "now accessing the dataset which we saved in previous block :P :D\n",
        "\"\"\"\n",
        "big5_continuous_with_lyrics_mean = pd.read_csv(\"/content/drive/My Drive/acads/MMT/MMT project/collected data/big5_continuous_with_lyrics_mean.csv\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F7BTkmW5GDdb",
        "colab_type": "code",
        "outputId": "5e832dbc-eec9-4649-f94f-903520873ecd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        }
      },
      "source": [
        "\"\"\"\n",
        "Now let's prepare the dataset for Spearman rho analysis. \n",
        "We need to label the categorical with numbers in order to use the algorithm. \n",
        "\"\"\"\n",
        "object_data = big5_continuous_with_lyrics_mean.select_dtypes(include=\"object\")\n",
        "numeric_data = big5_continuous_with_lyrics_mean.select_dtypes(exclude=\"object\")\n",
        "\n",
        "# Import label encoder \n",
        "from sklearn import preprocessing \n",
        "  \n",
        "# label_encoder object knows how to understand word labels. \n",
        "label_encoder = preprocessing.LabelEncoder() \n",
        "object_data = object_data.apply(label_encoder.fit_transform)\n",
        "data  = pd.merge(left=object_data, right=numeric_data, left_on=numeric_data.index, right_on=object_data.index)\n",
        "data.drop(columns=[\"id\"])"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>key_0</th>\n",
              "      <th>age group</th>\n",
              "      <th>sex</th>\n",
              "      <th>instumental or lyrical music</th>\n",
              "      <th>listen to english lyrics</th>\n",
              "      <th>extraversion</th>\n",
              "      <th>agreeableness</th>\n",
              "      <th>conscientiousness</th>\n",
              "      <th>neuroticism</th>\n",
              "      <th>openness</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>21</td>\n",
              "      <td>34</td>\n",
              "      <td>26</td>\n",
              "      <td>26</td>\n",
              "      <td>47</td>\n",
              "      <td>0.079186</td>\n",
              "      <td>0.152190</td>\n",
              "      <td>0.211465</td>\n",
              "      <td>0.221433</td>\n",
              "      <td>0.216763</td>\n",
              "      <td>0.101624</td>\n",
              "      <td>0.521714</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>24</td>\n",
              "      <td>41</td>\n",
              "      <td>35</td>\n",
              "      <td>23</td>\n",
              "      <td>43</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.072347</td>\n",
              "      <td>0.226194</td>\n",
              "      <td>0.286345</td>\n",
              "      <td>0.188826</td>\n",
              "      <td>0.059999</td>\n",
              "      <td>0.286870</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>27</td>\n",
              "      <td>34</td>\n",
              "      <td>24</td>\n",
              "      <td>28</td>\n",
              "      <td>33</td>\n",
              "      <td>0.079258</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.422219</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.265145</td>\n",
              "      <td>0.210948</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>30</td>\n",
              "      <td>41</td>\n",
              "      <td>38</td>\n",
              "      <td>18</td>\n",
              "      <td>36</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.057586</td>\n",
              "      <td>0.058399</td>\n",
              "      <td>0.269365</td>\n",
              "      <td>0.145375</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.276035</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>27</td>\n",
              "      <td>33</td>\n",
              "      <td>33</td>\n",
              "      <td>16</td>\n",
              "      <td>42</td>\n",
              "      <td>0.054502</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.197023</td>\n",
              "      <td>0.061493</td>\n",
              "      <td>0.226869</td>\n",
              "      <td>0.306120</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>57</th>\n",
              "      <td>57</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>25</td>\n",
              "      <td>29</td>\n",
              "      <td>28</td>\n",
              "      <td>26</td>\n",
              "      <td>34</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.201566</td>\n",
              "      <td>0.348010</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.150057</td>\n",
              "      <td>0.256812</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>58</th>\n",
              "      <td>58</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>19</td>\n",
              "      <td>28</td>\n",
              "      <td>24</td>\n",
              "      <td>29</td>\n",
              "      <td>25</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.319708</td>\n",
              "      <td>0.451885</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.147234</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>59</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>34</td>\n",
              "      <td>42</td>\n",
              "      <td>38</td>\n",
              "      <td>14</td>\n",
              "      <td>40</td>\n",
              "      <td>0.142029</td>\n",
              "      <td>0.112704</td>\n",
              "      <td>0.279541</td>\n",
              "      <td>0.315449</td>\n",
              "      <td>0.301320</td>\n",
              "      <td>0.176632</td>\n",
              "      <td>0.341260</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>60</th>\n",
              "      <td>60</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>23</td>\n",
              "      <td>33</td>\n",
              "      <td>29</td>\n",
              "      <td>20</td>\n",
              "      <td>32</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.616278</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.592273</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>61</th>\n",
              "      <td>61</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>25</td>\n",
              "      <td>33</td>\n",
              "      <td>23</td>\n",
              "      <td>26</td>\n",
              "      <td>36</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.168146</td>\n",
              "      <td>0.232925</td>\n",
              "      <td>0.227781</td>\n",
              "      <td>0.250559</td>\n",
              "      <td>0.403886</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>62 rows × 17 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    key_0  age group  sex  ...  analytical  confident  tentative\n",
              "0       0          1    0  ...    0.216763   0.101624   0.521714\n",
              "1       1          1    1  ...    0.188826   0.059999   0.286870\n",
              "2       2          1    0  ...    0.000000   0.265145   0.210948\n",
              "3       3          1    1  ...    0.145375   0.000000   0.276035\n",
              "4       4          2    1  ...    0.226869   0.306120   0.000000\n",
              "..    ...        ...  ...  ...         ...        ...        ...\n",
              "57     57          0    1  ...    0.000000   0.150057   0.256812\n",
              "58     58          0    1  ...    0.000000   0.000000   0.147234\n",
              "59     59          0    1  ...    0.301320   0.176632   0.341260\n",
              "60     60          0    1  ...    0.000000   0.592273   0.000000\n",
              "61     61          0    1  ...    0.227781   0.250559   0.403886\n",
              "\n",
              "[62 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aZ2RKPiY2LhZ",
        "colab_type": "text"
      },
      "source": [
        "## Correlation Analysis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yi8UV9o7Ikin",
        "colab_type": "code",
        "outputId": "dbb93aea-efff-48f2-a242-ecbf02c2772e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        }
      },
      "source": [
        "\"\"\"\n",
        "Now we have our desired df with us, let's do some analysis over it.\n",
        "In our data, some variables are numeric(like big-5 score, emotional tone scores),\n",
        "some are ordinal(like age, musical preference) and some are categorical(like sex, musical preferences).\n",
        "For this reason we will use \"Spearman's rho correlation\" which is a non-parametric analysis.\n",
        "\"\"\"\n",
        "\n",
        "\"\"\"\n",
        "H0: Big-5 personality type of a person is not assiciated with the emotion depicted\n",
        "    in lyrics of their favourite songs.\n",
        "H1: Big-5 personality type of a person is associated with the emotion depicted in \n",
        "    in lyrics of their favourite songs.\n",
        "\"\"\"\n",
        "\n",
        "from scipy.stats import pointbiserialr, spearmanr, chisquare, mannwhitneyu, kendalltau, pearsonr\n",
        "\n",
        "current_method = [pointbiserialr, spearmanr, chisquare, mannwhitneyu, pearsonr]\n",
        "\n",
        "big5_traits = ['agreeableness', 'extraversion',\t'conscientiousness',\\\n",
        "               'neuroticism',\t'openness']\n",
        "demographics = ['age group','sex', 'instumental or lyrical music',\t'listen to english lyrics']\n",
        "emotions = ['anger',\t'fear',\t'joy',\t'sadness',\t'analytical',\\\n",
        "                  'confident',\t'tentative']\n",
        "\n",
        "# Find correlations and p-value for aspects of data. For example: \n",
        "# between demographics and big5_traits\n",
        "for i in demographics:\n",
        "  for j in big5_traits:\n",
        "    if(i is not j):\n",
        "      i_score = data[i]\n",
        "      j_score = data[j]\n",
        "      pbc = current_method[1](i_score, j_score)\n",
        "      if(pbc[1]<=0.06 and (pbc[0]>=0.3 or pbc[0]<=-0.3)):\n",
        "        print(i, j, pbc)\n",
        "\n"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "age group agreeableness SpearmanrResult(correlation=0.39231765313308486, pvalue=0.0016120427154733164)\n",
            "age group openness SpearmanrResult(correlation=0.3738420572594634, pvalue=0.0027618559864112492)\n",
            "sex agreeableness SpearmanrResult(correlation=-0.3807540496254783, pvalue=0.002266142455208097)\n",
            "listen to english lyrics openness SpearmanrResult(correlation=-0.4612220943040446, pvalue=0.00016142593158098995)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MU7RtWOQYOzU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Plot correlation matrix\n",
        "matrix = data.corr(method=\"spearman\")\n",
        "matrix = matrix.drop(index=['id', 'key_0'],columns=['id', 'key_0'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EubRAwLJNXz2",
        "colab_type": "code",
        "outputId": "4f86355b-3b4c-415a-92a4-de28853b9bfb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 631
        }
      },
      "source": [
        "matrix"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age group</th>\n",
              "      <th>sex</th>\n",
              "      <th>instumental or lyrical music</th>\n",
              "      <th>listen to english lyrics</th>\n",
              "      <th>extraversion</th>\n",
              "      <th>agreeableness</th>\n",
              "      <th>conscientiousness</th>\n",
              "      <th>neuroticism</th>\n",
              "      <th>openness</th>\n",
              "      <th>anger</th>\n",
              "      <th>fear</th>\n",
              "      <th>joy</th>\n",
              "      <th>sadness</th>\n",
              "      <th>analytical</th>\n",
              "      <th>confident</th>\n",
              "      <th>tentative</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>age group</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.369054</td>\n",
              "      <td>-0.008177</td>\n",
              "      <td>-0.008551</td>\n",
              "      <td>0.125131</td>\n",
              "      <td>0.392318</td>\n",
              "      <td>0.130578</td>\n",
              "      <td>-0.088144</td>\n",
              "      <td>0.373842</td>\n",
              "      <td>-0.084278</td>\n",
              "      <td>0.261549</td>\n",
              "      <td>0.280185</td>\n",
              "      <td>-0.226027</td>\n",
              "      <td>0.184411</td>\n",
              "      <td>-0.062666</td>\n",
              "      <td>-0.035726</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sex</th>\n",
              "      <td>-0.369054</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.175454</td>\n",
              "      <td>-0.009522</td>\n",
              "      <td>-0.247968</td>\n",
              "      <td>-0.380754</td>\n",
              "      <td>0.145378</td>\n",
              "      <td>-0.267583</td>\n",
              "      <td>-0.198165</td>\n",
              "      <td>0.067060</td>\n",
              "      <td>-0.091735</td>\n",
              "      <td>-0.252283</td>\n",
              "      <td>-0.081420</td>\n",
              "      <td>0.082782</td>\n",
              "      <td>0.041211</td>\n",
              "      <td>-0.103257</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>instumental or lyrical music</th>\n",
              "      <td>-0.008177</td>\n",
              "      <td>-0.175454</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.127498</td>\n",
              "      <td>-0.086490</td>\n",
              "      <td>0.083320</td>\n",
              "      <td>-0.272411</td>\n",
              "      <td>0.150106</td>\n",
              "      <td>-0.259456</td>\n",
              "      <td>-0.193379</td>\n",
              "      <td>-0.026719</td>\n",
              "      <td>0.107828</td>\n",
              "      <td>-0.081924</td>\n",
              "      <td>0.034565</td>\n",
              "      <td>-0.051287</td>\n",
              "      <td>0.120720</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>listen to english lyrics</th>\n",
              "      <td>-0.008551</td>\n",
              "      <td>-0.009522</td>\n",
              "      <td>0.127498</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.156225</td>\n",
              "      <td>-0.052094</td>\n",
              "      <td>-0.103277</td>\n",
              "      <td>0.132858</td>\n",
              "      <td>-0.461222</td>\n",
              "      <td>-0.237323</td>\n",
              "      <td>-0.118935</td>\n",
              "      <td>-0.059845</td>\n",
              "      <td>-0.139058</td>\n",
              "      <td>0.029176</td>\n",
              "      <td>0.081838</td>\n",
              "      <td>0.288723</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>extraversion</th>\n",
              "      <td>0.125131</td>\n",
              "      <td>-0.247968</td>\n",
              "      <td>-0.086490</td>\n",
              "      <td>-0.156225</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.271803</td>\n",
              "      <td>0.149192</td>\n",
              "      <td>-0.256995</td>\n",
              "      <td>0.309172</td>\n",
              "      <td>-0.016635</td>\n",
              "      <td>-0.060768</td>\n",
              "      <td>0.168314</td>\n",
              "      <td>0.070242</td>\n",
              "      <td>-0.027151</td>\n",
              "      <td>0.077961</td>\n",
              "      <td>0.083890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>agreeableness</th>\n",
              "      <td>0.392318</td>\n",
              "      <td>-0.380754</td>\n",
              "      <td>0.083320</td>\n",
              "      <td>-0.052094</td>\n",
              "      <td>0.271803</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.218802</td>\n",
              "      <td>-0.036046</td>\n",
              "      <td>0.260074</td>\n",
              "      <td>-0.290760</td>\n",
              "      <td>0.136943</td>\n",
              "      <td>0.336636</td>\n",
              "      <td>-0.140496</td>\n",
              "      <td>0.109051</td>\n",
              "      <td>-0.130029</td>\n",
              "      <td>0.050162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conscientiousness</th>\n",
              "      <td>0.130578</td>\n",
              "      <td>0.145378</td>\n",
              "      <td>-0.272411</td>\n",
              "      <td>-0.103277</td>\n",
              "      <td>0.149192</td>\n",
              "      <td>0.218802</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.260458</td>\n",
              "      <td>0.254493</td>\n",
              "      <td>0.056270</td>\n",
              "      <td>-0.020349</td>\n",
              "      <td>0.154600</td>\n",
              "      <td>0.003839</td>\n",
              "      <td>0.150412</td>\n",
              "      <td>-0.024471</td>\n",
              "      <td>-0.011063</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>neuroticism</th>\n",
              "      <td>-0.088144</td>\n",
              "      <td>-0.267583</td>\n",
              "      <td>0.150106</td>\n",
              "      <td>0.132858</td>\n",
              "      <td>-0.256995</td>\n",
              "      <td>-0.036046</td>\n",
              "      <td>-0.260458</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.257184</td>\n",
              "      <td>-0.056400</td>\n",
              "      <td>-0.245074</td>\n",
              "      <td>0.009895</td>\n",
              "      <td>0.018358</td>\n",
              "      <td>-0.121589</td>\n",
              "      <td>-0.128516</td>\n",
              "      <td>0.156187</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>openness</th>\n",
              "      <td>0.373842</td>\n",
              "      <td>-0.198165</td>\n",
              "      <td>-0.259456</td>\n",
              "      <td>-0.461222</td>\n",
              "      <td>0.309172</td>\n",
              "      <td>0.260074</td>\n",
              "      <td>0.254493</td>\n",
              "      <td>-0.257184</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.133554</td>\n",
              "      <td>0.261617</td>\n",
              "      <td>0.123554</td>\n",
              "      <td>0.082832</td>\n",
              "      <td>0.232181</td>\n",
              "      <td>-0.075285</td>\n",
              "      <td>-0.030993</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>anger</th>\n",
              "      <td>-0.084278</td>\n",
              "      <td>0.067060</td>\n",
              "      <td>-0.193379</td>\n",
              "      <td>-0.237323</td>\n",
              "      <td>-0.016635</td>\n",
              "      <td>-0.290760</td>\n",
              "      <td>0.056270</td>\n",
              "      <td>-0.056400</td>\n",
              "      <td>0.133554</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.124853</td>\n",
              "      <td>0.077586</td>\n",
              "      <td>-0.101484</td>\n",
              "      <td>-0.098103</td>\n",
              "      <td>-0.113255</td>\n",
              "      <td>0.081835</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>fear</th>\n",
              "      <td>0.261549</td>\n",
              "      <td>-0.091735</td>\n",
              "      <td>-0.026719</td>\n",
              "      <td>-0.118935</td>\n",
              "      <td>-0.060768</td>\n",
              "      <td>0.136943</td>\n",
              "      <td>-0.020349</td>\n",
              "      <td>-0.245074</td>\n",
              "      <td>0.261617</td>\n",
              "      <td>0.124853</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.058849</td>\n",
              "      <td>-0.020859</td>\n",
              "      <td>0.224863</td>\n",
              "      <td>0.042863</td>\n",
              "      <td>-0.200257</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>joy</th>\n",
              "      <td>0.280185</td>\n",
              "      <td>-0.252283</td>\n",
              "      <td>0.107828</td>\n",
              "      <td>-0.059845</td>\n",
              "      <td>0.168314</td>\n",
              "      <td>0.336636</td>\n",
              "      <td>0.154600</td>\n",
              "      <td>0.009895</td>\n",
              "      <td>0.123554</td>\n",
              "      <td>0.077586</td>\n",
              "      <td>0.058849</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.207352</td>\n",
              "      <td>0.093479</td>\n",
              "      <td>0.004642</td>\n",
              "      <td>0.047391</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sadness</th>\n",
              "      <td>-0.226027</td>\n",
              "      <td>-0.081420</td>\n",
              "      <td>-0.081924</td>\n",
              "      <td>-0.139058</td>\n",
              "      <td>0.070242</td>\n",
              "      <td>-0.140496</td>\n",
              "      <td>0.003839</td>\n",
              "      <td>0.018358</td>\n",
              "      <td>0.082832</td>\n",
              "      <td>-0.101484</td>\n",
              "      <td>-0.020859</td>\n",
              "      <td>-0.207352</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.159411</td>\n",
              "      <td>-0.020450</td>\n",
              "      <td>0.098806</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>analytical</th>\n",
              "      <td>0.184411</td>\n",
              "      <td>0.082782</td>\n",
              "      <td>0.034565</td>\n",
              "      <td>0.029176</td>\n",
              "      <td>-0.027151</td>\n",
              "      <td>0.109051</td>\n",
              "      <td>0.150412</td>\n",
              "      <td>-0.121589</td>\n",
              "      <td>0.232181</td>\n",
              "      <td>-0.098103</td>\n",
              "      <td>0.224863</td>\n",
              "      <td>0.093479</td>\n",
              "      <td>0.159411</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.001663</td>\n",
              "      <td>0.000888</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>confident</th>\n",
              "      <td>-0.062666</td>\n",
              "      <td>0.041211</td>\n",
              "      <td>-0.051287</td>\n",
              "      <td>0.081838</td>\n",
              "      <td>0.077961</td>\n",
              "      <td>-0.130029</td>\n",
              "      <td>-0.024471</td>\n",
              "      <td>-0.128516</td>\n",
              "      <td>-0.075285</td>\n",
              "      <td>-0.113255</td>\n",
              "      <td>0.042863</td>\n",
              "      <td>0.004642</td>\n",
              "      <td>-0.020450</td>\n",
              "      <td>0.001663</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.453864</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tentative</th>\n",
              "      <td>-0.035726</td>\n",
              "      <td>-0.103257</td>\n",
              "      <td>0.120720</td>\n",
              "      <td>0.288723</td>\n",
              "      <td>0.083890</td>\n",
              "      <td>0.050162</td>\n",
              "      <td>-0.011063</td>\n",
              "      <td>0.156187</td>\n",
              "      <td>-0.030993</td>\n",
              "      <td>0.081835</td>\n",
              "      <td>-0.200257</td>\n",
              "      <td>0.047391</td>\n",
              "      <td>0.098806</td>\n",
              "      <td>0.000888</td>\n",
              "      <td>-0.453864</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                              age group       sex  ...  confident  tentative\n",
              "age group                      1.000000 -0.369054  ...  -0.062666  -0.035726\n",
              "sex                           -0.369054  1.000000  ...   0.041211  -0.103257\n",
              "instumental or lyrical music  -0.008177 -0.175454  ...  -0.051287   0.120720\n",
              "listen to english lyrics      -0.008551 -0.009522  ...   0.081838   0.288723\n",
              "extraversion                   0.125131 -0.247968  ...   0.077961   0.083890\n",
              "agreeableness                  0.392318 -0.380754  ...  -0.130029   0.050162\n",
              "conscientiousness              0.130578  0.145378  ...  -0.024471  -0.011063\n",
              "neuroticism                   -0.088144 -0.267583  ...  -0.128516   0.156187\n",
              "openness                       0.373842 -0.198165  ...  -0.075285  -0.030993\n",
              "anger                         -0.084278  0.067060  ...  -0.113255   0.081835\n",
              "fear                           0.261549 -0.091735  ...   0.042863  -0.200257\n",
              "joy                            0.280185 -0.252283  ...   0.004642   0.047391\n",
              "sadness                       -0.226027 -0.081420  ...  -0.020450   0.098806\n",
              "analytical                     0.184411  0.082782  ...   0.001663   0.000888\n",
              "confident                     -0.062666  0.041211  ...   1.000000  -0.453864\n",
              "tentative                     -0.035726 -0.103257  ...  -0.453864   1.000000\n",
              "\n",
              "[16 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    }
  ]
}