creating_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tGLYJuXmFBZI"
   },
   "source": [
    "# Web attack detection using CICIDS2017 dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sNPL1uYLFF1U"
   },
   "source": [
    "Training data: \"Intrusion Detection Evaluation Dataset\" (CICIDS2017). Description page: https://www.unb.ca/cic/datasets/ids-2017.html\n",
    "\n",
    "The data set is public. Download link: http://205.174.165.80/CICDataset/CIC-IDS-2017/Dataset/\n",
    "\n",
    "CICIDS2017 combines 8 files recorded on different days of observation (PCAP + CSV). Used archive: http://205.174.165.80/CICDataset/CIC-IDS-2017/Dataset/GeneratedLabelledFlows.zip\n",
    "\n",
    "In the downloaded archive GeneratedLabelledFlows.zip the file \"Thursday\" Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv is selected.\n",
    "\n",
    "Sources:\n",
    "\n",
    "* [Sharafaldin2018] Iman Sharafaldin, Arash Habibi Lashkari and Ali A. Ghorbani. Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization. 2018\n",
    "* [Kostas2018] Kahraman Kostas. Anomaly Detection in Networks Using Machine Learning. 2018 (error was found in assessing the importance of features)\n",
    "* https://github.com/bozbil/Anomaly-Detection-in-Networks-Using-Machine-Learning (error was found in assessing the importance of features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SCEkDfpOFPQX"
   },
   "source": [
    "## Data preprocessing\n",
    "\n",
    "Source: https://github.com/bozbil/Anomaly-Detection-in-Networks-Using-Machine-Learning/blob/master/01_preprocessing.ipynb [Kostas2018]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "hJVAToeYFV2Q"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PFtoDi5mFkut"
   },
   "source": [
    "Download the dataset from Github to Google Colab and unzip it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fX1ARK3i3xQs",
    "outputId": "088a276e-1fd0-495c-8d34-da7bfb3aec34"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/infosecdemos/ml-2020/master/ml-ids/datasets/Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.zip -O dataset.zip\n",
    "!unzip -u dataset.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i4vnx_OaGAl0"
   },
   "source": [
    "We use \"engine=python\" to avoid the \"UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 11: invalid start byte\" encoding error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "yIqeKHKYGCoz"
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv', engine='python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3U-gQIAmhamL"
   },
   "source": [
    "Thursday-WorkingHours-Morning-WebAttacks.pcap_ISCX.csv file contains 458968 records, 85 columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QMkibc_W4hpQ",
    "outputId": "7cfd3c75-7531-4a66-b4ee-d8876451de37"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(458968, 85)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E5kIR-daheeV"
   },
   "source": [
    "The columns \"Fwd Header Length\" and \"Fwd Header Length.1\" are identical, the second one is removed, 84 columns remain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "83UFLCVlhjKF",
    "outputId": "2feb7048-5acd-4d38-8767-5cf5ea9d91e2"
   },
   "outputs": [],
   "source": [
    "df.columns = df.columns.str.strip()\n",
    "df = df.drop(columns=['Fwd Header Length.1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BIQQvtxZhiTZ"
   },
   "source": [
    "When assessing the distribution of labels, it turns out that out of 458968 records there are many blank records (\"BENIGN\" - benign background traffic)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HrGLjk5nhrGE",
    "outputId": "ae3370b8-9cc5-4cd1-ae7f-a3e97b25653c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['BENIGN', 'Web Attack � Brute Force', 'Web Attack � XSS',\n",
       "       'Web Attack � Sql Injection', nan], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Label'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HnpQqQ8fhxwN",
    "outputId": "756fcca6-20dd-444b-bf29-e3a920381e7a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BENIGN                        168186\n",
       "Web Attack � Brute Force        1507\n",
       "Web Attack � XSS                 652\n",
       "Web Attack � Sql Injection        21\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l7lpytF7h6D4"
   },
   "source": [
    "Delete blank records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7ltfHW8dh7Tw",
    "outputId": "4c5303cd-0bdd-4d2c-b1e2-026d37fe76c0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(170366, 84)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop(df[pd.isnull(df['Flow ID'])].index)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Удалить NaN, Infinity values\n",
    "\n",
    "The \"Flow Bytes/s\" and \"Flow Packets/s\" columns have non-numerical values, replace them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(170231, 84)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.replace('Infinity',np.nan,inplace=True)\n",
    "df[[\"Flow Bytes/s\", \"Flow Packets/s\"]] = df[[\"Flow Bytes/s\", \"Flow Packets/s\"]].apply(pd.to_numeric)\n",
    "df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "df.dropna(inplace=True)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Flow ID', 'Source IP', 'Source Port', 'Destination IP',\n",
       "       'Destination Port', 'Protocol', 'Timestamp', 'Flow Duration',\n",
       "       'Total Fwd Packets', 'Total Backward Packets',\n",
       "       'Total Length of Fwd Packets', 'Total Length of Bwd Packets',\n",
       "       'Fwd Packet Length Max', 'Fwd Packet Length Min',\n",
       "       'Fwd Packet Length Mean', 'Fwd Packet Length Std',\n",
       "       'Bwd Packet Length Max', 'Bwd Packet Length Min',\n",
       "       'Bwd Packet Length Mean', 'Bwd Packet Length Std', 'Flow Bytes/s',\n",
       "       'Flow Packets/s', 'Flow IAT Mean', 'Flow IAT Std', 'Flow IAT Max',\n",
       "       'Flow IAT Min', 'Fwd IAT Total', 'Fwd IAT Mean', 'Fwd IAT Std',\n",
       "       'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Mean',\n",
       "       'Bwd IAT Std', 'Bwd IAT Max', 'Bwd IAT Min', 'Fwd PSH Flags',\n",
       "       'Bwd PSH Flags', 'Fwd URG Flags', 'Bwd URG Flags', 'Fwd Header Length',\n",
       "       'Bwd Header Length', 'Fwd Packets/s', 'Bwd Packets/s',\n",
       "       'Min Packet Length', 'Max Packet Length', 'Packet Length Mean',\n",
       "       'Packet Length Std', 'Packet Length Variance', 'FIN Flag Count',\n",
       "       'SYN Flag Count', 'RST Flag Count', 'PSH Flag Count', 'ACK Flag Count',\n",
       "       'URG Flag Count', 'CWE Flag Count', 'ECE Flag Count', 'Down/Up Ratio',\n",
       "       'Average Packet Size', 'Avg Fwd Segment Size', 'Avg Bwd Segment Size',\n",
       "       'Fwd Avg Bytes/Bulk', 'Fwd Avg Packets/Bulk', 'Fwd Avg Bulk Rate',\n",
       "       'Bwd Avg Bytes/Bulk', 'Bwd Avg Packets/Bulk', 'Bwd Avg Bulk Rate',\n",
       "       'Subflow Fwd Packets', 'Subflow Fwd Bytes', 'Subflow Bwd Packets',\n",
       "       'Subflow Bwd Bytes', 'Init_Win_bytes_forward',\n",
       "       'Init_Win_bytes_backward', 'act_data_pkt_fwd', 'min_seg_size_forward',\n",
       "       'Active Mean', 'Active Std', 'Active Max', 'Active Min', 'Idle Mean',\n",
       "       'Idle Std', 'Idle Max', 'Idle Min', 'Label'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2JMv7gKfi4nI"
   },
   "source": [
    "Convert string characters to numbers, use LabelEncoder, not OneHotEncoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XmYOMG0Ti8Wh",
    "outputId": "da9cacfc-4f7c-4b8b-f348-c86ed4b80dff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Flow ID', 'Source IP', 'Destination IP', 'Timestamp']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string_features = list(df.select_dtypes(include=['object']).columns)\n",
    "string_features.remove('Label')\n",
    "string_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "ySZ2eW_2jCfQ"
   },
   "outputs": [],
   "source": [
    "le = preprocessing.LabelEncoder()\n",
    "df[string_features] = df[string_features].apply(lambda col: le.fit_transform(col))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow ID</th>\n",
       "      <th>Source IP</th>\n",
       "      <th>Source Port</th>\n",
       "      <th>Destination IP</th>\n",
       "      <th>Destination Port</th>\n",
       "      <th>Protocol</th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Fwd Packets</th>\n",
       "      <th>Total Backward Packets</th>\n",
       "      <th>...</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Active Mean</th>\n",
       "      <th>Active Std</th>\n",
       "      <th>Active Max</th>\n",
       "      <th>Active Min</th>\n",
       "      <th>Idle Mean</th>\n",
       "      <th>Idle Std</th>\n",
       "      <th>Idle Max</th>\n",
       "      <th>Idle Min</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>150928</th>\n",
       "      <td>14347</td>\n",
       "      <td>1263</td>\n",
       "      <td>61239.0</td>\n",
       "      <td>1590</td>\n",
       "      <td>53.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>120</td>\n",
       "      <td>9735257.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>30898.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>30898.0</td>\n",
       "      <td>30898.0</td>\n",
       "      <td>9598535.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>9598535.0</td>\n",
       "      <td>9598535.0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30578</th>\n",
       "      <td>25669</td>\n",
       "      <td>1400</td>\n",
       "      <td>443.0</td>\n",
       "      <td>1594</td>\n",
       "      <td>40720.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>214</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106277</th>\n",
       "      <td>7718</td>\n",
       "      <td>920</td>\n",
       "      <td>443.0</td>\n",
       "      <td>1591</td>\n",
       "      <td>40608.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>57</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56871</th>\n",
       "      <td>28789</td>\n",
       "      <td>1260</td>\n",
       "      <td>59701.0</td>\n",
       "      <td>4284</td>\n",
       "      <td>80.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>236</td>\n",
       "      <td>115100459.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>...</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1507093.3</td>\n",
       "      <td>2332518.976</td>\n",
       "      <td>4954002.0</td>\n",
       "      <td>50394.0</td>\n",
       "      <td>9997912.2</td>\n",
       "      <td>66545.75937</td>\n",
       "      <td>10000000.0</td>\n",
       "      <td>9811232.0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146647</th>\n",
       "      <td>32436</td>\n",
       "      <td>1262</td>\n",
       "      <td>50525.0</td>\n",
       "      <td>2164</td>\n",
       "      <td>443.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>117</td>\n",
       "      <td>64111443.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>BENIGN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 84 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Flow ID  Source IP  Source Port  Destination IP  Destination Port  \\\n",
       "150928    14347       1263      61239.0            1590              53.0   \n",
       "30578     25669       1400        443.0            1594           40720.0   \n",
       "106277     7718        920        443.0            1591           40608.0   \n",
       "56871     28789       1260      59701.0            4284              80.0   \n",
       "146647    32436       1262      50525.0            2164             443.0   \n",
       "\n",
       "        Protocol  Timestamp  Flow Duration  Total Fwd Packets  \\\n",
       "150928      17.0        120      9735257.0                2.0   \n",
       "30578        6.0        214            1.0                2.0   \n",
       "106277       6.0         57            3.0                2.0   \n",
       "56871        6.0        236    115100459.0               21.0   \n",
       "146647       6.0        117     64111443.0                2.0   \n",
       "\n",
       "        Total Backward Packets  ...  min_seg_size_forward  Active Mean  \\\n",
       "150928                     2.0  ...                  20.0      30898.0   \n",
       "30578                      0.0  ...                  32.0          0.0   \n",
       "106277                     0.0  ...                  32.0          0.0   \n",
       "56871                     16.0  ...                  32.0    1507093.3   \n",
       "146647                     0.0  ...                  20.0          0.0   \n",
       "\n",
       "         Active Std  Active Max  Active Min  Idle Mean     Idle Std  \\\n",
       "150928        0.000     30898.0     30898.0  9598535.0      0.00000   \n",
       "30578         0.000         0.0         0.0        0.0      0.00000   \n",
       "106277        0.000         0.0         0.0        0.0      0.00000   \n",
       "56871   2332518.976   4954002.0     50394.0  9997912.2  66545.75937   \n",
       "146647        0.000         0.0         0.0        0.0      0.00000   \n",
       "\n",
       "          Idle Max   Idle Min   Label  \n",
       "150928   9598535.0  9598535.0  BENIGN  \n",
       "30578          0.0        0.0  BENIGN  \n",
       "106277         0.0        0.0  BENIGN  \n",
       "56871   10000000.0  9811232.0  BENIGN  \n",
       "146647         0.0        0.0  BENIGN  \n",
       "\n",
       "[5 rows x 84 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K94al8HAjJEN"
   },
   "source": [
    "### Undersampling against unbalance\n",
    "\n",
    "Dataset is unbalanced: total records = 170366, \"BENIGN\" records = 168186, records with attacks much less: 1507 + 652 + 21 = 2180."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7K7iWbNMjWrZ",
    "outputId": "09ddf90e-973c-496f-b888-ac82cd1cd730"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "168051"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "benign_total = len(df[df['Label'] == \"BENIGN\"])\n",
    "benign_total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nVs_xEHFjY5q",
    "outputId": "8a8cf8a3-493f-4cab-8c78-94f9801ffd3f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2180"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attack_total = len(df[df['Label'] != \"BENIGN\"])\n",
    "attack_total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PFvPOea2jhMt",
    "outputId": "26895dcc-74ef-4cb7-e7da-35c8b5ee4bd6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BENIGN                        168051\n",
       "Web Attack � Brute Force        1507\n",
       "Web Attack � XSS                 652\n",
       "Web Attack � Sql Injection        21\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.to_csv(\"web_attacks_unbalanced.csv\", index=False)\n",
    "df['Label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6A9-rS7OjyFl"
   },
   "source": [
    "We use **undersampling** to correct class imbalances: we remove most of the \"BENIGN\" records.\n",
    "\n",
    "Form a balanced dataset web_attacks_balanced.csv in proportion: 30% attack (2180 records), 70% benign data (2180 / 30 * 70 ~ = 5087 records).\n",
    "\n",
    "Algorithm to form a balanced df_balanced dataset:\n",
    "\n",
    "* All the records with the attacks are copied to the new dataset.\n",
    "* There are two conditions for copying \"BENIGN\" records to the new dataset:\n",
    " \n",
    "     1. The next record is copyied with the benign_inc_probability.\n",
    "     2. The total number of \"BENIGN\" records must not exceed the limit of 5087 records."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aVZutNOZj_N9"
   },
   "source": [
    "Сalculate the probability of copying a \"BENIGN\" record. The enlargement multiplier is used to get exactly 70% benign data (5087 records)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XaW7sEjHkB3K",
    "outputId": "79d5d5b5-d91b-4004-a3db-b4e8f8fe6dbe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5086.666666666667 0.03329544800883859\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('web_attacks_unbalanced.csv')\n",
    "enlargement = 1.1\n",
    "benign_included_max = attack_total / 30 * 70\n",
    "benign_inc_probability = (benign_included_max / benign_total) * enlargement\n",
    "print(benign_included_max, benign_inc_probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G3ZXh6fTkFK_"
   },
   "source": [
    "Copy records from df to df_balanced, save dataset **web_attacks_balanced.csv**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "0MdiMPmdkICk"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "indexes = []\n",
    "benign_included_count = 0\n",
    "for index, row in df.iterrows():\n",
    "    if (row['Label'] != \"BENIGN\"):\n",
    "        indexes.append(index)\n",
    "    else:\n",
    "        # Copying with benign_inc_probability\n",
    "        if random.random() > benign_inc_probability: continue\n",
    "        # Have we achieved 70% (5087 records)?\n",
    "        if benign_included_count > benign_included_max: continue\n",
    "        benign_included_count += 1\n",
    "        indexes.append(index)\n",
    "df_balanced = df.loc[indexes]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cqypBDIdkKN4",
    "outputId": "e5fe647e-d716-42b5-97ec-0353b887e2b2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BENIGN                        5087\n",
       "Web Attack � Brute Force      1507\n",
       "Web Attack � XSS               652\n",
       "Web Attack � Sql Injection      21\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_balanced['Label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hDPCkmutkNYD"
   },
   "source": [
    "If necessary, we can keep a single class of attack for experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1_q52wjdkPZ8"
   },
   "outputs": [],
   "source": [
    "# df_balanced = df_balanced[df_balanced['Label'] != \"Web Attack – Brute Force\"]\n",
    "# df_balanced = df_balanced[df_balanced['Label'] != \"Web Attack – Sql Injection\"]\n",
    "# df_balanced['Label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "qiYFzlGkkSXe"
   },
   "outputs": [],
   "source": [
    "df_balanced.to_csv(\"web_attacks_balanced.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### СТАРТ ОТСЮДА"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('web_attacks_balanced.csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/infosecdemos/ml-2020/master/ml-ids/datasets/mix_labelled.zip -O mix_labelled.zip\n",
    "!unzip -u mix_labelled.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real = pd.read_csv('mix_labelled.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing data for training\n",
    "\n",
    "7 features (Flow ID, Source IP, Source Port, Destination IP, Destination Port, Protocol, Timestamp) are excluded from the dataset. The hypothesis is that the \"shape\" of the data being transmitted is more important than these attributes. In addition, ports and addresses can be substituted by an attacker, so it is better that the ML algorithm does not take these features into account in training [Kostas2018]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "excluded = ['Flow ID', 'Source IP', 'Source Port', 'Destination IP', 'Destination Port', 'Protocol', 'Timestamp']\n",
    "df = df.drop(columns=excluded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below at the stage of importance estimation the \"Init_Win_bytes_backward\" feature has the maximum value. After viewing the source dataset, it seems that an inaccuracy was made in forming the dataset. \n",
    "\n",
    "It turns out that it is possible to make a fairly accurate classification by one feature.\n",
    "\n",
    "Description of features: http://www.netflowmeter.ca/netflowmeter.html\n",
    "\n",
    "     Init_Win_bytes_backward - The total number of bytes sent in initial window in the backward direction\n",
    "     Init_Win_bytes_forward - The total number of bytes sent in initial window in the forward direction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeEElEQVR4nO3df5QcZZ3v8feHBEIgQILAGEg0KFk1gAKZ5cdy0YmgBOQIq7InLkjYhc2u4ior/gjqWde9N/dkXeEq8kOzgIQfEiOoyYJRY9wBo2AICIZAsgSIEBMS+c0gIsHv/aOe2RSTnnk6M9M9XeHzOqdPVz9VT9W3JpP+TD1VXa2IwMzMrC87DHUBZmbW+hwWZmaW5bAwM7Msh4WZmWU5LMzMLMthYWZmWQ4LaypJiyRNb8J2uiS9YYDrCEkHDFZNg0lSp6SzB3mdE9I+Dx/M9fajjkHfNxs4h4UNCklrJR2XWy4iToiIuanPmZKW1rHuoyQ9K2lYqe0/emn7etrOqIh4qH97MzCSOiStG4ptmzWKw8KqYDkwDDis1HYMsL5H29uBW5tYlw2ACn4Pqgj/Q9mg6j5akPRlSU9JeljSCaX5nZLOlvQW4OvAUWnI6One1hkRLwG3U4QBkvYBdgK+3aPtz0hhUR5CknSVpEsk3SzpOUm/lPTGOnfpREkPSXpc0r9L2kHSCElPSjq4tF/7SHpB0uuBRcC+ab+6JO2b+s2U9KCkJyTNl7Rn6ruzpGtT+9OS7pDUVkdtb5S0TNIzkhZ0ry+t8zuSHkvzbpV0YGneSEkXSPpNmr9U0sieK5f0/nTEeJCkuZLOS+37pZ/vR9LrA9LPQ5LGSLpJ0u/Sv/9NksaV1tkpaZaknwO/B94g6V2SVqVaLgZU57+NNZHDwhrhCGA1sBfwJeAKSa94A4iI+4F/AG5LQ0ajM+u8lRQM6XlpepTbHo6I3oZ/Pgh8ERgDrAFm1bkvfwm0UxzBnAz8bUS8CMwDTu+x/p9ExG+AE4D1ab9GRcR64GPAKcA7gH2Bp4BLUt/pwB7AeOA1FD+XF+qo7Qzgb9P6NgMXleYtAiYC+wB3AdeV5n0ZmAz8BbAn8GngT+UVS/ob4N+A4yLiXuAWoCPNfgfwUHqG4mf/syjuHbQD8E3g9cDr0n5c3KPuDwEzgN2AZ4Abgc9T/L48CBxdx75bs0WEH34M+AGsBY4DzgTWlNp3AQJ4bXrdCZydps8Elta5/g7gCYq/Or8K/B0wCthYavtmafkADkjTVwGXl+adCKyqY5sBTC29/giwJE0fATwK7JBeLwf+qlTruh7ruh84tvR6LPASMJziDf8XwFu34efdCcwuvZ4E/BEYVmPZ0Wlf9qB4M38BeFuN5Sak5T4J3AeMK817I/B06v914O+79xGYC3yilzoPAZ7qUfe/ll6fAdxeei1gXffviB+t8/CRhTXCY90TEfH7NDlqgOu8Pa3jILb8JdtF8Ybd3dbX+YrHStO/34Z6Hi1N/4bir3gi4pfA88A7JL0ZOABY2Md6Xg98Lw0zPU0RHi8DbcA1wI+AeZLWS/qSpB37UduOwF6ShkmanYa8nqUIcij+ct8L2JniL/jefAq4JEpHaRHxINBF8eZ/DHATsF7SmyiOMG4BkLSLpG+kIa5nKf5NRpcvROhR977l11EkRnm+tQiHhQ2lum95HBF/AO4ATgLGRsSqNOtnqe2tNObk9vjS9OsoTqp3m0sxFPUh4IZUI9Ter0eBEyJidOmxc0T8NiJeiogvRsQkiqGhkyj+4t7W2l4CHgf+mmLI7DiKo4kJaRml+X+gOFLozbuBz0t6f4/2W4APADtFxG/T6zMohvbuTsucB7wJOCIidmfLMGF5GLL889lQ3o80XFneL2sRDgsbShuBcZJ2qnP5W4FzKYZsui1NbY+lv34H26fSSdvxwMcpTqp3u4binMbpwNWl9o3AayTtUWr7OjArnQBH0t6STk7TUyQdnP76fpbiTf/lOmo7XdIkSbsA/0oRWC9TnAt4kWLYbhfg/3Z3iIg/AVcCF6YT78NUXJo8orTelcBU4BJJ7y213wJ8lC2h3An8I8VQYne9u1EMcz2dTrh/IbMPNwMHSnqfis93fAx4bR37bk3msLCh9FOKN6bHJD1ex/K3UJywLX82Y2lqa9QlswuAOyn+cr4ZuKJ7RhqmuYviL+WfldpXAdcDD6Vhp30pzqksBH4s6TmKYbUjUpfXAjdQBMX9FPt5bR21XUNxPuYxiqGlj6X2qymGpX5Lce7h9h79PgmsoDhSe5LiRPYr3gsi4h6KI5z/0Jar2W6hCIPun/VSijAq/+y/AoykOIK5HfhhXzsQEY8DpwKzKcJtIvDzvvrY0FAxRGhm/SHpSoornz4/1LWYNdKQfqzfrMokTQDeBxw6xKWYNZyHoaxlSFpZ+iBb+XFag7Z3TC/b66qj7/8G7gX+PSIeblB9NWuTdEwjtmfWFw9DmZlZlo8szMwsa7s9Z7HXXnvFhAkT+tX3+eefZ9dddx3cgpqgqnVDdWuvat1Q3dqrWjdUo/Y777zz8YjYu2f7dhsWEyZMYPny5f3q29nZSUdHx+AW1ARVrRuqW3tV64bq1l7VuqEatUv6Ta12D0OZmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZ1nb7Ce6BWPHbZzhz5s1N3+7a2e9p+jbNzOrhIwszM8tyWJiZWVZDw0LSWkkrJN0taXlq21PSYkkPpOcxpeXPl7RG0mpJx5faJ6f1rJF0kSQ1sm4zM3ulZhxZTImIQyKiPb2eCSyJiInAkvQaSZOAacCBwFTgUknDUp/LgBkUX+Y+Mc03M7MmGYphqJOBuWl6LnBKqX1eRLyYvqZyDXC4pLHA7hFxWxRf63d1qY+ZmTVBQ79WVdLDwFNAAN+IiDmSno6I0aVlnoqIMZIuBm6PiGtT+xXAImAtMDsijkvtxwCfiYiTamxvBsURCG1tbZPnzZvXr7o3PfkMG1/oV9cBOXi/PQbUv6uri1GjRg1SNc1V1dqrWjdUt/aq1g3VqH3KlCl3lkaC/kejL509OiLWS9oHWCxpVR/L1joPEX20b90YMQeYA9De3h79/ZKRr123gAtWNP+q4rWndQyofxW+WKU3Va29qnVDdWuvat1Q7dobOgwVEevT8ybge8DhwMY0tER63pQWXweML3UfB6xP7eNqtJuZWZM0LCwk7Sppt+5p4N3AvcBCYHpabDqwIE0vBKZJGiFpf4oT2csiYgPwnKQj01VQZ5T6mJlZEzRyrKUN+F66ynU48K2I+KGkO4D5ks4CHgFOBYiIlZLmA/cBm4FzIuLltK4PA1cBIynOYyxqYN1mZtZDw8IiIh4C3laj/Qng2F76zAJm1WhfDhw02DWamVl9/AluMzPLcliYmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZlsPCzMyyHBZmZpblsDAzsyyHhZmZZTkszMwsy2FhZmZZDgszM8tyWJiZWZbDwszMshwWZmaW5bAwM7Msh4WZmWU5LMzMLMthYWZmWQ4LMzPLcliYmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCyr4WEhaZikX0m6Kb3eU9JiSQ+k5zGlZc+XtEbSaknHl9onS1qR5l0kSY2u28zMtmjGkcXHgftLr2cCSyJiIrAkvUbSJGAacCAwFbhU0rDU5zJgBjAxPaY2oW4zM0saGhaSxgHvAS4vNZ8MzE3Tc4FTSu3zIuLFiHgYWAMcLmkssHtE3BYRAVxd6mNmZk0wvMHr/wrwaWC3UltbRGwAiIgNkvZJ7fsBt5eWW5faXkrTPdu3ImkGxREIbW1tdHZ29qvotpFw3sGb+9V3IPpbb7eurq4Br2OoVLX2qtYN1a29qnVDtWtvWFhIOgnYFBF3Suqop0uNtuijfevGiDnAHID29vbo6Khns1v72nULuGBFo3N0a2tP6xhQ/87OTvq7z0OtqrVXtW6obu1VrRuqXXsj3xGPBt4r6URgZ2B3SdcCGyWNTUcVY4FNafl1wPhS/3HA+tQ+rka7mZk1ScPOWUTE+RExLiImUJy4/mlEnA4sBKanxaYDC9L0QmCapBGS9qc4kb0sDVk9J+nIdBXUGaU+ZmbWBM0fa4HZwHxJZwGPAKcCRMRKSfOB+4DNwDkR8XLq82HgKmAksCg9zMysSZoSFhHRCXSm6SeAY3tZbhYwq0b7cuCgxlVoZmZ98Se4zcwsy2FhZmZZDgszM8tyWJiZWZbDwszMshwWZmaW5bAwM7Msh4WZmWU5LMzMLMthYWZmWQ4LMzPLcliYmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZlsPCzMyyHBZmZpblsDAzsyyHhZmZZTkszMwsy2FhZmZZDgszM8tyWJiZWZbDwszMshwWZmaWVVdYSLpR0nskOVzMzF6F6n3zvwz4a+ABSbMlvTnXQdLOkpZJukfSSklfTO17Slos6YH0PKbU53xJayStlnR8qX2ypBVp3kWStI37aWZmA1BXWETETyLiNOAwYC2wWNIvJP2NpB176fYi8M6IeBtwCDBV0pHATGBJREwElqTXSJoETAMOBKYCl0oaltZ1GTADmJgeU7d1R83MrP/qHlaS9BrgTOBs4FfAVynCY3Gt5aPQlV7umB4BnAzMTe1zgVPS9MnAvIh4MSIeBtYAh0saC+weEbdFRABXl/qYmVkTqHj/zSwkfRd4M3ANcFVEbCjNWx4R7b30GwbcCRwAXBIRn5H0dESMLi3zVESMkXQxcHtEXJvarwAWURzJzI6I41L7McBnIuKkGtubQXEEQltb2+R58+bV8SPY2qYnn2HjC/3qOiAH77fHgPp3dXUxatSoQaqmuapae1XrhurWXtW6oRq1T5ky5c5a7+nD6+x/eUT8oNwgaUQ6CqgZFAAR8TJwiKTRwPckHdTHNmqdh4g+2mttbw4wB6C9vT06Ojr62FzvvnbdAi5YUe+PZvCsPa1jQP07Ozvp7z4PtarWXtW6obq1V7VuqHbt9Q5D/Z8abbfVu5GIeBropDjXsDENLZGeN6XF1gHjS93GAetT+7ga7WZm1iR9hoWk10qaDIyUdKikw9KjA9gl03fvdESBpJHAccAqYCEwPS02HViQphcC0ySNkLQ/xYnsZWnI6zlJR6aroM4o9TEzsybIjbUcT3FSexxwYan9OeCzmb5jgbnpvMUOwPyIuEnSbcB8SWcBjwCnAkTESknzgfuAzcA5aRgL4MPAVcBIivMYi+raOzMzGxR9hkVEzKV4w39/RNy4LSuOiF8Dh9ZofwI4tpc+s4BZNdqXA32d7zAzswbqMywknZ6uTpog6RM950fEhTW6mZnZdiY3DLVrem7ta73MzKyhcsNQ30jPX2xOOWZm1orqvZHglyTtLmlHSUskPS7p9EYXZ2ZmraHez1m8OyKeBU6i+NzDnwGfalhVZmbWUuoNi+6bBZ4IXB8RTzaoHjMza0H13tPiPyWtAl4APiJpb+APjSvLzMxaSb23KJ8JHAW0R8RLwPMUd4k1M7NXgW25W95bKD5vUe5z9SDXY2ZmLaiusJB0DfBG4G6g+xYc3d8tYWZm27l6jyzagUlRz5dfmJnZdqfeq6HuBV7byELMzKx11XtksRdwn6RlFN+tDUBEvLchVZmZWUupNyz+pZFFmJlZa6srLCLiFkmvByZGxE8k7QIMa2xpZmbWKuq9N9TfATcA30hN+wHfb1BNZmbWYuo9wX0OcDTwLEBEPADs06iizMystdQbFi9GxB+7X6QP5vkyWjOzV4l6w+IWSZ8FRkp6F/Ad4D8bV5aZmbWSesNiJvA7YAXw98APgM83qigzM2st9V4N9SdJ3we+HxG/a2xJZmbWavo8slDhXyQ9DqwCVkv6naR/bk55ZmbWCnLDUOdSXAX15xHxmojYEzgCOFrSPzW6ODMzaw25sDgD+GBEPNzdEBEPAaeneWZm9iqQC4sdI+Lxno3pvMWONZY3M7PtUC4s/tjPeWZmth3JXQ31NknP1mgXsHMD6jEzsxbUZ1hEhG8WaGZmdX8oz8zMXsUcFmZmluWwMDOzrIaFhaTxkv5L0v2SVkr6eGrfU9JiSQ+k5zGlPudLWiNptaTjS+2TJa1I8y6SpEbVbWZmW2vkkcVm4LyIeAtwJHCOpEkUNyVcEhETgSXpNWneNOBAYCpwqaTuE+yXATOAiekxtYF1m5lZDw0Li4jYEBF3penngPspvmHvZGBuWmwucEqaPhmYFxEvpk+MrwEOlzQW2D0ibouIAK4u9TEzsyZQ8f7b4I1IE4BbgYOARyJidGneUxExRtLFwO0RcW1qvwJYBKwFZkfEcan9GOAzEXFSje3MoDgCoa2tbfK8efP6Ve+mJ59h4wv96jogB++3x4D6d3V1MWrUqEGqprmqWntV64bq1l7VuqEatU+ZMuXOiGjv2V7XLcoHQtIo4Ebg3Ih4to/TDbVmRB/tWzdGzAHmALS3t0dHR8c21wvwtesWcMGKhv9otrL2tI4B9e/s7KS/+zzUqlp7VeuG6tZe1bqh2rU39GooSTtSBMV1EfHd1LwxDS2Rnjel9nXA+FL3ccD61D6uRruZmTVJI6+GEnAFcH9EXFiatRCYnqanAwtK7dMkjZC0P8WJ7GURsQF4TtKRaZ1nlPqYmVkTNHKs5WjgQ8AKSXents8Cs4H5ks4CHgFOBYiIlZLmA/dRXEl1TkS8nPp9GLgKGElxHmNRA+u2V4kJM28e8DrOO3gzZ27jetbOfs+At2vWbA0Li4hYSu3zDQDH9tJnFjCrRvtyipPjZmY2BPwJbjMzy3JYmJlZlsPCzMyyHBZmZpblsDAzsyyHhZmZZTkszMwsy2FhZmZZDgszM8tyWJiZWZbDwszMshwWZmaW5bAwM7Msh4WZmWU5LMzMLMthYWZmWQ4LMzPLcliYmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZlsPCzMyyHBZmZpblsDAzsyyHhZmZZTkszMwsq2FhIelKSZsk3Vtq21PSYkkPpOcxpXnnS1ojabWk40vtkyWtSPMukqRG1WxmZrU18sjiKmBqj7aZwJKImAgsSa+RNAmYBhyY+lwqaVjqcxkwA5iYHj3XaWZmDdawsIiIW4EnezSfDMxN03OBU0rt8yLixYh4GFgDHC5pLLB7RNwWEQFcXepjZmZN0uxzFm0RsQEgPe+T2vcDHi0tty617Zeme7abmVkTDR/qApJa5yGij/baK5FmUAxZ0dbWRmdnZ7+KaRsJ5x28uV99B6K/9Xbr6uoa8DqGylDUPhj/xv35XWmVf6Oq/r5UtW6odu3NDouNksZGxIY0xLQpta8DxpeWGwesT+3jarTXFBFzgDkA7e3t0dHR0a8iv3bdAi5Y0fwcXXtax4D6d3Z20t99HmpDUfuZM28e8DrOO3jzNv+uDPTfebBU9felqnVDtWtv9jDUQmB6mp4OLCi1T5M0QtL+FCeyl6WhquckHZmugjqj1MfMzJqkYX8+S7oe6AD2krQO+AIwG5gv6SzgEeBUgIhYKWk+cB+wGTgnIl5Oq/owxZVVI4FF6WFmZk3UsLCIiA/2MuvYXpafBcyq0b4cOGgQSzMzs23kT3CbmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZlsPCzMyyHBZmZpblsDAzsyyHhZmZZTkszMwsy2FhZmZZDgszM8tyWJiZWZbDwszMshwWZmaW5bAwM7Msh4WZmWU5LMzMLMthYWZmWQ4LMzPLcliYmVmWw8LMzLIcFmZmluWwMDOzLIeFmZllOSzMzCzLYWFmZlkOCzMzyxo+1AXUS9JU4KvAMODyiJg9xCWZmfVqwsybt2o77+DNnFmjfTCtnf2ehqy3EmEhaRhwCfAuYB1wh6SFEXHf0FY2uGr9cm2LgfwiNuoXzMy2D5UIC+BwYE1EPAQgaR5wMrBdhcVQGmhQ9derMaSG6mfdUzP+yu32avx33t4oIoa6hixJHwCmRsTZ6fWHgCMi4qM9lpsBzEgv3wSs7ucm9wIe72ffoVTVuqG6tVe1bqhu7VWtG6pR++sjYu+ejVU5slCNtq1SLiLmAHMGvDFpeUS0D3Q9zVbVuqG6tVe1bqhu7VWtG6pde1WuhloHjC+9HgesH6JazMxedaoSFncAEyXtL2knYBqwcIhrMjN71ajEMFREbJb0UeBHFJfOXhkRKxu4yQEPZQ2RqtYN1a29qnVDdWuvat1Q4dorcYLbzMyGVlWGoczMbAg5LMzMLMthUSJpqqTVktZImjmEdVwpaZOke0tte0paLOmB9DymNO/8VPNqSceX2idLWpHmXSRJqX2EpG+n9l9KmjBIdY+X9F+S7pe0UtLHq1C7pJ0lLZN0T6r7i1Wou7TNYZJ+JemmitW9Nm3zbknLq1K7pNGSbpC0Kv2uH1WFugcsIvwoztsMAx4E3gDsBNwDTBqiWt4OHAbcW2r7EjAzTc8E/i1NT0q1jgD2T/swLM1bBhxF8TmVRcAJqf0jwNfT9DTg24NU91jgsDS9G/Dfqb6Wrj1tY1Sa3hH4JXBkq9ddqv8TwLeAm6ryu5LWtxbYq0dby9cOzAXOTtM7AaOrUPeA93uoC2iVR/pH+1Hp9fnA+UNYzwReGRargbFpeiywuladFFeMHZWWWVVq/yDwjfIyaXo4xSdK1YB9WEBxP6/K1A7sAtwFHFGFuik+c7QEeCdbwqLl607rW8vWYdHStQO7Aw/3XE+r1z0YDw9DbbEf8Gjp9brU1iraImIDQHreJ7X3Vvd+abpn+yv6RMRm4BngNYNZbDp0PpTir/SWrz0N5dwNbAIWR0Ql6ga+Anwa+FOprQp1Q3EXhh9LulPFrXqqUPsbgN8B30xDf5dL2rUCdQ+Yw2KLum4p0oJ6q7uv/WnovkoaBdwInBsRz/a1aC91NL32iHg5Ig6h+Ev9cEkH9bF4S9Qt6SRgU0TcWW+XXmoYqt+VoyPiMOAE4BxJb+9j2VapfTjFEPFlEXEo8DzFsFNvWqXuAXNYbNHqtxTZKGksQHrelNp7q3tdmu7Z/oo+koYDewBPDkaRknakCIrrIuK7VaodICKeBjqBqRWo+2jgvZLWAvOAd0q6tgJ1AxAR69PzJuB7FHeXbvXa1wHr0pEnwA0U4dHqdQ+Yw2KLVr+lyEJgepqeTnE+oLt9WrqCYn9gIrAsHQo/J+nIdJXFGT36dK/rA8BPIw2QDkTazhXA/RFxYVVql7S3pNFpeiRwHLCq1euOiPMjYlxETKD4ff1pRJze6nUDSNpV0m7d08C7gXtbvfaIeAx4VNKbUtOxFF+V0NJ1D4qhPmnSSg/gRIoreB4EPjeEdVwPbABeovgr4yyKMcslwAPpec/S8p9LNa8mXVGR2tsp/gM+CFzMlk/s7wx8B1hDcUXGGwap7v9Fcbj8a+Du9Dix1WsH3gr8KtV9L/DPqb2l6+6xDx1sOcHd8nVTjP3fkx4ru/+/VaT2Q4Dl6ffl+8CYKtQ90Idv92FmZlkehjIzsyyHhZmZZTkszMwsy2FhZmZZDgszM8tyWJglkjrLdwVNbedKujTTp30Qtn2mpIt7mfeLga7fbKAcFmZbXE/x4bayaal9yETEXwzl9s3AYWFWdgNwkqQR8D83Q9wXWCrpMknLVfq+i54kdZWmPyDpqjS9t6QbJd2RHkf3sv3xkn6YvvfgCz3XK6kjHcl0f5fCdaXvQJgt6T5Jv5b05YH/KMxeafhQF2DWKiLiCUnLKO4LtYAt3yUQkj4XEU9KGgYskfTWiPh1nav+KvD/ImKppNdR3IL6LTWWOxw4CPg9cIekmyNieY9lDgUOpLiP0M+BoyXdB/wl8OZU6+ht2nGzOvjIwuyVykNR5SGov5J0F8VtQQ6k+FKbeh0HXJxugb4Q2L37vkg9LI6IJyLiBeC7FLdP6WlZRKyLiD9R3E5lAvAs8Afgcknvowgbs0HlIwuzV/o+cKGkw4CREXFXugHcJ4E/j4in0vDSzjX6lu+dU56/A8WX2byQ2XbPe+/UuhfPi6Xpl4HhEbFZ0uEUN7WbBnyU4suQzAaNjyzMSiKii+IW5Vey5ahid4rvLXhGUhvF9y/UslHSWyTtQDEs1O3HFG/gAEg6pJf+71LxXc4jgVMohpmy0veH7BERPwDOpbjRndmg8pGF2daupxgGmgYQEfdI+hXF3VEfovc38ZnATRTfcnYvMCq1fwy4RNKvKf7P3Qr8Q43+S4FrgAOAb9U4X9Gb3YAFknam+OKcf6qzn1ndfNdZMzPL8jCUmZllOSzMzCzLYWFmZlkOCzMzy3JYmJlZlsPCzMyyHBZmZpb1/wGcKjNlYkHEGwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "if 'Init_Win_bytes_backward' in df.columns:\n",
    "    df['Init_Win_bytes_backward'].hist(figsize=(6,4), bins=10);\n",
    "    plt.title(\"Init_Win_bytes_backward\")\n",
    "    plt.xlabel(\"Value bins\")\n",
    "    plt.ylabel(\"Density\")\n",
    "    plt.savefig('Init_Win_bytes_backward.png', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAi8UlEQVR4nO3dfZwddXn38c/XgBCJIUFgDUlqUGM1EA1kG2MpdqNUItIGW73vUB7Crd6xFO9qm1aT6l2xNi9TK6ggYKNQAkTWVNCkPLTG2AW5G4iJInmA3ERYISQmPJOlNCXx6h/z22bYnN052bPnYcj3/Xqd15lzzfxmrjk5OdfOb+bMTxGBmZnZQF7R7ATMzKz1uViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhZmZFXKxsJYh6XZJcxqwnR5Jr69xHSHpjUOV01CS1CbpTkm7JF3S7HwGIqlb0mnNzsOKuVhY3VX7hRAR742IJanNBZLuqmLd75D0nKRhudg3+ol9PW1nREQ8NLi9qY2kDklb67yZucATwMiImFfnbdlBwsXCym4tMAw4ORc7FdjWJ/ZO4M4G5tVMrwM2xSB+cSvpkDrkU7f1WuO4WFjD9B4tSPqSpKclPSzpvbn5XZI+IuktwNeBd6Quo2f6W2dEvAjcTVYMkHQs8Erg231ibyIVi3wXkqRrJV0h6dbUbXOPpDdUuUtnSHpI0hOS/k7SKyQdJukpSZNz+3WspBckvQ64HTgu7VePpONSu/mSfi7pSUnLJB2V2h4u6YYUf0bSjyW1DfAeXwvMAT6Z1n9ayukrkralx1ckHZaW75C0VdKnJP0S+AdJd0j6gzT/t9L7dUZ6fZqke9P0GyT9MOX2hKSlkkblculO670PeF7SIZLOk/SL1ObTVb7P1gJcLKzR3g5sBo4GvghcLUn5BSLifuCPgNWpy2hUwTrvJBWG9HxXeuRjD0dEf90/ZwOfA0YDW4CFVe7L+4F2siOYWcCHImI30Amc22f9P4iIXwDvBbal/RoREduAPwHOAn4bOA54GrgitZ0DHAmMB15D9r680F9CEXEBsBT4Ylr/D4BPA9OBKcDbgGnAZ3LNXgscRXZEMhe4A+hI894JPJRy6319R5oW8IWU81tSjhf3Sels4H3AKLKCfRVwXmrzGmBcf/tircXFwhrtFxHxjYjYCywBxgD9/qVcpTuA30pF51TgR8BqYHoudscA7W+OiDURsYfsi3ZKldv924h4KiIeAb5C9sUI2X79oaTe/1/nAdcPsJ6PAp+OiK2p2FwMfCB13bxI9qX6xojYGxHrIuK5KvPrdQ7w1xGxMyIeJyuM5+Xm/wr4bETsjogXyN6rfHH4Qu71b6f5RMSWiFiZ2j0OXJpbrtdlEfFoWu8HgFsi4s60n/83bdtKwMXCGu2XvRMR8e9pckSN67w7reNEsi+3H0VED/BoLjbQ+Ypf5qb//QDyeTQ3/Quyv5aJiHuA54HflvRm4I3AigHW8zrgu6mb6RngfmAvWRG9HvgXoDN1IX1R0qFV5tfruJTffrkmj0fEf+RerwbelLq7pgDXAeMlHU12VNLbnXespE5Jj0l6DriB7IgxL/8eHZd/HRHPA08e4L5Yk7hYWKuq+uRs+qL7MXAmMCYiHkizfpRib6U+J7fH56Z/jeykeq8lZF1R5wHfyX0ZV9qvR4H3RsSo3OPwiHgsIl6MiM9FxCTgN8n25/wDzHMbWUHqL9eX5JSK+Drg48CGiPhP4N+APwN+HhFPpEW/kNq+NSJGpv19SZdin3VvJ/eeSXoV2VGTlYCLhbWqHcA4Sa+scvk7gU+Qfan1uivFfhkRPx/S7DJ/IWm0pPFkX6zfzs27nuycxrlkf5n32gG8RtKRudjXgYXpBDiSjpE0K03PkDQ5XQb8HFm31N4DzPNG4DNpvUcDf0V2FDCQO4CPsa/7rqvPa4BXAz3AM5LGAn9RsM7vAGemk+avBP4afweVhv+hrFX9ENgI/FLSE0ULk32JHUtWIHrdlWL1umR2Odlf4PcCtwJX985IJ9N/QvaX9Y9y8QfIvrwfSt1OxwFfJeum+r6kXWTdam9PTV5L9iX7HFn31B0Uf9H39TdklxjfB6xPef1NQZs7yIrBnf28huzcx8nAs2T7f/NAK4yIjcBFwLfIjjKeBur9mxMbIvLgR2b1IekasiufPlO4sFmL8w9lzOpA0gTg94GTmpyK2ZBwN5SVgqSNuR+y5R/n1Gl7p/azvZ4q2n4e2AD8XUQ8XKf8KuYm6dR6bM/M3VBmZlbIRxZmZlboZXvO4uijj44JEyYMqu3zzz/PEUccMbQJNUBZ84by5l7WvKG8uZc1byhH7uvWrXsiIo7pG3/ZFosJEyawdu3aQbXt6uqio6NjaBNqgLLmDeXNvax5Q3lzL2veUI7cJf2iUtzdUGZmVsjFwszMCrlYmJlZoboXC0nDJP1U0i3p9VGSVkp6MD2Pzi27QNIWSZslnZ6LT5W0Ps27rO/4B2ZmVl+NOLL4ONk9bXrNB1ZFxERgVXqNpEnAbOAEYCZwpfaNoXwV2aAsE9NjZgPyNjOzpK7FQtI4slGyvpkLzyK7fTPp+axcvDMNpPIw2Yhl0ySNIRt4fnUaU/i6XBszM2uAel86+xXgk2R3q+zVFhHbASJiu7LxkQHGkt1ts9fWFHuRl96Zsje+H0lzyY5AaGtro6ura1BJ9/T0DLptM5U1byhv7mXNG8qbe1nzhnLnXrdiIelMYGdErJPUUU2TCrEYIL5/MGIxsBigvb09Bns9cxmuha6krHlDeXMva95Q3tzLmjeUO/d6HlmcAvyepDOAw4GRkm4Adkgak44qxgA70/JbeenIY+PIRvPayksHde+Nm5lZg9StWETEAmABQDqy+POIOFfS3wFzgEXpeXlqsgL4lqRLycbqnQisiYi9knZJmg7cQzak5OX1yhtg/WPPcsH8W+u5iYq6F72v4ds0M6tGM273sQhYJunDwCPAByEbRUvSMmATsAe4KCJ6h4+8ELgWGA7cnh5mZtYgDSkWEdFFNoYvEfEk8O5+llsILKwQXwucWL8MzcxsIP4Ft5mZFXKxMDOzQi4WZmZWyMXCzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhZmZFXKxMDOzQi4WZmZWyMXCzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhZmZFapbsZB0uKQ1kn4maaOkz6X4xZIek3RvepyRa7NA0hZJmyWdnotPlbQ+zbtMkuqVt5mZ7a+ew6ruBt4VET2SDgXuktQ7dvaXI+JL+YUlTQJmAycAxwE/kPSmNA73VcBc4G7gNmAmHofbzKxh6nZkEZme9PLQ9IgBmswCOiNid0Q8DGwBpkkaA4yMiNUREcB1wFn1ytvMzPan7Pu3TiuXhgHrgDcCV0TEpyRdDFwAPAesBeZFxNOSvgbcHRE3pLZXkx09dAOLIuK0FD8V+FREnFlhe3PJjkBoa2ub2tnZOai8dz71LDteGFTTmkwee2RN7Xt6ehgxYsQQZdNYZc29rHlDeXMva95QjtxnzJixLiLa+8br2Q1F6kKaImkU8F1JJ5J1KX2e7Cjj88AlwIeASuchYoB4pe0tBhYDtLe3R0dHx6Dyvnzpci5ZX9e3pqLuczpqat/V1cVg97nZypp7WfOG8uZe1ryh3Lk35GqoiHgG6AJmRsSOiNgbEb8CvgFMS4ttBcbnmo0DtqX4uApxMzNrkHpeDXVMOqJA0nDgNOCBdA6i1/uBDWl6BTBb0mGSjgcmAmsiYjuwS9L0dBXU+cDyeuVtZmb7q2dfyxhgSTpv8QpgWUTcIul6SVPIupK6gY8CRMRGScuATcAe4KLUjQVwIXAtMJzsPIavhDIza6C6FYuIuA84qUL8vAHaLAQWVoivBU4c0gTNzKxq/gW3mZkVcrEwM7NCLhZmZlbIxcLMzAq5WJiZWSEXCzMzK+RiYWZmhVwszMyskIuFmZkVcrEwM7NCLhZmZlbIxcLMzAq5WJiZWSEXCzMzK+RiYWZmhVwszMyskIuFmZkVqucY3IdLWiPpZ5I2Svpcih8laaWkB9Pz6FybBZK2SNos6fRcfKqk9WneZWksbjMza5B6HlnsBt4VEW8DpgAzJU0H5gOrImIisCq9RtIkYDZwAjATuDKN3w1wFTAXmJgeM+uYt5mZ9VG3YhGZnvTy0PQIYBawJMWXAGel6VlAZ0TsjoiHgS3ANEljgJERsToiArgu18bMzBrgkHquPB0ZrAPeCFwREfdIaouI7QARsV3SsWnxscDdueZbU+zFNN03Xml7c8mOQGhra6Orq2tQebcNh3mT9wyqbS0Gm2+vnp6emtfRLGXNvax5Q3lzL2veUO7c61osImIvMEXSKOC7kk4cYPFK5yFigHil7S0GFgO0t7dHR0fHAeXb6/Kly7lkfV3fmoq6z+moqX1XVxeD3edmK2vuZc0bypt7WfOGcufekKuhIuIZoIvsXMOO1LVEet6ZFtsKjM81GwdsS/FxFeJmZtYg9bwa6ph0RIGk4cBpwAPACmBOWmwOsDxNrwBmSzpM0vFkJ7LXpC6rXZKmp6ugzs+1MTOzBqhnX8sYYEk6b/EKYFlE3CJpNbBM0oeBR4APAkTERknLgE3AHuCi1I0FcCFwLTAcuD09zMysQepWLCLiPuCkCvEngXf302YhsLBCfC0w0PkOMzOrI/+C28zMCrlYmJlZIRcLMzMr5GJhZmaFXCzMzKyQi4WZmRVysTAzs0IuFmZmVsjFwszMCrlYmJlZIRcLMzMr5GJhZmaFXCzMzKyQi4WZmRVysTAzs0IuFmZmVsjFwszMCtVzDO7xkv5V0v2SNkr6eIpfLOkxSfemxxm5NgskbZG0WdLpufhUSevTvMvSWNxmZtYg9RyDew8wLyJ+IunVwDpJK9O8L0fEl/ILS5oEzAZOAI4DfiDpTWkc7quAucDdwG3ATDwOt5lZw9TtyCIitkfET9L0LuB+YOwATWYBnRGxOyIeBrYA0ySNAUZGxOqICOA64Kx65W1mZvtryDkLSROAk4B7Uuhjku6TdI2k0Sk2Fng012xrio1N033jZmbWIMr+WK/jBqQRwB3Awoi4WVIb8AQQwOeBMRHxIUlXAKsj4obU7mqyLqdHgC9ExGkpfirwyYj43QrbmkvWXUVbW9vUzs7OQeW886ln2fHCoJrWZPLYI2tq39PTw4gRI4Yom8Yqa+5lzRvKm3tZ84Zy5D5jxox1EdHeN17PcxZIOhS4CVgaETcDRMSO3PxvALekl1uB8bnm44BtKT6uQnw/EbEYWAzQ3t4eHR0dg8r78qXLuWR9Xd+airrP6aipfVdXF4Pd52ZrRu4T5t9a8zrmTd7LJXc9f0Btuhe9r+btDoWyfl7KmjeUO/d6Xg0l4Grg/oi4NBcfk1vs/cCGNL0CmC3pMEnHAxOBNRGxHdglaXpa5/nA8nrlbWZm+6vnn8+nAOcB6yXdm2J/CZwtaQpZN1Q38FGAiNgoaRmwiexKqovSlVAAFwLXAsPJroLylVBmZg1Ut2IREXcBlX4PcdsAbRYCCyvE1wInDl12ZmZ2IPwLbjMzK1RVsZB0k6T3SXJxMTM7CFX75X8V8IfAg5IWSXpzHXMyM7MWU1WxiIgfRMQ5wMlkJ6VXSvo3Sf8rXR5rZmYvY1V3K0l6DXAB8BHgp8BXyYrHygGamZnZy0BVV0NJuhl4M3A98Lvptw8A35a0tl7JmZlZa6j20tlvRsRLLnmVdFi66d9+Pws3M7OXl2q7of6mQmz1UCZiZmata8AjC0mvJbvD63BJJ7HvR3YjgVfVOTczM2sRRd1Qp5Od1B4HXJqL7yK7dYeZmR0EBiwWEbEEWCLpDyLipgblZGZmLaaoG+rcNL7EBEl/1nd+/m6yZmb28lXUDXVEem7t0TrMzKyuirqh/j49f64x6ZiZWSuq9kaCX5Q0UtKhklZJekLSufVOzszMWkO1v7N4T0Q8B5xJNszpm4C/qFtWZmbWUqotFr03CzwDuDEinqpTPmZm1oKqvd3HP0l6AHgB+GNJxwD/Ub+0zMyslVR7i/L5wDuA9oh4EXgemDVQG0njJf2rpPslbZT08RQ/StJKSQ+m59G5NgskbZG0WdLpufhUSevTvMskVRqu1czM6uRARr57C/A/JZ0PfAB4T8Hye4B5EfEWYDpwkaRJwHxgVURMBFal16R5s4ETgJnAlZKGpXVdBcwFJqbHzAPI28zMalTtLcqvB94A3AvsTeEAruuvTbqN+fY0vUvS/WT3mZoFdKTFlgBdwKdSvDMidgMPS9oCTJPUDYyMiNUpl+uAs4Dbq9tFMzOrlSKieKHsi35SVLNw5fYTgDuBE4FHImJUbt7TETFa0teAu9MvxpF0NVlB6AYWRcRpKX4q8KmIOLPCduaSHYHQ1tY2tbOzczDpsvOpZ9nxwqCa1mTy2CNrat/T08OIEeX8/WQzcl//2LM1r6NtOAf8Wan133molPXzUta8oRy5z5gxY12loSeqPcG9AXgt6UjhQEgaAdwEfCIinhvgdEOlGTFAfP9gxGJgMUB7e3t0dHQcaLoAXL50OZesr/atGTrd53TU1L6rq4vB7nOzNSP3C+bfWvM65k3ec8CflVr/nYdKWT8vZc0byp17tZ/yo4FNktYAu3uDEfF7AzVK43PfBCyNiJtTeIekMRGxXdIYYGeKbwXG55qPA7al+LgKcTMza5Bqi8XFB7ridMXS1cD9fW44uAKYAyxKz8tz8W9JuhQ4juxE9pqI2Ctpl6TpwD3A+cDlB5qPmZkNXlXFIiLukPQ6YGJE/EDSq4BhBc1OAc4D1ku6N8X+kqxILJP0YeAR4INpGxslLQM2kV1JdVFE9J5MvxC4FhhOdh7DJ7fNzBqo2quh/jfZieOjyK6KGgt8HXh3f20i4i4qn2+gv3YRsRBYWCG+luzkuJmZNUG1v7O4iOxI4TmAiHgQOLZeSZmZWWuptljsjoj/7H0h6RD6uSLJzMxefqotFndI+ktguKTfAf4R+Kf6pWVmZq2k2mIxH3gcWA98FLgN+Ey9kjIzs9ZS7dVQv5L0PeB7EfF4fVMyM7NWM+CRhTIXS3oCeADYLOlxSX/VmPTMzKwVFHVDfYLsKqjfiIjXRMRRwNuBUyT9ab2TMzOz1lBULM4Hzo6Ih3sDEfEQcG6aZ2ZmB4GiYnFoRDzRN5jOWxxaYXkzM3sZKioW/znIeWZm9jJSdDXU2yQ9VyEu4PA65GNmZi1owGIREUU3CzQzs4PAgYzBbWZmBykXCzMzK+RiYWZmhVwszMyskIuFmZkVqluxkHSNpJ2SNuRiF0t6TNK96XFGbt4CSVskbZZ0ei4+VdL6NO+yNLa3mZk1UD2PLK4FZlaIfzkipqTHbQCSJgGzgRNSmysl9V62exXZkK4T06PSOs3MrI7qViwi4k7gqSoXnwV0RsTudB+qLcA0SWOAkRGxOiICuA44qy4Jm5lZv6oaz2KIfUzS+cBaYF5EPA2MBe7OLbM1xV5M033jFUmaS3YUQltbG11dXYNKsG04zJu8Z1BtazHYfHv19PTUvI5maUbuQ/FvPJjPSqv8G5X181LWvKHcuTe6WFwFfJ5s/O7PA5cAHyK7fUhfMUC8oohYDCwGaG9vj46OjkElefnS5VyyvvF1tPucjprad3V1Mdh9brZm5H7B/FtrXse8yXsO+LNS67/zUCnr56WseUO5c2/o1VARsSMi9kbEr4BvANPSrK3A+Nyi44BtKT6uQtzMzBqoocUinYPo9X6g90qpFcBsSYdJOp7sRPaaiNgO7JI0PV0FdT6wvJE5m5lZHbuhJN0IdABHS9oKfBbokDSFrCupG/goQERslLQM2ATsAS6KiL1pVReSXVk1HLg9PczMrIHqViwi4uwK4asHWH4hsLBCfC1w4hCmZmZmB8i/4DYzs0IuFmZmVsjFwszMCrlYmJlZIRcLMzMr5GJhZmaFXCzMzKyQi4WZmRVysTAzs0IuFmZmVsjFwszMCrlYmJlZIRcLMzMr5GJhZmaFXCzMzKyQi4WZmRVysTAzs0J1KxaSrpG0U9KGXOwoSSslPZieR+fmLZC0RdJmSafn4lMlrU/zLktjcZuZWQPV88jiWmBmn9h8YFVETARWpddImgTMBk5Iba6UNCy1uQqYC0xMj77rNDOzOqtbsYiIO4Gn+oRnAUvS9BLgrFy8MyJ2R8TDwBZgmqQxwMiIWB0RAVyXa2NmZg2i7Du4TiuXJgC3RMSJ6fUzETEqN//piBgt6WvA3RFxQ4pfDdwOdAOLIuK0FD8V+FREnNnP9uaSHYXQ1tY2tbOzc1B573zqWXa8MKimNZk89sia2vf09DBixIghyqaxmpH7+seerXkdbcM54M9Krf/OQ6Wsn5ey5g3lyH3GjBnrIqK9b/yQZiRTQaXzEDFAvKKIWAwsBmhvb4+Ojo5BJXP50uVcsr7xb033OR01te/q6mKw+9xszcj9gvm31ryOeZP3HPBnpdZ/56FS1s9LWfOGcufe6KuhdqSuJdLzzhTfCozPLTcO2Jbi4yrEzcysgRpdLFYAc9L0HGB5Lj5b0mGSjic7kb0mIrYDuyRNT1dBnZ9rY2ZmDVK3vhZJNwIdwNGStgKfBRYByyR9GHgE+CBARGyUtAzYBOwBLoqIvWlVF5JdWTWc7DzG7fXK2czMKqtbsYiIs/uZ9e5+ll8ILKwQXwucOISpmZnZAfIvuM3MrJCLhZmZFXKxMDOzQi4WZmZWyMXCzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhZmZFXKxMDOzQq0ynoUBE2ocX2He5D2DHqOhe9H7atq2mb28+cjCzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrFBTioWkbknrJd0raW2KHSVppaQH0/Po3PILJG2RtFnS6c3I2czsYNbMI4sZETElItrT6/nAqoiYCKxKr5E0CZgNnADMBK6UNKwZCZuZHaxaqRtqFrAkTS8BzsrFOyNid0Q8DGwBpjU+PTOzg1ezikUA35e0TtLcFGuLiO0A6fnYFB8LPJpruzXFzMysQRQRjd+odFxEbJN0LLAS+D/AiogYlVvm6YgYLekKYHVE3JDiVwO3RcRNFdY7F5gL0NbWNrWzs3NQ+e186ll2vDCopk3VNpxB5z157JFDm8wB6unpYcSIEQ3d5vrHnq15HYN5z5v9Xvdqxns+FMqaN5Qj9xkzZqzLnR74b0253UdEbEvPOyV9l6xbaYekMRGxXdIYYGdafCswPtd8HLCtn/UuBhYDtLe3R0dHx6Dyu3zpci5ZX747ocybvGfQeXef0zG0yRygrq4uBvvvNViDvTVK3mDe82a/172a8Z4PhbLmDeXOveHdUJKOkPTq3mngPcAGYAUwJy02B1ieplcAsyUdJul4YCKwprFZm5kd3Jrx53Mb8F1Jvdv/VkT8s6QfA8skfRh4BPggQERslLQM2ATsAS6KiL1NyNvM7KDV8GIREQ8Bb6sQfxJ4dz9tFgIL65yamZn1o5UunTUzsxblYmFmZoVcLMzMrJCLhZmZFXKxMDOzQi4WZmZWyMXCzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhZmZFXKxMDOzQi4WZmZWqHzDwZmZlcCECiMxzpu8Z0hGaBxI96L31WW9PrIwM7NCLhZmZlbIxcLMzAqVplhImilps6QtkuY3Ox8zs4NJKYqFpGHAFcB7gUnA2ZImNTcrM7ODR1muhpoGbImIhwAkdQKzgE1NzcpsECpdJdMMjbgyp1e9rtCxxlFENDuHQpI+AMyMiI+k1+cBb4+Ij/VZbi4wN738dWDzIDd5NPDEINs2U1nzhvLmXta8oby5lzVvKEfur4uIY/oGy3JkoQqx/apcRCwGFte8MWltRLTXup5GK2veUN7cy5o3lDf3suYN5c69FOcsgK3A+NzrccC2JuViZnbQKUux+DEwUdLxkl4JzAZWNDknM7ODRim6oSJij6SPAf8CDAOuiYiNddxkzV1ZTVLWvKG8uZc1byhv7mXNG0qceylOcJuZWXOVpRvKzMyayMXCzMwKuVjktMotRSRdI2mnpA252FGSVkp6MD2Pzs1bkHLeLOn0XHyqpPVp3mWSlOKHSfp2it8jacIQ5T1e0r9Kul/SRkkfL0Pukg6XtEbSz1LenytD3rltDpP0U0m3lCzv7rTNeyWtLUvukkZJ+o6kB9Jn/R1lyLtmEeFHdt5mGPBz4PXAK4GfAZOalMs7gZOBDbnYF4H5aXo+8LdpelLK9TDg+LQPw9K8NcA7yH6ncjvw3hT/Y+DraXo28O0hynsMcHKafjXw/1N+LZ172saINH0ocA8wvdXzzuX/Z8C3gFvK8llJ6+sGju4Ta/ncgSXAR9L0K4FRZci75v1udgKt8kj/aP+Se70AWNDEfCbw0mKxGRiTpscAmyvlSXbF2DvSMg/k4mcDf59fJk0fQvaLUtVhH5YDv1Om3IFXAT8B3l6GvMl+c7QKeBf7ikXL553W183+xaKlcwdGAg/3XU+r5z0UD3dD7TMWeDT3emuKtYq2iNgOkJ6PTfH+8h6bpvvGX9ImIvYAzwKvGcpk06HzSWR/pbd87qkr515gJ7AyIkqRN/AV4JPAr3KxMuQN2V0Yvi9pnbJb9ZQh99cDjwP/kLr+vinpiBLkXTMXi32quqVIC+ov74H2p677KmkEcBPwiYh4bqBF+8mj4blHxN6ImEL2l/o0SScOsHhL5C3pTGBnRKyrtkk/OTTrs3JKRJxMdjfpiyS9c4BlWyX3Q8i6iK+KiJOA58m6nfrTKnnXzMVin1a/pcgOSWMA0vPOFO8v761pum/8JW0kHQIcCTw1FElKOpSsUCyNiJvLlDtARDwDdAEzS5D3KcDvSeoGOoF3SbqhBHkDEBHb0vNO4Ltkd5du9dy3AlvTkSfAd8iKR6vnXTMXi31a/ZYiK4A5aXoO2fmA3vjsdAXF8cBEYE06FN4laXq6yuL8Pm161/UB4IeROkhrkbZzNXB/RFxaltwlHSNpVJoeDpwGPNDqeUfEgogYFxETyD6vP4yIc1s9bwBJR0h6de808B5gQ6vnHhG/BB6V9Osp9G6yoRJaOu8h0eyTJq30AM4gu4Ln58Cnm5jHjcB24EWyvzI+TNZnuQp4MD0flVv+0ynnzaQrKlK8new/4M+Br7HvF/uHA/8IbCG7IuP1Q5T3b5EdLt8H3JseZ7R67sBbgZ+mvDcAf5XiLZ13n33oYN8J7pbPm6zv/2fpsbH3/1tJcp8CrE2fl+8Bo8uQd60P3+7DzMwKuRvKzMwKuViYmVkhFwszMyvkYmFmZoVcLMzMrJCLhVkiqSt/V9AU+4SkKwvatA/Bti+Q9LV+5v1bres3q5WLhdk+N5L9uC1vdoo3TUT8ZjO3bwYuFmZ53wHOlHQY/PfNEI8D7pJ0laS1yo130Zekntz0ByRdm6aPkXSTpB+nxyn9bH+8pH9O4x58tu96JXWkI5nesRSW5sZAWCRpk6T7JH2p9rfC7KUOaXYCZq0iIp6UtIbsvlDL2TeWQEj6dEQ8JWkYsErSWyPivipX/VXgyxFxl6RfI7sF9VsqLDcNOBH4d+DHkm6NiLV9ljkJOIHsPkL/DzhF0ibg/cCbU66jDmjHzargIwuzl8p3ReW7oP6HpJ+Q3RbkBLJBbap1GvC1dAv0FcDI3vsi9bEyIp6MiBeAm8lun9LXmojYGhG/IrudygTgOeA/gG9K+n2yYmM2pHxkYfZS3wMulXQyMDwifpJuAPfnwG9ExNOpe+nwCm3z987Jz38F2WA2LxRsu++9dyrdi2d3bnovcEhE7JE0jeymdrOBj5ENhmQ2ZHxkYZYTET1ktyi/hn1HFSPJxi14VlIb2fgLleyQ9BZJryDrFur1fbIvcAAkTemn/e8oG8t5OHAWWTdToTR+yJERcRvwCbIb3ZkNKR9ZmO3vRrJuoNkAEfEzST8luzvqQ/T/JT4fuIVslLMNwIgU/xPgCkn3kf2fuxP4owrt7wKuB94IfKvC+Yr+vBpYLulwsoFz/rTKdmZV811nzcyskLuhzMyskIuFmZkVcrEwM7NCLhZmZlbIxcLMzAq5WJiZWSEXCzMzK/Rf/4oO0UqprMYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "if 'Init_Win_bytes_forward' in df.columns:\n",
    "    df['Init_Win_bytes_forward'].hist(figsize=(6,4), bins=10);\n",
    "    plt.title(\"Init_Win_bytes_forward\")\n",
    "    plt.xlabel(\"Value bins\")\n",
    "    plt.ylabel(\"Density\")\n",
    "    plt.savefig('Init_Win_bytes_forward.png', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "excluded2 = ['Init_Win_bytes_backward', 'Init_Win_bytes_forward']\n",
    "df = df.drop(columns=excluded2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Удаление столбцов с нулевой дисперсией"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7267, 65)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zero_variance_columns = [col for col in df.columns[:-1] if df[col].std()==0]\n",
    "df = df.drop(columns=zero_variance_columns)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Flow Duration', 'Total Fwd Packets', 'Total Backward Packets',\n",
       "       'Total Length of Fwd Packets', 'Total Length of Bwd Packets',\n",
       "       'Fwd Packet Length Max', 'Fwd Packet Length Min',\n",
       "       'Fwd Packet Length Mean', 'Fwd Packet Length Std',\n",
       "       'Bwd Packet Length Max', 'Bwd Packet Length Min',\n",
       "       'Bwd Packet Length Mean', 'Bwd Packet Length Std', 'Flow Bytes/s',\n",
       "       'Flow Packets/s', 'Flow IAT Mean', 'Flow IAT Std', 'Flow IAT Max',\n",
       "       'Flow IAT Min', 'Fwd IAT Total', 'Fwd IAT Mean', 'Fwd IAT Std',\n",
       "       'Fwd IAT Max', 'Fwd IAT Min', 'Bwd IAT Total', 'Bwd IAT Mean',\n",
       "       'Bwd IAT Std', 'Bwd IAT Max', 'Bwd IAT Min', 'Fwd PSH Flags',\n",
       "       'Fwd Header Length', 'Bwd Header Length', 'Fwd Packets/s',\n",
       "       'Bwd Packets/s', 'Min Packet Length', 'Max Packet Length',\n",
       "       'Packet Length Mean', 'Packet Length Std', 'Packet Length Variance',\n",
       "       'FIN Flag Count', 'SYN Flag Count', 'RST Flag Count', 'PSH Flag Count',\n",
       "       'ACK Flag Count', 'URG Flag Count', 'ECE Flag Count', 'Down/Up Ratio',\n",
       "       'Average Packet Size', 'Avg Fwd Segment Size', 'Avg Bwd Segment Size',\n",
       "       'Subflow Fwd Packets', 'Subflow Fwd Bytes', 'Subflow Bwd Packets',\n",
       "       'Subflow Bwd Bytes', 'act_data_pkt_fwd', 'min_seg_size_forward',\n",
       "       'Active Mean', 'Active Std', 'Active Max', 'Active Min', 'Idle Mean',\n",
       "       'Idle Std', 'Idle Max', 'Idle Min', 'Label'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Признаки, которые есть в CICIDS2017, но которых нет в тестовой модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Idle Mean',\n",
       " 'Fwd PSH Flags',\n",
       " 'FIN Flag Count',\n",
       " 'Active Mean',\n",
       " 'SYN Flag Count',\n",
       " 'Avg Fwd Segment Size',\n",
       " 'Subflow Fwd Packets',\n",
       " 'Subflow Bwd Bytes',\n",
       " 'PSH Flag Count',\n",
       " 'Active Min',\n",
       " 'Avg Bwd Segment Size',\n",
       " 'Down/Up Ratio',\n",
       " 'Total Backward Packets',\n",
       " 'Idle Min',\n",
       " 'Subflow Bwd Packets',\n",
       " 'min_seg_size_forward',\n",
       " 'Active Max',\n",
       " 'ACK Flag Count',\n",
       " 'ECE Flag Count',\n",
       " 'Idle Max',\n",
       " 'RST Flag Count',\n",
       " 'Subflow Fwd Bytes',\n",
       " 'URG Flag Count',\n",
       " 'Idle Std',\n",
       " 'Active Std',\n",
       " 'act_data_pkt_fwd']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " list(set(df.columns) - set(df_real.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Признаки, которых нет в CICIDS2017, но которые есть в тестовой модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Source IP',\n",
       " 'Flow Key',\n",
       " 'Total Packets',\n",
       " 'Source Port',\n",
       " 'Protocol',\n",
       " 'Flow MB/s',\n",
       " 'Destination IP',\n",
       " 'Average Fwd Segment Size',\n",
       " 'Timestamp',\n",
       " 'Destination Port',\n",
       " 'Average Bwd Segment Size',\n",
       " 'Total Backword Packets',\n",
       " 'Packet Length List']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " list(set(df_real.columns) - set(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Общие признаки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = list(set.intersection(set(set(df.columns)), set(set(df_real.columns))))\n",
    "features.remove('Label')\n",
    "len(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7267, 38) (7267,)\n"
     ]
    }
   ],
   "source": [
    "y = df.Label.astype(\"category\").cat.codes\n",
    "X = df[features]\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "СОХРАНЕНИЕ И НОРМИРОВКА ДАННЫХ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Bwd IAT Min                   -1.819887e-15\n",
       " Packet Length Variance        -2.340329e-15\n",
       " Bwd IAT Total                 -1.376634e-15\n",
       " Fwd Packets/s                 -1.264259e-16\n",
       " Flow IAT Mean                 -2.347403e-16\n",
       " Fwd IAT Max                   -7.855741e-17\n",
       " Bwd Header Length              2.366538e-16\n",
       " Packet Length Mean            -3.471635e-15\n",
       " Total Length of Bwd Packets    5.754259e-17\n",
       " Fwd Header Length             -3.158261e-16\n",
       " Total Fwd Packets             -6.986655e-16\n",
       " Bwd Packet Length Std         -1.403079e-15\n",
       " Flow IAT Std                  -3.625481e-15\n",
       " Bwd Packet Length Mean         8.341798e-16\n",
       " Fwd Packet Length Mean        -2.294084e-16\n",
       " Average Packet Size           -3.471315e-15\n",
       " Min Packet Length              4.202164e-15\n",
       " Max Packet Length             -7.897525e-16\n",
       " Bwd Packets/s                 -8.252825e-16\n",
       " Fwd IAT Std                   -9.614498e-16\n",
       " Flow Duration                  4.170784e-18\n",
       " Bwd Packet Length Max         -2.505396e-15\n",
       " Fwd IAT Total                  6.610158e-16\n",
       " Flow Bytes/s                  -4.618714e-16\n",
       " Fwd Packet Length Std         -3.073242e-15\n",
       " Flow IAT Max                   2.261084e-17\n",
       " Total Length of Fwd Packets   -1.718806e-16\n",
       " Packet Length Std             -6.592589e-15\n",
       " Fwd IAT Mean                   3.269100e-16\n",
       " Bwd IAT Mean                  -1.063802e-15\n",
       " Bwd IAT Std                    1.085886e-15\n",
       " Bwd IAT Max                   -9.879946e-16\n",
       " Flow Packets/s                -7.450502e-17\n",
       " Fwd Packet Length Max          6.344863e-16\n",
       " Fwd IAT Min                    3.324329e-16\n",
       " Flow IAT Min                   3.705285e-16\n",
       " Bwd Packet Length Min         -3.305385e-15\n",
       " Fwd Packet Length Min         -2.731191e-15\n",
       " dtype: float64,\n",
       " Bwd IAT Min                    1.0\n",
       " Packet Length Variance         1.0\n",
       " Bwd IAT Total                  1.0\n",
       " Fwd Packets/s                  1.0\n",
       " Flow IAT Mean                  1.0\n",
       " Fwd IAT Max                    1.0\n",
       " Bwd Header Length              1.0\n",
       " Packet Length Mean             1.0\n",
       " Total Length of Bwd Packets    1.0\n",
       " Fwd Header Length              1.0\n",
       " Total Fwd Packets              1.0\n",
       " Bwd Packet Length Std          1.0\n",
       " Flow IAT Std                   1.0\n",
       " Bwd Packet Length Mean         1.0\n",
       " Fwd Packet Length Mean         1.0\n",
       " Average Packet Size            1.0\n",
       " Min Packet Length              1.0\n",
       " Max Packet Length              1.0\n",
       " Bwd Packets/s                  1.0\n",
       " Fwd IAT Std                    1.0\n",
       " Flow Duration                  1.0\n",
       " Bwd Packet Length Max          1.0\n",
       " Fwd IAT Total                  1.0\n",
       " Flow Bytes/s                   1.0\n",
       " Fwd Packet Length Std          1.0\n",
       " Flow IAT Max                   1.0\n",
       " Total Length of Fwd Packets    1.0\n",
       " Packet Length Std              1.0\n",
       " Fwd IAT Mean                   1.0\n",
       " Bwd IAT Mean                   1.0\n",
       " Bwd IAT Std                    1.0\n",
       " Bwd IAT Max                    1.0\n",
       " Flow Packets/s                 1.0\n",
       " Fwd Packet Length Max          1.0\n",
       " Fwd IAT Min                    1.0\n",
       " Flow IAT Min                   1.0\n",
       " Bwd Packet Length Min          1.0\n",
       " Fwd Packet Length Min          1.0\n",
       " dtype: float64)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = ((X-X.mean())/X.std())\n",
    "X.mean(), X.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разбивка на тестовую/вал/обуч выборки"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4096, 38) (990, 38)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, train_size=4096, random_state=42, stratify=y_train)\n",
    "print(X_train.shape, X_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Правильно ли соблюдены пропорции для каждой из выборок"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0.69995117, 0.20751953, 0.00292969, 0.08959961]),\n",
       " array([0.7       , 0.20707071, 0.0030303 , 0.08989899]),\n",
       " array([0.70013755, 0.20724438, 0.00275103, 0.08986703])]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_count = [np.unique(y, return_counts=True)[1] for y in [y_train, y_val, y_test] ]\n",
    "proportions = [a/a.sum() for a in y_count]\n",
    "proportions\n",
    "# np.unique(y_val, return_counts=True)[1], np.unique(y_test, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bwd IAT Min</th>\n",
       "      <th>Packet Length Variance</th>\n",
       "      <th>Bwd IAT Total</th>\n",
       "      <th>Fwd Packets/s</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Fwd IAT Max</th>\n",
       "      <th>Bwd Header Length</th>\n",
       "      <th>Packet Length Mean</th>\n",
       "      <th>Total Length of Bwd Packets</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>...</th>\n",
       "      <th>Fwd IAT Mean</th>\n",
       "      <th>Bwd IAT Mean</th>\n",
       "      <th>Bwd IAT Std</th>\n",
       "      <th>Bwd IAT Max</th>\n",
       "      <th>Flow Packets/s</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Bwd Packet Length Min</th>\n",
       "      <th>Fwd Packet Length Min</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4795</th>\n",
       "      <td>-0.0974</td>\n",
       "      <td>-0.293773</td>\n",
       "      <td>-0.304609</td>\n",
       "      <td>-0.185161</td>\n",
       "      <td>0.205616</td>\n",
       "      <td>0.097172</td>\n",
       "      <td>-0.042216</td>\n",
       "      <td>-0.525794</td>\n",
       "      <td>-0.024329</td>\n",
       "      <td>-0.046245</td>\n",
       "      <td>...</td>\n",
       "      <td>0.08995</td>\n",
       "      <td>-0.156323</td>\n",
       "      <td>-0.223877</td>\n",
       "      <td>-0.225833</td>\n",
       "      <td>-0.202216</td>\n",
       "      <td>-0.36587</td>\n",
       "      <td>-0.098749</td>\n",
       "      <td>-0.041573</td>\n",
       "      <td>-0.544157</td>\n",
       "      <td>-0.518141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Bwd IAT Min  Packet Length Variance  Bwd IAT Total  Fwd Packets/s  \\\n",
       "4795      -0.0974               -0.293773      -0.304609      -0.185161   \n",
       "\n",
       "      Flow IAT Mean  Fwd IAT Max  Bwd Header Length  Packet Length Mean  \\\n",
       "4795       0.205616     0.097172          -0.042216           -0.525794   \n",
       "\n",
       "      Total Length of Bwd Packets  Fwd Header Length  ...  Fwd IAT Mean  \\\n",
       "4795                    -0.024329          -0.046245  ...       0.08995   \n",
       "\n",
       "      Bwd IAT Mean  Bwd IAT Std  Bwd IAT Max  Flow Packets/s  \\\n",
       "4795     -0.156323    -0.223877    -0.225833       -0.202216   \n",
       "\n",
       "      Fwd Packet Length Max  Fwd IAT Min  Flow IAT Min  Bwd Packet Length Min  \\\n",
       "4795               -0.36587    -0.098749     -0.041573              -0.544157   \n",
       "\n",
       "      Fwd Packet Length Min  \n",
       "4795              -0.518141  \n",
       "\n",
       "[1 rows x 38 columns]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реальные данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1407, 38) (1407,)\n"
     ]
    }
   ],
   "source": [
    "y_real = df_real.Label\n",
    "X_real = df_real[features]\n",
    "print(X_real.shape, y_real.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Bwd IAT Min                   -0.085427\n",
       " Packet Length Variance         0.193969\n",
       " Bwd IAT Total                 -0.010453\n",
       " Fwd Packets/s                 -0.178292\n",
       " Flow IAT Mean                  0.175693\n",
       " Fwd IAT Max                   -0.039275\n",
       " Bwd Header Length              0.021730\n",
       " Packet Length Mean             0.488307\n",
       " Total Length of Bwd Packets    0.008164\n",
       " Fwd Header Length             -0.006146\n",
       " Total Fwd Packets              0.011853\n",
       " Bwd Packet Length Std          0.296452\n",
       " Flow IAT Std                  -0.149821\n",
       " Bwd Packet Length Mean         0.349549\n",
       " Fwd Packet Length Mean         0.282716\n",
       " Average Packet Size            0.395465\n",
       " Min Packet Length             -0.452151\n",
       " Max Packet Length              0.168065\n",
       " Bwd Packets/s                 -0.109963\n",
       " Fwd IAT Std                   -0.256956\n",
       " Flow Duration                 -0.001322\n",
       " Bwd Packet Length Max          0.183629\n",
       " Fwd IAT Total                 -0.061357\n",
       " Flow Bytes/s                  -0.039552\n",
       " Fwd Packet Length Std          0.326773\n",
       " Flow IAT Max                   0.060641\n",
       " Total Length of Fwd Packets    0.201883\n",
       " Packet Length Std              0.318875\n",
       " Fwd IAT Mean                  -0.033607\n",
       " Bwd IAT Mean                  -0.088503\n",
       " Bwd IAT Std                    0.091471\n",
       " Bwd IAT Max                    0.058659\n",
       " Flow Packets/s                -0.192934\n",
       " Fwd Packet Length Max          0.181308\n",
       " Fwd IAT Min                    0.059580\n",
       " Flow IAT Min                   0.489277\n",
       " Bwd Packet Length Min         -0.534467\n",
       " Fwd Packet Length Min         -0.390923\n",
       " dtype: float64,\n",
       " Bwd IAT Min                    0.316303\n",
       " Packet Length Variance         0.783639\n",
       " Bwd IAT Total                  0.853159\n",
       " Fwd Packets/s                  0.037438\n",
       " Flow IAT Mean                  2.290312\n",
       " Fwd IAT Max                    1.040111\n",
       " Bwd Header Length              0.465201\n",
       " Packet Length Mean             1.599348\n",
       " Total Length of Bwd Packets    0.244724\n",
       " Fwd Header Length              0.240596\n",
       " Total Fwd Packets              0.252165\n",
       " Bwd Packet Length Std          1.056845\n",
       " Flow IAT Std                   0.608691\n",
       " Bwd Packet Length Mean         1.413386\n",
       " Fwd Packet Length Mean         1.254750\n",
       " Average Packet Size            1.522293\n",
       " Min Packet Length              0.107742\n",
       " Max Packet Length              0.884702\n",
       " Bwd Packets/s                  0.117029\n",
       " Fwd IAT Std                    0.910806\n",
       " Flow Duration                  0.878281\n",
       " Bwd Packet Length Max          0.917912\n",
       " Fwd IAT Total                  0.866739\n",
       " Flow Bytes/s                   0.002609\n",
       " Fwd Packet Length Std          1.090266\n",
       " Flow IAT Max                   1.093159\n",
       " Total Length of Fwd Packets    1.422616\n",
       " Packet Length Std              1.185634\n",
       " Fwd IAT Mean                   1.033941\n",
       " Bwd IAT Mean                   0.339712\n",
       " Bwd IAT Std                    0.973610\n",
       " Bwd IAT Max                    0.918212\n",
       " Flow Packets/s                 0.057569\n",
       " Fwd Packet Length Max          0.934863\n",
       " Fwd IAT Min                    1.047806\n",
       " Flow IAT Min                   3.599581\n",
       " Bwd Packet Length Min          0.023329\n",
       " Fwd Packet Length Min          0.152309\n",
       " dtype: float64)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df[features]\n",
    "X_real = ((X_real-X.mean())/X.std())\n",
    "X_real.mean(), X_real.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сохранение данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "DATASET = 'cicids2017_webattacks_realdata'\n",
    "filepath = '/home/student/projects/data/' + DATASET +'/'\n",
    "Path(filepath).mkdir(parents=True, exist_ok=True)\n",
    "np.save(filepath + 'data-train',np.c_[X_train, y_train])\n",
    "np.save(filepath + 'data-val',np.c_[X_val, y_val])\n",
    "np.save(filepath + 'data-test',np.c_[X_test, y_test])\n",
    "np.save(filepath + 'data-real',np.c_[X_real, y_real])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4096, 990, 2181, 1407]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(y) for y in [y_train, y_val, y_test, y_real]] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4096, 990, 2181, 1407]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(x) for x in [X_train, X_val, X_test, X_real]] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'BENIGN',\n",
       " 1: 'Web Attack � Brute Force',\n",
       " 2: 'Web Attack � Sql Injection',\n",
       " 3: 'Web Attack � XSS'}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict( enumerate(pd.Categorical(df.Label).categories ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'Bwd IAT Min',\n",
       " 1: 'Packet Length Variance',\n",
       " 2: 'Bwd IAT Total',\n",
       " 3: 'Fwd Packets/s',\n",
       " 4: 'Flow IAT Mean',\n",
       " 5: 'Fwd IAT Max',\n",
       " 6: 'Bwd Header Length',\n",
       " 7: 'Packet Length Mean',\n",
       " 8: 'Total Length of Bwd Packets',\n",
       " 9: 'Fwd Header Length',\n",
       " 10: 'Total Fwd Packets',\n",
       " 11: 'Bwd Packet Length Std',\n",
       " 12: 'Flow IAT Std',\n",
       " 13: 'Bwd Packet Length Mean',\n",
       " 14: 'Fwd Packet Length Mean',\n",
       " 15: 'Average Packet Size',\n",
       " 16: 'Min Packet Length',\n",
       " 17: 'Max Packet Length',\n",
       " 18: 'Bwd Packets/s',\n",
       " 19: 'Fwd IAT Std',\n",
       " 20: 'Flow Duration',\n",
       " 21: 'Bwd Packet Length Max',\n",
       " 22: 'Fwd IAT Total',\n",
       " 23: 'Flow Bytes/s',\n",
       " 24: 'Fwd Packet Length Std',\n",
       " 25: 'Flow IAT Max',\n",
       " 26: 'Total Length of Fwd Packets',\n",
       " 27: 'Packet Length Std',\n",
       " 28: 'Fwd IAT Mean',\n",
       " 29: 'Bwd IAT Mean',\n",
       " 30: 'Bwd IAT Std',\n",
       " 31: 'Bwd IAT Max',\n",
       " 32: 'Flow Packets/s',\n",
       " 33: 'Fwd Packet Length Max',\n",
       " 34: 'Fwd IAT Min',\n",
       " 35: 'Flow IAT Min',\n",
       " 36: 'Bwd Packet Length Min',\n",
       " 37: 'Fwd Packet Length Min'}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict( enumerate(features ) )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}