tybalt_twohidden.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Tybalt models with two hidden layers\n",
    "\n",
    "**Gregory Way 2017**\n",
    "\n",
    "This script is an extension of [tybalt_vae.ipynb](tybalt_vae.ipynb). See that script for more details about the base model. Here, I train two alternative Tybalt models with different architectures. Both architectures have two hidden layers:\n",
    "\n",
    "1. **Model A**: 5000 input -> 100 hidden -> 100 latent -> 100 hidden -> 5000 input\n",
    "2. **Model B**: 5000 input -> 300 hidden -> 100 latent -> 300 hidden -> 5000 input\n",
    "\n",
    "This notebook trains _both_ models. The optimal hyperparameters were selected through a grid search for each model independently.\n",
    "\n",
    "The original tybalt model compressed 5000 input genes into 100 latent features in a single layer.\n",
    "\n",
    "Much of this script is inspired by the [keras variational_autoencoder.py example](https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py)\n",
    "\n",
    "## Output\n",
    "\n",
    "For both models, the script will output:\n",
    "\n",
    "1. The learned latent feature matrix\n",
    "2. Encoder and Decoder keras models with pretrained weights\n",
    "3. An abstracted weight matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "from keras.layers import Input, Dense, Lambda, Layer, Activation\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.models import Model, Sequential\n",
    "from keras import backend as K\n",
    "from keras import metrics, optimizers\n",
    "from keras.callbacks import Callback\n",
    "import keras\n",
    "\n",
    "import pydot\n",
    "import graphviz\n",
    "from keras.utils import plot_model\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1.2.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(keras.__version__)\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.style.use('seaborn-notebook')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Functions and Classes\n",
    "\n",
    "This will facilitate connections between layers and also custom hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Function for reparameterization trick to make model differentiable\n",
    "def sampling(args):\n",
    "    \n",
    "    import tensorflow as tf\n",
    "    # Function with args required for Keras Lambda function\n",
    "    z_mean, z_log_var = args\n",
    "\n",
    "    # Draw epsilon of the same shape from a standard normal distribution\n",
    "    epsilon = K.random_normal(shape=tf.shape(z_mean), mean=0.,\n",
    "                              stddev=epsilon_std)\n",
    "    \n",
    "    # The latent vector is non-deterministic and differentiable\n",
    "    # in respect to z_mean and z_log_var\n",
    "    z = z_mean + K.exp(z_log_var / 2) * epsilon\n",
    "    return z\n",
    "\n",
    "\n",
    "class CustomVariationalLayer(Layer):\n",
    "    \"\"\"\n",
    "    Define a custom layer that learns and performs the training\n",
    "\n",
    "    \"\"\"\n",
    "    def __init__(self, var_layer, mean_layer, **kwargs):\n",
    "        # https://keras.io/layers/writing-your-own-keras-layers/\n",
    "        self.is_placeholder = True\n",
    "        self.var_layer = var_layer\n",
    "        self.mean_layer = mean_layer\n",
    "        super(CustomVariationalLayer, self).__init__(**kwargs)\n",
    "\n",
    "    def vae_loss(self, x_input, x_decoded):\n",
    "        reconstruction_loss = original_dim * metrics.binary_crossentropy(x_input, x_decoded)\n",
    "        kl_loss = - 0.5 * K.sum(1 + self.var_layer - K.square(self.mean_layer) - \n",
    "                                K.exp(self.var_layer), axis=-1)\n",
    "        return K.mean(reconstruction_loss + (K.get_value(beta) * kl_loss))\n",
    "\n",
    "    def call(self, inputs):\n",
    "        x = inputs[0]\n",
    "        x_decoded = inputs[1]\n",
    "        loss = self.vae_loss(x, x_decoded)\n",
    "        self.add_loss(loss, inputs=inputs)\n",
    "        # We won't actually use the output.\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing Warm-up as described in Sonderby et al. LVAE\n",
    "\n",
    "This is modified code from https://github.com/fchollet/keras/issues/2595"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class WarmUpCallback(Callback):\n",
    "    def __init__(self, beta, kappa):\n",
    "        self.beta = beta\n",
    "        self.kappa = kappa\n",
    "    # Behavior on each epoch\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        if K.get_value(self.beta) <= 1:\n",
    "            K.set_value(self.beta, K.get_value(self.beta) + self.kappa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tybalt Model\n",
    "\n",
    "The following class implements a Tybalt model with given input hyperparameters. Currently, only a two hidden layer model is supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Tybalt():\n",
    "    \"\"\"\n",
    "    Facilitates the training and output of tybalt model trained on TCGA RNAseq gene expression data\n",
    "    \"\"\"\n",
    "    def __init__(self, original_dim, hidden_dim, latent_dim,\n",
    "                 batch_size, epochs, learning_rate, kappa, beta):\n",
    "        self.original_dim = original_dim\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.latent_dim = latent_dim\n",
    "        self.batch_size = batch_size\n",
    "        self.epochs = epochs\n",
    "        self.learning_rate = learning_rate\n",
    "        self.kappa = kappa\n",
    "        self.beta = beta\n",
    "\n",
    "    def build_encoder_layer(self):\n",
    "        # Input place holder for RNAseq data with specific input size\n",
    "        self.rnaseq_input = Input(shape=(self.original_dim, ))\n",
    "\n",
    "        # Input layer is compressed into a mean and log variance vector of size `latent_dim`\n",
    "        # Each layer is initialized with glorot uniform weights and each step (dense connections, batch norm,\n",
    "        # and relu activation) are funneled separately\n",
    "        # Each vector of length `latent_dim` are connected to the rnaseq input tensor\n",
    "        hidden_dense_linear = Dense(self.hidden_dim, kernel_initializer='glorot_uniform')(self.rnaseq_input)\n",
    "        hidden_dense_batchnorm = BatchNormalization()(hidden_dense_linear)\n",
    "        hidden_encoded = Activation('relu')(hidden_dense_batchnorm)\n",
    "\n",
    "        z_mean_dense_linear = Dense(self.latent_dim, kernel_initializer='glorot_uniform')(hidden_encoded)\n",
    "        z_mean_dense_batchnorm = BatchNormalization()(z_mean_dense_linear)\n",
    "        self.z_mean_encoded = Activation('relu')(z_mean_dense_batchnorm)\n",
    "\n",
    "        z_log_var_dense_linear = Dense(self.latent_dim, kernel_initializer='glorot_uniform')(hidden_encoded)\n",
    "        z_log_var_dense_batchnorm = BatchNormalization()(z_log_var_dense_linear)\n",
    "        self.z_log_var_encoded = Activation('relu')(z_log_var_dense_batchnorm)\n",
    "\n",
    "        # return the encoded and randomly sampled z vector\n",
    "        # Takes two keras layers as input to the custom sampling function layer with a `latent_dim` output\n",
    "        self.z = Lambda(sampling, output_shape=(self.latent_dim, ))([self.z_mean_encoded, self.z_log_var_encoded])\n",
    "    \n",
    "    def build_decoder_layer(self):\n",
    "        # The decoding layer is much simpler with a single layer glorot uniform initialized and sigmoid activation\n",
    "        self.decoder_model = Sequential()\n",
    "        self.decoder_model.add(Dense(self.hidden_dim, activation='relu', input_dim=self.latent_dim))\n",
    "        self.decoder_model.add(Dense(self.original_dim, activation='sigmoid'))\n",
    "        self.rnaseq_reconstruct = self.decoder_model(self.z)\n",
    "        \n",
    "    def compile_vae(self):\n",
    "        adam = optimizers.Adam(lr=self.learning_rate)\n",
    "        vae_layer = CustomVariationalLayer(self.z_log_var_encoded,\n",
    "                                           self.z_mean_encoded)([self.rnaseq_input, self.rnaseq_reconstruct])\n",
    "        self.vae = Model(self.rnaseq_input, vae_layer)\n",
    "        self.vae.compile(optimizer=adam, loss=None, loss_weights=[self.beta])\n",
    "        \n",
    "    def get_summary(self):\n",
    "        self.vae.summary()\n",
    "    \n",
    "    def visualize_architecture(self, output_file):\n",
    "        # Visualize the connections of the custom VAE model\n",
    "        plot_model(self.vae, to_file=output_file)\n",
    "        SVG(model_to_dot(self.vae).create(prog='dot', format='svg'))\n",
    "        \n",
    "    def train_vae(self):\n",
    "        self.hist = self.vae.fit(np.array(rnaseq_train_df),\n",
    "               shuffle=True,\n",
    "               epochs=self.epochs,\n",
    "               batch_size=self.batch_size,\n",
    "               validation_data=(np.array(rnaseq_test_df), np.array(rnaseq_test_df)),\n",
    "               callbacks=[WarmUpCallback(self.beta, self.kappa)])\n",
    "    \n",
    "    def visualize_training(self, output_file):\n",
    "        # Visualize training performance\n",
    "        history_df = pd.DataFrame(self.hist.history)\n",
    "        ax = history_df.plot()\n",
    "        ax.set_xlabel('Epochs')\n",
    "        ax.set_ylabel('VAE Loss')\n",
    "        fig = ax.get_figure()\n",
    "        fig.savefig(output_file)\n",
    "        \n",
    "    def compress(self, df):\n",
    "        # Model to compress input\n",
    "        self.encoder = Model(self.rnaseq_input, self.z_mean_encoded)\n",
    "        \n",
    "        # Encode rnaseq into the hidden/latent representation - and save output\n",
    "        encoded_df = self.encoder.predict_on_batch(df)\n",
    "        encoded_df = pd.DataFrame(encoded_df, columns=range(1, self.latent_dim + 1),\n",
    "                                  index=rnaseq_df.index)\n",
    "        return encoded_df\n",
    "    \n",
    "    def get_decoder_weights(self):\n",
    "        # build a generator that can sample from the learned distribution\n",
    "        decoder_input = Input(shape=(self.latent_dim, ))  # can generate from any sampled z vector\n",
    "        _x_decoded_mean = self.decoder_model(decoder_input)\n",
    "        self.decoder = Model(decoder_input, _x_decoded_mean)\n",
    "        weights = []\n",
    "        for layer in self.decoder.layers:\n",
    "            weights.append(layer.get_weights())\n",
    "        return(weights)\n",
    "    \n",
    "    def predict(self, df):\n",
    "        return self.decoder.predict(np.array(df))\n",
    "    \n",
    "    def save_models(self, encoder_file, decoder_file):\n",
    "        self.encoder.save(encoder_file)\n",
    "        self.decoder.save(decoder_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Gene Expression Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10459, 5000)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RPS4Y1</th>\n",
       "      <th>XIST</th>\n",
       "      <th>KRT5</th>\n",
       "      <th>AGR2</th>\n",
       "      <th>CEACAM5</th>\n",
       "      <th>KRT6A</th>\n",
       "      <th>KRT14</th>\n",
       "      <th>CEACAM6</th>\n",
       "      <th>DDX3Y</th>\n",
       "      <th>KDM5D</th>\n",
       "      <th>...</th>\n",
       "      <th>FAM129A</th>\n",
       "      <th>C8orf48</th>\n",
       "      <th>CDK5R1</th>\n",
       "      <th>FAM81A</th>\n",
       "      <th>C13orf18</th>\n",
       "      <th>GDPD3</th>\n",
       "      <th>SMAGP</th>\n",
       "      <th>C2orf85</th>\n",
       "      <th>POU5F1B</th>\n",
       "      <th>CHST2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0047-01</th>\n",
       "      <td>0.678296</td>\n",
       "      <td>0.289910</td>\n",
       "      <td>0.034230</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.084731</td>\n",
       "      <td>0.031863</td>\n",
       "      <td>0.037709</td>\n",
       "      <td>0.746797</td>\n",
       "      <td>0.687833</td>\n",
       "      <td>...</td>\n",
       "      <td>0.440610</td>\n",
       "      <td>0.428782</td>\n",
       "      <td>0.732819</td>\n",
       "      <td>0.634340</td>\n",
       "      <td>0.580662</td>\n",
       "      <td>0.294313</td>\n",
       "      <td>0.458134</td>\n",
       "      <td>0.478219</td>\n",
       "      <td>0.168263</td>\n",
       "      <td>0.638497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0055-01</th>\n",
       "      <td>0.200633</td>\n",
       "      <td>0.654917</td>\n",
       "      <td>0.181993</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.100606</td>\n",
       "      <td>0.050011</td>\n",
       "      <td>0.092586</td>\n",
       "      <td>0.103725</td>\n",
       "      <td>0.140642</td>\n",
       "      <td>...</td>\n",
       "      <td>0.620658</td>\n",
       "      <td>0.363207</td>\n",
       "      <td>0.592269</td>\n",
       "      <td>0.602755</td>\n",
       "      <td>0.610192</td>\n",
       "      <td>0.374569</td>\n",
       "      <td>0.722420</td>\n",
       "      <td>0.271356</td>\n",
       "      <td>0.160465</td>\n",
       "      <td>0.602560</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 5000 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   RPS4Y1      XIST      KRT5  AGR2  CEACAM5     KRT6A  \\\n",
       "TCGA-02-0047-01  0.678296  0.289910  0.034230   0.0      0.0  0.084731   \n",
       "TCGA-02-0055-01  0.200633  0.654917  0.181993   0.0      0.0  0.100606   \n",
       "\n",
       "                    KRT14   CEACAM6     DDX3Y     KDM5D    ...      FAM129A  \\\n",
       "TCGA-02-0047-01  0.031863  0.037709  0.746797  0.687833    ...     0.440610   \n",
       "TCGA-02-0055-01  0.050011  0.092586  0.103725  0.140642    ...     0.620658   \n",
       "\n",
       "                  C8orf48    CDK5R1    FAM81A  C13orf18     GDPD3     SMAGP  \\\n",
       "TCGA-02-0047-01  0.428782  0.732819  0.634340  0.580662  0.294313  0.458134   \n",
       "TCGA-02-0055-01  0.363207  0.592269  0.602755  0.610192  0.374569  0.722420   \n",
       "\n",
       "                  C2orf85   POU5F1B     CHST2  \n",
       "TCGA-02-0047-01  0.478219  0.168263  0.638497  \n",
       "TCGA-02-0055-01  0.271356  0.160465  0.602560  \n",
       "\n",
       "[2 rows x 5000 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnaseq_file = os.path.join('data', 'pancan_scaled_zeroone_rnaseq.tsv.gz')\n",
    "rnaseq_df = pd.read_table(rnaseq_file, index_col=0)\n",
    "print(rnaseq_df.shape)\n",
    "rnaseq_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Split 10% test set randomly\n",
    "test_set_percent = 0.1\n",
    "rnaseq_test_df = rnaseq_df.sample(frac=test_set_percent)\n",
    "rnaseq_train_df = rnaseq_df.drop(rnaseq_test_df.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize variables and hyperparameters for each model\n",
    "\n",
    "The hyperparameters provided below were determined through previous independent parameter searches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Set common hyper parameters\n",
    "original_dim = rnaseq_df.shape[1]\n",
    "latent_dim = 100\n",
    "beta = K.variable(0)\n",
    "epsilon_std = 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model A (100 hidden layer size)\n",
    "model_a_latent_dim = 100\n",
    "model_a_batch_size = 100\n",
    "model_a_epochs = 100\n",
    "model_a_learning_rate = 0.001\n",
    "model_a_kappa = 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model B (300 hidden layer size)\n",
    "model_b_latent_dim = 300\n",
    "model_b_batch_size = 50\n",
    "model_b_epochs = 100\n",
    "model_b_learning_rate = 0.0005\n",
    "model_b_kappa = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and process Tybalt Models\n",
    "\n",
    "### Model A Training and Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_a = Tybalt(original_dim=original_dim,\n",
    "                 hidden_dim=model_a_latent_dim,\n",
    "                 latent_dim=latent_dim,\n",
    "                 batch_size=model_a_batch_size,\n",
    "                 epochs=model_a_epochs,\n",
    "                 learning_rate=model_a_learning_rate,\n",
    "                 kappa=model_a_kappa,\n",
    "                 beta=beta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_1 (InputLayer)             (None, 5000)          0                                            \n",
      "____________________________________________________________________________________________________\n",
      "dense_1 (Dense)                  (None, 100)           500100      input_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNorm (None, 100)           400         dense_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "activation_1 (Activation)        (None, 100)           0           batch_normalization_1[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "dense_2 (Dense)                  (None, 100)           10100       activation_1[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "dense_3 (Dense)                  (None, 100)           10100       activation_1[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNorm (None, 100)           400         dense_2[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNorm (None, 100)           400         dense_3[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "activation_2 (Activation)        (None, 100)           0           batch_normalization_2[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "activation_3 (Activation)        (None, 100)           0           batch_normalization_3[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "lambda_1 (Lambda)                (None, 100)           0           activation_2[0][0]               \n",
      "                                                                   activation_3[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "sequential_1 (Sequential)        (None, 5000)          515100      lambda_1[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "custom_variational_layer_1 (Cust [(None, 5000), (None, 0           input_1[0][0]                    \n",
      "                                                                   sequential_1[1][0]               \n",
      "====================================================================================================\n",
      "Total params: 1,036,600\n",
      "Trainable params: 1,036,000\n",
      "Non-trainable params: 600\n",
      "____________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/gway/anaconda3/envs/vae_pancancer/lib/python2.7/site-packages/ipykernel_launcher.py:52: UserWarning: Output \"custom_variational_layer_1\" missing from loss dictionary. We assume this was done on purpose, and we will not be expecting any data to be passed to \"custom_variational_layer_1\" during training.\n"
     ]
    }
   ],
   "source": [
    "# Compile Model A\n",
    "model_a.build_encoder_layer()\n",
    "model_a.build_decoder_layer()\n",
    "model_a.compile_vae()\n",
    "model_a.get_summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_architecture_file = os.path.join('figures', 'twohidden_vae_architecture.png')\n",
    "model_a.visualize_architecture(model_architecture_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9413 samples, validate on 1046 samples\n",
      "Epoch 1/100\n",
      "9413/9413 [==============================] - 1s - loss: 2957.9805 - val_loss: 2950.8238\n",
      "Epoch 2/100\n",
      "9413/9413 [==============================] - 1s - loss: 2797.2792 - val_loss: 2805.8403\n",
      "Epoch 3/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.9802 - val_loss: 2762.6554\n",
      "Epoch 4/100\n",
      "9413/9413 [==============================] - 1s - loss: 2746.9909 - val_loss: 2754.8216\n",
      "Epoch 5/100\n",
      "9413/9413 [==============================] - 1s - loss: 2735.1533 - val_loss: 2740.8744\n",
      "Epoch 6/100\n",
      "9413/9413 [==============================] - 1s - loss: 2725.5900 - val_loss: 2726.4276\n",
      "Epoch 7/100\n",
      "9413/9413 [==============================] - 1s - loss: 2718.8524 - val_loss: 2724.1144\n",
      "Epoch 8/100\n",
      "9413/9413 [==============================] - 1s - loss: 2713.2076 - val_loss: 2716.0569\n",
      "Epoch 9/100\n",
      "9413/9413 [==============================] - 1s - loss: 2708.3757 - val_loss: 2711.0041\n",
      "Epoch 10/100\n",
      "9413/9413 [==============================] - 1s - loss: 2705.0074 - val_loss: 2707.2964\n",
      "Epoch 11/100\n",
      "9413/9413 [==============================] - 1s - loss: 2701.6027 - val_loss: 2704.6021\n",
      "Epoch 12/100\n",
      "9413/9413 [==============================] - 1s - loss: 2698.6276 - val_loss: 2702.7460\n",
      "Epoch 13/100\n",
      "9413/9413 [==============================] - 1s - loss: 2696.0863 - val_loss: 2696.5519\n",
      "Epoch 14/100\n",
      "9413/9413 [==============================] - 1s - loss: 2693.8027 - val_loss: 2695.8062\n",
      "Epoch 15/100\n",
      "9413/9413 [==============================] - 1s - loss: 2691.9961 - val_loss: 2694.1369\n",
      "Epoch 16/100\n",
      "9413/9413 [==============================] - 1s - loss: 2690.0699 - val_loss: 2691.5633\n",
      "Epoch 17/100\n",
      "9413/9413 [==============================] - 1s - loss: 2688.6888 - val_loss: 2690.7129\n",
      "Epoch 18/100\n",
      "9413/9413 [==============================] - 1s - loss: 2686.7476 - val_loss: 2689.8341\n",
      "Epoch 19/100\n",
      "9413/9413 [==============================] - 1s - loss: 2685.3433 - val_loss: 2687.7475\n",
      "Epoch 20/100\n",
      "9413/9413 [==============================] - 1s - loss: 2684.3288 - val_loss: 2685.7977\n",
      "Epoch 21/100\n",
      "9413/9413 [==============================] - 1s - loss: 2682.7771 - val_loss: 2684.1735\n",
      "Epoch 22/100\n",
      "9413/9413 [==============================] - 1s - loss: 2681.6806 - val_loss: 2685.5071\n",
      "Epoch 23/100\n",
      "9413/9413 [==============================] - 1s - loss: 2680.3572 - val_loss: 2683.1425\n",
      "Epoch 24/100\n",
      "9413/9413 [==============================] - 1s - loss: 2679.9060 - val_loss: 2684.7259\n",
      "Epoch 25/100\n",
      "9413/9413 [==============================] - 1s - loss: 2678.5832 - val_loss: 2683.2860\n",
      "Epoch 26/100\n",
      "9413/9413 [==============================] - 1s - loss: 2677.8243 - val_loss: 2682.6056\n",
      "Epoch 27/100\n",
      "9413/9413 [==============================] - 1s - loss: 2676.6036 - val_loss: 2679.3471\n",
      "Epoch 28/100\n",
      "9413/9413 [==============================] - 1s - loss: 2675.5763 - val_loss: 2678.8165\n",
      "Epoch 29/100\n",
      "9413/9413 [==============================] - 1s - loss: 2675.3098 - val_loss: 2678.9385\n",
      "Epoch 30/100\n",
      "9413/9413 [==============================] - 1s - loss: 2674.3548 - val_loss: 2676.9404\n",
      "Epoch 31/100\n",
      "9413/9413 [==============================] - 1s - loss: 2673.4457 - val_loss: 2676.3542\n",
      "Epoch 32/100\n",
      "9413/9413 [==============================] - 1s - loss: 2672.9657 - val_loss: 2676.0678\n",
      "Epoch 33/100\n",
      "9413/9413 [==============================] - 1s - loss: 2672.1828 - val_loss: 2676.0129\n",
      "Epoch 34/100\n",
      "9413/9413 [==============================] - 1s - loss: 2671.5003 - val_loss: 2675.2568\n",
      "Epoch 35/100\n",
      "9413/9413 [==============================] - 1s - loss: 2671.0813 - val_loss: 2675.5867\n",
      "Epoch 36/100\n",
      "9413/9413 [==============================] - 1s - loss: 2670.7040 - val_loss: 2673.9413\n",
      "Epoch 37/100\n",
      "9413/9413 [==============================] - 1s - loss: 2669.7803 - val_loss: 2673.5747\n",
      "Epoch 38/100\n",
      "9413/9413 [==============================] - 1s - loss: 2669.2723 - val_loss: 2673.5660\n",
      "Epoch 39/100\n",
      "9413/9413 [==============================] - 1s - loss: 2669.0598 - val_loss: 2673.2244\n",
      "Epoch 40/100\n",
      "9413/9413 [==============================] - 1s - loss: 2668.3699 - val_loss: 2675.0969\n",
      "Epoch 41/100\n",
      "9413/9413 [==============================] - 1s - loss: 2668.1428 - val_loss: 2671.7506\n",
      "Epoch 42/100\n",
      "9413/9413 [==============================] - 1s - loss: 2667.3772 - val_loss: 2670.8989\n",
      "Epoch 43/100\n",
      "9413/9413 [==============================] - 1s - loss: 2667.0446 - val_loss: 2670.1845\n",
      "Epoch 44/100\n",
      "9413/9413 [==============================] - 1s - loss: 2666.6668 - val_loss: 2669.2487\n",
      "Epoch 45/100\n",
      "9413/9413 [==============================] - 1s - loss: 2666.3487 - val_loss: 2671.3058\n",
      "Epoch 46/100\n",
      "9413/9413 [==============================] - 1s - loss: 2665.8814 - val_loss: 2669.0087\n",
      "Epoch 47/100\n",
      "9413/9413 [==============================] - 1s - loss: 2665.5644 - val_loss: 2669.0123\n",
      "Epoch 48/100\n",
      "9413/9413 [==============================] - 1s - loss: 2665.0417 - val_loss: 2668.4945\n",
      "Epoch 49/100\n",
      "9413/9413 [==============================] - 1s - loss: 2664.9909 - val_loss: 2668.5532\n",
      "Epoch 50/100\n",
      "9413/9413 [==============================] - 1s - loss: 2664.8526 - val_loss: 2667.8724\n",
      "Epoch 51/100\n",
      "9413/9413 [==============================] - 1s - loss: 2664.0827 - val_loss: 2668.5496\n",
      "Epoch 52/100\n",
      "9413/9413 [==============================] - 1s - loss: 2664.0727 - val_loss: 2669.2049\n",
      "Epoch 53/100\n",
      "9413/9413 [==============================] - 1s - loss: 2663.9176 - val_loss: 2668.2468\n",
      "Epoch 54/100\n",
      "9413/9413 [==============================] - 1s - loss: 2663.1647 - val_loss: 2666.8234\n",
      "Epoch 55/100\n",
      "9413/9413 [==============================] - 1s - loss: 2662.9519 - val_loss: 2667.1615\n",
      "Epoch 56/100\n",
      "9413/9413 [==============================] - 1s - loss: 2662.8080 - val_loss: 2666.0201\n",
      "Epoch 57/100\n",
      "9413/9413 [==============================] - 1s - loss: 2662.3003 - val_loss: 2666.1678\n",
      "Epoch 58/100\n",
      "9413/9413 [==============================] - 1s - loss: 2661.6830 - val_loss: 2666.0583\n",
      "Epoch 59/100\n",
      "9413/9413 [==============================] - 1s - loss: 2662.1333 - val_loss: 2666.6102\n",
      "Epoch 60/100\n",
      "9413/9413 [==============================] - 1s - loss: 2661.8820 - val_loss: 2665.8327\n",
      "Epoch 61/100\n",
      "9413/9413 [==============================] - 1s - loss: 2661.2620 - val_loss: 2665.5685\n",
      "Epoch 62/100\n",
      "9413/9413 [==============================] - 1s - loss: 2660.8630 - val_loss: 2665.3976\n",
      "Epoch 63/100\n",
      "9413/9413 [==============================] - 1s - loss: 2660.9776 - val_loss: 2663.9852\n",
      "Epoch 64/100\n",
      "9413/9413 [==============================] - 1s - loss: 2660.5806 - val_loss: 2666.0981\n",
      "Epoch 65/100\n",
      "9413/9413 [==============================] - 1s - loss: 2660.9485 - val_loss: 2665.4582\n",
      "Epoch 66/100\n",
      "9413/9413 [==============================] - 1s - loss: 2659.9776 - val_loss: 2663.9994\n",
      "Epoch 67/100\n",
      "9413/9413 [==============================] - 1s - loss: 2660.1682 - val_loss: 2664.4093\n",
      "Epoch 68/100\n",
      "9413/9413 [==============================] - 1s - loss: 2659.8602 - val_loss: 2665.4123\n",
      "Epoch 69/100\n",
      "9413/9413 [==============================] - 1s - loss: 2659.7320 - val_loss: 2664.2095\n",
      "Epoch 70/100\n",
      "9413/9413 [==============================] - 1s - loss: 2659.3986 - val_loss: 2663.3767\n",
      "Epoch 71/100\n",
      "9413/9413 [==============================] - 1s - loss: 2659.4631 - val_loss: 2662.3696\n",
      "Epoch 72/100\n",
      "9413/9413 [==============================] - 1s - loss: 2658.8762 - val_loss: 2663.2382\n",
      "Epoch 73/100\n",
      "9413/9413 [==============================] - 1s - loss: 2658.8785 - val_loss: 2662.9145\n",
      "Epoch 74/100\n",
      "9413/9413 [==============================] - 1s - loss: 2658.6419 - val_loss: 2662.6420\n",
      "Epoch 75/100\n",
      "9413/9413 [==============================] - 1s - loss: 2658.0485 - val_loss: 2662.2743\n",
      "Epoch 76/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.9589 - val_loss: 2662.9043\n",
      "Epoch 77/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.9868 - val_loss: 2661.9093\n",
      "Epoch 78/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.9815 - val_loss: 2662.5459\n",
      "Epoch 79/100\n",
      "9413/9413 [==============================] - 1s - loss: 2658.0705 - val_loss: 2661.9623\n",
      "Epoch 80/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.8526 - val_loss: 2662.4854\n",
      "Epoch 81/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.5689 - val_loss: 2661.9085\n",
      "Epoch 82/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.6613 - val_loss: 2661.5535\n",
      "Epoch 83/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.1739 - val_loss: 2661.2428\n",
      "Epoch 84/100\n",
      "9413/9413 [==============================] - 1s - loss: 2657.2324 - val_loss: 2662.6703\n",
      "Epoch 85/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.8949 - val_loss: 2661.0031\n",
      "Epoch 86/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.6910 - val_loss: 2660.5960\n",
      "Epoch 87/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.8326 - val_loss: 2661.4506\n",
      "Epoch 88/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.7350 - val_loss: 2660.4537\n",
      "Epoch 89/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.3786 - val_loss: 2661.3030\n",
      "Epoch 90/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.1899 - val_loss: 2660.8402\n",
      "Epoch 91/100\n",
      "9413/9413 [==============================] - 1s - loss: 2656.3453 - val_loss: 2660.1869\n",
      "Epoch 92/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.8668 - val_loss: 2661.4266\n",
      "Epoch 93/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.7987 - val_loss: 2661.0794\n",
      "Epoch 94/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.6291 - val_loss: 2660.4709\n",
      "Epoch 95/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.5421 - val_loss: 2659.9211\n",
      "Epoch 96/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.5208 - val_loss: 2659.5040\n",
      "Epoch 97/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.2294 - val_loss: 2659.9641\n",
      "Epoch 98/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.2495 - val_loss: 2659.8988\n",
      "Epoch 99/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.1405 - val_loss: 2661.1199\n",
      "Epoch 100/100\n",
      "9413/9413 [==============================] - 1s - loss: 2655.0629 - val_loss: 2659.8728\n",
      "CPU times: user 2min 14s, sys: 6.39 s, total: 2min 20s\n",
      "Wall time: 1min 53s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model_a.train_vae()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAFfCAYAAAA8vaR4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmYXHWd7/H3t9bes2+dPRAIgQxEkAkgMOICbuDgKDoK\n6FxxFB4FdJxRxLnoo+Mdncuocx24c0WRuaAwguIVERllWGZYTGIgkIQQQvZO0ll67671e//4nU4q\nIUt30tWVdH1ej/V01alTp751BOpTv+2YuyMiIiLVKVbpAkRERKRyFARERESqmIKAiIhIFVMQEBER\nqWIKAiIiIlVMQUBERKSKKQiIiIhUMQUBERGRKqYgICIiUsUSlS5gOIwfP95nzZpV6TJERESGzZIl\nS3a4+4TD7VcVQWDWrFksXry40mWIiIgMGzNbP5D91DUgIiJSxRQEREREqlhVdA2IiMjxJ5/PUywW\nK13GMS0Wi5FIHN1XuVoERETkmNPZ2Uk2m610Gce8bDZLZ2fnUR1DLQIiInJMyefzxONx6urqKl3K\nMS+VStHT00M+nz/ilgG1CIiIyDGlWCwedXN3NYnH40fVhaIgICIichwzs6N6vYKAiIhIFVMQEBER\nOYCGhoZKlzAsFARERESqmIKAiIjIIbg7n//85znttNNYsGAB9957LwAtLS1ccMEFnHHGGZx22mk8\n+eSTFAoFPvrRj+7Z9x//8R8rXP3hVcWwzBc3t9Pem2NUbbLSpYiIyCD92W3/RUt735Aec8qoGn76\nqXMHtO8DDzzAsmXLeP7559mxYwdvfOMbueCCC7jnnnu4+OKL+dKXvkShUKCnp4dly5axefNmXnzx\nRQDa2tqGtO5yqIoWAQeyea1OJSIig/fUU0/xoQ99iHg8zqRJk7jwwgv5/e9/zxvf+EZ++MMfcsst\nt7B8+XIaGxuZM2cOa9eu5dOf/jS//vWvaWpqqnT5h1UVLQIAmXyh0iWIiMgRGOgv9+F2wQUX8MQT\nT/DQQw/x0Y9+lM9+9rNcddVVPP/88zzyyCPcfvvt3HffffzgBz+odKmHVBUtAgAZtQiIiMgROP/8\n87n33nspFAq0trbyxBNPcPbZZ7N+/XomTZrENddcw8c//nGWLl3Kjh07KBaLvO997+NrX/saS5cu\nrXT5h1W2FgEzmw7cBUwitM7/i7t/x8xOB24HGoB1wIfdvcPMZgErgZejQzzj7p+MjnUmcCdQC/wK\nuN7dfTD1ZHIKAiIiMnh/+qd/ytNPP83pp5+OmfHNb36TyZMn86Mf/YhvfetbJJNJGhoauOuuu9i8\neTMf+9jH9qz0941vfKPC1R+eDfL7dOAHNpsCTHH3pWbWCCwB3gv8CPgrd3/czP4CmO3uX46CwC/d\n/bQDHOs54DPAs4Qg8F13f3igtaSnzPVnnn2OhTPGHPXnEhGR8uq/2FAqlapwJceHg50vM1vi7mcd\n7vVl6xpw9xZ3Xxrd7yT82p8KnAQ8Ee32KPC+Qx0nChRN7v5M1ApwFyFQDNgUdpLN9A7yE4iIiIx8\nwzJGIPq1v5Dwi/4l4LLoqfcD00t2nW1my8zscTM7P9o2FdhUss+maNuAjbd28r1Hd5lGERGRkajs\nQcDMGoD7gRvcvQP4C+BaM1sCNAL9F5xuAWa4+xnAZ4F7zOyI512Y2S1m5mbmAPlMz9F8DBERkRGp\nrEHAzJKEEHC3uz8A4O6r3P3t7n4m8GPg1Wh7xt13RveXRNtPAjYD00oOOy3adkjufou7m7sbQEFB\nQERE5HXKFgQsXBfxDmClu99asn1i9DcG3EyYQYCZTTCzeHR/DjAXWOvuLUCHmS2KjnkV8OBg68ln\nFQRERET2V84Fhc4DrgSWm9myaNtNwFwzuy56/ADww+j+BcBXzSwHFIFPuvuu6Llr2Tt98OHoNijF\nrAYLioiI7K9sQcDdnwLsIE9/5wD730/oRjjQsRYDr5tWOBjF7NCuUy0iIjISVM3KgsWcugZERKR8\nGhoaDvrcunXrOO20o/o9WzZVEwRcLQIiIiKvUzUXHfK8goCIyHHpjouhY8vQHrOpGf7bI4fc5Qtf\n+ALTp0/nuuvCsLZbbrmFRCLBY489xu7du8nlcnzta1/jsssuO+Rx9tfX18enPvUpFi9eTCKR4NZb\nb+XNb34zL730Eh/72MfIZrMUi0Xuv/9+mpub+cAHPsCmTZsoFAp8+ctf5oorrjjij30gVRMEyGuw\noIiIDNwVV1zBDTfcsCcI3HfffTzyyCN85jOfoampiR07drBo0SIuvfRSwqS2gfne976HmbF8+XJW\nrVrF29/+dlavXs3tt9/O9ddfz4c//GGy2SyFQoFf/epXNDc389BDDwHQ3t4+5J+zaoKA5dQiICJy\nXDrML/dyWbhwIdu3b2fLli20trYyZswYJk+ezI033sgTTzxBLBZj8+bNbNu2jcmTJw/4uE899RSf\n/vSnAZg3bx4zZ85k9erVnHPOOXz9619n06ZNXH755cydO5cFCxbwuc99jr/5m7/h3e9+N+eff/5h\njj54VTNGwNQ1ICIig/T+97+fn/70p9x7771cccUV3H333bS2trJkyRKWLVvGpEmT6Osbmu+XP//z\nP+cXv/gFtbW1vPOd7+R3v/sdJ510EkuXLmXBggXcfPPNfPWrXx2S9ypVPS0CBQUBEREZnCuuuIJr\nrrmGHTt28Pjjj3PfffcxceJEkskkjz32GOvXrx/0Mc8//3zuvvtuLrroIlavXs2GDRs4+eSTWbt2\nLXPmzOEzn/kMGzZs4IUXXmDevHmMHTuWj3zkI4wePZrvf//7Q/4ZqyYIxNQiICIig3TqqafS2dnJ\n1KlTmTJlCh/+8Id5z3vew4IFCzjrrLOYN2/eoI957bXX8qlPfYoFCxaQSCS48847SafT3Hffffzr\nv/4ryWSSyZMnc9NNN/H73/+ez3/+88RiMZLJJLfddtuQf0YLV/Yd2c5qjvvXP/sRLv6rH1W6FBER\nOYxsNlyLLpVKVbiS48PBzpeZLXH3sw73+qoZIxAvZCpdgoiIyDGnaroG4kUFARERKa/ly5dz5ZVX\n7rMtnU7z7LPPlu093X1Q0xf3VzVBIKEgICJyXIjFYmSz2eOya2DBggUsW7bs8DsOoUKhcFTnSkFA\nRESOKYlEgt7eXnp6eojH40f1a3ckc3cKhQKFQoFE4si/zqsmCCRdQUBE5HjR2NhIPp+nWCxWupRj\nlpmRSqWOKgRAlQQBx0gWs5UuQ0REBuFov+BkYKpi1oBjpFCLgIiIyP6qJAjESLtaBERERPZXHUHA\njBRZqmHxJBERkcGojiCAkSZLJq9BJyIiIqWqJAjEqCGnICAiIrKf6ggCZtSQJZMvVLoUERGRY0pV\nBAEwklYgk9GAQRERkVJVEQTcwsfM9vVUuBIREZFjS3UEAcLylLlMd4UrERERObZURRAgahHIZ9Qi\nICIiUqqqgkCur7fChYiIiBxbqiQIhK6BQlYtAiIiIqWqIwhEHzOfVYuAiIhIqeoIArHwMYsKAiIi\nIvuojiAQzRooaLCgiIjIPqojCEQtAp5Ti4CIiEip6ggC1h8E+ipciIiIyLGlKoKAmVoEREREDqQq\ngkB/iwAKAiIiIvuoiiCwp0Ugn6lwJSIiIseWqggCxMKsAcurRUBERKRU2YKAmU03s8fMbIWZvWRm\n10fbTzezp81suZn9PzNrKnnNF81sjZm9bGYXl2w/M9p/jZl91yxaKnDAtYSPaXkNFhQRESlVzhaB\nPPA5d58PLAKuM7P5wPeBL7j7AuBnwOcBouc+CJwKXAL8s5nFo2PdBlwDzI1ulwymkP4gECuoa0BE\nRKRU2YKAu7e4+9LofiewEpgKnAQ8Ee32KPC+6P5lwE/cPePurwFrgLPNbArQ5O7PuLsDdwHvHUwt\nFq0jEFOLgIiIyD6GZYyAmc0CFgLPAi8RvvQB3g9Mj+5PBTaWvGxTtG1qdH//7YN4/9CwEC8qCIiI\niJQqexAwswbgfuAGd+8A/gK41syWAI1Atkzve4uZuZn59tZWAOLqGhAREdlHWYOAmSUJIeBud38A\nwN1Xufvb3f1M4MfAq9Hum9nbOgAwLdq2Obq///ZDcvdb3N3c3aZMmQJAoqggICIiUqqcswYMuANY\n6e63lmyfGP2NATcDt0dP/QL4oJmlzWw2YVDgc+7eAnSY2aLomFcBDw6umPAx4woCIiIi+0iU8djn\nAVcCy81sWbTtJmCumV0XPX4A+CGAu79kZvcBKwgzDq5z90K037XAnUAt8HB0G5QccZIKAiIiIvso\nWxBw96fov/7v633nIK/5OvD1A2xfDJx2NPVkSJP0sgxHEBEROW5Vx8qCQIYUKVeLgIiISKmqCQJZ\nS5FSi4CIiMg+qiYI5CxNCrUIiIiIlKqeIBBLk1aLgIiIyD6qJwhYihqyeLFY6VJERESOGVUTBPLx\nGuLmZLJqFRAREelXNUGgEEsBkOnrqXAlIiIix44qCgJpALJ9XRWuRERE5NhRPUEgXgNArq+3wpWI\niIgcO6omCBQT/UFAXQMiIiL9qiYIeNQikM92V7gSERGRY0fVBIH+FoF8Rl0DIiIi/aomCBAFgUJG\nXQMiIiL9qi8IZNUiICIi0q96gkCyFlAQEBERKVU1QcCSoUWgmFMQEBER6VdFQSC0CLhaBERERPao\nmiAQS0VBQC0CIiIie1RNEIjvCQJ9Fa5ERETk2FE1QaC/RYC8WgRERET6VU0QSKTqATC1CIiIiOxR\nNUEgng4tAlbIVLgSERGRY0fVBIFkfxDIq0VARESkX9UEgUS6DoBYQUFARESkX9UEgZSCgIiIyOtU\nTRBI1oYgENcYARERkT2qJgika8KsgURRQUBERKRf9QSBZIKMJxUERERESlRPEEjE6CNJoqgxAiIi\nIv2qJgiYGRlSJDxb6VJERESOGVUTBACyliJVVBAQERHpV1VBIEOKFBojICIi0q+qgkDO0qTUNSAi\nIrJHVQWBbCxNmiy4V7oUERGRY0JVBYG8pYnhUFCrgIiICFRZEMjF0tGd3soWIiIicowoWxAws+lm\n9piZrTCzl8zs+mj7GWb2jJktM7PFZnZ2tH2WmfVG25eZ2e0lxzrTzJab2Roz+66Z2ZHUVIiCgCsI\niIiIAJAo47HzwOfcfamZNQJLzOxR4JvAV9z9YTN7Z/T4T6LXvOruZxzgWLcB1wDPAr8CLgEeHmxB\nhXgKgGyml/RgXywiIjICla1FwN1b3H1pdL8TWAlMBRxoinYbBWw51HHMbArQ5O7PuLsDdwHvPZKa\nCrEaALJ9PUfychERkRGnnC0Ce5jZLGAh4Rf9DcAjZvYPhCBybsmus81sGdAO3OzuTxLCw6aSfTZF\n2watGA/tALlM95G8XEREZMQp+2BBM2sA7gducPcO4FPAje4+HbgRuCPatQWYEXUNfBa4x8yaDnTM\nAb7vLWbmZuZbtoRGh2KiFoCcWgRERESAMgcBM0sSQsDd7v5AtPlqoP/+vwFnA7h7xt13RveXAK8C\nJwGbgWklh50WbTskd7/F3c3drbm5OWxLhBaBfEaDBUVERKC8swaM8Gt/pbvfWvLUFuDC6P5FwCvR\n/hPMLB7dnwPMBda6ewvQYWaLomNeBTx4JDV5IowRKKhrQEREBCjvGIHzgCuB5VG/P8BNhNH/3zGz\nBNAHfCJ67gLgq2aWA4rAJ919V/TctcCdQC1htsCgZwwAeNQ1UMiqRUBERATKGATc/SngYPP9zzzA\n/vcTuhEOdKzFwGlHXVR/i4CCgIiICFBlKwtaMgSBooKAiIgIUGVBIJYMXQPFXF+FKxERETk2VFUQ\nsCgIaIlhERGRoKqCQCylICAiIlKqqoJAPF0X7igIiIiIANUWBKIWAfIaIyAiIgJVGgRMQUBERASo\nsiCQSNcDCgIiIiL9qioIJGtCi0CskKlwJSIiIseG6goC0WDBWEEtAiIiIjDIIGBmE81sUbmKKbdU\nFATiahEQEREBBhAEzOxJMxtlZqOBPwB3mNm3yl/a0Esn4/R5krhaBERERICBtQg0uHs78G7gbmAB\ncElZqyqTdDJGHykSRbUIiIiIwMCCQDr6+2bgUXcvAvnylVQ+6URcQUBERKTEQILAf5jZCuD86P5o\noFDessojnYjR5ykSriAgIiICkBjAPtcBpwNr3T1nZnHgmvKWVR7pROgaGF/srnQpIiIix4SBtAjM\nBVa5e4eZXUwIAevKWlWZmBk5S5L0XKVLEREROSYMJAjcBxTMbDbwv4E5wI/KWlUZZS1Nmgy4V7oU\nERGRihtIECi6ew54F/DP7v4JYEZ5yyqfrKXCnbzGCYiIiAwkCNSY2STgPcDvom1WvpLKK2fRJIi8\nLkUsIiIykCDwbeBloMvdF5vZHKC9vGWVTz4WBYGcFhUSERE57KwBd/8X4F9KNq0D3lqugsptTxBQ\ni4CIiMjhg4CZGfAJ9n75Pwr8n3IWVU5qERAREdlrIOsIfBNYCPwwenw1cCLw1+UqqpwK8RoAPNd7\n/A50EBERGSIDCQIXA29w9zyAmd0HLOE4DQLFeGgRyGZ696ydLCIiUq0GMljQgNJJ985xPGugGLUI\n5DM9Fa5ERESk8gbSIvAI8LCZ3Rk9vjradlzyRGgHyGW0zLCIiMhAgsBfEwYLXh49/hlhhcHjUjFe\nC0A+o1kDIiIiA5k+WARuj24AmNlXgP9exrrKJxm6BgoKAiIiIgMaI3AgHxvSKoZTIgoCWY0REBER\nOdIgcNwOFiQZugaKWbUIiIiIHGkQOG4v3WcKAiIiInscdIyAmf0bB/7CN2Bs2Soqs1g0RsC1xLCI\niMghBwv+8gifO6bFUqFFwLXEsIiIyMGDgLv/aDgLGS6Wagh/M50VrkRERKTyjnSMwHGr0DAZgFTP\n1gpXIiIiUnllCwJmNt3MHjOzFWb2kpldH20/w8yeMbNlZrbYzM4uec0XzWyNmb1sZheXbD/TzJZH\nz303uiLiEYnXNtHuddT2thzdBxQRERkBytkikAc+5+7zgUXAdWY2n3A1w6+4+xnA30aPiZ77IHAq\ncAnwz2YWj451G3ANMDe6XXKkRaUTMbb4eOp6W8CP28kPIiIiQ+KgQcDMZhziuTcc7sDu3uLuS6P7\nncBKYCphJkJTtNsoYEt0/zLgJ+6ecffXgDXA2WY2BWhy92fc3YG7gPce9pMdRDoRY7OPI1Xogb72\nIz2MiIjIiHCoFoGf998xs+f2e+77g3kTM5sFLASeBW4AvmVmG4F/AL4Y7TYV2Fjysk3RtqnR/f23\nH5F0Is4WHx8etG869M4iIiIj3KGCQGk/fPIQzx2SmTUA9wM3uHsH8CngRnefDtwI3DHQYw2Gmd1i\nZm5mvmXLlj3b08kYLT4uPFAQEBGRKneoIOAHuX+gxwdkZklCCLjb3R+INl8N9N//N6B/sOBmYHrJ\ny6dF2zZH9/fffkjufou7m7tbc3Pznu39XQMAdCgIiIhIdTtUEKgxs1OiQXx77vc/PtyBo5H9dwAr\n3f3Wkqe2ABdG9y8CXonu/wL4oJmlzWw2YVDgc+7eAnSY2aLomFcBDw7mQ5YaXZdii1oEREREgEOv\nLFgH/Krk8a8OtuNBnAdcCSw3s2XRtpsIo/+/Y2YJoA/4BIC7v2Rm9wErCDMOrnP3QvS6a4E7gVrg\n4eh2RKY01bAzMTE8UBAQEZEqd6iVBWcdzYHd/SkOPpbgzIO85uvA1w+wfTFw2tHU0y8WM2rHTqPQ\nZsTaNx3Hl1EUERE5eoNaRyBqtv+wmf22XAUNh1kTm9jGGIq7Nx5+ZxERkRFsQEHAzN5oZrcT+vc/\nQmimP27NGd/AFh9PrKsFioXDv0BERGSEOtSCQuPN7EYzWw78K7AB6Hb3d7j7vw5bhWUwZ0I9W3wc\n5gXo1DUHRESkeh2qRWAz8C7gL919nrv/HWEQ33Fv9vh6LSokIiLCoYPAt4FTgG+Y2dVmVj9MNZVd\n6BoYGx60a5yAiIhUr4MGAXf/G8ICP98iXAdgIzDezN48TLWVzai6JJ3pcDliOg67NpGIiMiIdah1\nBHD3IvBL4JdmNoGwmM93zWyMu0871GuPdYkx02EnFHZvJH743UVEREakQw0WPL/0sbu3uvv/dPcF\nwOVlr6zM6ifMBqB3x/oKVyIiIlI5hxojcKeZrTazL5pZc+kT7r7/1QiPO5MnT6bb0xTbNEZARESq\n16HGCJwA/CUwD1hpZg+Z2fuipYGPe3MmNNDi40h1bTn8ziIiIiPUIRcUcvfH3P1qYCrwM8Jlgzeb\n2a2Het3xYM6EBrb4OGry7ZDtrnQ5IiIiFTGglQXdvYtwJcFvEBYW+styFjUcZoyto4X+tQQ0c0BE\nRKrTYYOAmZ1sZn8PbAJuAX5IaCE4rqUSMbpqoimEWktARESq1KFmDXzCzJ4GngRSwDvc/Y3u/s/u\n3jZsFZaRN4UZkD2t6ypbiIiISIUcauDfe4H/CTzo7rlhqmdYpcfNgJ3Qvm0ddZUuRkREpAIOGgTc\n/Z3DWUglNE2eDashs3NDpUsRERGpiAENFhypJk6bA4DpwkMiIlKlqjoIzJk8jlYfRW1vS6VLERER\nqYiqDgITG9NsYxyjc9vBvdLliIiIDLuqDgJmRnt6MilyFLtaK12OiIjIsKvqIACQrQ+XUdi+aU2F\nKxERERl+VR8EbNR0AHZuWVvhSkRERIZf1QeBugkzAeje9lqFKxERERl+VR8ExjWHKYT53VpmWERE\nqk/VB4HJM04EIKHLEYuISBWq+iBQP2YKWRLU922tdCkiIiLDruqDALEYu+ITmFBspSebr3Q1IiIi\nw0pBAMjUTWGitbF4jVYYFBGR6qIgACSnvwGAzJP/q8KViIiIDC8FAWDCO7/EDkZxQcsd+A4tLCQi\nItVDQQBINozl51NuJE2O3geug2Kx0iWJiIgMCwWByLg3/hm/KZxJ3ZZn4A93VbocERGRYaEgELnw\n5En8bf5j9Fgd/OZvoUMDB0VEZORTEIiMrU8xdcYJ/F32g5Bph4c/X+mSREREyk5BoMRF8yZyd+Ei\ndow7E1b+v3ATEREZwRQESlw0byJOjNsbPwMY/JemE4qIyMhWtiBgZtPN7DEzW2FmL5nZ9dH2e81s\nWXRbZ2bLou2zzKy35LnbS451ppktN7M1ZvZdM7Ny1DxvciPNo2q4b10txVnnw8ZnYJeuSigiIiNX\nOVsE8sDn3H0+sAi4zszmu/sV7n6Gu58B3A88UPKaV/ufc/dPlmy/DbgGmBvdLilHwWbGRadMpKMv\nz7qp7wkbl/+0HG8lIiJyTChbEHD3FndfGt3vBFYCU/ufj37VfwD48aGOY2ZTgCZ3f8bdHbgLeG+5\n6n7LvEkA/KzvDZCogRd+Au7lejsREZGKGpYxAmY2C1gIPFuy+Xxgm7u/UrJtdtQt8LiZnR9tmwps\nKtlnEyWBYqidc8I4apIxfv1KN5z8Tti5BrYsLdfbiYiIVFTZg4CZNRC6AG5w946Spz7Evq0BLcCM\nqMvgs8A9ZtZ0FO97i5m5mfmWLVsG/LqaZJzzThjPK9u7aJ0TNTy8cN+RliEiInJMK2sQMLMkIQTc\n7e4PlGxPAJcD9/Zvc/eMu++M7i8BXgVOAjYD00oOOy3adkjufou7m7tbc3PzoOq+6JSJAPyq5xSo\nGxfGCRRygzqGiIjI8aCcswYMuANY6e637vf0W4FV7r6pZP8JZhaP7s8hDApc6+4tQIeZLYqOeRXw\nYLnqhjBOIGZw//Pb8VMvh54d8Opj5XxLERGRiihni8B5wJXARSVTAt8ZPfdBXj9I8ALghWg64U+B\nT7r7rui5a4HvA2sILQUPl7FuJo+q4a2nTOKFTe28PCkq+YV7D/0iERGR41CiXAd296eAA873d/eP\nHmDb/YRuhAPtvxg4bSjrO5yrz53Fb1Zs4/ZXRvPtsXNg1UOQ6YR043CWISIiUlZaWfAgzj1hHHMn\nNvDQi1vpOvl9kO+Flb+sdFkiIiJDSkHgIMyMq86dRa7g/DR7Xtj4wk8qW5SIiMgQUxA4hMsXTqUx\nneCfXyhQnPZGWPu4Lk8sIiIjioLAIdSnE/zZWdPY3pnhpbEXAw4v/azSZYmIiAwZBYHDuOqcWQB8\ne8t8sBi8qGsPiIjIyKEgcBizx9dz4UkT+O0m6Go+DzYvgV1rK12WiIjIkFAQGICPnjsLgF8RDRp8\n8YCzHEVERI47CgIDcOFJE5g5ro5vrp+Lx1OwXEFARERGBgWBAYjFjI+fP4cd+VpWNSyC1pWw7aVK\nlyUiInLUFAQG6IqzpjN9bC2371wYNizXoEERETn+KQgMUCoR47NvO4lH8mfQF6sN4wTcK12WiIjI\nUVEQGIRLT5/KjEnj+HXuDdC2HjYtrnRJIiIiR0VBYBDiMeOv3n4yPy+cGzZoTQERETnOKQgM0tvm\nT6Kz+Xx2eQO5F+6HYqHSJYmIiBwxBYFBMjM+d8mpPFz4Y5K9rbDuyUqXJCIicsQUBI7AuSeOZ+2U\nSwDI3vcX8MS3oGdXhasSEREZPAWBI/Se97yff8i9n2xfH/zua/CPp8HDX4C2DZUuTUREZMAUBI7Q\nGTPG0HfOZ1nU9x0enHgtXjsanr0NvrcIWldXujwREZEBURA4Cn99yTxOmN7M9RvexP3n/RIu/gbk\nuuGXN2qNAREROS4oCByFVCLG//rQQppqEtz8y5dZPedKOPldsP4pWHZPpcsTERE5LAWBozR9bB3f\nev/p9OWKXHv3Unrf9g1I1sNvbobunZUuT0RE5JAUBIbAxadO5qPnzmLN9i6+/Fgb/uaboHcXPPrl\nSpcmIiJySAoCQ+SL75zHgqmj+OmSTdyRfTtMXgDL7obXtM6AiIgcuxQEhkg6Eef2K89kUlOar/36\nFR4/+WbA4Jc3QD5T6fJEREQOSEFgCE0dXcudHzubxnSCj/97kZaTPwI718C/f0WzCERE5JikIDDE\nTpnSxP++6kwA/nTVW8iMmg3PfC8sOqQwICIixxgFgTI494Tx/M8PnMHWTIrLu79IbtRsePIf4LG/\nUxgQEZFjioJAmVx6ejM3v+sUXupq4IOZL5EbNQue+Cb8xzcqXZqIiMgeCgJl9PHz53D9W+aypK2O\n9/feRK5R2pcPAAAgAElEQVRpJjz+92oZEBGRY4aCQJnd+LaT+MI75rGso4E/7fkiucYZIQz829XQ\n21bp8kREpMopCAyDT154Al+59FRe7Gri3d1fonvy2bDiQfjf58OmxZUuT0REqpiCwDC5+txZ/I/L\nF7C6t5Hztt7Iyyd9Em/bCD+4GP7zu1AsVrpEERGpQgoCw+iDZ8/gOx9cSLYY4+IXLuDWKd+iUDs2\nLEX8w0tg/dOVLlFERKqMgsAwu/T0Zh654QIWzRnLP73WzFt7/o4tU94GG58NYeCeD8K2FZUuU0RE\nqoSCQAVMH1vHPR9fxFcuPZWt+UbOfe1jfHPqd8lNWwSrH4bbzoUH/hI2PKPZBSIiUlbmVfBFc9ZZ\nZ/nixcfmoLz1O7v5/L+9wHPrdjGuLsm/nLOTM1/5J9j+Uthh7Bw4/UPwR1fAmJmVLVZERI4bZrbE\n3c863H5laxEws+lm9piZrTCzl8zs+mj7vWa2LLqtM7NlJa/5opmtMbOXzeziku1nmtny6LnvmpmV\nq+7hNnNcPT/5xCK+/O75dGULvO+3Tdww5rt0feDfYMEHoKMFHvs6fOeP4De6rLGIiAytcnYN5IHP\nuft8YBFwnZnNd/cr3P0Mdz8DuB94AMDM5gMfBE4FLgH+2czi0bFuA64B5ka3S8pY97CLxYz/9qbZ\nPPSZ8zl9+mh+/vw23vyzGA+ecAv+V6vhsu/B2BPgv74LS+6sdLkiIjKClC0IuHuLuy+N7ncCK4Gp\n/c9Hv+o/APw42nQZ8BN3z7j7a8Aa4GwzmwI0ufszHvox7gLeW666K+nEiQ3c/8lz+PzFJ9Pem+P6\nnyzjijtfYsWkS+HKB6B2DDz0V2HsgIiIyBAYlsGCZjYLWAg8W7L5fGCbu78SPZ4KbCx5flO0bWp0\nf//th3vPW8zMzcy3bNly5MUPs0Q8xnVvPpHffvZC3j5/Es+t28W7/+lJ/vaJLrouvQO8CPdeCe2b\nK12qiIiMAGUPAmbWQOgCuMHdO0qe+hB7WwOGnLvf4u7m7tbc3Fyutymb6WPr+JerzuJHf3E2s8bV\nc9fT6znnJ3kem3UDdG+Hez8Mud5KlykiIse5sgYBM0sSQsDd7v5AyfYEcDlwb8num4HpJY+nRds2\nR/f3314VLjxpAr++4QJuftcppBIxPrbyDfyMP4EtfyB3/1/Chmehe6emGYqIyBFJlOvA0RiAO4CV\n7n7rfk+/FVjl7qVN/r8A7jGzW4FmwqDA59y9YGYdZraI0LVwFfBP5ar7WJRKxPj4+XP48z+ewV1P\nr+cb//EJZhU2sXDVg7DqwbBTzWgYdyLMWASn/ilMPRNGzuQKEREpk7KtI2BmbwKeBJYD/Qvp3+Tu\nvzKzO4Fn3P32/V7zJeAvCDMObnD3h6PtZwF3ArXAw8CnfRCFH8vrCByJrkyee554ka3/9WMm5TZy\nQmwrC2pbmZhrwYq5sNOo6TD/Mjjt8hAKRESkqgx0HQEtKHQc68sV+OmSTdzx1Gu8tqObNFk+3rye\n99f+npk7HscynWHH2RfCRTfD9LMrW7CIiAwbBYESIzUI9CsUnX9fuY07nnyN59btAqAhUeC6GRv4\nQOFhxm17Kux44tvgzTfB1DdUsFoRERkOCgIlRnoQKLVhZw8PLtvMz5ZtZm1rNwBvq3+Vm+seYGbn\nH8JOE+fDxFPCbUL0d8wsiMUPfmARETmuKAiUqKYg0M/dWb65nfuXbOLny7bQ3pvlnNgKvtjwEKcU\nXiZZ2G/qYaIWJs6DiafCpPkw583hr4iIHJcUBEpUYxAo1Zcr8OiKbdy3eCNPrdkBXmSq7eCchu1c\nNHYnp6e3MLnvNWI7V0Mhu/eFM86Fsz8O894DiVTlPoCIiAyagkCJag8CpVrae3n85VaefGUHT63Z\nQXtvmGUwui7J5adP4s9PzHNi4VV4/sfw6u/Ci+onwplXw8IrdQVEEZHjhIJACQWBAysUnRc2tfHr\nF7dy/9JN7OgKrQELpo7i0tObecvETmav+wm27G7oawcM5lwIb7gK5r0bEunKfgARETkoBYESCgKH\nlysUeWzVdu5bvJHHXm6lUAz/XExuquEtJzbwZzWLmb/156S3PBdeUDsGTr0c5l8KM98E8ZK1qdxh\n11pY9xSMmgonvEWLG4mIDDMFgRIKAoPT2pnhidWtPPFKK0+sbmV3T27Pcwtrt/Pxhv/kT/r+nfrc\n7rCxdgyc/C6YdhZsXgxrH4f2kutHTToN3nQjzH/v3sDQvQNefhhW/xqStXDiW0NgaJgwjJ9URGTk\nUhAooSBw5ApF58XN7Ty9difLN7ezfFM7G3b1EKfA2bFVvCe5mHcmFjO6sHPvi2rHhEWMZr0JNj4L\nL94frpo4Zhac9r5wGeUNT4dt+zBoXggnXAQTToYxs2HsbKgbpxYFEZFBUhAooSAwtNp6sjy/qZ3/\neHk7j67Yxubd3Sy0NZwaW0/b2NNpmv0GzpgxloUzxjBnfD2xtnXwX/8Ef/i/UMgAFlY5nPdumPcu\nyGfgld/Amn8PAaGY3/cNU40w921wyTegcXIlPrKIyHFHQaCEgkD5uDurtnby6IptPLG6leWb28nk\n9/7SH1ef4pwTxnHeieO5oNmZ2vkCTDsbGicd+IB9HbDpOdj1GuxeF27bV4QxBzWj4OK/gzM+rBYC\nEZHDUBAooSAwfHKFIqtaOvnDxt38YUMbT7+6k60dfXuebx5Vw5TRtYxvSDG+Ic34hjQnTWrknBPG\nMbb+IGsVFIuw5Afw6H+HbBfM+RN497dDt4GIiByQgkAJBYHKcXfW7ujmP9fs4D/X7OAPG9rY2Z3d\nMyuhnxmcMrmJN80dzzlzxjF3UgPNo2qJxUp++bdthF/eCGsehVgSRs+ApmZonAJNUyBZFxZEKmSh\nkAOLh66HmeeqBUFEqo6CQAkFgWNLsejs7smyoyvL9s4+nt/Yxn+u2cmS9bvJFvZ2K9QkY8waV88J\nExqY39zE2bPH8kdTm0ivuB+evQ3aN0F36+HfcNxcOPOjcPqHoH5c+T6YiMgxREGghILA8aE3W2Dx\n+l38ft1uXtvRzdrWLl7b0U1PtrBnn3QixhnTR3P27LGcMX00C6bUMpE26GgJAxHjKYgnw9/uVlh2\nD6x4MLQSxFMw/Y9h1HQYNS3cmqZC7WhIN0G6EWqaIFEDFlMrgogc1xQESigIHL/cna0dfSzb0Maz\nr+3iudd2sXJrB6X/2E4ZVcOCqaOYOa6OhnSS+nScxpoEo2qTzJ3UyOzaPmLL74Wl/wqtKwf+5hYL\n3QuJdOiGGHdiuI2fC6n6sNpibxv0tYWZD7PeFC7WlKwZuhOQ64NYYt8Fm0REBkBBoISCwMjS3ptj\n6YbdLN/Uzgub2nh+UzutnZmD7l+XijNvciOnNo/itElpTmvs5oTUbmp6WqBjc5ipkOmATGe4n+8L\naxwUC+FvvjfMYsh2Hb64VCOcdDHMvwxmnhdaGwZ7eediEdY/Bct+HFoz4kl4xzfhjz6gVgoRGTAF\ngRIKAiPfto4+trb30ZXJh1tfnp3dGVa1dPLSlg7WtHbtM0DRDGaNq+ekSQ1MGVXLhMY0ExrTTGxM\nM21MLTPG1pNKxPa+gTt0bYMdr8DOV8Iv9drRUDM6/PUirH4kfHG3rS+pzMLztWPCvrHE3gIgtDb0\nH6NmNODw0oPQviE8P2oG9OyEXDec/E549z9qLQURGRAFgRIKAtKXK7B6WycrWzpY2dLJqq3hb//V\nF/cXjxkzx9YxZ0IDJ0yoZ+a4emaMrWPmuDqmjKohEY8d8HW4w9YXQiBofRl6dkHvLujdHboRvAiU\n/Du3/+JJAMl6OPW9cMafh0tBt62HX3wa1j0ZwsI7/j6s0BhPHv2JEZERS0GghIKAHIi7s6Mry7aO\nPlo7M7R2Ztje2ceGXT282trNq61dtPW8PigkYsakphrG1qcY15AKf+tTjKlPMaau/5akeXQt08bU\nYodqzi/k9o4z6N0NuR6Y9sYwBqFUsQiL7whrKeS6w/TJ8XNhwjyYOB8mnARj54Tb/q/dX64PVvw8\ntGDMPDcs0JSqO4IzKCLHMgWBEgoCcqR2dWd5tbWLDTt7WL+rhw07u1m/q4dt7X3s7M7us4rigTSm\nE8yb0sgpU5qYN7mJ5tE1TGoKtzF1yUOHhAPZvQ6e+jZsXQ6tqw48bqFhMow7ASaeApMXhNvE+dCx\nBRb/AJbdHUJHv9qxcPY1cPYnoH784OoRkWOWgkAJBQEpB3enJ1tgZ1eWnd0Z2npy7O7Jsrsnx+7u\nLOt39bCypYO1rV0UD/CvWSoeY1RdkrpUnNpknLpUnPp0gqmja5k5rp6Z4+qYMbaOSU011Kfj1CTi\n+y6wVCxCxybYvjKMXdi1Nrq9GtZYKL2ok8X2Pq4bBwuvDN0Pq34Fv/8/IRgkasLFokq7HMzC/o1T\n9t5qRoEXosGUhdAd0jgFxswMV5IsVSyEAZn9y0XvXh/+tq0PUzonnQaT/wim/FG4X9M0RP/viIiC\nQAkFAamk/vEJL2/tZFtHH9s6MuFvZ4b2niw92QK9uQK92QL5AyWGEv1hYfqYWuZMaGDOhHrmjG9g\n2phaRtUmGVWXpCGVIFbIhKmSW5fD1hfDuIV4KgSA+ZeGQYr9st3wh7vh6f+130DHI9A4JVw1MlkT\nvvTbNkDxAOMwYokwNbOw32yPKWeES1LPfRtMPev10yaLRYgdZHyGiOxDQaCEgoAcL3qzBTbt7mH9\nzh7W7exmw64ednRl6MkW6MkU6Mnl6ejNs7mt93XLNPeLGTTVJhlTl2J0XXLPuIXRdUmaapI01YY1\nFppqkoxrSDGhMVzzoSZuYaxCKS9C9w7obIHOreFvpiN8icfi4S++91f/rnWhlcKLUDc+XHp67GwY\nPXPv3zGzwtLQ7mEGRssLIahs+QNsfG5vcKgZBRNPjaZ1toWxFNnO8Prpf7z3NuHkMNaikAnrOeQz\nobZYMlpcKhme79gS3TaHz1I/PloX4qRQz1BMzXQPLR2lQUukQhQESigIyEiTzRfZsKuHta1drN3R\nzdb2Pjp6c7SX3Hb35GjryR62laFfY02CcfWpEBJq+0NDkgmNaSY31TCpKR3GNtSniJsRMzAz4jEj\nGTdSiRipeAwr5MKXYbph8B8s0wmvPREuSf3Kv4dplKnGaArm6DCjonXV6wPL0UrWh66NdFOoO90I\nqYZw/YpkDSRqw5d7/fjQfTJm5r6vz3bDC/fBc/8Htr8UulNGzwjBZ/QMmHZWWGzqcF0f7iGstCwL\n01Wnnhm6TAa7FoUICgL7UBCQauXudGXy7O7O0dabpbMvT3tvjo7eHG29OXZ2ZdjRld0za2Jnd5bO\nvtxhB0EeSv/Yh9nj6zkh6rqYPb6eulT0ZRb98I6bkU7GSSdipBMxapJxJjSmSfZPzXQPLQv7fwkW\ni6ElYeOz4bZ7ffiSTtSE7o9EOryu/+JThVw4RlNzdJsKDZNCS8eO1eFYO14JF7XKdu47tuJgxs0N\nXRiz3gTr/zN0rWTaQ5fH1LOgZ0foFilk974mloAZ58Dct4eZIZnOsF/PzrAc9rYVIQDsf/2MdBNM\nPzu8tnlhmCkyVC0YMqIpCJRQEBAZnL5cgY6+HO09OVo7M2wtGdvQ3puj6E6h6KElvOjkCkWyhSKZ\nXJFMociu7gybdvcy2P+8JGLGjLF1zB5fz+zx9UwdU0t9KkFtKgymrE3FSSdCeEhFAaIuFVoy9hlI\neaTcwxTOTFeYkZHrCdMt872hy2H3Oljz29Bqkeve+7qGSXDmx8LFrZqmhG3FYvhVv2tt2P+VR0L3\nx6GMmg5TTg9f+PUTYNPvYcPTsHPNvvulm0KXyMRT9u4/8dS9y1vnesM6FttXQPvm0MqxZ+GqUeGW\nilo+0o2HXpOi/3P0tcHYEyBxkMuFyzFHQaCEgoDI8OvLFVi/M3RfrNvZQzZfxEsWUyoUnWy+SCZf\nJJMv0JMtsHFXD6/t6Gb3AdZvOJREzMLKkE01TGpM0zy6lqnROg5Tx9TSVJNkZ3eG1s4sO7oy7OrO\nEo8Z9dHgy4Z0GDcxdUwtU0bV7ruq5IHkM7DhmdAaMP4kOOXSgX1Bdm0P3R7bV4bVJuvHh26EuvFh\nyufBpm92bQ/vt+2l0DXSuiqEg9IFqWIJmHBKCC+7XxtYy8aeE1gb6qkbG/7WjglBaPd6aN+4t2Uj\nloRJ80P4mHJGaGnpHxTatiG0cDRM2ntRr1HTwsyRzpZwYbDOLWGsR8MkGDU1uvDXNGiYuPd968aG\noLN/i0f/Yl393UbbXwoLbp36Xjj5HSHcyD4UBEooCIgcX3Z3Z/eMfQgzKvL05kJYyOSLUYAokM0X\n6ezLs70ztFZs78jscynrI2EGkxprmDYmLD09pj7F2LqwcFRDOkGuWCSXL5IvOtlCkYZ0gjHR82Pr\nU9Qm47T1hqmkbT1Z2ntyTGqq4dTmUUwfe5gFpgajkAtdG1uWhZaGlmVhlkiiBiadGtaOmHhKGKeQ\n644WrmoPv+wznSW3jnCNjd7d4Zbp2PsedeP2jnNIN4YWhq0vvn62B4QgUjsmdHUcKoQkakMLy6FY\nLISB/taLmlHhs3Zt698hdPF0bAoP4yk44aLQfVI/PgSr+vFhOuvOV0Nw2r4itJLE4nuDzJQzYPyJ\noVuodVUIaK2rQotKw6QQUBomhrU5Rk8P56JhUpi54h5qWvdUCIQtL4Rz37+keO1oqJ8YXjdqejiH\no6a9fort0cj1weYlsP6/wjVSZp4D0xftGZ+jIFBCQUCkOrg7u3tybGnrZdPuXja39bJpdw9dfXnG\nNaQZ35BifEOacQ0pCkWnO1OgO5OnO5tnd3eWTf2v291LS3vvAdd/OBqNNQlObW5i9vh6snmnL1eg\nLxemj6YTMRqjWR1NNUkaahKk4rE9gzCT8Rj1UctF6VTRojsFd4pFp1AsMKo2RV36KJafLuRCIEjW\nHXjAZyEXvixbXgipqT8oNE4J0z0LudAC0L45rGcRi0Fjc+gyaZgcui/6OsLsjfbNocWhZ0cIKj27\n9g0kfe3hlu0KX+79U0vnvBnqx0Hr6rCc94qfw7YXD//ZUo2hFeVwQeRQEjXhi72vbd/xHDWjQjgo\nDVIHUjdu71iVpuZwTuqjVqH6CSFA9LZB9/bQEtS1PbTyWCyaDROd481LYNPi14cyi4euollvwt7+\nVQWBfgoCIjJY+UIx/LLvzrIrunVnCyTjRjL6Yk7EjO5sfs/zu7vDuhCj65KMjqZtNtUm2Ly7lxe3\ndPDSlnZe29E96LETgzW6LknzqFqaR4dWjf2HT6QSMepTCerScepTCRprEkwZFbpTJo+qOXzXyHAr\n5KK1Jw7RmrLz1fALvXvH3kGYma4wbXXi/DDIsr+rYsdqaHk+tKLsfDX8au/fZ+IpYZnu/i/h7u0h\n2LRt2HdBrERtWKJ71nkw801hyW+zUGtfewgznVtD0GnbGGbAtG3cG5JKx5gcEQurhs48L9SQqA0t\nE+uegi1LoZjHvtKhINBPQUBEjhXdmTxb2nqpScapSfYPgIyRzRfp6MvR0Zunsy9HZyZPNl8kVwi3\nTK5IVya/zzTRrkyBeAwSsRixWJjS2d8isqWtl55sYdD1mcGEhjRNtUkSMSMRNxKxEHocKLqHlpID\nfHc4kC+EgaT5Yug+iceMulScumQY9FmfjtOQTtBYk4z+JkjEjHzRyRWcfNS1M7YhxcTGGiY2ppnY\nlKY2GadQ7G/9CO/XWJOgLhUfuu6W4dLfctC+OXR39M8c6d4RWhpqRoUuiPoJ4W+qPlrNs7h3Nc+J\np4TWgwPJdsPGZ7ET36Ig0E9BQESqjbvT0Zuntatvv+2QyRfpyRbozubpyRRo783R0h66Ura0hb/d\nmQL5QvgyzxfCF3vMjJgZhP8d8Ad6IhaLwkNYYyJf8D2rZ5ZDImZ7uksaahLURuGqLpphkisU94wr\n6R+w2t+ak4i6XppqkqEVpzb8baxJUpMMU1prk3FSiRj5gpPJF+nLhXEqqUSMCQ0hpIytG6JZK0Ns\noGMEEofbQUREjj9mxqi6MJbgWFAsOn35At2ZAl2ZPF19eTozOTr78hSLTmLPl7OFRSe7M2zvyLC9\nM9wyuQLxWAgi/V+6nX1R60hP+Lu5rfeo1sA4UvGYMbY+RSoeIx4LISgWC11INcm962SkEzES8RjJ\nmBGPxUjGw379ocoIrTr16QQNNaHVpDGdoCYZC8tqsLchJh4zUonQWhO6qix676iG+MCDiYKAiIiU\nXSxm1KUS1KUSTGgs3xLM/YGjf4ZJMm6k4+FXfSoRw4BcsRhaOQpOplCgozdPe282unBYju5MmKXS\nP5AzkyvuWbeifw2LTK5Ia1cIK61dGXZ2ZcgVwpTY3midjVy+SF++QK5wbLe8KwiIiMiIURo4DiYd\ni5Pe83SSiY3lralQdDL5An25IvliMYyhKDj5olMoFvf5tV8oOj3ZPJ19eTr6QotJJl8MS3rDnvEQ\nocumGC3mFe4X3CnsOa7ztQHWV7YgYGbTgbuASYTP+C/u/p3ouU8D1wEF4CF3/2szmwWsBF6ODvGM\nu38y2v9M4E6gFvgVcL1Xw+AGERE57sX3hJPhfd+KBwEgD3zO3ZeaWSOwxMweJQSDy4DT3T1jZhNL\nXvOqu59xgGPdBlwDPEsIApcAD5exdhERkapQtsmi7t7i7kuj+52EX/tTgU8B/8PdM9Fz2w91HDOb\nAjS5+zNRK8BdwHvLVbeIiEg1GZZVI6Jm/4WEX/QnAeeb2bNm9riZvbFk19lmtizafn60bSqwqWSf\nTdG2w73nLWbmZuZbtmwZks8hIiIy0pQ9CJhZA3A/cIO7dxC6I8YCi4DPA/dZGP3QAsyIugY+C9xj\nZoe5ePfBufst7m7ubs3NzUf9OUREREaisgYBM0sSQsDd7v5AtHkT8IAHzwFFYLy7Z9x9J4C7LwFe\nJbQebAamlRx2WrRNREREjlLZgkD0K/8OYKW731ry1M+BN0f7nASkgB1mNsHM4tH2OcBcYK27twAd\nZrYoOuZVwIPlqltERKSalHPWwHnAlcByM1sWbbsJ+AHwAzN7EcgCV7u7m9kFwFfNLEdoJfiku++K\nXncte6cPPoxmDIiIiAyJsgUBd3+KsP7BgXzkAPvfT+hGONCxFgOnDV11IiIiAsM0a0BERESOTQoC\nIiIiVUxBQEREpIpZNSzZb2bdhJUNpbyaAa3eNDx0roeHzvPw0Hkuj5nuPuFwO1VLEHB3H/jFmeWI\n6DwPH53r4aHzPDx0nitLXQMiIiJVTEFARESkilVLEPhKpQuoEjrPw0fnenjoPA8PnecKqooxAiIi\nInJg1dIiICIiIgegICAiIlLFFARERESqmIKAiIhIFVMQEBERqWIjPgiY2SVm9rKZrTGzL1S6npHC\nzKab2WNmtsLMXjKz66PtY83sUTN7Jfo7ptK1jgRmFjezP5jZL6PHOs9DzMxGm9lPzWyVma00s3N0\nnoeemd0Y/TfjRTP7sZnV6DxX1ogOAmYWB74HvAOYD3zIzOZXtqoRIw98zt3nA4uA66Jz+wXgt+4+\nF/ht9FiO3vXse70Mneeh9x3g1+4+DzidcL51noeQmU0FPgOc5e6nAXHgg+g8V9SIDgLA2cAad1/r\n7lngJ8BlFa5pRHD3FndfGt3vJPxHcyrh/P4o2u1HwHsrU+HIYWbTgHcB3y/ZrPM8hMxsFHABcAeA\nu2fdvQ2d53JIALVmlgDqCBcb0nmuoJEeBKYCG0seb4q2yRAys1nAQuBZYJK7t0RPbQUmVaiskeTb\nwF8DxZJtOs9DazbQCvww6oL5vpnVo/M8pNx9M/APwAagBWh399+g81xRIz0ISJmZWQNwP3CDu3eU\nPudh2UotXXkUzOzdwHZ3X3KwfXSeh0QCeANwm7svBLrZr3la5/noRX3/lxGCVzNQb2YfKd1H53n4\njfQgsBmYXvJ4WrRNhoCZJQkh4G53fyDavM3MpkTPTwG2V6q+EeI84FIzW0fo2rrIzP4vOs9DbROw\nyd2fjR7/lBAMdJ6H1luB19y91d1zwAPAueg8V9RIDwK/B+aa2WwzSxEGpfyiwjWNCGZmhP7Ule5+\na8lTvwCuju5fDTw43LWNJO7+RXef5u6zCP/8/s7dP4LO85By963ARjM7Odr0FmAFOs9DbQOwyMzq\nov+GvIUwvkjnuYJG/EWHzOydhD7WOPADd/96hUsaEczsTcCTwHL29l3fRBgncB8wA1gPfMDdd1Wk\nyBHGzP7k/7d3By9WlWEcx78/GSLDWc1QLVyMWNDOkQQJhFy4cDGBkODCJtGC2gWDKwNhApFWLfwT\npChw2yIyKkQNNKY0glkULaN2TWDBzDwuzjtxmZhxjKs35nw/8MK999xzznvP5jzned/7PsDZqppJ\nMoHXeaiSTNNNyHwC+Bk4Tfew5HUeoiTzwAm6fx4tAG8Cu/A6j8y2DwQkSdLGtvvQgCRJ2oSBgCRJ\nPWYgIElSjxkISJLUYwYCkiT12NioOyDp/6ctYPRXa2uOVdUvQzzHFHC7qiaHdUxJD89AQNJGjlfV\nD6PuhKRHy6EBSVuWpJLMJ/kuyWKSVwe2HW0Fe+4k+SLJcwPbziT5vrVbSZ4Z2Hah7bfYFqoiydNJ\nria529oHj/eXSv1hRkDSRq4kWRsaWK6qA+31SlVNt+V4byS51j6/DLxcVT8meQP4EDjYVkQ8Bxyq\nql9boaplYCcwAdysqneTnATep6uvcBL4qaqOwD/FaiQ9Aq4sKOlf2hyBmfVDA0kK2N3KyZLkc+AS\nXbW4dwZu3DuAe8AkcB5Yqqr31h1rCrhbVePt/V7gelU9m+Ql4JPWvgY+a0VqJA2ZQwOSRunvgdcr\ntCxlVd0E9gPfArPAl4+/a1I/GAhIelinAZI8T3ez/qa1fUleaN85BSxU1RLwKfD62ryAJLuSPLnZ\nCZLsAf6oqo+BOeDFlmWQNGTOEZC0kcE5AtBViQMYS7IAPAW8VVW/ASSZBT5KMgb8DrwGUFVfJbkI\nXOJyLHYAAABfSURBVE2ySpcFeOUB5z4MzCVZoXtgebuqVjffRdJ/4RwBSVvW5giMV9Wfo+6LpOEw\n1SZJUo+ZEZAkqcfMCEiS1GMGApIk9ZiBgCRJPWYgIElSjxkISJLUYwYCkiT12H0S5tSALqqoBAAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff6227933d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_a_training_file = os.path.join('figures', 'twohidden_100hidden_training.pdf')\n",
    "model_a.visualize_training(model_a_training_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>91</th>\n",
       "      <th>92</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0047-01</th>\n",
       "      <td>1.804567</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.082371</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.628737</td>\n",
       "      <td>0.882983</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.976136</td>\n",
       "      <td>1.912838</td>\n",
       "      <td>3.621609</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.947124</td>\n",
       "      <td>1.840908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0055-01</th>\n",
       "      <td>0.635178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.591518</td>\n",
       "      <td>0.029515</td>\n",
       "      <td>1.855888</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.964176</td>\n",
       "      <td>1.741375</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.160538</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.639663</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.046312</td>\n",
       "      <td>0.304179</td>\n",
       "      <td>6.382465</td>\n",
       "      <td>0.919127</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      1    2         3         4         5    6    7    \\\n",
       "TCGA-02-0047-01  1.804567  0.0  0.000000  0.000000  0.000000  0.0  0.0   \n",
       "TCGA-02-0055-01  0.635178  0.0  1.591518  0.029515  1.855888  0.0  0.0   \n",
       "\n",
       "                      8         9    10     ...          91        92   93   \\\n",
       "TCGA-02-0047-01  2.082371  0.000000  0.0    ...     5.628737  0.882983  0.0   \n",
       "TCGA-02-0055-01  4.964176  1.741375  0.0    ...     1.160538  0.000000  0.0   \n",
       "\n",
       "                      94        95        96        97        98        99   \\\n",
       "TCGA-02-0047-01  0.000000  1.976136  1.912838  3.621609  0.000000  1.947124   \n",
       "TCGA-02-0055-01  1.639663  0.000000  0.000000  4.046312  0.304179  6.382465   \n",
       "\n",
       "                      100  \n",
       "TCGA-02-0047-01  1.840908  \n",
       "TCGA-02-0055-01  0.919127  \n",
       "\n",
       "[2 rows x 100 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_a_compression = model_a.compress(rnaseq_df)\n",
    "model_a_file = os.path.join('data', 'encoded_rnaseq_twohidden_100model.tsv.gz')\n",
    "model_a_compression.to_csv(model_a_file, sep='\\t', compression='gzip')\n",
    "model_a_compression.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_a_weights = model_a.get_decoder_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_model_a_file = os.path.join('models', 'encoder_twohidden100_vae.hdf5')\n",
    "decoder_model_a_file = os.path.join('models', 'decoder_twohidden100_vae.hdf5')\n",
    "model_a.save_models(encoder_model_a_file, decoder_model_a_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model B Training and Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_b = Tybalt(original_dim=original_dim,\n",
    "                 hidden_dim=model_b_latent_dim,\n",
    "                 latent_dim=latent_dim,\n",
    "                 batch_size=model_b_batch_size,\n",
    "                 epochs=model_b_epochs,\n",
    "                 learning_rate=model_b_learning_rate,\n",
    "                 kappa=model_b_kappa,\n",
    "                 beta=beta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_3 (InputLayer)             (None, 5000)          0                                            \n",
      "____________________________________________________________________________________________________\n",
      "dense_6 (Dense)                  (None, 300)           1500300     input_3[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNorm (None, 300)           1200        dense_6[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "activation_4 (Activation)        (None, 300)           0           batch_normalization_4[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "dense_7 (Dense)                  (None, 100)           30100       activation_4[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "dense_8 (Dense)                  (None, 100)           30100       activation_4[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNorm (None, 100)           400         dense_7[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNorm (None, 100)           400         dense_8[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "activation_5 (Activation)        (None, 100)           0           batch_normalization_5[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "activation_6 (Activation)        (None, 100)           0           batch_normalization_6[0][0]      \n",
      "____________________________________________________________________________________________________\n",
      "lambda_2 (Lambda)                (None, 100)           0           activation_5[0][0]               \n",
      "                                                                   activation_6[0][0]               \n",
      "____________________________________________________________________________________________________\n",
      "sequential_2 (Sequential)        (None, 5000)          1535300     lambda_2[0][0]                   \n",
      "____________________________________________________________________________________________________\n",
      "custom_variational_layer_2 (Cust [(None, 5000), (None, 0           input_3[0][0]                    \n",
      "                                                                   sequential_2[1][0]               \n",
      "====================================================================================================\n",
      "Total params: 3,097,800\n",
      "Trainable params: 3,096,800\n",
      "Non-trainable params: 1,000\n",
      "____________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/gway/anaconda3/envs/vae_pancancer/lib/python2.7/site-packages/ipykernel_launcher.py:52: UserWarning: Output \"custom_variational_layer_2\" missing from loss dictionary. We assume this was done on purpose, and we will not be expecting any data to be passed to \"custom_variational_layer_2\" during training.\n"
     ]
    }
   ],
   "source": [
    "# Compile Model B\n",
    "model_b.build_encoder_layer()\n",
    "model_b.build_decoder_layer()\n",
    "model_b.compile_vae()\n",
    "model_b.get_summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9413 samples, validate on 1046 samples\n",
      "Epoch 1/100\n",
      "9413/9413 [==============================] - 2s - loss: 2976.0963 - val_loss: 2909.1827\n",
      "Epoch 2/100\n",
      "9413/9413 [==============================] - 1s - loss: 2859.1518 - val_loss: 2852.9602\n",
      "Epoch 3/100\n",
      "9413/9413 [==============================] - 1s - loss: 2835.7822 - val_loss: 2827.6078\n",
      "Epoch 4/100\n",
      "9413/9413 [==============================] - 1s - loss: 2822.8338 - val_loss: 2822.6562\n",
      "Epoch 5/100\n",
      "9413/9413 [==============================] - 1s - loss: 2814.4313 - val_loss: 2811.0809\n",
      "Epoch 6/100\n",
      "9413/9413 [==============================] - 1s - loss: 2807.8876 - val_loss: 2807.1923\n",
      "Epoch 7/100\n",
      "9413/9413 [==============================] - 1s - loss: 2803.3193 - val_loss: 2801.9632\n",
      "Epoch 8/100\n",
      "9413/9413 [==============================] - 1s - loss: 2800.3531 - val_loss: 2801.8731\n",
      "Epoch 9/100\n",
      "9413/9413 [==============================] - 1s - loss: 2796.8033 - val_loss: 2795.0505\n",
      "Epoch 10/100\n",
      "9413/9413 [==============================] - 1s - loss: 2793.8498 - val_loss: 2792.7511\n",
      "Epoch 11/100\n",
      "9413/9413 [==============================] - 1s - loss: 2792.0580 - val_loss: 2791.4050\n",
      "Epoch 12/100\n",
      "9413/9413 [==============================] - 1s - loss: 2789.9623 - val_loss: 2790.5441\n",
      "Epoch 13/100\n",
      "9413/9413 [==============================] - 1s - loss: 2788.6482 - val_loss: 2786.7315\n",
      "Epoch 14/100\n",
      "9413/9413 [==============================] - 1s - loss: 2787.1517 - val_loss: 2786.8334\n",
      "Epoch 15/100\n",
      "9413/9413 [==============================] - 1s - loss: 2785.5486 - val_loss: 2782.8495\n",
      "Epoch 16/100\n",
      "9413/9413 [==============================] - 1s - loss: 2784.6404 - val_loss: 2783.8522\n",
      "Epoch 17/100\n",
      "9413/9413 [==============================] - 1s - loss: 2782.7719 - val_loss: 2783.0421\n",
      "Epoch 18/100\n",
      "9413/9413 [==============================] - 1s - loss: 2781.9863 - val_loss: 2781.9717\n",
      "Epoch 19/100\n",
      "9413/9413 [==============================] - 1s - loss: 2781.1133 - val_loss: 2779.6485\n",
      "Epoch 20/100\n",
      "9413/9413 [==============================] - 2s - loss: 2780.4245 - val_loss: 2780.3831\n",
      "Epoch 21/100\n",
      "9413/9413 [==============================] - 2s - loss: 2779.1302 - val_loss: 2778.6101\n",
      "Epoch 22/100\n",
      "9413/9413 [==============================] - 2s - loss: 2778.6055 - val_loss: 2777.9476\n",
      "Epoch 23/100\n",
      "9413/9413 [==============================] - 2s - loss: 2777.6820 - val_loss: 2777.0861\n",
      "Epoch 24/100\n",
      "9413/9413 [==============================] - 2s - loss: 2775.9326 - val_loss: 2775.4428\n",
      "Epoch 25/100\n",
      "9413/9413 [==============================] - 2s - loss: 2776.2956 - val_loss: 2776.0726\n",
      "Epoch 26/100\n",
      "9413/9413 [==============================] - 2s - loss: 2775.1535 - val_loss: 2775.9854\n",
      "Epoch 27/100\n",
      "9413/9413 [==============================] - 2s - loss: 2775.5803 - val_loss: 2775.4990\n",
      "Epoch 28/100\n",
      "9413/9413 [==============================] - 2s - loss: 2773.7269 - val_loss: 2774.2552\n",
      "Epoch 29/100\n",
      "9413/9413 [==============================] - 2s - loss: 2773.2844 - val_loss: 2773.6002\n",
      "Epoch 30/100\n",
      "9413/9413 [==============================] - 2s - loss: 2773.0693 - val_loss: 2771.3928\n",
      "Epoch 31/100\n",
      "9413/9413 [==============================] - 2s - loss: 2772.3376 - val_loss: 2771.8702\n",
      "Epoch 32/100\n",
      "9413/9413 [==============================] - 2s - loss: 2772.1800 - val_loss: 2772.0828\n",
      "Epoch 33/100\n",
      "9413/9413 [==============================] - 2s - loss: 2771.9286 - val_loss: 2770.8244\n",
      "Epoch 34/100\n",
      "9413/9413 [==============================] - 2s - loss: 2771.4918 - val_loss: 2769.4966\n",
      "Epoch 35/100\n",
      "9413/9413 [==============================] - 1s - loss: 2770.4796 - val_loss: 2770.5737\n",
      "Epoch 36/100\n",
      "9413/9413 [==============================] - 1s - loss: 2770.3200 - val_loss: 2769.5569\n",
      "Epoch 37/100\n",
      "9413/9413 [==============================] - 1s - loss: 2769.6836 - val_loss: 2769.1391\n",
      "Epoch 38/100\n",
      "9413/9413 [==============================] - 1s - loss: 2769.3641 - val_loss: 2767.8874\n",
      "Epoch 39/100\n",
      "9413/9413 [==============================] - 1s - loss: 2768.3753 - val_loss: 2769.2286\n",
      "Epoch 40/100\n",
      "9413/9413 [==============================] - 1s - loss: 2768.1522 - val_loss: 2768.8916\n",
      "Epoch 41/100\n",
      "9413/9413 [==============================] - 1s - loss: 2767.7560 - val_loss: 2767.6186\n",
      "Epoch 42/100\n",
      "9413/9413 [==============================] - 1s - loss: 2767.4286 - val_loss: 2766.8226\n",
      "Epoch 43/100\n",
      "9413/9413 [==============================] - 1s - loss: 2766.9297 - val_loss: 2766.3980\n",
      "Epoch 44/100\n",
      "9413/9413 [==============================] - 1s - loss: 2766.3725 - val_loss: 2767.5063\n",
      "Epoch 45/100\n",
      "9413/9413 [==============================] - 1s - loss: 2766.4936 - val_loss: 2766.4490\n",
      "Epoch 46/100\n",
      "9413/9413 [==============================] - 1s - loss: 2766.0341 - val_loss: 2766.3559\n",
      "Epoch 47/100\n",
      "9413/9413 [==============================] - 1s - loss: 2765.7350 - val_loss: 2764.8949\n",
      "Epoch 48/100\n",
      "9413/9413 [==============================] - 1s - loss: 2765.1965 - val_loss: 2766.1944\n",
      "Epoch 49/100\n",
      "9413/9413 [==============================] - 1s - loss: 2765.0791 - val_loss: 2766.0875\n",
      "Epoch 50/100\n",
      "9413/9413 [==============================] - 1s - loss: 2764.7780 - val_loss: 2763.8616\n",
      "Epoch 51/100\n",
      "9413/9413 [==============================] - 1s - loss: 2764.0595 - val_loss: 2763.8204\n",
      "Epoch 52/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.9017 - val_loss: 2765.7488\n",
      "Epoch 53/100\n",
      "9413/9413 [==============================] - 1s - loss: 2764.2699 - val_loss: 2764.0976\n",
      "Epoch 54/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.4282 - val_loss: 2763.4282\n",
      "Epoch 55/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.1986 - val_loss: 2764.0629\n",
      "Epoch 56/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.2241 - val_loss: 2763.3203\n",
      "Epoch 57/100\n",
      "9413/9413 [==============================] - 1s - loss: 2762.3466 - val_loss: 2762.2408\n",
      "Epoch 58/100\n",
      "9413/9413 [==============================] - 1s - loss: 2763.0907 - val_loss: 2763.5929\n",
      "Epoch 59/100\n",
      "9413/9413 [==============================] - 1s - loss: 2761.6403 - val_loss: 2762.6243\n",
      "Epoch 60/100\n",
      "9413/9413 [==============================] - 1s - loss: 2762.1639 - val_loss: 2763.6944\n",
      "Epoch 61/100\n",
      "9413/9413 [==============================] - 1s - loss: 2761.9182 - val_loss: 2763.8620\n",
      "Epoch 62/100\n",
      "9413/9413 [==============================] - 1s - loss: 2761.4355 - val_loss: 2761.3378\n",
      "Epoch 63/100\n",
      "9413/9413 [==============================] - 1s - loss: 2760.8157 - val_loss: 2761.7237\n",
      "Epoch 64/100\n",
      "9413/9413 [==============================] - 1s - loss: 2761.3420 - val_loss: 2760.0974\n",
      "Epoch 65/100\n",
      "9413/9413 [==============================] - 1s - loss: 2760.7411 - val_loss: 2763.7263\n",
      "Epoch 66/100\n",
      "9413/9413 [==============================] - 1s - loss: 2760.1829 - val_loss: 2758.5567\n",
      "Epoch 67/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.7191 - val_loss: 2761.6695\n",
      "Epoch 68/100\n",
      "9413/9413 [==============================] - 1s - loss: 2760.0463 - val_loss: 2762.2958\n",
      "Epoch 69/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.4546 - val_loss: 2758.3040\n",
      "Epoch 70/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.8041 - val_loss: 2759.9196\n",
      "Epoch 71/100\n",
      "9413/9413 [==============================] - 1s - loss: 2760.5095 - val_loss: 2760.8132\n",
      "Epoch 72/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.6557 - val_loss: 2758.9026\n",
      "Epoch 73/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.4091 - val_loss: 2758.6991\n",
      "Epoch 74/100\n",
      "9413/9413 [==============================] - 1s - loss: 2759.1058 - val_loss: 2760.5453\n",
      "Epoch 75/100\n",
      "9413/9413 [==============================] - 2s - loss: 2758.9385 - val_loss: 2758.7857\n",
      "Epoch 76/100\n",
      "9413/9413 [==============================] - 1s - loss: 2758.2842 - val_loss: 2760.3362\n",
      "Epoch 77/100\n",
      "9413/9413 [==============================] - 1s - loss: 2758.7999 - val_loss: 2758.8773\n",
      "Epoch 78/100\n",
      "9413/9413 [==============================] - 1s - loss: 2758.6116 - val_loss: 2758.2635\n",
      "Epoch 79/100\n",
      "9413/9413 [==============================] - 1s - loss: 2757.9021 - val_loss: 2759.7957\n",
      "Epoch 80/100\n",
      "9413/9413 [==============================] - 1s - loss: 2757.8185 - val_loss: 2758.8604\n",
      "Epoch 81/100\n",
      "9413/9413 [==============================] - 1s - loss: 2758.2601 - val_loss: 2758.0821\n",
      "Epoch 82/100\n",
      "9413/9413 [==============================] - 1s - loss: 2757.5211 - val_loss: 2759.0933\n",
      "Epoch 83/100\n",
      "9413/9413 [==============================] - 1s - loss: 2757.1283 - val_loss: 2757.9790\n",
      "Epoch 84/100\n",
      "9413/9413 [==============================] - 1s - loss: 2757.3255 - val_loss: 2759.0767\n",
      "Epoch 85/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.2826 - val_loss: 2756.8941\n",
      "Epoch 86/100\n",
      "9413/9413 [==============================] - 2s - loss: 2756.2938 - val_loss: 2759.5235\n",
      "Epoch 87/100\n",
      "9413/9413 [==============================] - 2s - loss: 2756.8478 - val_loss: 2758.8628\n",
      "Epoch 88/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.7957 - val_loss: 2757.2729\n",
      "Epoch 89/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.4703 - val_loss: 2757.6702\n",
      "Epoch 90/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.3907 - val_loss: 2756.1392\n",
      "Epoch 91/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.1861 - val_loss: 2757.5099\n",
      "Epoch 92/100\n",
      "9413/9413 [==============================] - 1s - loss: 2756.2406 - val_loss: 2756.3702\n",
      "Epoch 93/100\n",
      "9413/9413 [==============================] - 1s - loss: 2755.4179 - val_loss: 2758.0628\n",
      "Epoch 94/100\n",
      "9413/9413 [==============================] - 1s - loss: 2755.9896 - val_loss: 2756.5850\n",
      "Epoch 95/100\n",
      "9413/9413 [==============================] - 1s - loss: 2755.6997 - val_loss: 2757.0054\n",
      "Epoch 96/100\n",
      "9413/9413 [==============================] - 1s - loss: 2754.9688 - val_loss: 2757.4058\n",
      "Epoch 97/100\n",
      "9413/9413 [==============================] - 1s - loss: 2755.1529 - val_loss: 2757.3296\n",
      "Epoch 98/100\n",
      "9413/9413 [==============================] - 1s - loss: 2754.5996 - val_loss: 2756.8905\n",
      "Epoch 99/100\n",
      "9413/9413 [==============================] - 1s - loss: 2755.0214 - val_loss: 2755.1875\n",
      "Epoch 100/100\n",
      "9413/9413 [==============================] - 1s - loss: 2754.6306 - val_loss: 2755.2204\n",
      "CPU times: user 3min 51s, sys: 16.1 s, total: 4min 7s\n",
      "Wall time: 3min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model_b.train_vae()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAFfCAYAAAA8vaR4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVPW9//HXZ8r2xtJ36UgRQUAQsaCxxBZbisFuzFWv\n5ZdozE3zmhuTX3Jzfyma5CZXk2ii5qpXVBJ7wa5XQUHBpSjSYWm7lO27szPz/f1xzsLQF9jZgT3v\n5+Mxj505c+bMZ44J855vO+acQ0RERIIplOkCREREJHMUBERERAJMQUBERCTAFAREREQCTEFAREQk\nwBQEREREAkxBQEREJMAUBERERAJMQUBERCTAIpkuoDP06NHDDRo0KNNliIiIdJo5c+ZUO+d67mu/\nQASBQYMGMXv27EyXISIi0mnMbGV79lPXgIiISIApCIiIiARYILoGRETk8BOPx0kmk5ku45AWCoWI\nRA7uq1wtAiIicsipq6sjFotluoxDXiwWo66u7qCOoRYBERE5pMTjccLhMHl5eZku5ZCXlZVFY2Mj\n8Xj8gFsG1CIgIiKHlGQyedDN3UESDocPqgtFQUBEROQwZmYH9XoFARERkQBTEBAREdmNgoKCTJfQ\nKRQEREREAkxBQEREZC+cc3znO99h9OjRjBkzhsceewyAdevWcfLJJzNu3DhGjx7N22+/TSKR4Gtf\n+9q2fe++++4MV79vgRiWOb+yhpqmVopzo5kuRURE9tNX7nmXdTXNHXrMvsU5PHHjCe3ad/r06cyd\nO5d58+ZRXV3Nsccey8knn8wjjzzCWWedxb/+67+SSCRobGxk7ty5VFZWMn/+fAC2bt3aoXWnQyBa\nBBzQEk9kugwRETkMvfPOO1x66aWEw2F69+7NKaecwgcffMCxxx7LX//6V+68804qKiooLCxkyJAh\nLFu2jG984xu8+OKLFBUVZbr8fQpEiwBAS6uWqRQRORy195d7Zzv55JN56623eO655/ja177Gbbfd\nxlVXXcW8efN46aWXuPfee5k2bRp/+ctfMl3qXgWiRQCgJa4gICIi+2/KlCk89thjJBIJqqqqeOut\nt5g0aRIrV66kd+/eXHfddVx77bV8+OGHVFdXk0wm+fKXv8xPf/pTPvzww0yXv0/BaRFQ14CIiByA\nL37xi7z33nuMHTsWM+MXv/gFffr04cEHH+SXv/wl0WiUgoICHnroISorK7nmmmu2rfT385//PMPV\n75s55zJdQ9pl9x3m3pv1PscM6JbpUkREZB/aLjaUlZWV4UoOD3s6X2Y2xzk3cV+vD07XgMYIiIiI\n7CI4QUBdAyIiIrsIUBBQi4CIiMjOFAREREQCLDhBoFVdAyIiIjsLThBQi4CIiMguFAREREQCLEBB\nQF0DIiKSPgUFBXt8bsWKFYwePboTq2m/4AQBrSMgIiKyiwAtMawgICJyWLr/LKhd27HHLCqDf3pp\nr7t8//vfp3///tx8880A3HnnnUQiEV5//XW2bNlCa2srP/3pT7nwwgv3662bm5u58cYbmT17NpFI\nhLvuuotTTz2VBQsWcM011xCLxUgmkzz55JOUlZXx1a9+lTVr1pBIJPjhD3/I1KlTD/hj706AgoC6\nBkREpP2mTp3Krbfeui0ITJs2jZdeeolvfvObFBUVUV1dzeTJk7ngggsws3Yf9w9/+ANmRkVFBZ98\n8glnnnkmixcv5t577+WWW27h8ssvJxaLkUgkeP755ykrK+O5554DoKampsM/Z4CCgFoEREQOS/v4\n5Z4u48ePZ+PGjaxdu5aqqiq6detGnz59+Na3vsVbb71FKBSisrKSDRs20KdPn3Yf95133uEb3/gG\nACNHjmTgwIEsXryY448/np/97GesWbOGL33pSwwbNowxY8bw7W9/m+9973ucd955TJkypcM/p8YI\niIiI7MHFF1/ME088wWOPPcbUqVN5+OGHqaqqYs6cOcydO5fevXvT3NzcIe912WWX8fTTT5Obm8u5\n557La6+9xvDhw/nwww8ZM2YMd9xxBz/5yU865L1SBahFQF0DIiKyf6ZOncp1111HdXU1b775JtOm\nTaNXr15Eo1Fef/11Vq5cud/HnDJlCg8//DCnnXYaixcvZtWqVYwYMYJly5YxZMgQvvnNb7Jq1So+\n/vhjRo4cSWlpKVdccQUlJSXcd999Hf4ZAxQE1CIgIiL756ijjqKuro7y8nL69u3L5Zdfzvnnn8+Y\nMWOYOHEiI0eO3O9j3nTTTdx4442MGTOGSCTCAw88QHZ2NtOmTeNvf/sb0WiUPn36cPvtt/PBBx/w\nne98h1AoRDQa5Z577unwz2jOuQ4/6KEmu+8wN/VnD/PQ1ydluhQREdmHWCwGQFZWVoYrOTzs6XyZ\n2Rzn3MR9vT5AYwTUNSAiIrKzQHQNGNCsrgEREUmziooKrrzyyh22ZWdnM2vWrLS9p3Nuv6Yv7iwY\nQcBMLQIiIoeJUChELBY7LLsGxowZw9y5czv1PROJxEGdq0AEgZBBTC0CIiKHhUgkQlNTE42NjYTD\n4YP6tduVOedIJBIkEgkikQP/Og9EEDAzzRoQETmMFBYWEo/HSSb1b/eemBlZWVkHFQIgIEEgZFpH\nQETkcHOwX3DSPoGYNeCNEVCqFBER2VnagoCZ9Tez181soZktMLNb/O1jzew9M6sws2fMrMjfPsjM\nmsxsrn+7N+VYE/z9l5jZ72w/O4y8FgEFARERkZ2ls0UgDnzbOTcKmAzcbGajgPuA7zvnxgB/B76T\n8pqlzrlx/u2GlO33ANcBw/zb2ftTiGHEEkmSya6/eJKIiMj+SFsQcM6tc8596N+vAxYB5cBw4C1/\ntxnAl/d2HDPrCxQ552Y6bxnEh4CL9qeWkN9+EEuoVUBERCRVp4wRMLNBwHhgFrAAuNB/6mKgf8qu\ng/1ugTfNrO1ai+XAmpR91vjb9uf9AV2BUEREZGdpDwJmVgA8CdzqnKsFvg7cZGZzgEIg5u+6Dhjg\nnBsH3AY80jZ+4ADf904zc2bmmpubAM0cEBER2Vlag4CZRfFCwMPOuekAzrlPnHNnOucmAI8CS/3t\nLc65Tf79Of724UAl0C/lsP38bXvlnLvTOWfOOcvPywM0YFBERGRn6Zw1YMD9wCLn3F0p23v5f0PA\nHcC9/uOeZhb27w/BGxS4zDm3Dqg1s8n+Ma8Cntq/Wry/ahEQERHZUTpXazgRuBKoMLO2hZdvB4aZ\n2c3+4+nAX/37JwM/MbNWIAnc4Jzb7D93E/AAkAu84N/aLeQngWaNERAREdlB2oKAc+4dvAv/7c5v\nd7P/k3jdCLs71mxg9IHWsr1FQEFAREQkVTBWFvT/qmtARERkR4EIAm1dA2oREBER2VEggsC2rgGN\nERAREdlBIILA9hYBdQ2IiIikCkQQ0GBBERGR3QtEENAYARERkd0LRBDYNmugVV0DIiIiqYIRBNQi\nICIisluBCAKhbbMG1CIgIiKSKhBBQC0CIiIiuxeIIBDSrAEREZHdCkQQMK0jICIisluBCAIhrSwo\nIiKyW4EIAhojICIisnuBCALbxwioa0BERCRVIIKAoRYBERGR3QlGEDDvpjECIiIiOwpEEADIjoTU\nNSAiIrKTAAWBsLoGREREdhKgIBBSEBAREdlJcIJANKRrDYiIiOwkOEFAXQMiIiK7CFAQUNeAiIjI\nzgIWBNQ1ICIikipAQSBMa8KRSLpMlyIiInLICE4QiHofNabuARERkW2CEwQi3kdV94CIiMh2AQoC\nYUDXGxAREUkVoCDgtwjoegMiIiLbBCcIRNU1ICIisrPgBAF1DYiIiOwiQEHA+6jNWmZYRERkmwAF\nAbUIiIiI7CwwQSBHYwRERER2EZggoFkDIiIiuwpOEIiqa0BERGRnwQkCWllQRERkFwEKAmoREBER\n2VmAgoDGCIiIiOwsOEFAswZERER2EZwgoK4BERGRXQQoCLS1CCgIiIiItAlOEGjrGtASwyIiItsE\nJwioa0BERGQXwQgCGxaQ45oABQEREZFUwQgCiRg58XpAswZERERSBSMIANmJOkDrCIiIiKQKTBDI\naq0B1DUgIiKSKjBBINxSQ8jUNSAiIpIqMEHAmmvJjoTVIiAiIpIiMEGA5q1kR0MaIyAiIpIiOEGg\naSvZkZC6BkRERFIEJwg0b1XXgIiIyE7SFgTMrL+ZvW5mC81sgZnd4m8fa2bvmVmFmT1jZkUpr/mB\nmS0xs0/N7KyU7RP8/ZeY2e/MzPa7oOYasiMhmrXEsIiIyDbpbBGIA992zo0CJgM3m9ko4D7g+865\nMcDfge8A+M9dAhwFnA38l5mF/WPdA1wHDPNvZ+93NU1byYmqRUBERCRV2oKAc26dc+5D/34dsAgo\nB4YDb/m7zQC+7N+/EPgf51yLc245sASYZGZ9gSLn3EznnAMeAi7av2rM7xoIKQiIiIik6JQxAmY2\nCBgPzAIW4H3pA1wM9PfvlwOrU162xt9W7t/fefu+3vNOM3Nm5pKY1zUQDZFIOuIJhQERERHohCBg\nZgXAk8Ctzrla4OvATWY2BygEYul4X+fcnc45c85ZKBL1Zw3oCoQiIiKpIuk8uJlF8ULAw8656QDO\nuU+AM/3nhwNf8HevZHvrAEA/f1ulf3/n7ftRSHhb1wB4QSA/e38/jYiISNeTzlkDBtwPLHLO3ZWy\nvZf/NwTcAdzrP/U0cImZZZvZYLxBge8759YBtWY22T/mVcBT+1VMKAytjeSFvRkDWktARETEk84W\ngROBK4EKM5vrb7sdGGZmN/uPpwN/BXDOLTCzacBCvBkHNzvn2r6xbwIeAHKBF/xb+4XCQJxiawJ0\nBUIREZE2aQsCzrl3gD3N9//tHl7zM+Bnu9k+Gxh9wMVYBC8I1AMaIyAiItImGCsLhrxBgkU0AOoa\nEBERaROsIODagoBaBERERCAoQcBfoDDf+V0DGiMgIiICBCUI+C0CBW1BQF0DIiIiQMCCQF5SgwVF\nRERSBSMImDc5IjehFgEREZFUwQgCfotAbqIO0BgBERGRNoEKAjnxWkBdAyIiIm0CFQSy4n6LgLoG\nREREgKAEAQyyCshqVdeAiIhIqoAEASCnhGhrDaCuARERkTbBCQK5JURibWME1DUgIiICQQoCOcWE\nY7WESKpFQERExLdfQcDMepnZ5HQVk1Y5JQAU0KgxAiIiIr59BgEze9vMis2sBPgIuN/Mfpn+0jpY\nrhcEiq2BZnUNiIiIAO1rEShwztUA5wEPA2OAs9NaVTrkFANQTINaBERERHztCQLZ/t9TgRnOuSQQ\nT19JaeJ3DRRZowYLioiI+CLt2OcNM1vo73uD30Vw+H2TtnUN0ECNBguKiIgA7QsCNwNjgWXOuVYz\nCwPXpbesNPBbBEpDDWxUEBAREQHa1zUwDPjEOVdrZmfhhYAVaa0qHfwxAqUhdQ2IiIi0aU8QmAYk\nzGww8EdgCPBgWqtKB79roFu4SYMFRUREfO0JAknnXCvwBeC/nHPXAwPSW1Ya+F0DJdaoBYVERER8\n7QkCOWbWGzgfeM3fZukrKU1y24JAg7oGREREfO0JAr8BPgXqnXOzzWwIUJPestLAHyNQZA1qERAR\nEfHtc9aAc+5PwJ9SNq0AzkhXQWkTzYVwNoVaUEhERGSbfQYBMzPgerZ/+c8A/pzOotImt4TCJq9r\nwDmH99FERESCqz3rCPwCGA/81X98NXAE8N10FZU2OSUUNK4n6SCedETDCgIiIhJs7QkCZwHHOOfi\nAGY2DZjDYRkEislLLgEcLfEk0XBwrsIsIiKyO+35JjTApTx2HI6zBgByS4iQII8WWlo1c0BERKQ9\nLQIvAS+Y2QP+46v9bYefnO3XG9DMARERkfYFge/iDRb8kv/473grDB5+2i48pCmEIiIiQPumDyaB\ne/0bAGb2Y+BHaawrPdrWEkCLComIiED7xgjszjUdWkVnyUlpEdBaAiIiIgccBA7bwYKgrgEREZE2\nBxoE3L53OQTtMFhQXQMiIiJ7HCNgZo+z+y98A0rTVlE6bbveQKO6BkRERNj7YMFnD/C5Q5ffNVBE\nA81qERAREdlzEHDOPdiZhXQKDRYUERHZQbDW2N1h+qCCgIiISLCCQHYhSQv7swbUNSAiIhKsIGBG\nPFqkJYZFRER8ewwCZjZgL88dk55y0i+RXaQxAiIiIr69tQj8o+2Omb2/03P3paec9EtmF1NEo7oG\nRERE2HsQSF09MLqX5w4rLruYXIsRjzVnuhQREZGM21sQcHu4v7vHhw9/LYF4/eYMFyIiIpJ5e1tQ\nKMfMjsT79Z96HyAn7ZWlSW5RdwA2bdqY4UpEREQyb29BIA94PuXx83va8XAS9lsEtm6uynAlIiIi\nmbe3lQUHdWIdnccPAtZSw+aGGKX5WRkuSEREJHP2ax0BM8s2s8vN7NV0FZR2KVcgXFpVn+FiRERE\nMqtdQcDMjjWze4G1wBXAA+ksKq3aLjxkjSzZqCAgIiLBtrfLEPcArgS+jjd98CGgwTl3TifVlh7+\n9QaKaWCpgoCIiATc3gYLVgJvA//snHsXwMyu7ZSq0inlCoQL1DUgIiIBt7eugd8ARwI/N7OrzSx/\nfw5sZv3N7HUzW2hmC8zsFn/7ODObaWZzzWy2mU3ytw8ysyZ/+1y/K6LtWBPMrMLMlpjZ78zswBc0\nKuwDwKDoFpZWNRzwYURERLqCPQYB59z3gP7AL4ELgdVADzM7tZ3HjgPfds6NAiYDN5vZKOAXwI+d\nc+OAf/Mft1nqnBvn325I2X4PcB0wzL+d3c4adlXYF6J5HBHeyOotjTS3aqlhEREJrr0OFnTOJZ1z\nzzrnvgSMAH4M/M7M1uzrwM65dc65D/37dcAioBxvVcIif7divAGIe2RmfYEi59xM55zDG6tw0b7e\nfy8HhNIhlCXX4pxjebVaBUREJLj2dvXBKamPnXNVzrlfO+fGAF/anzcxs0HAeGAWcCvwSzNbDfwK\n+EHKroP9boE3U96/HEgNHmv8bQeudAjZySZ6slVTCEVEJND21iLwgJktNrMfmFlZ6hPOuZ2vRrhH\nZlYAPAnc6pyrBW4EvuWc6w98C7jf33UdMMDvMrgNeMTMinZ3zHa+751m5szMrV27U6ND96EADLb1\nLN2oFgEREQmuvY0RGAr8MzASWGRmz5nZl81sbzMNdmBmUbwQ8LBzbrq/+Wqg7f7jwCT//Vqcc5v8\n+3OApcBwvNkL/VIO28/ftlfOuTudc+acs7Kysh2fLPWCwKDQepaoRUBERAJsX2MEXnfOXY3XFP93\nvF/wlWZ2174O7I/svx9Y5JxL3X8tcIp//zTgM3//nmYW9u8PwRsUuMw5tw6oNbPJ/jGvAp7aj8+4\nK79F4IjwBq0lICIigdauX/fOuXozux+v+f5OvJaC2/bxshPxFiSqMLO5/rbb8Ub//9ZvWWgGrvef\nOxn4iZm1AkngBudc27WCb8JbzTAXeMG/HTi/ReCo7Gruqq4nmXSEQgc+I1FERORwtc8gYGYj8FYX\nvALv1/xfgUf29Trn3Dtsv2zxzibsZv8n8boRdnes2cDofb1nuxX0gqxCBrGe5tYka2ua6Nctr8MO\nLyIicrjY26yB683sPbzVBbOAc5xzxzrn/ss5t7XTKkwHM+g+hF7xSoykrjkgIiKBtbcxAhcBvwbK\nnXPfcs593Ek1dY7SoUSTLfRBKwyKiEhw7bFrwDl3bmcW0um6b585oLUEREQkqNp1GeIuqTR1LQEF\nARERCabgBgG/ReDo3Gq1CIiISGAFNwj4LQIjohupro+xtTGW4YJEREQ6X3CDQF4p5BTTz60D0IBB\nEREJpOAGATMoHUpprJIQSXUPiIhIIAU3CAB0H0o42UqZbdKAQRERCaRgB4G2iw+ZphCKiEgwBTsI\n+DMHRmdv1BgBEREJJAUBYExuNSs3NdAST2S4IBERkc4V7CDgdw0Mi2wk6WDuqsP7EgoiIiL7K9hB\nILcE8rrTL+lNIXxjcVWGCxIREelcwQ4CAKVDyW1cQ27Y8canCgIiIhIsCgLdh2LJOOf2j7FoXS0b\napszXZGIiEinURDwxwl8vrc3ffBNdQ+IiEiAKAh0HwLAhMLNALyp7gEREQkQBQG/RaBHyxrKS3J5\n+7Mq4olkhosSERHpHAoC/loCtnkZp4zoSW1znLmrNY1QRESCQUEguxDye8GmJXxueE8AzR4QEZHA\nUBAAr1WgZjUnDMwjGjYNGBQRkcBQEAAYeAK4JAVv3MnEgaVUVNZQVdeS6apERETSTkEA4OTvQK+j\nYPb9fL3bXADeUquAiIgEgIIAQDQXLn4Aonmc9tlP6W8b1D0gIiKBoCDQpudw+MJdhGN1/DHn97y7\neC2JpMt0VSIiImmlIJBq3KUw9jJGuaXc0Po35q3RNEIREenaFAR2du4vqS8cwrWRF1jx7vRMVyMi\nIpJWCgI7yy4g9NUHABjw6V9o1SqDIiLShSkI7EZe/7GsyT+KccmFzJizKNPliIiIpI2CwB7kj72Q\niCX57K1pOKdBgyIi0jUpCOxBt2O+DMComrf4cJUGDYqISNekILAnPY6gsXgYJ4cqePjthZmuRkRE\nJC0UBPYi9+gLybZWWha9zNqtTZkuR0REpMMpCOyFHXk+AGeG3ueh91ZmuBoREZGOpyCwN33H4or7\nc3p4Lo/PWkpjLJ7pikRERDqUgsDemGFHnk8BjYyOzWP6h5WZrkhERKRDKQjsy8jzADgn8gF/+d/l\nJHX9ARER6UIUBPZlwGTI68EXoh+xoqqOp+etzXRFIiIiHUZBYF9CYRh5LoWJLRwXWcL/e/ETmmKJ\nTFclIiLSIRQE2mOkN3vgW/0Xs66mmfvfWZbhgkRERDqGgkB7DDkFsgqZ2PgO3fOi3PPGUjbWNWe6\nKhERkYOmINAekWwYcQ6hmlU8NPA5GmJx7p6xONNViYiIHDQFgfY669+h+zCOWv4APyp+gcc+WM2i\ndbWZrkpEROSgKAi0V0FPuOofUDyAa1r+m6tCL/Lvzy/SlQlFROSwpiCwP4r7eWGgoDd3Rh+iz7In\neP3TjZmuSkRE5IApCOyv7kPhyn+QyC7hPyJ/5rlp97GuRhckEhGRw5OCwIHoPYrwVdNJhrP5t8Tv\nueOBF2lu1doCIiJy+FEQOFDlE4h84RcUWyP/VP0L7pj+scYLiIjIYUdB4CDYMVeRGHYOJ4QXUvLx\nn3nw3RWZLklERGS/KAgcDDPCF/2eRF5Pvht9jMeff5H3lm7KdFUiIiLtpiBwsPJ7EL7ov8gizt3h\n33Prf7/HnJVbMl2ViIhIu6QtCJhZfzN73cwWmtkCM7vF3z7OzGaa2Vwzm21mk1Je8wMzW2Jmn5rZ\nWSnbJ5hZhf/c78zM0lX3ARl+Jhx7LcNDa7gt/md+++f7ePWlf0DlHFg/HxKtma5QRERktyxdA9zM\nrC/Q1zn3oZkVAnOAi4DfAHc7514ws3OB7zrnPmdmo4BHgUlAGfAKMNw5lzCz94FvArOA54HfOede\naG8tEydOdLNnz+7Qz7eLWCP88WTY9Nmuzx09Fb70p/S+v4iISAozm+Ocm7iv/SLpKsA5tw5Y59+v\nM7NFQDnggCJ/t2JgrX//QuB/nHMtwHIzWwJMMrMVQJFzbiaAmT2EFyjaHQQ6RVYeXP00VDzO1ppa\nXpi3ktqGRs7PnkvZx4/B5JugbFymqxQREdlB2oJAKjMbBIzH+0V/K/CSmf0Kr2viBH+3cmBmysvW\n+Nta/fs7bz/0FJXBibdQAlxwepzvPDGPNxe8wiNZ/07Li/9G9tefznSFIiIiO0j7YEEzKwCeBG51\nztUCNwLfcs71B74F3J+m973TzJyZubVr1+77BR0sPzvCHy47hqOnXMhbiTFkr3qTjXMPrUYMERGR\ntAYBM4vihYCHnXPT/c1XA233H8cbEwBQCfRPeXk/f1ulf3/n7XvlnLvTOWfOOSsrKzvwD3EQzIzv\nnT2CNRO+B8Dmp25neVVdRmoRERHZnXTOGjC8X/uLnHN3pTy1FjjFv38a0Da67mngEjPLNrPBwDDg\nfX+sQa2ZTfaPeRXwVLrq7mhmxmUXnc9nvc5mpFvGn++9i882KAyIiMihIZ0tAicCVwKn+VMF5/qz\nBK4Dfm1m84B/B64HcM4tAKYBC4EXgZudc20L+N8E3AcsAZZyqA0UbIdhl/wHCYtwfevDXP7Ht3l/\n+eZMlyQiIpK+6YOHkk6ZPtgez38X3v8jP4pfw8PJM/nRBUdxxXEDONSWRRARkcNfe6cPamXBznTy\ndyCrgDvyn2Zyzkp++I/5fP/JClriunKhiIhkhoJAZyroCafdQbRlE39zt/Pb4kd5bvanTP3jTNZs\nacx0dSIiEkAKAp1t8o1w9bNY6RAubHmGdwp+QK/KGZz2qze5/e8VrN6sQCAiIp1HYwQyJd4Cb9+F\ne+cuLBFjQWgEj7acwAvuBE4bP4KbTj2CwT3yM12liIgcpto7RkBBINOqFsPLd+CWzMBckhgRXk2M\n55nkiQw85vNcf/YkuuVnZbpKERE5zCgIpDikg0Cb2nVQ8Thu3qPYxoXbNi+lnHj5cQyd8Hkio86D\nnKK9HERERMSjIJDisAgCbZyD9R8TX/Q86+e/QbfNc8mnGYDagqGE/ul5Crr1yXCRIiJyqFMQSHFY\nBYGdbK5r5H+eeYFui/7GpeHXWeQG8uCw/+TMiSOZMqwn0bDGe4qIyK4UBFIczkGgzcrqerY+8U3G\nrn+SuckhXBm7naz8Er4ysR+XTRrAwO4aWCgiItspCKToCkEAgGQS9/T/weY+zKr8o7mk8V9Y2+Rd\nSfrk4T25/LgBnD6yFxG1EoiIBF57g0CkM4qRDhIKYRf8J8RbGDD/Cd7p82vWRAayumorDcsbCC9P\ncmf2afSfcjmXHDuA4rxopisWEZFDnILA4SYUhi/eC8lWQgufYgDzGAAQ9p4+tXUu33ipld+8ciJf\nnlDO1ccP4oheBbqegYiI7Ja6Bg5XzsHWlYBBJBvCWbB5Ge6hi0i2NvEvkX/l73UjACgrzuG4Id05\nbnApk4d0Z2D3PAUDEZEuTmMEUnTJILAnK96Bv30JFwrz3ol/4aHVvZi1fBNbGlsBiBBnVFk3bj1z\nBKeO6KV565OIAAAaiElEQVRAICLSRSkIpAhUEAD45Hl47ArILoRrXiCZaGXTvBeJf/YqPbZ8SEMy\ni7nJI1hXOJrRk05n9HGnYbndMl21iIh0IAWBFIELAgBzH4F/3AgYkPLfuPdoYk31ZNWu2LYpQYj1\nhaNpGnQ6PcafR8ngCaCWAhGRw5pmDQTduMugpR5m/hcMmAxDT4Mhn4OCXmQBNGxi5cdvUDHzVXpv\n+YBjaisIV3wMFXdTRSmze1zAlvE3M2FoX4b1KiAUUjAQEemK1CIgbGmIsWDpCmoXvEzJmtcY1TCL\nEupZmuzL7a3X8mnu0UwY0I2jyosZ1beIo8qK6NctV+MLREQOYeoaSKEgsH9ccy01z99J8cd/wXA8\nGz6DOxouptTqGG3LGRNazpjIKpoLB5M8/v9w/MRjyc0KZ7psERFJoSCQQkHgAK2ZA8/cAhsq9rhL\nwhnPcRKLhl7LxGOPZ2TfIvoW5agrQUQkwxQEUigIHIREK7z3B1j0NJQOhbLxUDYOeo1i7exniL57\nNz2blpJ0xivJY5iVHMmS0CAaikdS0rOMo8qKmDykO+MHlJATVauBiEhnURBIoSCQRskk7tPnaHz1\nF+RXf7zDUxtdCR8nBzM7OYK5diSR/uM57ogyph7bn15FORkqWEQkGBQEUigIdALnoPoz2DAfNizA\nbagguW4+4brKbbu0uCgfuSN43J1O/jFf4bpTRtC/NC+DRYuIdF0KAikUBDKodi2smgmrZhJf8S7h\njfMxHGtdKQ8lzqZm1GWMGjKALVu2YtWfkrv1U0pa1lFems+QPt3oVVKIRXIgrxS6DYKSQd59zVgQ\nEdkrBYEUCgKHkM3LSc68h+Sch4gkmqh3OVS7YgbYRkLWvv8tNpLL6lBfavqdzlFnfo38fqPTXLSI\nyOFHQSCFgsAhqGkLydkPEnvvj1i8keZuI3C9RpFdNopkt6F8vHorc1dUsXDNJhKtzfSwGgbYRgbY\nRgaGqhjEOrLNu37CpvyhFEz4KtkjzoReR0I0N8MfTkQk8xQEUigIHL5i8SQfrNhMUyxBWUku5SW5\nFOVGqK/byrvP/zfRT57iRPcR2RYHIEmIxoIBtPYYRaTfOKLHXkN2UU8tfiQigaMgkEJBoOuqbW7l\nkTfns/K9JxgZ/5SRodWMtFUUWyMAW1wBv05cwgvRM8jPzaG8JJcx/YoZXV7MmPJiBpbmac0DEemS\nFARSKAh0fU2xBMuq66nc0sTaLY3UblxJv7Uvcc6mB8l1jXwSGsZ/hK7lzYb+mEvSmy30t40Mjm6h\nZ7SZbuEmikNNFFkz9Vm9+KT0NGLdjqAoN0q3vCh9i3MoL8mjrCSH0vwstTCIyCFPQSCFgkCA1a6D\nGT+EiscBI1kyEGorCSVb9/nSBcmBPJM4nmeTx7PG9dy2PScaYkTvQqYM68nJw3syfkAJ0XAojR9C\nRGT/KQikUBAQlr8NL98BNWug20AoGej9Le4Pud0gpwiyi3HZBbSs/gg3fzrZK9/YFhjWF49jdtHn\neSV0AotroyzeUEcymWCsLeWsrHmcmLea6vzhrOx2PNXdxpGTk0PPwmyG9y5keO9C8rN1oU8R6VwK\nAikUBOSANG6GT56Fiidg+VuAg1AUhp9FayQP99kMslq27PKyepfDe8lRfJQcxlrXnXWuO664jJLe\ng+hRUkiP/Cy6F2TToyCbQT3yOLJP0fZxCg2bICtPMx9E5KApCKRQEJCDVrvW616Y9xhsXOBtK+gD\nwz4Pw89ic+lYWld/RGTF6+StepPc2mW7HCLuQrybPIpnk5N5KXEsNRQAMDSviRu6f8TpsdcprfGP\nnVUIBb28W8+RcNQXYdBJENL1GkSkfRQEUigISIfauMi7GFOfMXte4XDrKm+/mjVQWwk1lcQ3LCKy\nYR4ASYtS2f04apuTjKifRYQEcRdiZvJIkoToFaqhl9VQQi0hvP+P1kV7sKrvmdQOuYDogIl0L8qj\nR0EWBdkRDV4UkV0oCKRQEJBDxpYVsODvMH86rPcu0uT6jqV6yBeZEZrC/64PsaUxRm1zK7VNcRqa\nmhneUsH5oXc5N/w+JdYAeN0PHyeHMNcdwXwbzqaCYeR270959yIGds9jQGk+Q0qjDGqYR9aSl2Hx\nCxBrhKGnwrAzYehp3lLNItJlKQikUBCQQ9Kmpd7FmnocsdfdWhNJNta1sGFzLYnPXqNo5Ut03zKX\n0qYV21oLABLO2Eg31rru1Lo8JoQWU2RNADRZHq3hHIrimwFv4aWl0eE0ZnUnKzuX7JwccnNyiXYf\niI27lOI+Q4jsPBPCOVj9PlQvhnAWRLIhkuONaeg3CaK6oqTIoURBIIWCgHRJzTVQ+SGsmQ3Vi0ls\nXU1i6xoi9esIuTibs/oyMzKJZ5vHMqPxCFoJM8pW8rnQXD4XnscEW0x4N9d3SDrjzeTRPB05kwX5\nk5lYuJmzk28zvuYVCpvX7r6WboPh3F96Yyb2ZetqePtX3t9jroSR50NYsypEOpqCQAoFAQmUZMIL\nCbndto1hqGlqpbaplZxomOxoiJxImHAyRtWWLazbVMuGLTVs3FJHwfr3GVf9NEOaFwLQQA75NANe\nd8RLyYnMTI4iQoIs4mTRymBbz1fDbxCxJO9mHc/jPf4POT0GclRZEaPLixnZp5CcaBjqq+DtX8Ps\n+yER215vyQCYfBOJsZcTzi3q9NMl0lUpCKRQEBDZT+vnw4cPwqcv0NrzKNb2P4+K/ONZsjXJhtoW\nYvEkLfEELfEkza0JSmoXc23t7xnrPqHRZTMtcQr15JIghCPEwLwY57TOIMc1Ux3tyyu9vs7KnJFM\nXP8YJzW8TDYxal0ecyNjaOo2kux+Y+k3ciKD+/UjXP0JbFzoDb7c9BkUlkHZOCgb7w3YzMrf+2eJ\nNUBDlbd2xO4GVTrnddOsn+ft23aLN8OIc6B8wsGdy9UfQPehGpMhnU5BIIWCgEgnSCZx8x6BGT/C\nGqt3eXqjK+F38S/yWOJUWtneFdA/p5GvZ7/GBa0v0T25af/ekhBrsoZQ3WcKeaPPZci4U8nKikIy\nSf2nr9P4wd/otvJFookm4vl9iAz9HAw5BQaeAJuWwGczYPFLsGX57t/AQnDSbXDK9yCStV+1AfDO\nb+CVH0Gvo+C61zSOQjqVgkAKBQGRTtRS5/16TybAJby/ZsT7HkMT2TTFEjTGEgD0KsomL2t7KEjW\nrmf94g+oWjKHxLr5JJpqWEo/FiXK+ThWxqLWPpTZJsbYMo4OLWdMaBljbdm2S1JvcQV8mjOWwbFP\n6e28MLIq2ZMFbhDHhj6lh9XuUm4iWsCGniewqnAc4dwSsvIKyckvosia6T3zp4RqV3stD1/8E/Qe\n1f7z8Pav4dWfgIW98zD5Jjj753veP5mAzcuhapF3/izkvSYrr/3vKZJCQSCFgoBI1xCLJ3E4QmaE\nzDCgastmVn3wAnz2MoM2v0NPt4k6cpmd/zkqB1xI6ZGn0Jp0THt/JZuWz+PEUAWTo5+xlt68GDua\n2ckRO7RQpCqgkZ/kPMKXeI1WizKv95fplhumh9tMYWwjocYqKDuGhgn/zPKco1he3UBdc5yT1v6V\nAfPuwhX3wy57HKZd5XVrXPEkHHHGjm9SswaeuhlWzfS6I1INPxum/jeEo+k5odKlKQikUBAQCQjn\naKlaSlZJGbabX9Irqht4bPZqnpm3lqxIiEHd871bjzy652dT19xKTZN329rUytqtTaza3MjwrW/z\nf0N/omdKi0LMhWmwArpRA8BHySO4P34OR4QquTUynTWuB9eHfkx+7yFMKajk5qU3EM/uxrrLXqW8\nvL93oarKOfDopVC/AXqPgT6jvZUkex0Js+6Fpa/B0ZfARfdAaA8XtmptIr7qA7YseoPEqveJdRtO\nyRn/QlHPsvads+olXh3DPr/ncQyJVm91zZIBe15ESw45CgIpFARE5GAkk46qqvVsWjyTZY05VNTm\nM7c6zJKqBk7K+oSr7TnGNc3atq7D1uwy/qPXr3h/Sz4rNjWQdHB9+Blujz7KjMQEbkh8m0sLPuKH\nrb8j6lp5a/CtzO49leqGGNX1LVTXxyBWz93NP2JwyyKWDL2a5lN/AmZU1bWwecsWui99koFrX6Bf\n40KixHeot8Fl83TOBXx2xDWMHDSA7gVZFOdGKcmLUpQbpSQ3i6ya5fDmL6BiGrgkZBXAsf8Ek2+G\nwt7egeo2eINGZ/8F6tZBzyNhwtVw9NR9D35s2uJdpwO85bF7juzcEOGcN/YjvxdkF3Te+x5CFARS\nKAiISNpVL4FZ93gzEC78PRT3A6C5NcGqzY0s21jL6Fevpt/WD5idNYmJsfepdzl8o/UbvJ4cv8Oh\nomEjGg6RFdvK41k/YViokl+0TuXviZO4OvIyl4ZfpdgaSThjoRvM0ryjqe09iZxBk+i+6mWOWXkf\n3ZKbqXF5/C3xeVa5XtS5POrJpZUIXwm/xUXhd4iQZGVkEBX5JzCl/kWKE5tptSwW9L6AklAjA9bP\n8K7AmVUI/Y/1ruKZbMWFs9nY/yw2lp9BsvtIIr2GUpCb64WNrQvhgz9DxZMQb9r+ofJ7eoFg0BTo\ndyz0GrXj+hHOwYb5XivIxk+g/ySva6So747neetqWPyit7DVuMu9GSQ7q98I/7gJlszwHncbDL2P\ngt6jYfAUGHhiIFo2FARSKAiIyCGhphLuOQGat0JRP2JTH6UyeyhrtjQSDYfoUZBNz4JsinK9L8iN\ndS2sXP4Zo164mILmdSQJEyJBc1Yp64dfTvyYaxg4cLDXzZAq1khi1p9w79xNpGXrbktZHhrIPXYx\n05vGE08a2cT4Svgtbgg/Q/9QFQCLk+U8lDiT2cVnUtqtlMatGziu5iWmhl5lSGj99rdzYZa7vrQS\nYXRoBQBbs8tZP/xy8ou7U7h+JgXr3iXSsGHba+KhHDYUjGRpdDjFrpYRDR+Q07KbWSNl42H4OZBo\n8WZ4bJif8qTBxK/DaXdsb6FY/DI8dZM3ZbT/ZG+2x/r50LR5+8t6j4bj/hnGXJyeK30654/5aILB\np2TsYmEKAikUBETkkLH8Lfh4mvflVdinfa+pXgIPnu992U2+EUZ/pX1TEZtrYcXb3gJTLXXQUuv9\nLZ/gregYCpFMOupa4jTG4jTGEjQ2NRNd9gprm6O8HRvBJ+vr+WR9LVsaW+men8XA7nkMKs1jcvQz\n+jUupKhuCSX1S+nRvJysZDPv2jHc13I6byaPxpEaUByDbT2TQp8w1pYwPrSU4bZ62+qWVa6It5NH\nM8vG0drjSE6MfMIxze8xsG4uIed1fSRCWawvncSy0pOojfTg1DX3kFe7FHJL4fQfei0J7//RWwL7\njB/DcTd4Yyuc88ZhrPsY5j0KC5/yZnLklsLYS7znayu9cRB167z/LhO+5p3n1LEmbV/wFdMgHoPh\nZ8ERp29fyyIRh0VPwf/+FtZ5FxijqByOuQrGXwnF5Tseq7nGf99129+/3g9L4aj3OUIRyCmC4gFe\nK1NJfyjs265woSCQQkFARA57zmWsOds5R0s86a0QueedoLURsvLZVN/CgrW1zF9bw9bGVmLxJK0J\n7+Yc9CnOobwkl34FjkGtS6lNZjGroS8Va+uYX1nDko31JP2vpiIaOClUQSsR/jc5mka2B6Aoca7P\nnsE3Qk+Q47xuiMbiYSyd8hsaS48klkiyYlMjy6rqWVbVwPLqBupb4vS1zVycfJGLkjMooW7b8RKE\nacjqQUGsihBJWqJFLO93EevLz6J8yyz6rXqK3LqVO3zsZDiHxn4n0dJzDKVLpmNbVwIGoy7wgkbF\n4xCr96aDDjnVe1HbVUlj9Qf2HyScBaMughO/6U1t3QMFgRQKAiIih4+2C22tr2lifU0L62ubSSST\nFGRHKciJUJgdoTWR5N2lm3jj0400bqrk1sgT1JPHXfGv0Ez2bo/boyCLkrwskklHPOkIJ5oZFV/I\nmuZsKhOlbKIQR4i+bOKyyKtcEn5th5kiTS6LF5KTeDIxhTqXx+fDc/h8aA4jQ6sBaHFR3sz7PMuG\nfY3BI8ZSmBNh5boq8hf/nTHrpzM49hkA9aFCaqK9aMjpTaKgL/k9B9GzfDC53QdAQW8v8CVavaW4\nk3Fv4GXNam98RM0ar7Vhk3csjjgDTrzVG3+xU1BUEEihICAi0nWtqG7grc+q2FjbAnjfhwaEQyEG\ndM9lcI8CBvfIpzh39+sxJJOO2uZWqutbqKqL0dASJ55MEo+10GvNS/TY8A5risYzv/hzbI7nUNcc\npzWZJBIywiGjZ+taetct5Nm6I5i1MbytNWNnI/Lq2BTPoTq2+zoG98jnyL6FANS3JGhoidPQEicr\nEmJwj3yG9ChgSM98+nfLJbz0FXpV3Evvzd53W124GxYKETZHCEfYHNHbVykItFEQEBGRzlDfEufj\n1Vv5cNUWmluTDO2Vz9CeXhApzPECQHNrgs0NMTY3xFhe3cD8yhoqKmuYX1lDbfP2qaAhg/zsCC2t\nSWKJ5G7fb5wt4frIs4yylf61PYwkRpIQR/7fBQoCbRQERETkUOecY2NdC+GQUZAdITsSwsxIJB1r\ntzaxtKqe5dUNrNnSRElulL4luZSV5FBWnEs0EmJ9TTMbapu3/b39C6MUBNooCIiISNC0d4zAHtas\n7JAC+pvZ62a20MwWmNkt/vbHzGyuf1thZnP97YPMrCnluXtTjjXBzCrMbImZ/c4sACtBiIiIdILd\nX2mjY8SBbzvnPjSzQmCOmc1wzk1t28HMfg3+Qt2epc653SwTxT3AdcAs4HngbOCF9JUuIiISDGlr\nEXDOrXPOfejfrwMWAdtWU/B/1X8VeHRvxzGzvkCRc26m8/oxHgIuSlfdIiIiQZK2IJDKzAYB4/F+\n0beZAmxwzn2Wsm2w3y3wpplN8beVA2tS9llDSqDYy3veaWbOzNzatWsPqn4REZGuKu1BwMwKgCeB\nW51ztSlPXcqOrQHrgAF+18BtwCNmVnSg7+ucu9M5Z845Kytr5+U4RUREAiadYwQwsyheCHjYOTc9\nZXsE+BIwoW2bc64FaPHvzzGzpcBwoBLol3LYfv42EREROUjpnDVgwP3AIufcXTs9fQbwiXNuTcr+\nPc0s7N8fAgwDljnn1gG1ZjbZP+ZVwFPpqltERCRI0tk1cCJwJXBaypTAc/3nLmHXQYInAx/70wmf\nAG5wzrVdN/Im4D5gCbAUzRgQERHpEFpQSEREpAvK+IJCIiIicuhTEBAREQmwQHQNmFkD3oJGkl5l\ngBZt6Bw6151D57lz6Dynx0DnXM997RSUIOCcc7o+QZrpPHcenevOofPcOXSeM0tdAyIiIgGmICAi\nIhJgQQkCP850AQGh89x5dK47h85z59B5zqBAjBEQERGR3QtKi4CIiIjshoKAiIhIgCkIiIiIBJiC\ngIiISIApCIiIiARYlw8CZna2mX1qZkvM7PuZrqerMLP+Zva6mS00swVmdou/vdTMZpjZZ/7fbpmu\ntSsws7CZfWRmz/qPdZ47mJmVmNkTZvaJmS0ys+N1njuemX3L/zdjvpk9amY5Os+Z1aWDgJmFgT8A\n5wCjgEvNbFRmq+oy4sC3nXOjgMnAzf65/T7wqnNuGPCq/1gO3i3seL0MneeO91vgRefcSGAs3vnW\nee5AZlYOfBOY6JwbDYSBS9B5zqguHQSAScAS59wy51wM+B/gwgzX1CU459Y55z7079fh/aNZjnd+\nH/R3exC4KDMVdh1m1g/4AnBfymad5w5kZsXAycD9AM65mHNuKzrP6RABcs0sAuThXWxI5zmDunoQ\nKAdWpzxe42+TDmRmg4DxwCygt3Nunf/UeqB3hsrqSn4DfBdIpmzTee5Yg4Eq4K9+F8x9ZpaPznOH\ncs5VAr8CVgHrgBrn3MvoPGdUVw8CkmZmVgA8CdzqnKtNfc55y1Zq6cqDYGbnARudc3P2tI/Oc4eI\nAMcA9zjnxgMN7NQ8rfN88Py+/wvxglcZkG9mV6Tuo/Pc+bp6EKgE+qc87udvkw5gZlG8EPCwc266\nv3mDmfX1n+8LbMxUfV3EicAFZrYCr2vrNDP7b3SeO9oaYI1zbpb/+Am8YKDz3LHOAJY756qcc63A\ndOAEdJ4zqqsHgQ+AYWY22Myy8AalPJ3hmroEMzO8/tRFzrm7Up56Grjav3818FRn19aVOOd+4Jzr\n55wbhPe/39ecc1eg89yhnHPrgdVmNsLfdDqwEJ3njrYKmGxmef6/IafjjS/Sec6gLn/RITM7F6+P\nNQz8xTn3swyX1CWY2UnA20AF2/uub8cbJzANGACsBL7qnNuckSK7GDP7HPAvzrnzzKw7Os8dyszG\n4Q3IzAKWAdfg/VjSee5AZvZjYCrezKOPgGuBAnSeM6bLBwERERHZs67eNSAiIiJ7oSAgIiISYAoC\nIiIiAaYgICIiEmAKAiIiIgEWyXQBInLo8RcwavZvbS5yzq3owPcYBMx2zvXoqGOKyP5TEBCRPfmK\nc25+posQkfRS14CItJuZOTP7sZnNNbNPzezLKc+d7V+w52Mze9XMjkh57utmNs+/fWBmvVOe+5n/\nuk/9haows15m9oqZVfi3uzv3k4oEh1oERGRPnjCztq6BuHNuon8/4Zwb5y/H+66Zve1v/xtwinNu\noZn9E/AwcJy/IuLtwEnOufX+hariQC7QHXjPOfevZnY58P/wrq9wObDUOXcGbLtYjYikgVYWFJFd\n+GMEztu5a8DMHNDPv5wsZjYD+E+8q8XdkvLFHQKagB7AvwF1zrmf7HSsQUCFc67QfzwU+F/nXB8z\nOx54zL+9CbzkX6RGRDqYugZEJJNaUu4n8FspnXPvAeOBOcCVwOudX5pIMCgIiMj+ugbAzIbhfVnP\n9G9jzWykv8/VwEfOuTrgOeCqtnEBZlZgZjl7ewMzGwzUOuf+B7gNmOC3MohIB9MYARHZk9QxAuBd\nJQ4gYmYfAXnAPzvnNgKY2ZXAI2YWAaqAKwCcc2+Y2c+BV8wsidcKcP4+3vtzwG1mlsD7wXKDcy65\n95eIyIHQGAERaTd/jEChc64+07WISMdQU5uIiEiAqUVAREQkwNQiICIiEmAKAiIiIgGmICAiIhJg\nCgIiIiIBpiAgIiISYAoCIiIiAfb/Aa57+MsevCJpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7ff584b37e50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_b_training_file = os.path.join('figures', 'twohidden_300hidden_training.pdf')\n",
    "model_b.visualize_training(model_b_training_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>91</th>\n",
       "      <th>92</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0047-01</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.250155</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.95250</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TCGA-02-0055-01</th>\n",
       "      <td>0.556497</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.056864</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.13956</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      1    2         3    4    5    6    7    8    9    10   \\\n",
       "TCGA-02-0047-01  0.000000  0.0  1.250155  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "TCGA-02-0055-01  0.556497  0.0  0.056864  0.0  0.0  0.0  0.0  0.0  0.0  0.0   \n",
       "\n",
       "                ...   91       92   93   94   95   96   97   98   99   100  \n",
       "TCGA-02-0047-01 ...   0.0  0.95250  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "TCGA-02-0055-01 ...   0.0  0.13956  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "\n",
       "[2 rows x 100 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_b_compression = model_b.compress(rnaseq_df)\n",
    "model_b_file = os.path.join('data', 'encoded_rnaseq_twohidden_300model.tsv.gz')\n",
    "model_b_compression.to_csv(model_b_file, sep='\\t', compression='gzip')\n",
    "model_b_compression.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_b_weights = model_b.get_decoder_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_model_b_file = os.path.join('models', 'encoder_twohidden300_vae.hdf5')\n",
    "decoder_model_b_file = os.path.join('models', 'decoder_twohidden300_vae.hdf5')\n",
    "model_b.save_models(encoder_model_b_file, decoder_model_b_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract the abstract connections between samples and genes\n",
    "\n",
    "In a two layer model, there are two sets of learned weights between samples and latent features and from latent features to reconstructed samples. The first set connects the genes to the hidden layer and the second set connects the hidden layer to the latent feature activation. The two layers can be connected by matrix multiplication, which provides a direct connection from gene to latent feature. It is likely that the two layers learn different biological features, but the immediate connection is easiest to currently analyze and intuit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_weights(weights, weight_file):\n",
    "    # Multiply hidden layers together to obtain a single representation of gene weights\n",
    "    intermediate_weight_df = pd.DataFrame(weights[1][0])\n",
    "    hidden_weight_df = pd.DataFrame(weights[1][2])\n",
    "    abstracted_weight_df = intermediate_weight_df.dot(hidden_weight_df)\n",
    "\n",
    "    abstracted_weight_df.index = range(1, 101)\n",
    "    abstracted_weight_df.columns = rnaseq_df.columns\n",
    "    abstracted_weight_df.to_csv(weight_file, sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Model A\n",
    "model_a_weight_file = os.path.join('data', 'tybalt_gene_weights_twohidden100.tsv')\n",
    "extract_weights(model_a_weights, model_a_weight_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Model B\n",
    "model_b_weight_file = os.path.join('data', 'tybalt_gene_weights_twohidden300.tsv')\n",
    "extract_weights(model_b_weights, model_b_weight_file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (tybalt)",
   "language": "python",
   "name": "vae_pancancer"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}