1-intro-to-linear-algebra.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jonkrohn/ML-foundations/blob/master/notebooks/1-intro-to-linear-algebra.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# From https://github.com/jonkrohn/ML-foundations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aTOLgsbN69-P"
   },
   "source": [
    "# Intro to Linear Algebra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yqUB9FTRAxd-"
   },
   "source": [
    "This topic, *Intro to Linear Algebra*, is the first in the *Machine Learning Foundations* series. \n",
    "\n",
    "It is essential because linear algebra lies at the heart of most machine learning approaches and is especially predominant in deep learning, the branch of ML at the forefront of today’s artificial intelligence advances. Through the measured exposition of theory paired with interactive examples, you’ll develop an understanding of how linear algebra is used to solve for unknown values in high-dimensional spaces, thereby enabling machines to recognize patterns and make predictions. \n",
    "\n",
    "The content covered in *Intro to Linear Algebra* is itself foundational for all the other topics in the Machine Learning Foundations series and it is especially relevant to *Linear Algebra II*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d4tBvI88BheF"
   },
   "source": [
    "Over the course of studying this topic, you'll: \n",
    "\n",
    "* Understand the fundamentals of linear algebra, a ubiquitous approach for solving for unknowns within high-dimensional spaces. \n",
    "\n",
    "* Develop a geometric intuition of what’s going on beneath the hood of machine learning algorithms, including those used for deep learning. \n",
    "* Be able to more intimately grasp the details of machine learning papers as well as all of the other subjects that underlie ML, including calculus, statistics, and optimization algorithms. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z68nQ0ekCYhF"
   },
   "source": [
    "**Note that this Jupyter notebook is not intended to stand alone. It is the companion code to a lecture or to videos from Jon Krohn's [Machine Learning Foundations](https://github.com/jonkrohn/ML-foundations) series, which offer detail on the following:**\n",
    "\n",
    "*Segment 1: Data Structures for Algebra*\n",
    "\n",
    "* What Linear Algebra Is  \n",
    "* A Brief History of Algebra \n",
    "* Tensors \n",
    "* Scalars \n",
    "* Vectors and Vector Transposition\n",
    "* Norms and Unit Vectors\n",
    "* Basis, Orthogonal, and Orthonormal Vectors\n",
    "* Arrays in NumPy  \n",
    "* Matrices \n",
    "* Tensors in TensorFlow and PyTorch\n",
    "\n",
    "*Segment 2: Common Tensor Operations* \n",
    "\n",
    "* Tensor Transposition\n",
    "* Basic Tensor Arithmetic\n",
    "* Reduction\n",
    "* The Dot Product\n",
    "* Solving Linear Systems\n",
    "\n",
    "*Segment 3: Matrix Properties*\n",
    "\n",
    "* The Frobenius Norm\n",
    "* Matrix Multiplication\n",
    "* Symmetric and Identity Matrices\n",
    "* Matrix Inversion\n",
    "* Diagonal Matrices\n",
    "* Orthogonal Matrices\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2khww76J5w9n"
   },
   "source": [
    "## Segment 1: Data Structures for Algebra\n",
    "\n",
    "**Slides used to begin segment, with focus on introducing what linear algebra is, including hands-on paper and pencil exercises.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NgGMhK4B51oe"
   },
   "source": [
    "### Scalars (Rank 0 Tensors) in Base Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZXnTHDn_EW6b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = 25\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VF8Jam76R4KJ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x) # if we'd like more specificity (e.g., int16, uint8), we need NumPy or another numeric library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZBzYlL0mRd-P"
   },
   "outputs": [],
   "source": [
    "y = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1i-hW0bcReyy"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_sum = x + y\n",
    "py_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CpyUxB6XRk6y",
    "outputId": "09cc2e50-fec3-4d18-af9e-bbf496794a62"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(py_sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V2UiLj-JR8Ij",
    "outputId": "20f1f6e8-87c0-4251-b056-de0331a791d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_float = 25.0\n",
    "float_sum = x_float + y\n",
    "float_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ikOwjp6ASCaf",
    "outputId": "c76e5e4a-29ee-4a44-8b28-f3d755eea9cd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "float"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(float_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SgUvioyUz8T2"
   },
   "source": [
    "### Scalars in PyTorch\n",
    "\n",
    "* PyTorch tensors are designed to be pythonic, i.e., to feel and behave like NumPy arrays\n",
    "* The advantage of PyTorch tensors relative to NumPy arrays is that they easily be used for operations on GPU (see [here](https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html) for example) \n",
    "* Documentation on PyTorch tensors, including available data types, is [here](https://pytorch.org/docs/stable/tensors.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A9Hhazt2zKeD"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a211IRW_0-iY",
    "outputId": "5754b012-20fb-4d78-f136-664d48be0335"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(25)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_pt = torch.tensor(25) # type specification optional, e.g.: dtype=torch.float16\n",
    "x_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LvxzMa_HhUNB",
    "outputId": "9b9b55f6-4328-4d2e-8080-b9f4037447b4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_pt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eUyuZXlWS8T9"
   },
   "source": [
    "### Scalars in TensorFlow (ver 2.0 or later)\n",
    "\n",
    "Tensors created with a wrapper, all of which [you can read about here](https://www.tensorflow.org/guide/tensor):  \n",
    "\n",
    "* `tf.Variable`\n",
    "* `tf.constant`\n",
    "* `tf.placeholder`\n",
    "* `tf.SparseTensor`\n",
    "\n",
    "Most widely-used is `tf.Variable`, which we'll use here. \n",
    "\n",
    "As with TF tensors, in PyTorch we can similarly perform operations, and we can easily convert to and from NumPy arrays\n",
    "\n",
    "Also, a full list of tensor data types is available [here](https://www.tensorflow.org/api_docs/python/tf/dtypes/DType)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CHBYse_MEqZM"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sDv92Nh-NSOU",
    "outputId": "9d924e9d-4d6e-4794-8afe-cb986daffe57"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=int16, numpy=25>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tf = tf.Variable(25, dtype=tf.int16) # dtype is optional\n",
    "x_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EmPMBIV9RQjS",
    "outputId": "20e01030-8493-4709-c123-d8daad4ecb77"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mEILtO9pPctO"
   },
   "outputs": [],
   "source": [
    "y_tf = tf.Variable(3, dtype=tf.int16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dvvWuaw6Ph_D",
    "outputId": "c94dfddb-fc28-4f09-8a4c-10d7045ae6af"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int16, numpy=28>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tf + y_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JZVhRnX9RUGW",
    "outputId": "4f1a01f4-67f3-4164-aa05-1c28f7d99844"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int16, numpy=28>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_sum = tf.add(x_tf, y_tf)\n",
    "tf_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sVbMxT1Ey6Y3",
    "outputId": "8212034d-88d0-495e-e4ca-46ec15cac809"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_sum.numpy() # note that NumPy operations automatically convert tensors to NumPy arrays, and vice versa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LXpv69t0y-f6",
    "outputId": "b77e90ba-426a-4156-8a5a-4c803a816f5f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.int16"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(tf_sum.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VszuTUAg1uXk",
    "outputId": "ca1506e0-0d9f-4cd5-a045-cb3d02f8b611"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float16, numpy=25.0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_float = tf.Variable(25, dtype=tf.float16)\n",
    "tf_float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B5VRGo1H6010"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4CURG9Er6aZI"
   },
   "source": [
    "### Vectors (Rank 1 Tensors) in NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2GfV14gThgO0"
   },
   "outputs": [],
   "source": [
    "import numpy as np "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "T9ME4kBr4wg0",
    "outputId": "eb8ffdbd-732c-4a7b-cc00-c0a1fa9c4ff2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([25, 2, 5]) # type argument is optional, e.g.: dtype=torch.float16\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZuotxmlZL2wp",
    "outputId": "5ba8e820-7ee7-4403-ed15-64a9c501469f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OlPYy6GOaIVy",
    "outputId": "27a421b5-70ef-4069-d66f-abe101e2f74a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ME_xuvD_oTPg",
    "outputId": "9863e71d-cfc5-4ab1-96c5-a0920720acbc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[0] # zero-indexed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hXmBHZQ-nxFw",
    "outputId": "a63dabf7-bb29-4f47-9eb0-f7a6e81b93e3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NiEofCzYZBrQ"
   },
   "source": [
    "### Vector Transposition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hxGFNDx6V95l",
    "outputId": "8d063619-378a-4a34-8ee8-5f0acc5d4217"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can't transpose a regular 1D array...\n",
    "x_t = x.T\n",
    "x_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_f8E9ExDWw4p",
    "outputId": "022d4dcb-155b-459a-f806-f0c1fb170af9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_t.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2,  5]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...but can if we use nested \"matrix-style\" brackets: \n",
    "y = np.array([[25, 2, 5]])\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 3)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SPi1JqGEXXUc",
    "outputId": "d384104b-33f9-4bfc-db9d-651ebda4817d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25],\n",
       "       [ 2],\n",
       "       [ 5]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...but can transpose a matrix with a dimension of length 1, which is mathematically equivalent: \n",
    "y_t = y.T\n",
    "y_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6rzUv762Yjis",
    "outputId": "58e5c2cf-31ff-4d34-cb62-7a4eb7b98a3a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 1)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_t.shape # this is a column vector as it has 3 rows and 1 column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xVnQMLOrYtra",
    "outputId": "04840ac5-a327-4756-ab86-8be6a7fb95ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2,  5]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Column vector can be transposed back to original row vector: \n",
    "y_t.T "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QIAA2NLRZIXC",
    "outputId": "0ba30792-7b1f-4766-eb4f-8475eae88862"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 3)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_t.T.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Voj26mSpZLuh"
   },
   "source": [
    "### Zero Vectors\n",
    "\n",
    "Have no effect if added to another vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-46AbOdkZVn_",
    "outputId": "5430603b-1b57-44ad-b492-bc14d93950b3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0.])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = np.zeros(3) \n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c6xyYiwSnSGC"
   },
   "source": [
    "### Vectors in PyTorch and TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s2TGDeqXnitZ",
    "outputId": "95c7c44e-fa89-4518-eb58-4845eb394963"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([25,  2,  5])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_pt = torch.tensor([25, 2, 5])\n",
    "x_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-0jbHgc5iijG",
    "outputId": "fb0dbc48-96fc-423f-aed1-d5f6d138d7b4",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(3,) dtype=int32, numpy=array([25,  2,  5], dtype=int32)>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tf = tf.Variable([25, 2, 5])\n",
    "x_tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rTDDta1Ro4Pf"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8fU5qVTI6SLD"
   },
   "source": [
    "### $L^2$ Norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lLc2FbGG6SLD",
    "outputId": "8f8ee383-6fff-4041-da63-2c6e9a76ad09"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AN43hsl86SLG",
    "outputId": "4c2f8c9b-9556-48fc-be9f-9e2a06b5e9b2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.573423705088842"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(25**2 + 2**2 + 5**2)**(1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D9CyWo-l6SLI",
    "outputId": "f4185d9c-9cb4-4fe8-b109-be10daffa6b5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.573423705088842"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TNEMRi926SLK"
   },
   "source": [
    "So, if units in this 3-dimensional vector space are meters, then the vector $x$ has a length of 25.6m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ugQC6k4h6SLK"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PwiRlMuC6SLK"
   },
   "source": [
    "### $L^1$ Norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lcYKyc5H6SLL",
    "outputId": "dfb352cb-5dc2-4add-871c-84f90524b540"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8jNb6nYl6SLM",
    "outputId": "243613c6-f0e8-4b15-84f3-8bf4e0e2e6d4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.abs(25) + np.abs(2) + np.abs(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lQP73B916SLP"
   },
   "source": [
    "### Squared $L^2$ Norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qv1ouJ8r6SLP",
    "outputId": "7e860eb5-a223-4f34-b612-fc3ad74c0101"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eG3WSB5R6SLT",
    "outputId": "eb56aa28-02a0-474c-f2b1-1348a9dd30ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "654"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(25**2 + 2**2 + 5**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bXwzSudS6SLV",
    "outputId": "0271a41c-74b6-45b6-f7c5-5a25a6373532"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "654"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we'll cover tensor multiplication more soon but to prove point quickly: \n",
    "np.dot(x, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q3CIH9ba6SLX"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BHWxVPFC6SLX"
   },
   "source": [
    "### Max Norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vO-zfvDG6SLX",
    "outputId": "9d73f920-77f6-4ef2-c1e3-900aae9e372f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vXXLgbyW6SLZ",
    "outputId": "fe3a43e5-0281-4474-8169-694558b4cdb5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max([np.abs(25), np.abs(2), np.abs(5)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Orthogonal Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = np.array([1, 0])\n",
    "i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "j = np.array([0, 1])\n",
    "j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(i, j) # detail on the dot operation coming up..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mK3AZH53o8Br"
   },
   "source": [
    "### Matrices (Rank 2 Tensors) in NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "stk57cmaESW1",
    "outputId": "79f86778-9e66-4ad4-d1ed-7441a2945d0c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2],\n",
       "       [ 5, 26],\n",
       "       [ 3,  7]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use array() with nested brackets: \n",
    "X = np.array([[25, 2], [5, 26], [3, 7]])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IhDL4L8S6SLc",
    "outputId": "a685591f-8cd5-4700-ca95-d0e9fb00647d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 2)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q3oyaAK36SLe",
    "outputId": "a9a107e4-dc16-43c6-fd1f-7ea576a5ca48"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YN9CHzja6SLg",
    "outputId": "828e67b0-64f1-4f17-ded4-e8761032a706"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  5,  3])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select left column of matrix X (zero-indexed)\n",
    "X[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ih7nh4qC6SLi",
    "outputId": "f6d5795f-d9e3-46a2-b39c-887dcab218b7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5, 26])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select middle row of matrix X: \n",
    "X[1,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pg7numxP6SLl",
    "outputId": "b7ce7dbf-9830-4dcf-dca7-c9e8b418ee4c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2],\n",
       "       [ 5, 26]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Another slicing-by-index example: \n",
    "X[0:2, 0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HGEfZiBb6SLt"
   },
   "source": [
    "### Matrices in PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-bibT9ye6SLt",
    "outputId": "60ce2da0-6546-4438-df5a-7bffc26b3413"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[25,  2],\n",
       "        [ 5, 26],\n",
       "        [ 3,  7]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pt = torch.tensor([[25, 2], [5, 26], [3, 7]])\n",
    "X_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TBPu1L7P6SLv",
    "outputId": "a28072ab-3476-480f-d448-479e5c201463"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pt.shape # more pythonic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mTj56M16SLw",
    "outputId": "01353acd-35f1-4be2-8f7d-f1eb4c9cf53c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 5, 26])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pt[1,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E026fQlD6SLn"
   },
   "source": [
    "### Matrices in TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1gtGH6oA6SLn",
    "outputId": "fd57de2c-196b-456f-db0a-5f9caf3d597b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(3, 2) dtype=int32, numpy=\n",
       "array([[25,  2],\n",
       "       [ 5, 26],\n",
       "       [ 3,  7]], dtype=int32)>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tf = tf.Variable([[25, 2], [5, 26], [3, 7]])\n",
    "X_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4CV_KiTP6SLp",
    "outputId": "f5ca895c-724f-42a7-da8a-8342b85f073a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=2>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.rank(X_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vUsce8tC6SLq",
    "outputId": "c3d0002d-3886-4ee3-c873-470206278ac9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([3, 2], dtype=int32)>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.shape(X_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QNpfvNPj6SLr",
    "outputId": "c53f6515-bc49-436d-a3c3-60ba733571d6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([ 5, 26], dtype=int32)>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tf[1,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CodS4evY6SLy"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cMpfujF_6SLy"
   },
   "source": [
    "### Higher-Rank Tensors\n",
    "\n",
    "As an example, rank 4 tensors are common for images, where each dimension corresponds to: \n",
    "\n",
    "1. Number of images in training batch, e.g., 32\n",
    "2. Image height in pixels, e.g., 28 for [MNIST digits](http://yann.lecun.com/exdb/mnist/)\n",
    "3. Image width in pixels, e.g., 28\n",
    "4. Number of color channels, e.g., 3 for full-color images (RGB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KSZlICRR6SL1"
   },
   "outputs": [],
   "source": [
    "images_pt = torch.zeros([32, 28, 28, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6Dqj0vmh6SL2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "        [[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "        [[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "        ...,\n",
       "\n",
       "\n",
       "        [[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "        [[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "        [[[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]],\n",
       "\n",
       "         [[0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          ...,\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.],\n",
       "          [0., 0., 0.]]]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7TASTVD96SLy"
   },
   "outputs": [],
   "source": [
    "images_tf = tf.zeros([32, 28, 28, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ftOliyru6SL0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(32, 28, 28, 3), dtype=float32, numpy=\n",
       "array([[[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       ...,\n",
       "\n",
       "\n",
       "       [[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]],\n",
       "\n",
       "        [[0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         ...,\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.],\n",
       "         [0., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O3sgkdXZ6SL3"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lmG3LEZK6SL4"
   },
   "source": [
    "## Segment 2: Common Tensor Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iSHGMCxd6SL4"
   },
   "source": [
    "### Tensor Transposition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1YN1narR6SL4",
    "outputId": "b0d7d51b-5c6b-44b8-f940-72bcd568c5b8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2],\n",
       "       [ 5, 26],\n",
       "       [ 3,  7]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5hf3M_NL6SL5",
    "outputId": "86cb6394-b247-4daf-aacf-6990013326eb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  5,  3],\n",
       "       [ 2, 26,  7]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K2DuDJc_6SL6",
    "outputId": "63525788-2042-426d-f497-9ad347a803ce"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=int32, numpy=\n",
       "array([[25,  5,  3],\n",
       "       [ 2, 26,  7]], dtype=int32)>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.transpose(X_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vyBFN_4g6SL9",
    "outputId": "64d41f17-1f9a-41bc-e475-601b5937c5a3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[25,  5,  3],\n",
       "        [ 2, 26,  7]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pt.T # more pythonic!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Hp9P1jx76SL_"
   },
   "source": [
    "### Basic Arithmetical Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WxaImEUc6SMA"
   },
   "source": [
    "Adding or multiplying with scalar applies operation to all elements and tensor shape is retained: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yhXGETii6SMA",
    "outputId": "ef81375b-7b8a-4233-f8c8-e982155b9e0f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[50,  4],\n",
       "       [10, 52],\n",
       "       [ 6, 14]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X*2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KnPULtDO6SMC",
    "outputId": "48a1fdde-9324-4ddf-b304-7ea611a89b32"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[27,  4],\n",
       "       [ 7, 28],\n",
       "       [ 5,  9]])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X+2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MkfC0Gsb6SMD",
    "outputId": "5b93df73-b926-4204-8299-1fd5784b845b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[52,  6],\n",
       "       [12, 54],\n",
       "       [ 8, 16]])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X*2+2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "04bIDpGj6SMH",
    "outputId": "f5042ddc-d422-485a-f176-7dd898141492"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[52,  6],\n",
       "        [12, 54],\n",
       "        [ 8, 16]])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pt*2+2 # Python operators are overloaded; could alternatively use tf.multiply() or tf.add()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2oRBSmRL6SMI",
    "outputId": "a0a605c9-e035-4921-cda9-9e87e6a2d2e7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[52,  6],\n",
       "        [12, 54],\n",
       "        [ 8, 16]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.add(torch.mul(X_pt, 2), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OMSb9Otd6SMF",
    "outputId": "1dd22926-5187-434f-de3a-3e1f24a0fc3f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[52,  6],\n",
       "       [12, 54],\n",
       "       [ 8, 16]], dtype=int32)>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tf*2+2  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5ya2xZ4u6SMG",
    "outputId": "d8f811f9-18ae-4ee7-94b3-c36ba6d6a38f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[52,  6],\n",
       "       [12, 54],\n",
       "       [ 8, 16]], dtype=int32)>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.add(tf.multiply(X_tf, 2), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wt8Ls4076SMK"
   },
   "source": [
    "If two tensors have the same size, operations are often by default applied element-wise. This is **not matrix multiplication**, which we'll cover later, but is rather called the **Hadamard product** or simply the **element-wise product**. \n",
    "\n",
    "The mathematical notation is $A \\odot X$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KUMyU1t46SMK",
    "outputId": "e3ee77b8-9876-42c9-ccc1-d2f7abf41018"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2],\n",
       "       [ 5, 26],\n",
       "       [ 3,  7]])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UNIbp0P36SML",
    "outputId": "afd003b4-a4a4-46fd-c054-907b128dc665"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[27,  4],\n",
       "       [ 7, 28],\n",
       "       [ 5,  9]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = X+2\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[52,  6],\n",
       "       [12, 54],\n",
       "       [ 8, 16]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A + X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xKyCwGia6SMP",
    "outputId": "51292499-478b-42de-a96d-384ee70fc821"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[675,   8],\n",
       "       [ 35, 728],\n",
       "       [ 15,  63]])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A * X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B5jXGIBp6SMT"
   },
   "outputs": [],
   "source": [
    "A_pt = X_pt + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A7k6yxu36SMU",
    "outputId": "3154b393-1da8-489b-a90b-29a4bd8c05d0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[52,  6],\n",
       "        [12, 54],\n",
       "        [ 8, 16]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_pt + X_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r8vOul0m6SMW",
    "outputId": "1c9b34b2-f771-4a53-eec7-0f22279d7c2e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[675,   8],\n",
       "        [ 35, 728],\n",
       "        [ 15,  63]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_pt * X_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rQcBMSb76SMQ"
   },
   "outputs": [],
   "source": [
    "A_tf = X_tf + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x6s1wtNj6SMR",
    "outputId": "0d563e5b-0e74-4e8c-cb3d-5370a5742bc9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[52,  6],\n",
       "       [12, 54],\n",
       "       [ 8, 16]], dtype=int32)>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_tf + X_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J1D7--296SMS",
    "outputId": "7348bf95-31d6-452d-c32c-3033d74ed3a7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[675,   8],\n",
       "       [ 35, 728],\n",
       "       [ 15,  63]], dtype=int32)>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_tf * X_tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FE5f-FEq6SMY"
   },
   "source": [
    "### Reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WPJ9FVQF6SMY"
   },
   "source": [
    "Calculating the sum across all elements of a tensor is a common operation. For example: \n",
    "\n",
    "* For vector ***x*** of length *n*, we calculate $\\sum_{i=1}^{n} x_i$\n",
    "* For matrix ***X*** with *m* by *n* dimensions, we calculate $\\sum_{i=1}^{m} \\sum_{j=1}^{n} X_{i,j}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rXi2stvz6SMZ",
    "outputId": "2a41efe1-2e05-4e12-95ce-19d769e65182"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[25,  2],\n",
       "       [ 5, 26],\n",
       "       [ 3,  7]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W9FKaJbf6SMZ",
    "outputId": "78c34847-b30a-47c9-eb40-14481f5632bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "68"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3y9aw7t66SMc",
    "outputId": "b651aa38-f7b5-4096-97b4-114113b9a4fe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(68)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(X_pt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wcjRtFml6SMb",
    "outputId": "a3e3cbef-75bd-4c3b-b882-d0539cce3195"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=68>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_sum(X_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "awjH9bOz6SMc",
    "outputId": "f2d009aa-6413-423e-f704-efd5af69dff3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([33, 35])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can also be done along one specific axis alone, e.g.:\n",
    "X.sum(axis=0) # summing all rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n2SASjsn6SMd",
    "outputId": "dfbec7c7-8869-4e9a-e821-69b400998e5e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([27, 31, 10])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.sum(axis=1) # summing all columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uVnSxvSJ6SMh",
    "outputId": "1f829bed-87f8-4c70-d741-c5b35cda72a1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([33, 35])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(X_pt, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IO8drxz36SMe",
    "outputId": "0f26f716-da46-4c2b-9609-04ea31229fe1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=int32, numpy=array([27, 31, 10], dtype=int32)>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_sum(X_tf, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gdAe8S4A6SMj"
   },
   "source": [
    "Many other operations can be applied with reduction along all or a selection of axes, e.g.:\n",
    "\n",
    "* maximum\n",
    "* minimum\n",
    "* mean\n",
    "* product\n",
    "\n",
    "They're fairly straightforward and used less often than summation, so you're welcome to look them up in library docs if you ever need them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r2eW8S_46SMj"
   },
   "source": [
    "### The Dot Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LImETgD76SMj"
   },
   "source": [
    "If we have two vectors (say, ***x*** and ***y***) with the same length *n*, we can calculate the dot product between them. This is annotated several different ways, including the following: \n",
    "\n",
    "* $x \\cdot y$\n",
    "* $x^Ty$\n",
    "* $\\langle x,y \\rangle$\n",
    "\n",
    "Regardless which notation you use (I prefer the first), the calculation is the same; we calculate products in an element-wise fashion and then sum reductively across the products to a scalar value. That is, $x \\cdot y = \\sum_{i=1}^{n} x_i y_i$\n",
    "\n",
    "The dot product is ubiquitous in deep learning: It is performed at every artificial neuron in a deep neural network, which may be made up of millions (or orders of magnitude more) of these neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([25,  2,  5])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = np.array([0, 1, 2])\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "25*0 + 2*1 + 5*2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([25,  2,  5])"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 1, 2])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pt = torch.tensor([0, 1, 2])\n",
    "y_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(x_pt, y_pt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(12.)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.dot(torch.tensor([25, 2, 5.]), torch.tensor([0, 1, 2.]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(3,) dtype=int32, numpy=array([25,  2,  5], dtype=int32)>"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(3,) dtype=int32, numpy=array([0, 1, 2], dtype=int32)>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_tf = tf.Variable([0, 1, 2])\n",
    "y_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=12>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_sum(tf.multiply(x_tf, y_tf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mSmvC1cc6SMj"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bYDhomCP6SMj"
   },
   "source": [
    "## Segment 3: Matrix Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-HGU_an66SMk"
   },
   "source": [
    "### Frobenius Norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pNQHvAqN6SMk",
    "outputId": "64961322-b92c-40d3-b0f2-0c5a49c9626b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4]])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.array([[1, 2], [3, 4]])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "T-q-Tzn26SMm",
    "outputId": "20640fd4-3891-4fae-815f-9e0e8765e26e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.477225575051661"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1**2 + 2**2 + 3**2 + 4**2)**(1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YVG8qiFw6SMn",
    "outputId": "73c605f1-5829-4df6-d0a8-c480a6ed18b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.477225575051661"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(X) # same function as for vector L2 norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4c6rjVAf6SMo"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OLN-MMIe6SMo"
   },
   "source": [
    "### Matrix Multiplication (with a Vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XJw0j8cr6SMo",
    "outputId": "7e828770-57d0-45ac-9057-29ecd39a01e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 4],\n",
       "       [5, 6],\n",
       "       [7, 8]])"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[3, 4], [5, 6], [7, 8]])\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zZQ1Aupc6SMq",
    "outputId": "21f21fee-65ac-4df1-df64-a229d6c2f923"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = np.array([1, 2])\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZbeVtNyW6SMq",
    "outputId": "5f426061-c14c-461d-d3d9-b653fa50ad93"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11, 17, 23])"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(A, b) # even though technically dot products are between vectors only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "srVI55X96SMu",
    "outputId": "bf6bb74d-ed3a-48f8-a59c-b234c38dd44c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[3, 4],\n",
       "        [5, 6],\n",
       "        [7, 8]])"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_pt = torch.tensor([[3, 4], [5, 6], [7, 8]])\n",
    "A_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5SDn71Xc6SMv",
    "outputId": "e3df455d-fb57-4380-9490-0b7cdf8e7ecd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1, 2])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_pt = torch.tensor([1, 2])\n",
    "b_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OIeoJlsh6SMx",
    "outputId": "ec62cf90-1db2-426b-df96-a8d5a12e1df1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([11, 17, 23])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.matmul(A_pt, b_pt) # like np.dot(), automatically infers dims in order to perform dot product, matvec, or matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pnob9GkB6SMs",
    "outputId": "0dfb0b2e-387d-4b4f-8fa1-e0c0726ebbc5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(3, 2) dtype=int32, numpy=\n",
       "array([[3, 4],\n",
       "       [5, 6],\n",
       "       [7, 8]], dtype=int32)>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_tf = tf.Variable([[3, 4], [5, 6], [7, 8]])\n",
    "A_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vYtWxf8K6SMt",
    "outputId": "1663a520-b48d-4552-e5ff-00cadaa9922b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([1, 2], dtype=int32)>"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_tf = tf.Variable([1, 2])\n",
    "b_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NGBImWRH6SMt",
    "outputId": "c7b1509a-a044-4177-cd39-9b3d50a02304"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=int32, numpy=array([11, 17, 23], dtype=int32)>"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.linalg.matvec(A_tf, b_tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kzjZmdRR6SMy"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "21ySqay36SM5"
   },
   "source": [
    "### Matrix Multiplication (with Two Matrices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 4],\n",
       "       [5, 6],\n",
       "       [7, 8]])"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DyOEZk_c6SM5",
    "outputId": "4d25e1ef-c90b-4c46-97e7-ba7f02d98802"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 9],\n",
       "       [2, 0]])"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B = np.array([[1, 9], [2, 0]])\n",
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SfKuNxH-6SM6",
    "outputId": "2fedfbcf-0699-4e88-c560-f25e621df163"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[11, 27],\n",
       "       [17, 45],\n",
       "       [23, 63]])"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(A, B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WcnQMF0s6SNB"
   },
   "source": [
    "Note that matrix multiplication is not \"commutative\" (i.e., $AB \\neq BA$) so uncommenting the following line will throw a size mismatch error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_mwBGOXO6SNB"
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "shapes (2,2) and (3,2) not aligned: 2 (dim 1) != 3 (dim 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-119-831079463ca7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mB\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mA\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mdot\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: shapes (2,2) and (3,2) not aligned: 2 (dim 1) != 3 (dim 0)"
     ]
    }
   ],
   "source": [
    "np.dot(B, A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JrrvPoNE6SM9",
    "outputId": "d06e0ec6-477a-414d-e78f-9929af777cbb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 9],\n",
       "        [2, 0]])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B_pt = torch.from_numpy(B) # much cleaner than TF conversion\n",
    "B_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z6PfwCvX6SM-",
    "outputId": "3406e66e-a4f8-47c0-c475-adc4eddce477"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 9],\n",
       "        [2, 0]])"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# another neat way to create the same tensor with transposition: \n",
    "B_pt = torch.tensor([[1, 2], [9, 0]]).T\n",
    "B_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "16ZNRaVe6SM_",
    "outputId": "76fbd274-71eb-48bd-9ae1-81f61e8ac79f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[11, 27],\n",
       "        [17, 45],\n",
       "        [23, 63]])"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.matmul(A_pt, B_pt) # no need to change functions, unlike in TF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rkymNjE46SM8",
    "outputId": "9b049e00-757e-4b1c-beee-518e164da163"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[1, 9],\n",
       "       [2, 0]], dtype=int32)>"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B_tf = tf.convert_to_tensor(B, dtype=tf.int32)\n",
    "B_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rslTzFRk6SM8",
    "outputId": "6bbbea28-985c-4f05-fc34-458389861750"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=int32, numpy=\n",
       "array([[11, 27],\n",
       "       [17, 45],\n",
       "       [23, 63]], dtype=int32)>"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.matmul(A_tf, B_tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0eBiTmPp6SNC"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L2H9F-DQ6SMz"
   },
   "source": [
    "### Symmetric Matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5YsPoWo76SMz",
    "outputId": "71d60661-da03-4528-9ec7-30f80308095e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2],\n",
       "       [1, 7, 8],\n",
       "       [2, 8, 9]])"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_sym = np.array([[0, 1, 2], [1, 7, 8], [2, 8, 9]])\n",
    "X_sym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2],\n",
       "       [1, 7, 8],\n",
       "       [2, 8, 9]])"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_sym.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Jv40-i9H6SM1",
    "outputId": "c5e07a6b-bf88-498f-d9b0-72e0ec263dbf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ True,  True,  True],\n",
       "       [ True,  True,  True],\n",
       "       [ True,  True,  True]])"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_sym.T == X_sym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QZFoUFkq6SM2"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mq_c3ftZ6SM2"
   },
   "source": [
    "### Identity Matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KVSNbH-Z6SM2",
    "outputId": "c4f2205a-b74a-4f93-8bf1-fc58f050ffb9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 0, 0],\n",
       "        [0, 1, 0],\n",
       "        [0, 0, 1]])"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "I = torch.tensor([[1, 0, 0], [0, 1, 0], [0, 0, 1]])\n",
    "I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wcoPDhvR6SM3",
    "outputId": "35d8cb2d-8ea9-4a7b-fbf6-0f72f1b2e463"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([25,  2,  5])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_pt = torch.tensor([25, 2, 5])\n",
    "x_pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tuA4RsMv6SM4",
    "outputId": "8f3efdea-b1da-4f23-bf26-29c2398161fa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([25,  2,  5])"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.matmul(I, x_pt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bgDiOYLk6SM5"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answers to Matrix Multiplication Qs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pINsKNxH6SNC",
    "outputId": "35c5c33d-a1d0-4cad-821f-9fc25226037d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5],\n",
       "        [6, 7, 8]])"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M_q = torch.tensor([[0, 1, 2], [3, 4, 5], [6, 7, 8]])\n",
    "M_q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gfjWd8OO6SNE",
    "outputId": "3a5a63b9-8215-4081-b9fa-c0aea00557f2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1,  0],\n",
       "        [ 1,  1],\n",
       "        [-2,  2]])"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "V_q = torch.tensor([[-1, 1, -2], [0, 1, 2]]).T\n",
    "V_q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "boSkaV2M6SNF",
    "outputId": "b1ecf179-111d-4f22-b7ab-c2e63707bdb9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ -3,   5],\n",
       "        [ -9,  14],\n",
       "        [-15,  23]])"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.matmul(M_q, V_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "slSNKUcN6SNG"
   },
   "source": [
    "### Matrix Inversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EW0i5ZRk6SNG",
    "outputId": "74a5959f-4868-42d0-b2dd-8021ce0aed9b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4,  2],\n",
       "       [-5, -3]])"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.array([[4, 2], [-5, -3]])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hTYpxaWR6SNI",
    "outputId": "723533a0-7ee9-41d4-84de-70ab1432519c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.5,  1. ],\n",
       "       [-2.5, -2. ]])"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xinv = np.linalg.inv(X)\n",
    "Xinv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q5sQqFaz6SNK",
    "outputId": "6e56c889-eb99-407c-d0bb-289cd204a19f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4, -7])"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = np.array([4, -7])\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PK7m6F1I6SNL",
    "outputId": "7a8ebc4f-ef10-4d43-e2ab-9a43087a5258"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.,  4.])"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w = np.dot(Xinv, y)\n",
    "w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show that $y = Xw$: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4., -7.])"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(X, w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AMxROg326SNN"
   },
   "source": [
    "**Return to slides here.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N8ZxpgcN6SNO"
   },
   "source": [
    "### Matrix Inversion Where No Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RYORHY4E6SNO",
    "outputId": "2f0aaacc-d0b6-47f1-b5f8-74598842a170"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-4,  1],\n",
       "       [-8,  2]])"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.array([[-4, 1], [-8, 2]])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o7GcISdI6SNP"
   },
   "outputs": [
    {
     "ename": "LinAlgError",
     "evalue": "Singular matrix",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLinAlgError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-141-e5f69e400042>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Uncommenting the following line results in a \"singular matrix\" error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mXinv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36minv\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/ML/lib/python3.7/site-packages/numpy/linalg/linalg.py\u001b[0m in \u001b[0;36minv\u001b[0;34m(a)\u001b[0m\n\u001b[1;32m    545\u001b[0m     \u001b[0msignature\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'D->D'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misComplexType\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'd->d'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    546\u001b[0m     \u001b[0mextobj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_linalg_error_extobj\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_raise_linalgerror_singular\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 547\u001b[0;31m     \u001b[0mainv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_umath_linalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msignature\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msignature\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mextobj\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mextobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    548\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mwrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mainv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult_t\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    549\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/ML/lib/python3.7/site-packages/numpy/linalg/linalg.py\u001b[0m in \u001b[0;36m_raise_linalgerror_singular\u001b[0;34m(err, flag)\u001b[0m\n\u001b[1;32m     95\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     96\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_raise_linalgerror_singular\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 97\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLinAlgError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Singular matrix\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     98\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     99\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_raise_linalgerror_nonposdef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLinAlgError\u001b[0m: Singular matrix"
     ]
    }
   ],
   "source": [
    "# Uncommenting the following line results in a \"singular matrix\" error\n",
    "Xinv = np.linalg.inv(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FPb5cW8r6SNQ"
   },
   "source": [
    "Feel free to try inverting a non-square matrix; this will throw an error too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "1-intro-to-linear-algebra.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "ML",
   "language": "python",
   "name": "ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}