Skip to content

Instantly share code, notes, and snippets.

@sagarmainkar

sagarmainkar/GradientDescent.ipynb

Created August 25, 2018 04:40
Show Gist options
  • Save sagarmainkar/41d135a04d7d3bc4098f0664fe20cf3c to your computer and use it in GitHub Desktop.
Save sagarmainkar/41d135a04d7d3bc4098f0664fe20cf3c to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
GradientDescent.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:23.993097Z",
"start_time": "2018-08-25T03:56:23.138209Z"
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:24.006246Z",
"start_time": "2018-08-25T03:56:23.998204Z"
}
},
"outputs": [],
"source": [
"plt.style.use(['ggplot'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create Data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5> Generate some data with:\n",
"\\begin{equation} \\theta_0= 4 \\end{equation} \n",
"\\begin{equation} \\theta_1= 3 \\end{equation} \n",
"\n",
"Add some Gaussian noise to the data"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:24.027559Z",
"start_time": "2018-08-25T03:56:24.016052Z"
}
},
"outputs": [],
"source": [
"X = 2 * np.random.rand(100,1)\n",
"y = 4 +3 * X+np.random.randn(100,1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot our data to check the relation between X and Y"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:24.780299Z",
"start_time": "2018-08-25T03:56:24.032204Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAESCAYAAAD5d3KwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Wt0VNXZB/D/yWQCSSeESUbAcFkICVqUQDVCUEDEsQLS1sXCgIpW0dXFxbooiiBvq75almMliChZaHUhS0uLWsBLAV3ToKHcyhBDuAiKUoU3ljgkQCKBmcmc90OcMSeZSeZM9rlk5v/7onPdz5wc9nP2c87eR5JlWQYREdEPUowOgIiIzIWJgYiIFJgYiIhIgYmBiIgUmBiIiEiBiYGIiBSYGIiISIGJgYiIFJgYiIhIgYmBiIgUUo0OoD3V1dVGh9Auh8MBr9drdBgdYpxiMU6xGKc4ubm5Qr6HIwYiIlJgYiAiIgUmBiIiUmBiICIiBSYGIiJSYGIgIiIFJgYiIlJgYiAiIgUmBiIiUhCaGEpLS/HAAw/g4YcfbvPae++9h+LiYpw7d05kk0REJJjQxDB+/HgsWbKkzfNerxcHDhyAw+EQ2RwREWlAaGIYOnQobDZbm+fXrl2Lu+66C5IkiWyOiIg0oPk5Bo/Hg+zsbAwcOFDrpoiISABNV1e9ePEiNmzYgN///vcxvd/tdsPtdgMAXC6X6UtPqamppo8RYJyiMU6xGKf5aJoYTp06hZqaGixcuBAAcPr0aSxatAjPPPMMevbs2eb9TqcTTqcz/NjsS9x2hWV4AcYpGuMUi3GKI2rZbU0Tw4ABA/Dqq6+GH8+bNw/PPPMMevTooWWzRETUCUITw4oVK3D48GHU19dj9uzZKC4uxoQJE0Q2QUREGhOaGObPn9/u66tWrRLZHBERaYAzn4mISIGJgYiIFJgYiIhIgYmBiIgUmBiIiEiBiYGIiBSYGIiISIGJgYiIFJgYiIhIgYmBiIgUmBiIiEiBiYGIiBSYGIiISIGJgYiIFJgYiIhIgYmBiIgUmBiIiEiBiYGIiBSYGIiISIGJgYiIFFJFfllpaSkqKiqQlZWFkpISAMAbb7yBffv2ITU1Fb1798bcuXPxk5/8RGSzREQkkNARw/jx47FkyRLFcwUFBSgpKcGyZctw6aWXYuPGjSKbJCIiwYQmhqFDh8JmsymeGz58OCwWCwBgyJAhqK2tFdkkEREJpus5hrKyMowYMULPJomISCWh5xjas2HDBlgsFowdOzbqe9xuN9xuNwDA5XLB4XDoFV5cUlNTTR8jwDhFY5xiMU7z0SUxfPzxx9i3bx8ef/xxSJIU9X1OpxNOpzP82Ov16hFe3BwOh+ljBBinaIxTLMYpTm5urpDv0byUVFlZiXfffReLFi1Ct27dtG6OiIg6SeiIYcWKFTh8+DDq6+sxe/ZsFBcXY+PGjQgEAnj66acBAPn5+fjNb34jslkiIhJIaGKYP39+m+cmTJggsgkiItIYZz4TEZECEwMRJS2Px4oXX7TB47EaHYqp6Ha5KhGRmXg8VkyfngO/X4LVasP69adRWOg3OixT4IiBiJLSrl3d4PdLaGqS4PdL2LWra101qeVohyMGIkpKo0dfhNXavISP1Spj9OiLBkcUO61HO0wMRJSUCgv9WL/+NHbt6obRoy92qTJSy9FO6DETAxGRAIWF/i6VEEK0Hu0wMRARdTFaj3aYGIiIuiAtRzu8KomIyOT0nm/BEQMRkUl4PNY25SEj5lswMRARmUDLBGCx2DB9+nlMm9ao+RVIkbCURERkAi0TgM8n4c03f4Lp03NgtzfBapVhsci6zbfgiIGIyARCl6AGg4AsA7Iswe8H6uosus+3YGIgIjKB0CWo77yTgfXrM9DU9OMIQe/5FkwMREQm0ZwAzmLatPOGzshmYiAiMhmjZ2Tz5DMRESkwMRARkQITAxERKQg9x1BaWoqKigpkZWWhpKQEANDQ0IDnn38e3333HS655BL87ne/g81mE9ksESWJSDODSTyhI4bx48djyZIliuc2bdqEYcOGYeXKlRg2bBg2bdokskkiShKhmcHPPZeJ6dNzeJ9mDQlNDEOHDm0zGti7dy9uuOEGAMANN9yAvXv3imySiJKE3rfi1HvhOjPR/HLVs2fPwm63AwDsdjvOnTundZNElID0vBVnpIXrJk7UrDlVcelRSjPVPAa32w232w0AcLlccDgcBkfUvtTUVNPHCDBO0RinWLHG2bOnhLvvDgKQMHNmEEVFWZrFVFWVoli4rqrKjilTUoRsz927JZSXSxg3TkZRkdzh8y1fnzEjFT4fkJaWia1bAxHfJ4LmiSErKwt1dXWw2+2oq6tDjx49or7X6XTC6XSGH3u9Xq3D6xSHw2H6GAHGKRrjFCuWOJVH8DKmTDkDr1e7I+aCAius1hwAzaOTgoI6BAJZnd6erX9HaAntaM+3tGWLDT5f5g+L7MnYsqUReXkNivfk5uZ2Kr4QzS9XLSwsxCeffAIA+OSTT3Dttddq3SQRJRi9zy+E1i1auLBe6P0Pov2OWH5fcylNn1VWhY4YVqxYgcOHD6O+vh6zZ89GcXExbrvtNjz//PMoKyuDw+HAggULRDZJRElAz/MLIVosSxHtd8Ty+7S+z3NLkizL2hSpBKiurjY6hHYl0lDdDBinWIkWp9FzGERtz9a/I/TYbm9CXZ2lU79PVCnJVCefiYii6cwRvJ5JpaO2Wv6OWM4tGIGJgYgSlsdjxTvvpP9wfwPx90yOdPSv5v7MRty2MxZMDERkOiKO8EOd9MWLEpoL5mI730hJQG1Hb8S5k1gwMRCRalqWZtQedUcT6qRlWQIgQ5LEXs0TKQmo7ej1PKGsBhMDEakiquOORlR5pWUnbbEA06efx7Rp54XFGikJxNPRG31TnkiYGIhIFa3r4qLKK6KPxnfvlrBli01xD+ZI39/Zjt7oq68AJgYiUknrurjIDl3U0bjHY/1hOYpMxShJ9NF+tNGY3smCiYGIVNGjLm628squXd3g80HVKCmezjzSaAyApqW7SJgYiEg1s3XcWhs9+iLS0jLh88V2Ajve8zCRRmMtk4UsA8uXZ2LBgvrkWV2ViMRpXROn+BUW+rF1awBbtjTGtD3jPQ8TbTRmtdogy0AwCGzf3g179qRpOnJgYiBKQNFq4hQfj8eKqiop5iTbmfMwrUdjoWSxfHkmtm/vhmBQgt+v7WQ4JgaiBBRPTZwiU5aFcmJKsqLPwxQW+rFgQT327EmD36/9ZDgmBqJOMsPlha2prYlTdJ0pC4ncH/ScDMfEQNSKmo5e68le8VJbEwfMmeDMwEzLVuh10p+JgaiFRFkEDQCKiuQ2d/iKxkwJTm2C0jqhhY7Uq6rsKCioM83fV0sx3cHtlVdeQXFxMWpra9u8Vl1djTvuuANr1qwRHhyR3tTeKUzPu2ppSe87pEUTSlDPPZeJ6dNz4PFYhb4/XoWFfjz6aDApkgIQY2IYMmQIAODYsWNtXlu7di0yMjJw++23i42MyABqO3qtbgGpN7MkOLUJyiwJLdHEVEpqmRhGjhwZfr6iogKffvop7r//fthsNm0iJNJRoiyCppZZVvlUW883U/0/kcSUGHJzc2Gz2fDll1+GnwsEAli7di369++Pm2++WbMAifSWCB19PMzwu9UmKLMktEQT88nn/Px8HD16FLIsQ5IkbN68Gd9++y3+8Ic/ICUlpooUEVGH1CYoMyS0RKMqMXz66aeorq6GzWbD3//+d1x77bUYNmxYTJ//4IMPUFZWBkmS0L9/f8ydOxdpaWlxB05ERNqI+VC/5XmGv/71rwgEArjnnnti+mxtbS22bNkCl8uFkpISBINB7Ny5M76IiX7g8Vjx4os2za5EIRKtq+yzqkYMkiShrKwMR44cwS9/+Uv07t075oaCwSB8Ph8sFgt8Ph/sdntcAZsZJwjpR4vr7rX6+3G/IEDMPqvXvhRzYkhPT0e/fv3w2WefoWfPnpg6dWrMjWRnZ+MXv/gF5syZg7S0NAwfPhzDhw+PK2CzMtMEoWQgemKZVn8/7hcU0tl9Vs99SdXM57y8PJw4cQJ33nkn0tPTY/5cQ0MD9u7di1WrViEjIwPLly9HeXk5xo0bp3if2+2G2+0GALhcLjgcDjXh6S41NTUcY1VViuKPXlVlx8SJQSPDC2sZp5mpiXPSJAkvvAD4fDLS0oBJk9LhcHSPu201fz81cRq5XyTi3z2a3bsllJdLGDdORlGRLCgypc7G2dl9Vs99KebEEAgEcPjwYQwePBg33HCDqkYOHDiAXr16oUePHgCAUaNG4fPPP2+TGJxOJ5xOZ/ix1+tV1Y7eHA5HOMaCAius1hwAzddTFxTUwes1x5FhyzjNTE2ceXnA3/7247A6L8+PzvxENX8/NXEauV8k4t89EuWRtKzZkXRn4+zsPhvLvpSbmxt3fC3FnBjef/991NTU4Le//S0kSVLViMPhwBdffIGLFy8iLS0NBw4cwODBg1UHa2a8nlp/Ii9T1Orvx/1Ce2Zer6q1zuyzplldtaGhAZWVlfjmm2/w3nvv4dZbbw1fnaRGfn4+ioqKsGjRIlgsFgwcOFAxMkgUvJ66a9Pq75fs+4WIE6btfUcyzX42xeqqlZWVWLlyJbKysnDrrbfirrvuiruh4uJiFBcXx/15Iup6RF2J0953cFQmXruJYcyYMRgzZoxesRBRghFR5onlO5J9VCYa17KghNZVJhQlKhGrtppl5ddkwhv1UMJqrwSRbJPOjPq9Iso8LBXpj4mBEla0EkSyTToT9XvjTS4iyjwsFemLiYESVrSrVVonjHfeSU/oo1ERdf5kS6bJjomBEla0EkTLhGGxAOvXZ6CpKXE7PBGXc3aluQLUeUwMlNAilSBaJoz/+z8L1q3LSOgOT0SNPpnmChATAyWpUMLweKx4++3mdb9i6fA8HiuqqlJQUGA1PIGoqfl3tkbPE8DJhYmBkpqaDk9ZZ88xtOxkRM2fJ4CTBxMDJb1YOzwz1dnNFAslHk5wI4pRLBOt9JpQx0lfpCWOGJJUsk3waunNN9OxeXM6Jk9uxMyZjTF/LlR2qqqyo6Cgrs1207O8w5o/aYmJIQkl8zXpb76ZjkWLegIAPvmkGwCoTg4TJwYj3lNB7/IOa/6kFZaSEoSaEkbLDszvl7BrVzcdIjSHzZtDdx6UWj1Wr/U2N2t5h+tFkVocMSQAtSOAaNekJ0N5afLkxh9GCnL4cTyibXOzlXeSeXRI8WNiSABqSxiROjDRHYhZk0yobBTPOYaWWm5zWQaWL8/EggX1pivv8OoligcTQwKIZ1Zq6w5MZAdi9qPUmTPjTwghoW0uy0AwCGzf3g179qSZ7rdyxjLFg+cYEkBoBLBwYX3cHZPI+ngynMMIbfOxYy8iJQUIBtX9Vr3q/iL2DUo+HDGYjFZLG3f0vSLr48lylFpY6MeCBfXYsycNfn/sv1XvEZXZyltkfkwMJqJVh7F7txTT94rqQMx4ElYr8fxW1v3J7HRLDN9//z1Wr16NEydOQJIkzJkzB0OGDNGr+S5Bqw6jvFzSvSNKpqNUtb81WUZU1HXplhjWrFmDESNG4OGHH0YgEMDFi/zH0Fq8HUbrMlHrx+PGNZ83UPu9icJsV0gl04iKuiZdEsP58+fx2WefYd68ec2NpqYiNdW4KpbZOoqQaB1Ge/G2Lj/97/+exRNPZCnKRhMnyknbEZn1CqlkGlFR16NL71xTU4MePXqgtLQUX3/9NQYNGoR7770X3bt316N5BbN2FCGtO4yO4m1dftq8Ob1N2WjixOTtiFjPJ1JPl8TQ1NSE48ePY9asWcjPz8eaNWuwadMmzJgxQ/E+t9sNt9sNAHC5XHA4HMJjqapKUXQUVVV2TJwYDL++e7eE8nIJ48bJKCqS2/2u1NRUTWJUE++kSRJeeAHw+WSkpQHFxan4979/fDxpUjpSUy2axymCFtuz9faZNCkdDkfnDkj0+LuLoEWcav59xCqZt6dZ6ZIYcnJykJOTg/z8fABAUVERNm3a1OZ9TqcTTqcz/Njr9QqPpaDACqs1B0Bzvb2goC68IJry6FzucDThcDg0iTHWeAEgLw/429+UpaZ+/X58nJfnRyAQX5xaltwifbcW27P19snL86OzTejxdxdBdJxq/33EKlm3pxZyc3OFfI8uiaFnz57IyclBdXU1cnNzceDAAfTr10+Pptto78SfGcsOsZyo1KJMpGXJjdfxd01m/PdB2tDtDPCsWbOwcuVKBAIB9OrVC3PnztWr6TaidRRmvYxQTccWqdOdOFF9m1p2Auxguiaz/vsg8XRLDAMHDoTL5dKrubgkwmWEkTrdeBJDR51AZ8pM7X23Wa8Yo8T490Gx4cznVvQoO2jZ+UXudKOfbI0WS3udQKRLZOvqLO3+ntbtRPruWGdok3FYlksOTAydsHu3hC1bbKo6eK3r62qO6jqKJVon0HrJ6f/5nyzIcvTfE62d1u/Tc4a2kSMTvdqOZ/8kApgY4ubxWDFjRip8vkxVHXykUk/ov6L+Acd6VBdvrb/lqESSmlcWDQajf0es7eg1Q9vIuSx6tR3v/kkEMDHEbdeubvD50KlO1WqVYbc3GdZJqT2Z2PJINzQqsdubfphpHf07Ym2nqEifGdpGnvzWq+14908igIkhbqNHX0RaWiZ8PnX3L2hd6jGyk2oZi93eFB69RGo/0pHub3/bAAC44oqAsCW99ahhG3l1jV5tx7t/EgFMDHErLPRj69YAtmxp7PS9E4y8BDAUR0ejlvYSWCyduZlOWhp5dY1ebXdm/yRiYuiEoiIZeXkNnfoOoy8B9HisWL48Ez5f++cJEu0adiMTlV5ti9g/KTkxMZiAUZ1UqDzUnBSAlJToZQejExgR6YeJIYmFykPBoISUFBljx17EggX1UTt9M5WDiEg7KUYHYEYd3ag99Pru3ZLOkYnVXB6SYbHISEuT200KRJQ8usyIQa9JQR1dZ97y9RdeaF6508jOtDPbheUhIoqkSySGWDprUZ1bR5ePtnzd55MNvT5cxGQploeIqLUuUUpq2Rn7/VL4envgx87xuecyMX16TtTyT6xallcinYhVll9g6NU57W0X0ToqrxFR4ugSI4b2LpXUYoLY7bc3ApAxbVpj+LsizfqdNCkdeXnGHW3rdQmp2W+HSkRidYnE0F4tXGTn2PoOVdOmNUZ4/sdZvw5H93bvBib6vEjr79PrHAHvn0CUXLpEYgCi18JFdo7ROsB4OkbRR9mRvi8Ui9YnjhNtchsRta/LJIb2iDqBGq0DjKdjFH2U3fr73nknHW+/naHLKp27dnWL6Z4LRJQYEiIxxKqj0k600Uc8oxLRR9mtvw/Q/t4FWt38nYjMLWkSQ6ylnfZKVmoXyhNZ/2/9fQDw9tvpALQr7/DcAlFySprEYEQnJ3qOQOvv0/rEM88tECUnXRNDMBjE4sWLkZ2djcWLF+vZdEJ2clpPTuPMaKLkpGti2Lx5M/r27YvGxkbVn+3spZ/s5OLDmdFEyUe3xHD69GlUVFRg6tSp+OCDD1R9VtSln+zkiIg6ptuSGK+//jpmzpwJSVK/IqmeSz8QESU7XUYM+/btQ1ZWFgYNGoRDhw5FfZ/b7Ybb7QYAuFwuOBwOAMCkSc0rmfp8zesTTZqUDoejux6htys1NTUco5kxzsh275ZQXi5h3DgZRUVyzJ/j9hSLcZqPJMty7P8i4rRu3TqUl5fDYrHA5/OhsbERI0eOxEMPPdTu56qrq8P/r9ey22o4HA5421sTwyQYZ1udmaPB7SkW4xQnNzdXyPfoMmK48847ceeddwIADh06hPfff7/DpNAazw+QSJyjQRRdl1h2m0i0jpZXJ0pmuk9wu/LKK3HllVfq3SyRAi9fJoouaWY+E7XG8iRRZCwlERGRAhMDEREpMDEQEZECEwMRESkwMRARkQITAxERKXS5xODxWPHiizZ4PFajQyEiSkhdah6DqOW3iYgoui41YuDy20RE2utSiYHr2xARaa9LlZK4vg0Rkfa6VGIAuL4NEZHWulQpiYiItMfEQERECkwMRESkwMRAREQKTAxERKSQkImBy2YQEcWvy12u2hEum0FE1Dm6JAav14tVq1bhzJkzkCQJTqcTkydP1qStlstmhB4zMRARxU6XxGCxWHD33Xdj0KBBaGxsxOLFi1FQUIB+/foJb6t52QwbAHDZDCKiOOiSGOx2O+x2OwAgPT0dffv2RW1trSaJgctmEBF1ju7nGGpqanD8+HHk5eVp1gaXzSAiip8ky7KsV2MXLlzAE088galTp2LUqFFtXne73XC73QAAl8sFn8+nV2hxSU1NRSAQMDqMDjFOsRinWIxTnLS0NCHfo1tiCAQCePbZZzF8+HBMmTIlps9UV1drHFXnOBwOeL1eo8PoEOMUi3GKxTjFyc3NFfI9usxjkGUZq1evRt++fWNOCkREZAxdzjEcPXoU5eXlGDBgABYuXAgAuOOOO3D11Vfr0TwREamgS2K44oor8NZbb+nRFBERdVJCLolBRETxY2IgIiIFJgYiIlJgYiAiIgUmBiIiUmBiICIiBSYGIiJSYGIgIiIFJgYiIlJgYiAiIgUmBiIiUmBiICIiBSYGIiJSYGIgIiIFJgYiIlJgYiAiIgUmBiIiUmBiICIiBSYGIiJSYGIgIiKFVL0aqqysxJo1axAMBnHTTTfhtttu06tpIiJSQZcRQzAYxGuvvYYlS5bg+eefx44dO3Dy5Ek9miYiIpV0SQzHjh1Dnz590Lt3b6SmpuK6667D3r179WiaiIhU0iUx1NbWIicnJ/w4JycHtbW1ejRNREQq6XKOQZblNs9JktTmObfbDbfbDQBwuVzIzc3VPLbO6goxAoxTNMYpFuM0F11GDDk5OTh9+nT48enTp2G329u8z+l0wuVyweVyYfHixXqE1ildIUaAcYrGOMVinOKIilGXxDB48GB8++23qKmpQSAQwM6dO1FYWKhH00REpJIupSSLxYJZs2Zh6dKlCAaDuPHGG9G/f389miYiIpUsTz755JN6NHTppZdi0qRJmDx5Mn7605/G9JlBgwZpHFXndYUYAcYpGuMUi3GKIyJGSY50ZpiIiJIWl8QgIiIF3ZbECOloaQy/34+XXnoJX331FTIzMzF//nz06tULALBx40aUlZUhJSUF9913H0aMGGFYnB988AH++c9/wmKxoEePHpgzZw4uueQSAMD06dMxYMAAAIDD4cCiRYsMi/Pjjz/GG2+8gezsbADAxIkTcdNNN4Vf27BhAwBg6tSpGD9+vGFxvv766zh06BAAwOfz4ezZs3j99dcB6Lc9S0tLUVFRgaysLJSUlLR5XZZlrFmzBp9++im6deuGuXPnhoftem7LjuLcvn073n33XQBA9+7d8cADD2DgwIEAgHnz5qF79+5ISUmBxWKBy+UyLM5Dhw7hT3/6U/jf96hRozBt2jQA+i2h01GM7733HrZv3w6geQWHkydP4rXXXoPNZtN1W3q9XqxatQpnzpyBJElwOp2YPHmy4j1C909ZR01NTfKDDz4o//e//5X9fr/8yCOPyCdOnFC8Z+vWrfLLL78sy7Is/+tf/5KXL18uy7IsnzhxQn7kkUdkn88nnzp1Sn7wwQflpqYmw+I8cOCAfOHCBVmWZfnDDz8MxynLsjxz5kxN4oonzm3btsmvvvpqm8/W19fL8+bNk+vr6xX/b1ScLW3evFletWpV+LFe2/PQoUPyl19+KS9YsCDi6/v27ZOXLl0qB4NB+ejRo/Jjjz0my7K+2zKWOI8cORJuv6KiIhynLMvy3Llz5bNnz2oWm5o4Dx48KD/zzDNtnle7v2gZY0t79+6Vn3zyyfBjPbdlbW2t/OWXX8qyLMvnz5+XH3rooTbbROT+qWspKZalMTweTzibFRUV4eDBg5BlGXv37sV1110Hq9WKXr16oU+fPjh27JhhcV511VXo1q0bACA/P9+QmdydWWqksrISBQUFsNlssNlsKCgoQGVlpSni3LFjB8aMGaNJLO0ZOnQobDZb1Nc9Hg/GjRsHSZIwZMgQfP/996irq9N1W8YS5+WXXx5+PT8/XzGHSE8dxRmNnkvoqIlxx44duP766zWJoyN2uz189J+eno6+ffu26XNE7p+6lpIiLY3xxRdfRH2PxWJBRkYG6uvrUVtbi/z8/PD7srOzNeuMY4mzpbKyMkVZy+/3Y/HixbBYLPjVr36FkSNHGhrnnj178Nlnn+HSSy/Fr3/9azgcjjafNcv2/O6771BTU4Orrroq/Jxe27MjtbW1cDgc4cehpV303JZqlZWV4Wc/+5niuaVLlwIAbr75ZjidTiPCCvv888+xcOFC2O123H333ejfv7/qf396uHjxIiorK3H//fcrnjdiW9bU1OD48ePIy8tTPC9y/9Q1McgxLI0R7T2RntdKLHGGlJeX46uvvkLLq35LS0uRnZ2NU6dO4amnnsKAAQPQp08fQ+K85pprcP3118NqteKjjz7CqlWr8MQTT0T8vmi/UY84Q3bs2IGioiKkpPw4mNVre3ZEze/QaluqcfDgQWzbtg1PPfVU+Lmnn34a2dnZOHv2LP74xz8iNzcXQ4cONSS+yy67DKWlpejevTsqKirw3HPPYeXKlaq2s1727dunGIkBxmzLCxcuoKSkBPfeey8yMjIUr4ncP3UtJcWyNEbL9zQ1NeH8+fOw2WxtPltbWxs+oWpEnABQVVWFjRs34tFHH4XVag0/H4qrd+/eGDp0KP7zn/8YFmdmZmY4NqfTia+++iocY+vtGek36hVnyM6dO9sM1/Xanh3JycmB1+sNPw79Dj23Zay+/vprvPzyy1i4cCEyMzPDz4e2ZVZWFq699lrNyrGxyMjIQPfu3QEAV199NZqamnDu3DlV+4teIpU39d6WgUAAJSUlGDt2LEaNGtXmdZH7p66JIZalMa655hp8/PHHAIDdu3fjyiuvhCRJKCwsxM6dO+H3+1FTU4Nvv/22zVBKzziPHz+OP//5z3j00UeRlZUVfr6hoQF+vx8AcO7cORw9ehT9+vUzLM66urrw/3s8nnAsI0aMwP79+9HQ0ICGhgb4Y7vWAAADlklEQVTs379fs6u8Yl0Spbq6Gt9//z2GDBkSfk7P7dmRwsJClJeXQ5ZlfP7558jIyIDdbtd1W8bC6/Vi2bJlePDBBxWLvl24cAGNjY3h/6+qqgpf7WWEM2fOhI9yjx07hmAwiMzMTNMtoXP+/HkcPnxYEYPe21KWZaxevRp9+/bFlClTIr5H5P6p+wS3iooKrF27Nrw0xtSpU7F+/XoMHjwYhYWF8Pl8eOmll3D8+HHYbDbMnz8fvXv3BgBs2LAB27ZtQ0pKCu699942tVM943z66afxzTffoGfPngB+vIzy6NGjeOWVV5CSkoJgMIhbb70VEyZMMCzOdevWwePxwGKxwGaz4YEHHkDfvn0BNNefN27cCKD5ErYbb7zRsDgB4K233oLf78ddd90V/pye23PFihU4fPgw6uvrkZWVheLiYgQCAQDAz3/+c8iyjNdeew379+9HWloa5s6di8GDBwPQd1t2FOfq1auxZ8+ecL05dCnlqVOnsGzZMgDNo/ExY8Zg6tSphsW5detWfPTRR7BYLEhLS8M999yDyy+/HEDk/cWIGIHmSz0rKysxf/788Of03pZHjhzB448/jgEDBoTLQHfccUd4hCB6/+TMZyIiUuDMZyIiUmBiICIiBSYGIiJSYGIgIiIFJgYiIlJgYiAiIgUmBiIiUmBiICIiBSYGIiJSYGIgaofP58Ps2bMxZ86c8JpNIatXr8b06dOxY8cOg6Ij0gYTA1E70tLSUFxcjNOnT+PDDz8MP79u3TqUlZVh1qxZht28hUgrTAxEHRg/fjz69++PTZs24cKFC/jHP/6BTZs2obi4GLfccovR4REJx0X0iGKwb98+PPvssxg2bBgOHjyIW265BbNmzTI6LCJNcMRAFINrrrkGl112GQ4cOIDRo0fjvvvuMzokIs0wMRDFYOfOneE7x6Wnpxt+q0kiLTExEHVg//79eOmllzBy5Ehcd9112LZtG06ePGl0WESaYWIgascXX3yBZcuW4fLLL8dDDz2EGTNmQJIkrFu3zujQiDTDxEAUxcmTJ+FyuZCbm4uFCxfCarWiT58+mDBhAjweD44cOWJ0iESaYGIgisDr9WLp0qXIyMjAY489hoyMjPBr06ZNQ1paGv7yl78YGCGRdni5KhERKXDEQERECkwMRESkwMRAREQKTAxERKTAxEBERApMDEREpMDEQERECkwMRESkwMRAREQKTAxERKTw/xLEDYkYkZB1AAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"\n",
"plt.plot(X,y,'b.')\n",
"plt.xlabel(\"$x$\", fontsize=18)\n",
"plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
"_ =plt.axis([0,2,0,15])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Analytical way of Linear Regression"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:24.792752Z",
"start_time": "2018-08-25T03:56:24.783380Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[4.14660597]\n",
" [2.87479146]]\n"
]
}
],
"source": [
"X_b = np.c_[np.ones((100,1)),X]\n",
"theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)\n",
"print(theta_best)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5>This is close to our real thetas 4 and 3. It cannot be accurate due to the noise I have introduced in data"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:24.805370Z",
"start_time": "2018-08-25T03:56:24.795517Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[4.14660597],\n",
" [9.89618888]])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_new = np.array([[0],[2]])\n",
"X_new_b = np.c_[np.ones((2,1)),X_new]\n",
"y_predict = X_new_b.dot(theta_best)\n",
"y_predict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5>Let's plot prediction line with calculated:theta"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:25.168502Z",
"start_time": "2018-08-25T03:56:24.808124Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[0, 2, 0, 15]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAESCAYAAAD5d3KwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8FOW9P/DP7C0XEsKSlUu4HIQEFCEgBNhFiIhRudjWUhqQiyLaHi5qKYhQz2n1aPmZngKiFYqtvtRqOcVyhCoF9KyIoeRCNiGE+00UFCSGBEhCktnNzu+PTTaZXHc3M7Ob5PN+vXxJ9jbfnWyezz7PzPOMIEmSBCIiohq6YBdAREShhcFAREQyDAYiIpJhMBARkQyDgYiIZBgMREQkw2AgIiIZBgMREckwGIiISIbBQEREMoZgF9CSS5cuBbuEFlksFhQVFQW7jFaxTmWxTmWxTuXExcUp8jrsMRARkQyDgYiIZBgMREQkw2AgIiIZBgMREckwGIiISIbBQEREMgwGIiKSYTAQEZGMosGwadMmPPHEE1ixYkWj+z766COkpqbixo0bSm6SiIgUpmgwTJo0Cc8991yj24uKinDkyBFYLBYlN0dERCpQNBiGDh2KqKioRre/++67mDt3LgRBUHJzRESkAtWPMTgcDnTv3h0DBgxQe1NERKQAVVdXraqqwocffoj//M//9OnxdrsddrsdAJCWlhbyQ08GgyHkawRYp9JYp7JYZ+hRNRiuXLmCwsJCrFy5EgBw9epVrFq1Ci+//DK6devW6PEpKSlISUnx/hzqS9y2h2V4AdapNNapLNapHKWW3VY1GPr3748333zT+/PSpUvx8ssvo2vXrmpuloiI2kDRYNiwYQOOHz+O0tJSLFq0CKmpqZg8ebKSmyAiIpUpGgzLli1r8f6NGzcquTkiIlIBZz4TEZEMg4GIiGQYDEREJMNgICIiGQYDERHJMBiIiEiGwUBERDIMBiIikmEwEBGRDIOBiIhkGAxERCTDYCAiIhkGAxERyTAYiIhIhsFAREQyDAYiIpJhMBARkQyDgYiIZBgMREQkw2AgIiIZg5IvtmnTJuTl5SEmJgbr1q0DALz33nvIzc2FwWBAz549sWTJEnTp0kXJzRIRkYIU7TFMmjQJzz33nOy2xMRErFu3DmvXrkXv3r2xfft2JTdJREQKUzQYhg4diqioKNltI0aMgF6vBwAMHjwYxcXFSm6SiIgUpukxhr1792LkyJFabpKIiPyk6DGGlnz44YfQ6/WYOHFis4+x2+2w2+0AgLS0NFgsFq3KC4jBYAj5GgHWqTTWqSzWGXo0CYZ9+/YhNzcXv/nNbyAIQrOPS0lJQUpKivfnoqIiLcoLmMViCfkaAdapNNapLNapnLi4OEVeR/WhpPz8fPzjH//AqlWrEBYWpvbmiIiojRTtMWzYsAHHjx9HaWkpFi1ahNTUVGzfvh0ulwsvvfQSACAhIQE///nPldwsEREpSNFgWLZsWaPbJk+erOQmiIhIZZz5TEREMgwGIuq0HA4j/vCHKDgcxmCXElI0O12ViCiUOBxGzJoVC6dTgNEYha1bryIpyRnsskICewxE1CllZobB6RRQXS3A6RSQmdm+zppUs7fDHgMRdUo2WxWMRs8SPkajBJutKsgV+U7t3g6DgYg6paQkJ7ZuvYrMzDDYbFXtahipfm+n9mcGAxGRApKSnO0qEGqp3dthMBARtTNq93YYDERE7ZCavR2elUREFOK0nm/BHgMRUYhwOIyNhoeCMd+CwUBEFALqB4BeH4VZs25i5swK1c9AagqHkoiIQkD9ABBFAe+/3wWzZsXCbK6G0ShBr5c0m2/BHgMRUQioPQXV7QYkCZAkAU4nUFKi13y+BYOBiCgE1J6Cum1bJLZujUR1dV0PQev5FgwGIqIQ4QmA65g582ZQZ2QzGIiIQkywZ2Tz4DMREckwGIiISIbBQEREMooeY9i0aRPy8vIQExODdevWAQDKysrwyiuv4Pvvv8ctt9yCX/7yl4iKilJys0TUSTQ1M5iUp2iPYdKkSXjuuedkt+3YsQPDhw/Ha6+9huHDh2PHjh1KbpKIOonamcG//300Zs2K5XWaVaRoMAwdOrRRbyAnJwd33303AODuu+9GTk6Okpskok5C60txar1wXShR/XTV69evw2w2AwDMZjNu3Lih9iaJqAPS8lKcTS1cN2WKapvzqy4thtJCah6D3W6H3W4HAKSlpcFisQS5opYZDIaQrxFgnUpjncrytc5u3QTMn+8GIGDePDes1hjVaioo0MkWrisoMOPBB3WK7M+sLAHp6QKSkyVYrVKrt9e/f/ZsA0QRMJmisWePq+5xLheE/HwgLq7N9QEaBENMTAxKSkpgNptRUlKCrl27NvvYlJQUpKSkeH8uKipSu7w2sVgsIV8jwDqVxjqV5Uud8m/wEh588BqKitT7xpyYaITRGAvA0ztJTCyByxXT5v3Z8H3ULqHd3O317d4dBVGMrllkT8Knb53DyE/fhykzE6aDB6ErK/MssqQA1U9XTUpKwhdffAEA+OKLLzBmzBi1N0lEHYzWxxdq1y1aubJU0esfNPc+Wn1/TicmxhbAJDihhwum6gpM/8tj6LpmDfQXLqDixz9G8aZNitQIKNxj2LBhA44fP47S0lIsWrQIqampeOihh/DKK69g7969sFgsWL58uZKbJKJOQMvjC7XUWJaiuffR6PYx5TDm5iIsM9PTI8jJwbTycnwGKz6LnYnxY8sx5IeP4jvrH+Hu0UPRGgFAkCSF+h4quHTpUrBLaFFH6qqHAtaprI5WZ7DnMCi1Pxu+D4fDiMx/GWAp/xo3jn2He65/hOTT70J38yYAwDl4MESbDVU2G0SrFe5bbmn2tePayzEGIiIltOUbvJah0tq2kpKcSEosh/HwYRxe9S1mb1kI0W2ACYPxGRYjaUgJKlJT64IgCCcQMBiIqMNyOIzYti2i5voGyl8zualv/01en1kUYTp8GKaMDIRlZsKYkwNdZSVysBoiDKiGAaJOh4+f3I5/W+VWpLa2YDAQUchR4ht+bSNdVSXUnKyj7DWTmwoB2fWZJQm5ax14QPp/MDoc0FVWAgCct9+Om3PmQLTZMDz8Hhh/pgOcEoxGwHqvHgCDgYjaITWHZpr91u2n2kZakgQAEgRB2Wsmy0IAwMEPruBeaR9elZ6ACD1Mbiem7H8RuqE3cHPuXM9xgnHjIHXv7n2NUYDml+30BYOBiPyiVMPdnIYNbqDf8Ouf6aPXA7Nm3cTMmTeVqbWyEsnRp/CqcBdE6GCqFjH9rz+DVcjGJwMy8VnsDNjuM2HAnD/j+3pB0JRgX5SnKQwGIvKLUg13c5Q6NbV2LoIi38YrK5H9x3zYtxVh8rUdSD7zF8RVVcEOG/b2TIVtXCWGPPRzfDf2TQwymzGo5mmBnPIZ7LOvAAYDEflJ7TkFSjboAX8br6iAKS/PM48gKwu5DiMecO6GCBN+L9yDj384CCMf6olbx47Fwm7dAACVAVdZp7nemNZhwWAgIr8o+k28hW1o+m25ogKm2gllWVkw5eVBEEVIOh2cw4bh0ztfgOgIR7VbB1Gnh/32xbj9/rIWXzKQxryp3hgAVYfumsJgICK/heK4uD+EigoYHY66IDh0qC4Ihg9H+cKFqLJaIY4dCykmBqMcRphme9Yo8qWXFOhxmKZ6Y/XDQpKA9eujsXx5aedZXZWIlJOVJWD37qiQOtslWISbNxsHgdPpCYLERJQ//rhnQtmYMZCaWOgzKcmJPXtc2L27wqf9GehxmOZ6Y0ZjFCQJcLuB/fvDkJ1tUrXnwGAg6oAcDmPNEs3Rmg0/hBLh5k2YHI66CWWHD3uCQK+HMzERZT/7GcTaIIiObvX1HA4jCgoEn0O2LcdhGvbGasNi/fpo7N8fBrdbgNOp/EH/+hgMRB1QZmYYRBGqnTkUaoTy8sZB4HLVBcHPf14XBH5ec14+LBTrU8gqfRwmKcmJ5ctLkZ1tgtOp/kKCDAaiNgqF0wsbstmqYDJF+zwm3t4I5eUw5eTAlJmJsIwMGAsKPEFgMHiCYNEiTxAkJfkdBA21ZVhIyc+DFgf9azEYiBrwp6FXe7JXoPwdEwdCM+BqCWVljYOgutoTBCNGeIJg/HhPEHTpoui2g7Hkd3O0OujPYCCqx9+GXu3JXm1htUqIj2/5lMpaoRRwDocRWfuA5Ohc3FX0sWdoqH4QjByJsiVLvEGQczzGE2hdqpDURZ1TZ7duvYqCAjMSE0tC5verJp+C4U9/+hPsdjs2b96M7g2md1+6dAkrVqzA/fffj8cee0yVIom04m9DH0rfJtsi2AEn3LgB08GDyN9xBbN3PAVRMmADkmHXr8Ho0UaULV2KKpsNzqQkSJGR3udpFWhJSU5MmeJW9XKiocSnYBg8eDDsdjvOnj2LsWPHyu579913ERkZiZ/+9KeqFEikJX8bei3HfdWkdcAJN27AlJ2NsKwsGHJy0OvQIQhuN7J1/wFRqluGeueyHRiwvPlagh1oHZXPwQCgUTDk5eXh0KFDePzxxxHVxgM8RKEgkIa+vU/2AtQPOOH6dW8QmDIzYTx6FILbDclkgjRuHMqefhpVNhuGC+NhfKTeMtTJLS9B3VF6bKHGp2CIi4tDVFQUzp07573N5XLh3XffRb9+/XDfffepViCR1jpCQx8IJd+3cO0aTAcPeq9ZbDx6FIIkQTKZII4ejbJf/MIzoWzUKFj69UNpzSUzR8O/Zag7So8t1Ph88DkhIQGnTp2CJEkQBAG7du3C5cuX8etf/xo6nU7NGokoxAklJQg7eBCmjAyYsrJgPHbMEwRhYRBHjULp8uUQrVaId94JRES0+Fr+BlRnDXI1+RUMhw4dwqVLlxAVFYX//d//xZgxYzB8+HCfnr9z507s3bsXgiCgX79+WLJkCUwmU8CFE1HwCCUlCMvO9kwoy8qC4fjxuiAYPRqlK1bUBUF4eLDLJT/5HAz1jzOcOHECLpcLjzzyiE/PLS4uxu7du/HKK6/AZDJh/fr1yMjIwKRJkwIqmggI7fPuOxqhuNgTBJmZCMvMhOHECU8QhIfXBYHNBnHkSAZBC9rLZ9avHoMgCNi7dy9OnjyJH/7wh+jZs6fPG3K73RBFEXq9HqIowmw2B1RwKGsvv/SOQI3TFNX6/bXHz4WuuNiz2FxNEBhPnAAAuMPD4UxKQukzz3jmEYwYAYSFBbna9kGJz6xWnyWfgyEiIgJ9+/bFiRMn0K1bN8yYMcPnjXTv3h0/+MEPsHjxYphMJowYMQIjRowIqOBQFUoThDoDpU9TVOv3114+F7qrV+VBcPIkgJogGDMGN559ti4IOAQckLZ+ZrX8LPk18zk+Ph4XL17EnDlzENHKAaT6ysrKkJOTg40bNyIyMhLr169Heno6kpOTZY+z2+2w2+0AgLS0NFgsFn/K05zBYPDWWFCgk/3SCwrMmDKl5VPttFK/zlDmT51Tpwp49VVAFCWYTMDUqRGwWAIfwvDn9+dPncH8XLRYZ2EhhH/9C7ovvoCQng7d8eMAACkyEpLNBtecOZCSkyGNHg3BZEI4ALUGiJT4fGZlCUhPF5CcLMFqDeSCmq1ra51t/cxq+VnyORhcLheOHz+OQYMG4e677/ZrI0eOHEGPHj3QtWad83HjxuH06dONgiElJQUpKSnen4tqTmELVRaLxVtjYqIRRmMsAM/51ImJJSEzS7J+naHMnzrj44G//a2uWx0f70Rb3qI/vz9/6gzm56J+nbqiIm9vwJSZCePp0wAAd2QkxDFjIK5e7ZlZnJgo7xHcuKFpnYGQf5OWVPsm3dY62/qZ9eWzFBcXF3B99fkcDB9//DEKCwvx1FNPQRAEvzZisVhw5swZVFVVwWQy4ciRIxg0aFDrT2xHeD619pQ8TVGt31+wPhe677+Hbt8+xHzyiScIzpwBUBMEY8ei4ic/qQsCo1GTmtTSnmY/t+UzGzKrq5aVlSE/Px8XLlzARx99hOnTp3vPTvJHQkICrFYrVq1aBb1ejwEDBsh6Bh0Fz6du39T6/WnxudAVFsp7BGfPem7v0sUTBD/9qScIhg/XPAiUOGDa0mt0ptnPIbG6an5+Pl577TXExMRg+vTpmDt3bsAbSk1NRWpqasDPJ6I6uitXYMrKQlhGhicIalYlcEdFeYJg1ixETJ2K7/v1AwzBW0RZqTNxWnoN9taV1+InZsKECZgwYYJWtRBRM3TffeftDYRlZsLw5ZcAaoJg3DjcfPhhiDYbnMOGeYMg3GJBmw68KECJYR5fXoO9dWXxegzUobXHOQQAoLt82Xvh+rCMDBjOnwcAuKOjIY4bh/K5cz1BcMcdQe0RtEaJYZ7ONFQUKkL3E0XURi0NQYRaYOguXZIHwVdfAQDcXbt6gmD+/Log0Ov9fv1gvV8lhnk4VKQ9BgN1WM0NQYTCpDPdt9/WDQ1lZdUFQUwMqsaNQ/mjj3qCYOjQgIKgPqXeb6DhosQwD4eKtMVgoA6ruSGIhoGxbVuE6t9G9d9+61lwrqZXYPj6awCAu1s3TxAsWIAqmw2u229vcxA0pMQ4fyiEKWmHwUAdVnNDEPUDQ68Htm6NRHW1sg2e/ptv5EFw4QKAmiCwWlG+cGFdEKi8bL0SY/Ttaa4AtR2DgTq0poYg6gfGt9/qsWVLZJsbPP3Fi3VBkJkJwzffAKgJApsN5U884QmC225TPQgaUmKMngeAOxcGA3VKtYHhcBjx97971v3ypcFzOIwoOKzDnb2KMb7s/zzzCLKyvEFQbTZDtNlQ/u//7gmCIUNUCwJ/xvzbOkbPA8CdC4OBOjWfGjxJgv7CBeT/z1eYvWkGxGrAhIH4DB9gbPfTEK1WlC1aBNFmg2vwYE16BMEY8+cB4M6DwUCdXqMGT5Kg//prz7BQzfCQ/vJl5GE1RPwU1TBAFATsfPx9/NvzYZoPDQEc8yd1MRiIJAn6r76qO300IwP6774DAFRbLBCtVlTZbEiMmQLjMzrAKcFoFDDuB10Bnbwx1mq+AMf8SU0Mhk4q1CZ4aen99yOwe5uEH/XPwc/db3h6BLVBcMstEG02VFmtEMePhys+HqhZTfhOAFv7XUVBgRmJiSWN9puWwzsc8yc1MRg6oU53TrokQX/uHMKysvD+/0RhVf5iAMC+nPsRHv0pHpns9ly4fvx4uAYN8gZBU5KSnJgyxd3kNRW0Ht7hmD+phcHQQfjTA+jw49OSBMO5czDVW3ROX1gIAPjItLfmQQIACX8btQY/3FQc0GYa7vNQHd7pzL1DCgyDoQPwtwfQXAPWbhsQSYLh7FnPgeKa6xbrv/8eAFDdqxeq7rrLOzyUkjkM/7cKADyXf5w2rSKgTTa3z0NteKfT9Q5JEQyGDsDfHkBTDZjSDYiqISNJMJw5Iw+CmuWlq3v1QtXEid4gqL71VtnQ0LxBniDYtSsC06ZVYN68wIKh/j6XJGD9+mgsX14acsM7Hb53SKpgMHQAgQxhNGzAlGxAFP+WKkkwnD4tu0KZ/upVAEB1796oSk6GOH68JwgGDGjxGAEAzJsXeCDUqt3nkgS43cD+/WHIzjaF3DfyUB3eotDGYOgAQm3JgzaHjNtdFwQZGTBlZ3uDwBUXh6pJk1A1fjxEmw3V/fu3GgRqqN3n69dHY//+MLjdApxO39+rVsN2oTi8RaGPwRBi1FrauLXXVbIB8Ttk3G4YTp2CrqAAZrsdpqws6Is9B4Rdffqg6p576oKgX7+gBEFTkpKcWL68FNnZJjidvgeq1uP+oTa8RaGPwRBC1GowsrIEn15XqQak1ZBxu2E4edI7LGTKyoK+pMRzX9++qLr3XlTZbBDHj/cEQQgLJFA57k+hTrNgKC8vx+bNm3Hx4kUIgoDFixdj8ODBWm2+XVCrwUhPFzRviGQh43bDcOKE7MI0umvXAACufv1Qdd99qLLZ0GXaNBRFRalalxr8DVSO+1Oo0ywY3n77bYwcORIrVqyAy+VCVRX/GBoKtMFoOEzU8OfkZAlGo+T36wbM7Ybh+PG6IMjOrguC/v1R+cADnh6BzYbqvn29T+ui0sXrQ+00XI77U6jTJBhu3ryJEydOYOnSpZ6NGgwwBPEC5qHWUNRqrsFoqd6Gw0//9V/X8fzzMbJhoylTJHUboupqT48gI6MuCK5fBwC4BgxAxZQpEGuDoE8fZbfdilA9j5/j/hTKNGmdCwsL0bVrV2zatAlff/01Bg4ciAULFiA8PFyLzcuEakNRq2GD0Vq9DYefdu2KaDRsNGWKwg1RdTWMx47VnT6anQ3djRsAaoJg2jTvPAK3xkHQEMfzifynSTBUV1fj/PnzWLhwIRISEvD2229jx44dmD17tuxxdrsddrsdAJCWlgaLxaJ4LQUFOllDUVBgxpQpbu/9WVkC0tMFJCdLsFqlFl/LYDCoUqM/9U6dKuDVVwFRlGAyAampBhw8WPfz1KkRMBj0bavT5YJw+DCE9HTo0tMhHDgAoaZHIMXHwz1zJlzJyXBPnAj07QsjACOALn5uRo392XD/TJ0aAYulbV9ItPi9K0GNOv35+/BVZ96foUqTYIiNjUVsbCwSEhIAAFarFTt27Gj0uJSUFKSkpHh/LlJhvDkx0QijMRaAZ7w9MbHEuyCa/Nu51GpvwmKxqFKjr/UCQHw88Le/yYea+vat+zk+3gmXy886XS4Yjx5F/rZvkZWuw+TL/4O7bn7muWvgQFQ9+GBdj6B3b/lzfdxOU8NjauzPhvsnPt7Z5sMYWvzelaB0nf7+ffiqs+5PNcTFxSnyOpoEQ7du3RAbG4tLly4hLi4OR44cQd96Bx211NKBv1AcdvDlQGWbh4lcLhiPHKk7ffTgQWSXDcOP8BlEmPCyfh62r9iJEXNuhbtXrza8Gw+ex98+heLfB6lDsyPACxcuxGuvvQaXy4UePXpgyZIlWm26keYailA9jdCfhq2pRnfKlAYPcjrrgiAry3OMoLzcc1d8PCp+/GPsur4E4s4IVLsFiNBjn/E+DO9Vpsj7YQPTPoXq3wcpT7NgGDBgANLS0rTaXEA6wmmETTW6U+51wpibWxcEBw/WBUFCAip+8hPP6aNWK9w9egAAxjiMMH4qAc3M6G3LmV0tNTChesYYdYy/D/INZz43oMWwg5qNn7fRlSSYBBce2LMaxj/8BbfUBsHgwaiYOdMbBAe/jvPUEleFpB51tbTUCDR1imxJib7F99PwPTf12r7O0Kbg4bBc58BgaIOsLAG7d0f51cCrMr4uijAePoywzEzcn5kJu1tAutuGSe59SKoogXv+fFy/805Pj6DeWRWt1dJcI9Bwyen/+I8YSFLz76e57TR8nJYztIPZM9Fq24F8PokABkPAHA4jZs82QBSj/Wrgmxrqqf2/z3/AogjT4cOe6xFkZsLocEBX4VlG2nnbbRgx14bbbQMhWufg+9hYWCwWVDZxNkWgY/31h4IEAXC7Bbjdzb+Gr9vRaoZ2MOeyaLXtQD+fRACDIWCZmWEQRbSpUTUaJZjN1a03FFVVjYOgshIA4Lz9dtx8+GHPzGKrFe7u3X1+D/4eTKz/Tbd2KMhsrq6Zad38a/i6HatV5RnaNYJ58FurbQf6+SQCGAwBs9mqYDJFQxQlv77dNhxfb7KhGF4GU36+NwhMubkQ6gfB3LmeIBg3zq8gaKkWs7na23tpqgFp6pvuU095zlK67TaXYkt6azGGHcyza7TadqCfTyKAwRCwpCQn9uxxYffuijZfO6H+weIp/3wGvTe8B6GyEpIgwDV0KMrnzvVcoWzsWEhtCILmagHQaq+lpW+6vjTmoXTQMphn12i17bZ8PokYDG1gtUqIjw/g3P7KSpjy8mDKysIDGRn4rNqIL9zjcbf7CyRJpSifN68uCMxm5Quvx+EwYv36aIhiy8cJOto57MEMKq22HfDnkzo9BoMWKipgysvzXrjelJcHoaoKkiDAOWwYEhdYMWT8EIhj56OoWzfNyqodHvKEAqDTNT/swHPYiToPBoMaKipgys2VB4EoQtLp4Bw2DOULFqDKaoU4bhykmJiglVk7POR2C9DpJEycWIXly0ubbfRDaTiIiNTDYGhCa+eZ194/daqA+HhAqKiA0eGom1l86FBdEAwfjvKFCz1BMHZsUIOgoYbDQy2FAhF1Hu0mGLSaFNTaeebe+0Xg1bUu7E74NZLPvgfB6fQEQWIiyh9/vC4IunZVrdbaegLdLxweIqKmtItg8KWxVqpxa+rsmzFDr8PkcMCUkYHDHw6Bs/LfUQ0DRLeA/ddGYtTPwiHWBkF0dJu27w8lJktxeIiIGmoXwdDSqZJKzyRtuNbQ1B3L0GvtexBcLkh6Pe4eNA8mw88gut0wmQxI3DwbpUFqWLWcqMXF7Yg6j3YRDC2dKqlE4yiUl8OUkwNTZia6fVqKR6usECQJ8/FXJEVWoGzRIhyw/ADpN+6E9W7gb7hec4whAvHxwWsktTqFNNQvh0pEymoXwdDSWHggjaNQVuYNgrCMDBgLCiBUVyNDPwFT3P8HEUYYTRKmvfcgiibo5Q3jJsk769diCW/xamBKf8tu+HpaHSPg9ROIOpd2EQxA82PhvjSOQmlpXRBkZnqDQDIY4Bw5EmVLlkAcPx47HfdB3BDmaQCrJWQcisHoCWUBNYxKf8tu6vVqa1F7eKejTW4jopa1m2BoScPQEEpLYTp40HupSmNBAQS3G5LRCHHkSJQtXYoqmw3OpCRIkZHe51kj3TBubLy6ZyANo9Lfshu+3rZtEfj73yM1WaUzMzPMp2suEFHH0CGCQbhxQx4ER47UBcGoUSh76ilU2WzI1N2FjLyufi/2FsiQjdLfshu+HqD+tQvUuvg7EYW2dhkMwvXrMGVne2cWG48e9QSByeQJgqef9vQIRo+GFBEBwPehnZaGrPxdKE/J8f+GrwcAf/+7572pNbzDYwtEnVO7CAbh2jV5j+DoUQiS5AmC0aNR9otfeC5VOWoUUBMEDQWjkVN6jkDD11P7wDOPLRB1TpoGg9vtxurVq9G9e3esXr261cd3feEFTxAcO+YJgrAwT4/gl7/0BMGddzYbBA11xEZO7clpnBlN1DlpGgy7du1Cnz59UFFzGcrWdPnLXyBuPRqMAAAOoUlEQVSOHo3SFStwwDwN6SUjYJtYHVADxUYuMJwZTdT5aBYMV69eRV5eHmbMmIGdO3f69JzLx48D4eHy4wOvB34QlI0cEVHrdFpt6J133sG8efMgCILvTwoPByA/PuB0Ct5LUBIRkfI06THk5uYiJiYGAwcOxLFjx5p9nN1uh91uBwCkpaXBYrEAAKZOFfDqq4AoSjCZgKlTI2CxhGtReosMBoO3xlDGOpuWlSUgPV1AcrIEq1Xy+Xncn8pinaFHkCTJ97+IAG3ZsgXp6enQ6/UQRREVFRUYO3Ysnn766Rafd+nSJe+/Q3ERN4vFgqKW1sQIEayzsbbM0eD+VBbrVE5cXJwir6NJj2HOnDmYM2cOAODYsWP4+OOPWw2Fhnh8gJTEORpEzdPsGANRKPGcvixBr2/+OtdEnZXmE9zuuOMO3HHHHVpvlkiGpy8TNa9dzHwmUgOHJ4maxqEkIiKSYTAQEZEMg4GIiGQYDEREJMNgICIiGQYDERHJtLtgcDiM+MMfouBwGINdChFRh9Su5jH4enlOIiIKXLvqMXD5bSIi9bWrYOD6NkRE6mtXQ0lc34aISH3tKhgArm9DRKS2djWURERE6mMwEBGRDIOBiIhkGAxERCTDYCAiIpkOGQxcNoOIKHDt7nTV1nDZDCKittEkGIqKirBx40Zcu3YNgiAgJSUF06ZNU2Vb9ZfNqP2ZwUBE5DtNgkGv12P+/PkYOHAgKioqsHr1aiQmJqJv376Kb8uzbEYUAHDZDCKiAGgSDGazGWazGQAQERGBPn36oLi4WJVg4LIZRERto/kxhsLCQpw/fx7x8fGqbYPLZhARBU6QJEnSamOVlZV4/vnnMWPGDIwbN67R/Xa7HXa7HQCQlpYGURS1Ki0gBoMBLpcr2GW0inUqi3Uqi3Uqx2QyKfI6mgWDy+XC7373O4wYMQIPPvigT8+5dOmSylW1jcViQVFRUbDLaBXrVBbrVBbrVE5cXJwir6PJPAZJkrB582b06dPH51AgIqLg0OQYw6lTp5Ceno7+/ftj5cqVAICHH34Yo0aN0mLzRETkB02C4bbbbsMHH3ygxaaIiKiNOuSSGEREFDgGAxERyTAYiIhIhsFAREQyDAYiIpJhMBARkQyDgYiIZBgMREQkw2AgIiIZBgMREckwGIiISIbBQEREMgwGIiKSYTAQEZEMg4GIiGQYDEREJMNgICIiGQYDERHJMBiIiEiGwUBERDIGrTaUn5+Pt99+G263G/feey8eeughrTZNRER+0KTH4Ha78dZbb+G5557DK6+8ggMHDuCbb77RYtNEROQnTYLh7Nmz6NWrF3r27AmDwYDx48cjJydHi00TEZGfNAmG4uJixMbGen+OjY1FcXGxFpsmIiI/aXKMQZKkRrcJgtDoNrvdDrvdDgBIS0tDXFyc6rW1VXuoEWCdSmOdymKdoUWTHkNsbCyuXr3q/fnq1aswm82NHpeSkoK0tDSkpaVh9erVWpTWJu2hRoB1Ko11Kot1KkepGjUJhkGDBuHy5csoLCyEy+VCRkYGkpKStNg0ERH5SZOhJL1ej4ULF2LNmjVwu92455570K9fPy02TUREftK/8MILL2ixod69e2Pq1KmYNm0abr/9dp+eM3DgQJWrarv2UCPAOpXGOpXFOpWjRI2C1NSRYSIi6rS4JAYREclotiRGrdaWxnA6nXj99dfx5ZdfIjo6GsuWLUOPHj0AANu3b8fevXuh0+nw2GOPYeTIkUGrc+fOnfjss8+g1+vRtWtXLF68GLfccgsAYNasWejfvz8AwGKxYNWqVUGrc9++fXjvvffQvXt3AMCUKVNw7733eu/78MMPAQAzZszApEmTglbnO++8g2PHjgEARFHE9evX8c477wDQbn9u2rQJeXl5iImJwbp16xrdL0kS3n77bRw6dAhhYWFYsmSJt9uu5b5src79+/fjH//4BwAgPDwcTzzxBAYMGAAAWLp0KcLDw6HT6aDX65GWlha0Oo8dO4b//u//9v59jxs3DjNnzgSg3RI6rdX40UcfYf/+/QA8Kzh88803eOuttxAVFaXpviwqKsLGjRtx7do1CIKAlJQUTJs2TfYYRT+fkoaqq6ulJ598Uvruu+8kp9MpPfPMM9LFixdlj9mzZ4/0xhtvSJIkSf/617+k9evXS5IkSRcvXpSeeeYZSRRF6cqVK9KTTz4pVVdXB63OI0eOSJWVlZIkSdInn3zirVOSJGnevHmq1BVInZ9//rn05ptvNnpuaWmptHTpUqm0tFT272DVWd+uXbukjRs3en/Wan8eO3ZMOnfunLR8+fIm78/NzZXWrFkjud1u6dSpU9KvfvUrSZK03Ze+1Hny5Env9vPy8rx1SpIkLVmyRLp+/bpqtflT59GjR6WXX3650e3+fl7UrLG+nJwc6YUXXvD+rOW+LC4uls6dOydJkiTdvHlTevrppxvtEyU/n5oOJfmyNIbD4fCmmdVqxdGjRyFJEnJycjB+/HgYjUb06NEDvXr1wtmzZ4NW57BhwxAWFgYASEhICMpM7rYsNZKfn4/ExERERUUhKioKiYmJyM/PD4k6Dxw4gAkTJqhSS0uGDh2KqKioZu93OBxITk6GIAgYPHgwysvLUVJSoum+9KXOIUOGeO9PSEiQzSHSUmt1NkfLJXT8qfHAgQO46667VKmjNWaz2fvtPyIiAn369GnU5ij5+dR0KKmppTHOnDnT7GP0ej0iIyNRWlqK4uJiJCQkeB/XvXt31RpjX+qsb+/evbJhLafTidWrV0Ov1+NHP/oRxo4dG9Q6s7OzceLECfTu3RuPPvooLBZLo+eGyv78/vvvUVhYiGHDhnlv02p/tqa4uBgWi8X7c+3SLlruS3/t3bsXd955p+y2NWvWAADuu+8+pKSkBKMsr9OnT2PlypUwm82YP38++vXr5/ffnxaqqqqQn5+Pxx9/XHZ7MPZlYWEhzp8/j/j4eNntSn4+NQ0GyYelMZp7TFO3q8WXOmulp6fjyy+/RP2zfjdt2oTu3bvjypUrePHFF9G/f3/06tUrKHWOHj0ad911F4xGIz799FNs3LgRzz//fJOv19x71KLOWgcOHIDVaoVOV9eZ1Wp/tsaf96HWvvTH0aNH8fnnn+PFF1/03vbSSy+he/fuuH79On77298iLi4OQ4cODUp9t956KzZt2oTw8HDk5eXh97//PV577TW/9rNWcnNzZT0xIDj7srKyEuvWrcOCBQsQGRkpu0/Jz6emQ0m+LI1R/zHV1dW4efMmoqKiGj23uLjYe0A1GHUCQEFBAbZv345nn30WRqPRe3ttXT179sTQoUPx1VdfBa3O6Ohob20pKSn48ssvvTU23J9NvUet6qyVkZHRqLuu1f5sTWxsLIqKirw/174PLfelr77++mu88cYbWLlyJaKjo7231+7LmJgYjBkzRrXhWF9ERkYiPDwcADBq1ChUV1fjxo0bfn1etNLU8KbW+9LlcmHdunWYOHEixo0b1+h+JT+fmgaDL0tjjB49Gvv27QMAZGVl4Y477oAgCEhKSkJGRgacTicKCwtx+fLlRl0pLes8f/48/vznP+PZZ59FTEyM9/aysjI4nU4AwI0bN3Dq1Cn07ds3aHWWlJR4/+1wOLy1jBw5EocPH0ZZWRnKyspw+PBh1c7y8nVJlEuXLqG8vByDBw/23qbl/mxNUlIS0tPTIUkSTp8+jcjISJjNZk33pS+Kioqwdu1aPPnkk7JF3yorK1FRUeH9d0FBgfdsr2C4du2a91vu2bNn4Xa7ER0dHXJL6Ny8eRPHjx+X1aD1vpQkCZs3b0afPn3w4IMPNvkYJT+fmk9wy8vLw7vvvutdGmPGjBnYunUrBg0ahKSkJIiiiNdffx3nz59HVFQUli1bhp49ewIAPvzwQ3z++efQ6XRYsGBBo7FTLet86aWXcOHCBXTr1g1A3WmUp06dwp/+9CfodDq43W5Mnz4dkydPDlqdW7ZsgcPhgF6vR1RUFJ544gn06dMHgGf8efv27QA8p7Ddc889QasTAD744AM4nU7MnTvX+zwt9+eGDRtw/PhxlJaWIiYmBqmpqXC5XACA+++/H5Ik4a233sLhw4dhMpmwZMkSDBo0CIC2+7K1Ojdv3ozs7GzveHPtqZRXrlzB2rVrAXh64xMmTMCMGTOCVueePXvw6aefQq/Xw2Qy4ZFHHsGQIUMANP15CUaNgOdUz/z8fCxbtsz7PK335cmTJ/Gb3/wG/fv39w4DPfzww94egtKfT858JiIiGc58JiIiGQYDERHJMBiIiEiGwUBERDIMBiIikmEwEBGRDIOBiIhkGAxERCTDYCBqQBRFLFq0CIsXL/Yux1Fr8+bNmDVrFg4cOBCk6ojUx2AgasBkMiE1NRVXr17FJ5984r19y5Yt2Lt3LxYuXBi0dfmJtMBgIGrCpEmT0K9fP+zYsQOVlZX45z//iR07diA1NRUPPPBAsMsjUhXXSiJqRm5uLn73u99h+PDhOHr0KB544AEsXLgw2GURqU7TC/UQtSejR4/GrbfeiiNHjmD8+PF47LHHGj1mz549+OKLL3DhwgUkJCTILthE1F4xGIiakZGR4b0oUERERJNXvTKbzXjooYdw7tw5nD59WuMKidTBYCBqwuHDh/H6669j7Nix0Ov1+PzzzzF9+vRGFwmqvZJW/StnEbV3PPhM1MCZM2ewdu1aDBkyBE8//TRmz54NQRCwZcuWYJdGpAkGA1E933zzDdLS0hAXF4eVK1fCaDSiV69emDx5MhwOB06ePBnsEolUx2AgqlFUVIQ1a9YgMjISv/rVrxAZGem9b+bMmTCZTPjrX/8axAqJtMFjDEQ1LBYL/vjHPzZ5n9lsxvvvv69xRUTBwWAgaoPq6mrvf5IkQRRF6HQ6GAz806L2ixPciNrggw8+wLZt22S3DR06lPMZqF1jMBARkQwPPhMRkQyDgYiIZBgMREQkw2AgIiIZBgMREckwGIiISIbBQEREMgwGIiKS+f/jQ8o9Dyw1WQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(X_new,y_predict,'r-')\n",
"plt.plot(X,y,'b.')\n",
"plt.xlabel(\"$x_1$\", fontsize=18)\n",
"plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
"plt.axis([0,2,0,15])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gradient Descent"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cost Function & Gradients\n",
"\n",
"<h4> The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have to be derived from the cost functions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"<b>Cost</b>\n",
"\\begin{equation}\n",
"J(\\theta) = 1/2m \\sum_{i=1}^{m} (h(\\theta)^{(i)} - y^{(i)})^2 \n",
"\\end{equation}\n",
"\n",
"<b>Gradient</b>\n",
"\n",
"\\begin{equation}\n",
"\\frac{\\partial J(\\theta)}{\\partial \\theta_j} = 1/m\\sum_{i=1}^{m}(h(\\theta^{(i)} - y^{(i)}).X_j^{(i)}\n",
"\\end{equation}\n",
"\n",
"<b>Gradients</b>\n",
"\\begin{equation}\n",
"\\theta_0: = \\theta_0 -\\alpha . (1/m .\\sum_{i=1}^{m}(h(\\theta^{(i)} - y^{(i)}).X_0^{(i)})\n",
"\\end{equation}\n",
"\\begin{equation}\n",
"\\theta_1: = \\theta_1 -\\alpha . (1/m .\\sum_{i=1}^{m}(h(\\theta^{(i)} - y^{(i)}).X_1^{(i)})\n",
"\\end{equation}\n",
"\\begin{equation}\n",
"\\theta_2: = \\theta_2 -\\alpha . (1/m .\\sum_{i=1}^{m}(h(\\theta^{(i)} - y^{(i)}).X_2^{(i)})\n",
"\\end{equation}\n",
"\n",
"\\begin{equation}\n",
"\\theta_j: = \\theta_j -\\alpha . (1/m .\\sum_{i=1}^{m}(h(\\theta^{(i)} - y^{(i)}).X_0^{(i)})\n",
"\\end{equation}"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:25.180630Z",
"start_time": "2018-08-25T03:56:25.172904Z"
}
},
"outputs": [],
"source": [
"\n",
"def cal_cost(theta,X,y):\n",
" '''\n",
" \n",
" Calculates the cost for given X and Y. The following shows and example of a single dimensional X\n",
" theta = Vector of thetas \n",
" X = Row of X's np.zeros((2,j))\n",
" y = Actual y's np.zeros((2,1))\n",
" \n",
" where:\n",
" j is the no of features\n",
" '''\n",
" \n",
" m = len(y)\n",
" \n",
" predictions = X.dot(theta)\n",
" cost = (1/2*m) * np.sum(np.square(predictions-y))\n",
" return cost\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:25.204130Z",
"start_time": "2018-08-25T03:56:25.188272Z"
}
},
"outputs": [],
"source": [
"def gradient_descent(X,y,theta,learning_rate=0.01,iterations=100):\n",
" '''\n",
" X = Matrix of X with added bias units\n",
" y = Vector of Y\n",
" theta=Vector of thetas np.random.randn(j,1)\n",
" learning_rate \n",
" iterations = no of iterations\n",
" \n",
" Returns the final theta vector and array of cost history over no of iterations\n",
" '''\n",
" m = len(y)\n",
" cost_history = np.zeros(iterations)\n",
" theta_history = np.zeros((iterations,2))\n",
" for it in range(iterations):\n",
" \n",
" prediction = np.dot(X,theta)\n",
" \n",
" theta = theta -(1/m)*learning_rate*( X.T.dot((prediction - y)))\n",
" theta_history[it,:] =theta.T\n",
" cost_history[it] = cal_cost(theta,X,y)\n",
" \n",
" return theta, cost_history, theta_history\n",
" \n",
" \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3> Let's start with 1000 iterations and a learning rate of 0.01. Start with theta from a Gaussian distribution"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T03:56:27.126960Z",
"start_time": "2018-08-25T03:56:27.062811Z"
},
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Theta0: 4.117,\n",
"Theta1: 2.899\n",
"Final cost/MSE: 5208.660\n"
]
}
],
"source": [
"lr =0.01\n",
"n_iter = 1000\n",
"\n",
"theta = np.random.randn(2,1)\n",
"\n",
"X_b = np.c_[np.ones((len(X),1)),X]\n",
"theta,cost_history,theta_history = gradient_descent(X_b,y,theta,lr,n_iter)\n",
"\n",
"\n",
"print('Theta0: {:0.3f},\\nTheta1: {:0.3f}'.format(theta[0][0],theta[1][0]))\n",
"print('Final cost/MSE: {:0.3f}'.format(cost_history[-1]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3> Let's plot the cost history over iterations"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T04:01:31.912400Z",
"start_time": "2018-08-25T04:01:31.604459Z"
},
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAu0AAAHmCAYAAADHkbA5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3X90VPWd//HXzcwkkAw/JhN+LBFb+XW+C4UGvRwytOWHZK2FbpelfGVVbMH6xUN6cIE9HhG7ZT2tlBYxbAJUVzjUrq5NSzXds6fWNs2GdCXYzEJo1VZAtC0LbMjMABkSksnM/f6RMhIhGBBm7sx9Ps7xwFxnct/Dm9FXPnnfzzUsy7IEAAAAwLZy0l0AAAAAgCsjtAMAAAA2R2gHAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtAMAAAA25053AXZ1/PjxlJ+zqKhIra2tKT8vUos+OwN9dgb67Az02RnS1edRo0b163mstAMAAAA2R2gHAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtAMAAAA2R2gHAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJtzp7sA9AgGPfrNb3I0ZYpHphlLdzkAAACwEUK7DQSDHi1e7FcsZsjj8au6OkRwBwAAQBLjMTbQ2JinWMxQPG4oFjPU2JiX7pIAAABgI4R2GwgEOuXxWHK5LHk8lgKBznSXBAAAABthPMYGTDOm6uqQfvMbn6ZMiTAaAwAAgF4I7TZhmjHdeWdCra0EdgAAAPTGeAwAAABgc4R2AAAAwOYI7QAAAIDNEdoBAAAAmyO0AwAAADZHaAcAAABsjtAOAAAA2ByhHQAAALA5QjsAAABgc4R2AAAAwOYI7QAAAIDNEdptZN8+Q1VVXgWDnnSXAgAAABtxp7sA9AgGPfq7v3Orq2uQPB6vqqtDMs1YussCAACADbDSbhONjXnq6pLicUOxmKHGxrx0lwQAAACbILTbRCDQqdxcyeWy5PFYCgQ6010SAAAAbILxGJswzZh+9rNuvfJKhwKBTkZjAAAAkERot5HSUkvjxkXTXQYAAABshvEYAAAAwOYI7QAAAIDNEdoBAAAAmyO0AwAAADZHaAcAAABsjtAOAAAA2ByhHQAAALC5lOzT3tXVpfXr16u7u1vxeFylpaW66667tG3bNr311lvKz8+XJH31q1/Vxz/+cVmWpV27dunAgQPKy8tTeXm5xowZI0mqr6/XSy+9JElauHChZs+eLUk6evSotm3bpq6uLk2dOlXLli2TYRiKRqOqqKjQqVOnNGzYMK1evVperzcVbxsAAAC4LlIS2j0ej9avX68BAwaou7tbX//611VSUiJJuu+++1RaWtrr+QcOHNDJkydVWVmpw4cPa8eOHdqwYYOi0ah2796tjRs3SpLWrl0r0zTl9Xr17LPP6sEHH9T48eP1rW99S83NzZo6dapqamo0efJkLViwQDU1NaqpqdGSJUtS8bYBAACA6yIl4zGGYWjAgAGSpHg8rng8LsMw+nx+MBjUzJkzZRiGJkyYoHPnzikSiai5uVlTpkyR1+uV1+vVlClT1NzcrEgkoo6ODk2YMEGGYWjmzJlqamqSJDU1NWnWrFmSpFmzZiWPAwAAAJkiJSvtkpRIJPTII4/o5MmT+uxnP6vx48fr5z//uV588UXt3r1bn/jEJ3TvvffK4/EoHA6rqKgo+Vq/369wOKxwOCy/3588XlhYeNnjF54vSWfOnJHP55Mk+Xw+nT179rL11dbWqra2VpK0cePGXudPFbfbrSNHhqmhwdDMmZZKS62U14Abz+12p+XvF1KLPjsDfXYG+uwMdu9zykJ7Tk6ONm3apHPnzunJJ5/UH//4R91zzz0aOnSouru79cwzz+gnP/mJFi1aJMu6NKz2tTJvGMZln3+1ysrKVFZWlnzc2tr6kb/m1TpyZJg++1mXYjFDHo+l6uqQTDOW8jpwYxUVFaXl7xdSiz47A312BvrsDOnq86hRo/r1vJTvHlNQUKCJEyequblZPp9PhmHI4/Fozpw5OnLkiKSelfKL/9BCoZB8Pp8KCwsVCoWSx8PhsHw+n/x+f6/joVBIhYWFkqQhQ4YoEolIkiKRiAYPHpyKt3lNGhoMxWKG4vGeXxsb89JdEgAAAGwgJaH97NmzOnfunKSenWR++9vfqri4OBmmLctSU1OTRo8eLUkyTVMNDQ2yLEuHDh1Sfn6+fD6fSkpKdPDgQUWjUUWjUR08eFAlJSXy+XwaOHCgDh06JMuy1NDQINM0k19rz549kqQ9e/Zo2rRpqXjL12TmTEsejyWXq+fXQKAz3SUBAADABlIyHhOJRLRt2zYlEglZlqVAIKDbbrtNjz/+eHLG/GMf+5iWL18uSZo6dar279+vhx56SLm5uSovL5ckeb1effGLX9Sjjz4qSVq0aFFy+8YHHnhA27dvV1dXl0pKSjR16lRJ0oIFC1RRUaG6ujoVFRVpzZo1qXjL16S0tGckprExT4FAJ6MxAAAAkCQZ1vUYCM9Cx48fT/k5mZlzBvrsDPTZGeizM9BnZ2CmHQAAAMBHQmgHAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtAMAAAA2R2gHAAAAbI7QbjPBoEdVVV4Fg550lwIAAACbcKe7ALxv3z5Dixf7FYsZ8ni8qq4OyTRj6S4LAAAAacZKu400NBiKxQzF4z2/NjbmpbskAAAA2AAr7TYyc6Ylj8eSJHk8lgKBzjRXBAAAADsgtNtIaaml6uqQGhvzFAh0MhoDAAAASYR22zHNGGEdAAAAvTDTDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtAMAAAA2R2gHAAAAbI7QDgAAANgcod2GgkGPqqq8CgY96S4FAAAANuBOdwHoLRj0aPFiv2IxQx6PV9XVIZlmLN1lAQAAII1YabeZxsY8xWKG4nFDsZihxsa8dJcEAACANCO020wg0CmPx5LLZcnjsRQIdKa7JAAAAKQZ4zE2Y5oxVVeH1NiYp0Cgk9EYAAAAENrtyDRjhHUAAAAkMR4DAAAA2ByhHQAAALA5QjsAAABgc4R2AAAAwOYI7QAAAIDNEdoBAAAAmyO0AwAAADZHaAcAAABsjtAOAAAA2Byh3YaCQY+qqrwKBj3pLgUAAAA24E53AegtGPRo8WK/YjFDHo9X1dUhmWYs3WUBAAAgjVhpt5nGxjzFYobicUOxmKHGxrx0lwQAAIA0I7TbTCDQKY/HkstlyeOxFAh0prskAAAApFlKxmO6urq0fv16dXd3Kx6Pq7S0VHfddZdaWlq0ZcsWRaNR3XLLLVq5cqXcbrdisZi2bt2qo0ePatCgQVq1apWGDx8uSXr55ZdVV1ennJwcLVu2TCUlJZKk5uZm7dq1S4lEQnPnztWCBQskqc9z2JVpxlRdHVJjY54CgU5GYwAAAJCalXaPx6P169dr06ZN+s53vqPm5mYdOnRIzz//vObPn6/KykoVFBSorq5OklRXV6eCggJVVVVp/vz5euGFFyRJx44d0969e/XUU0/pscce086dO5VIJJRIJLRz506tW7dOFRUVeu2113Ts2DFJ6vMcdmaaMa1cGSWwAwAAQFKKQrthGBowYIAkKR6PKx6PyzAMvfnmmyotLZUkzZ49W01NTZKkYDCo2bNnS5JKS0v1xhtvyLIsNTU1acaMGfJ4PBo+fLhGjhypI0eO6MiRIxo5cqRGjBght9utGTNmqKmpSZZl9XkOAAAAIFOkbE4kkUjokUce0cmTJ/XZz35WI0aMUH5+vlwulySpsLBQ4XBYkhQOh+X3+yVJLpdL+fn5amtrUzgc1vjx45Nf8+LXXHj+hd8fPnxYbW1tfZ7jg2pra1VbWytJ2rhxo4qKiq7zn8CHc7vdaTkvUos+OwN9dgb67Az02Rns3ueUhfacnBxt2rRJ586d05NPPqn/+Z//6fO5lmVdcswwjMsev9Lzr0ZZWZnKysqSj1tbW6/q9ddDUVFRWs6L1KLPzkCfnYE+OwN9doZ09XnUqFH9el7Kd48pKCjQxIkTdfjwYbW3tysej0vqWV0vLCyU1LNSHgqFJPWM07S3t8vr9fY6fvFrPng8FArJ5/Np0KBBfZ4DAAAAyBQpCe1nz57VuXPnJPXsJPPb3/5WxcXFmjRpkvbt2ydJqq+vl2makqTbbrtN9fX1kqR9+/Zp0qRJMgxDpmlq7969isViamlp0YkTJzRu3DiNHTtWJ06cUEtLi7q7u7V3716ZpinDMPo8BwAAAJApUjIeE4lEtG3bNiUSCVmWpUAgoNtuu0033XSTtmzZoh/84Ae65ZZbdPvtt0uSbr/9dm3dulUrV66U1+vVqlWrJEmjR49WIBDQmjVrlJOTo6985SvKyen5vuP+++/XE088oUQioTlz5mj06NGSpHvvvfey5wAAAAAyhWH1NSjucMePH0/5OZmZcwb67Az02RnoszPQZ2dgph3XJBj0qKrKq2DQk+5SAAAAkGb2vTWogwWDHi1e7FcsZsjj8aq6OsSNlgAAAByMlXYbamzMUyxmKB43FIsZamzMS3dJAAAASCNCuw0FAp3yeCy5XJY8HkuBQGe6SwIAAEAaMR5jQ6YZU3V1SI2NeQoEOhmNAQAAcDhCu02ZZoywDgAAAEmMxwAAAAC2R2gHAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtNtYMOhRVZVXwaAn3aUAAAAgjbgjqk0Fgx4tXuxXLGbI4/GqujrEHVIBAAAcipV2m2pszFMsZigeNxSLGWpszEt3SQAAAEgTQrtNBQKd8ngsuVyWPB5LgUBnuksCAABAmjAeY1OmGVN1dUiNjXkKBDoZjQEAAHAwQruNmWaMsA4AAADGYwAAAAC7I7QDAAAANkdoBwAAAGyO0A4AAADYHKEdAAAAsDlCOwAAAGBzhHYbCwY9qqryKhj0pLsUAAAApBH7tNtUMOjR4sV+xWKGPB6vqqtD7NkOAADgUKy021RjY55iMUPxuKFYzFBjY166SwIAAECaENptKhDolMdjyeWy5PFYCgQ6010SAAAA0oTxGJsyzZiqq0NqbMxTINDJaAwAAICDEdptzDRjhHUAAAAwHgMAAADYHaEdAAAAsDlCOwAAAGBzhHYAAADA5gjtAAAAgM0R2gEAAACbI7TbXDDoUVWVV8GgJ92lAAAAIE3Yp93GgkGPFi/2KxYz5PF4VV0dYt92AAAAB2Kl3cYaG/MUixmKxw3FYoYaG/PSXRIAAADSgNBuY4FApzweSy6XJY/HUiDQme6SAAAAkAaMx9iYacZUXR1SY2OeAoFORmMAAAAcitBuc6YZI6wDAAA4XEpCe2trq7Zt26bTp0/LMAyVlZVp3rx5+uEPf6hf/vKXGjx4sCTp7rvv1q233ipJevnll1VXV6ecnBwtW7ZMJSUlkqTm5mbt2rVLiURCc+fO1YIFCyRJLS0t2rJli6LRqG655RatXLlSbrdbsVhMW7du1dGjRzVo0CCtWrVKw4cPT8XbBgAAAK6LlIR2l8ul++67T2PGjFFHR4fWrl2rKVOmSJLmz5+vL3zhC72ef+zYMe3du1dPPfWUIpGIvvGNb+if//mfJUk7d+7U1772Nfn9fj366KMyTVM33XSTnn/+ec2fP1+f+tSn9C//8i+qq6vTHXfcobq6OhUUFKiqqkqvvfaaXnjhBa1evToVbxsAAAC4LlJyIarP59OYMWMkSQMHDlRxcbHC4XCfz29qatKMGTPk8Xg0fPhwjRw5UkeOHNGRI0c0cuRIjRgxQm63WzNmzFBTU5Msy9Kbb76p0tJSSdLs2bPV1NQkSQoGg5o9e7YkqbS0VG+88YYsy7qxbxgAAAC4jlI+097S0qJ3331X48aN0+9//3u9+uqramho0JgxY/SlL31JXq9X4XBY48ePT76msLAwGfL9fn/yuN/v1+HDh9XW1qb8/Hy5XK5Lnh8Oh5Ovcblcys/PV1tbW3Ik54La2lrV1tZKkjZu3KiioqIb94fQB7fbnZbzIrXoszPQZ2egz85An53B7n1OaWg/f/68Nm/erKVLlyo/P1933HGHFi1aJEmqrq7W97//fZWXl/e5En6544ZhXPGc/X1NWVmZysrKko9bW1uv+HVvhKKiorScF6lFn52BPjsDfXYG+uwM6erzqFGj+vW8lO3T3t3drc2bN+szn/mMpk+fLkkaOnSocnJylJOTo7lz5+qdd96R1LOCHgqFkq8Nh8MqLCy85HgoFJLP59OgQYPU3t6ueDze6/kf/FrxeFzt7e3yer0pec8AAADA9ZCS0G5Zlp5++mkVFxfr85//fPJ4JBJJ/v7Xv/61Ro8eLUkyTVN79+5VLBZTS0uLTpw4oXHjxmns2LE6ceKEWlpa1N3drb1798o0TRmGoUmTJmnfvn2SpPr6epmmKUm67bbbVF9fL0nat2+fJk2a9KGr83YSDHpUVeVVMOhJdykAAABIk5SMx7z99ttqaGjQzTffrIcfflhSz/aOr732mt577z0ZhqFhw4Zp+fLlkqTRo0crEAhozZo1ysnJ0Ve+8hXl5PR8f3H//ffriSeeUCKR0Jw5c5JB/95779WWLVv0gx/8QLfccotuv/12SdLtt9+urVu3auXKlfJ6vVq1alUq3vJ1EQx6tHixX7GYIY/Hq+rqEHu2AwAAOJBhsZXKZR0/fjzl5/zgLFVVlVebNg1SPG7I5bL08MNtWrkymvK6cH0xG+kM9NkZ6LMz0GdnYKYd1ywQ6JTHY8nlsuTxWAoEOtNdEgAAANIg5Vs+ov9MM6bq6pAaG/MUCHQyGgMAAOBQhHabM80YYR0AAMDhGI8BAAAAbI7QDgAAANgcoR0AAACwOUI7AAAAYHOEdgAAAMDmCO0ZIBj0qKrKq2DQk+5SAAAAkAZs+WhzwaBHixf7FYsZ8ni8qq4OsQUkAACAw7DSbnONjXmKxQzF44ZiMUONjXnpLgkAAAApRmi3uUCgUx6PJZfLksdjKRDoTHdJAAAASDHGY2zONGOqrg6psTFPgUAnozEAAAAORGjPAKYZI6wDAAA4GOMxAAAAgM0R2gEAAACbI7QDAAAANkdoBwAAAGyO0J4huCsqAACAc7F7TAbgrqgAAADOxkp7BuCuqAAAAM5GaM8A3BUVAADA2RiPyQDcFRUAAMDZCO0ZgruiAgAAOBfjMQAAAIDNEdoBAAAAmyO0AwAAADZHaAcAAABsjtCeIbgjKgAAgHOxe0wG4I6oAAAAzsZKewbgjqgAAADORmjPANwRFQAAwNkYj8kA3BEVAADA2T40tHd3d+vw4cP6wx/+oHPnzqmgoEAf+9jHNH78eLndZP5U4Y6oAAAAztVn6j579qxqamq0Z88eeb1eFRcXa8CAATp//rxeeeUVRaNRzZo1SwsWLNDgwYNTWTMAAADgKH2G9vXr12vOnDnatGmTCgsLL/n34XBY//Vf/6X169eroqLihhYJAAAAOFmfoX3Tpk1XHH8pLCzUF77wBc2bN++GFAYAAACgR5+7x/R3Xp25dgAAAODG6lfijsfjevXVV/XWW2+pra2t1797/PHHb0hhuFQw6GEHGQAAAAfq1z7tzz33nGprazVx4kQdPXpU06dP15kzZzRp0qQbXR/+7MJdUTdtGqTFi/0KBj3pLgkAAAAp0q/Q/vrrr2vdunWaN2+eXC6X5s2bp4cfflhvvvnmja4Pf8ZdUQEAAJyrX6G9q6tLfr9fkpSbm6vOzk4VFxfrvffeu5G14SLcFRUAAMC5+jXTXlxcrHfeeUfjxo3TmDFj9KMf/UgDBw687FaQuDG4KyoAAIBz9Su0L126VDk5PYvyX/7yl7Vjxw51dHRo+fLl/TpJa2urtm3bptOnT8swDJWVlWnevHmKRqOqqKjQqVOnNGzYMK1evVper1eWZWnXrl06cOCA8vLyVF5erjFjxkiS6uvr9dJLL0mSFi5cqNmzZ0uSjh49qm3btqmrq0tTp07VsmXLZBhGn+fIRNwVFQAAwJn6FdqLioo0dOhQSdJf/MVf6B//8R8lSadPn+7XSVwul+677z6NGTNGHR0dWrt2raZMmaL6+npNnjxZCxYsUE1NjWpqarRkyRIdOHBAJ0+eVGVlpQ4fPqwdO3Zow4YNikaj2r17tzZu3ChJWrt2rUzTlNfr1bPPPqsHH3xQ48eP17e+9S01Nzdr6tSpqqmpuew5AAAAgEzRr5n2v//7v7/s8dWrV/frJD6fL7lSPnDgQBUXFyscDqupqUmzZs2SJM2aNUtNTU2SpGAwqJkzZ8owDE2YMEHnzp1TJBJRc3OzpkyZIq/XK6/XqylTpqi5uVmRSEQdHR2aMGGCDMPQzJkzk1+rr3MAAAAAmaJfod2yrEuOtbe3J0dmrkZLS4veffddjRs3TmfOnJHP55PUE+zPnj0rSQqHwyoqKkq+xu/3KxwOKxwOJy+IlXruynq54xeeL6nPcwAAAACZ4orjMStWrJDUs3vMhd9fEI1G9alPfeqqTnb+/Hlt3rxZS5cuVX5+fp/Pu9w3CYZhXPa5hmFc9vlXq7a2VrW1tZKkjRs39vqmIVXcbveHnnffPkMNDYZmzrRUWvrR3zdSrz99Ruajz85An52BPjuD3ft8xdC+cuVKWZalb33rW1q5cmWvfzd06FCNGjWq3yfq7u7W5s2b9ZnPfEbTp0+XJA0ZMkSRSEQ+n0+RSESDBw+W1LNS3tramnxtKBSSz+dTYWGh3nrrreTxcDisiRMnyu/3KxQK9Xr+hZ1t+jrHB5WVlamsrCz5+OLzp0pRUdEVz3vhBkuxmCGPx1J1dYgLUzPQh/UZ2YE+OwN9dgb67Azp6nN/8/QV51smTpyoSZMmaefOnZo4cWKvf64msFuWpaefflrFxcX6/Oc/nzxumqb27NkjSdqzZ4+mTZuWPN7Q0CDLsnTo0CHl5+fL5/OppKREBw8eVDQaVTQa1cGDB1VSUiKfz6eBAwfq0KFDsixLDQ0NMk3ziufIRNxgCQAAwJn6tXtMTk6OXnzxRb322mtqa2vTc889p4MHD+rEiRO68847P/T1b7/9thoaGnTzzTfr4YcfliTdfffdWrBggSoqKlRXV6eioiKtWbNGkjR16lTt379fDz30kHJzc1VeXi5J8nq9+uIXv6hHH31UkrRo0aLk9o0PPPCAtm/frq6uLpWUlGjq1KmS1Oc5MlHPDZZ63i83WAIAAHAOw+rHQPizzz6rSCSiBQsWaMOGDfre976ncDisb37zm3rqqadSUWfKHT9+POXn7M+PZYJBDzdYynD8mNUZ6LMz0GdnoM/OYPfxmH6ttDc1NamyslIDBgxIXhB6YecWpBY3WAIAAHCefu3Z6Ha7lUgkeh07e/asBg0adEOKAgAAAPC+foX20tJSbd26VS0tLZKkSCSinTt3asaMGTe0OAAAAAD9DO333HOPhg8frn/4h39Qe3u7HnroIfl8Pi1atOhG1wcAAAA4Xr9m2t1ut5YuXaqlS5cmx2L6utkRbiwuRAUAAHCefoV2SWpvb9fx48d1/vz5Xsc/8YlPXPeicHm9b67k5eZKAAAADtGv0F5fX6+dO3dqwIABys3NTR43DENbt269YcWht4tvrnThMaEdAAAg+/UrtL/44otas2ZN8oZFSA9urgQAAOBM/QrtiURCn/zkJ290LfgQphlTdXWImXYAAACH6dfuMX/zN3+jH//4x5fs1Y7UM82YVq6MEtgBAAAcpM+V9hUrVvR6fPr0af37v/+7vF5vr+Pf/e53b0xlAAAAACRdIbSvXLkylXUAAAAA6EOfoX3ixInasWOHHnjggVTWAwAAAOADrjjT/qtf/SpVdeAqBIMeVVV5FQx60l0KAAAAUuCKu8dYlpWqOtBP3GAJAADAea4Y2ru7u1VdXX3FL7B48eLrWhCujBssAQAAOM+HrrSHQqFU1YJ+4AZLAAAAznPF0J6bm6vy8vJU1YJ+4AZLAAAAzsNMewYyzRhhHQAAwEGuuHvMX/7lX6aqDgAAAAB96DO0nzlzRo8++uiHfoHTp09f14IAAAAA9NbneMzjjz+uiRMnaubMmRo3bpxyct7P94lEQkeOHFFDQ4N+97vfafPmzSkpFj2CQQ8z7QAAAA7SZ2j/zne+o1/84hd65pln1NLSouHDh2vgwIHq6OhQS0uLRo4cqbKyMi1dujSF5YJ92gEAAJynz9Dudrv1uc99Tp/73OfU2tqqP/7xj2pvb1dBQYE+9rGPqbCwMJV14s/Ypx0AAMB5rrh7TF1d3SXHurq6FIlEJEmGYWjQoEH6+Mc/rqKiohtTIXphn3YAAADnuWJo/9WvfvWhX6C9vV3Hjx/XvffeqzvvvPO6FYbLY592AAAA57liaF+/fn2/vsif/vQnbdiwgdCeIuzTDgAA4CxX3Ke9v0aPHq1Pf/rT1+NLAQAAAPiA6xLaJenee++9Xl8K/RAMelRV5VUw6El3KQAAALjBrjgeA3ti20cAAABnuW4r7Uidi7d9jMUMNTbmpbskAAAA3ECE9gzUs+2jJZfLYttHAAAAB2A8JgOx7SMAAICzENozFNs+AgAAOAfjMQAAAIDNEdoBAAAAmyO0ZzD2agcAAHAGZtozFHu1AwAAOAcr7RmKvdoBAACcg9CeodirHQAAwDkYj8lQ7NUOAADgHIT2DMZe7QAAAM7AeAwAAABgc4T2DMaWjwAAAM6QkvGY7du3a//+/RoyZIg2b94sSfrhD3+oX/7ylxo8eLAk6e6779att94qSXr55ZdVV1ennJwcLVu2TCUlJZKk5uZm7dq1S4lEQnPnztWCBQskSS0tLdqyZYui0ahuueUWrVy5Um63W7FYTFu3btXRo0c1aNAgrVq1SsOHD0/FW77h2PIRAADAOVKy0j579mytW7fukuPz58/Xpk2btGnTpmRgP3bsmPbu3aunnnpKjz32mHbu3Knut1WZAAAgAElEQVREIqFEIqGdO3dq3bp1qqio0GuvvaZjx45Jkp5//nnNnz9flZWVKigoUF1dnSSprq5OBQUFqqqq0vz58/XCCy+k4u2mBFs+AgAAOEdKQvvEiRPl9Xr79dympibNmDFDHo9Hw4cP18iRI3XkyBEdOXJEI0eO1IgRI+R2uzVjxgw1NTXJsiy9+eabKi0tldTzDUJTU5MkKRgMavbs2ZKk0tJSvfHGG7Is64a8x1Rjy0cAAADnSOvuMa+++qoaGho0ZswYfelLX5LX61U4HNb48eOTzyksLFQ4HJYk+f3+5HG/36/Dhw+rra1N+fn5crlclzw/HA4nX+NyuZSfn6+2trbkSM7FamtrVVtbK0nauHGjioqKbsybvgK3293v8955p/Tqq3E1NBiaOdNSaemQG1wdrper6TMyF312BvrsDPTZGeze57SF9jvuuEOLFi2SJFVXV+v73/++ysvL+1wJv9xxwzCueI6reU1ZWZnKysqSj1tbW6/4tW+EoqKiqzrvuHE9/0hSGsrFNbraPiMz0WdnoM/OQJ+dIV19HjVqVL+el7bdY4YOHaqcnBzl5ORo7ty5eueddyT1rKCHQqHk88LhsAoLCy85HgqF5PP5NGjQILW3tysej/d6/ge/VjweV3t7e7/HdAAAAAC7SFtoj0Qiyd//+te/1ujRoyVJpmlq7969isViamlp0YkTJzRu3DiNHTtWJ06cUEtLi7q7u7V3716ZpinDMDRp0iTt27dPklRfXy/TNCVJt912m+rr6yVJ+/bt06RJkz50dT7TsO0jAABA9jOsFFyZuWXLFr311ltqa2vTkCFDdNddd+nNN9/Ue++9J8MwNGzYMC1fvlw+n0+S9NJLL+k///M/lZOTo6VLl2rq1KmSpP379+u5555TIpHQnDlztHDhQknS//7v/16y5aPH41FXV5e2bt2qd999V16vV6tWrdKIESP6VfPx48dvzB/GFVztj2V6b/tose1jhuDHrM5An52BPjsDfXYGu4/HpCS0Z6JMCO1VVV5t2jRI8bghl8vSww+3aeXK6A2sENcD//F3BvrsDPTZGeizM9g9tHNH1AzGto8AAADOkNYtH/HRmGZM1dUhNTbmKRDoZDQGAAAgSxHaM5xpxgjrAAAAWY7xmAzH7jEAAADZj5X2DNZ79xgvu8cAAABkKVbaM1hjY55iMUPxuKFYzFBjY166SwIAAMANQGjPYOweAwAA4AyMx2Qwdo8BAABwBkJ7hmP3GAAAgOzHeEwWYAcZAACA7MZKe4ZjBxkAAIDsx0p7hmMHGQAAgOxHaM9w7CADAACQ/RiPyXDsIAMAAJD9CO1ZgB1kAAAAshvjMVmCHWQAAACyFyvtWYAdZAAAALIbK+1ZgB1kAAAAshuhPQuwgwwAAEB2YzwmC7CDDAAAQHYjtGcJdpABAADIXozHZAl2jwEAAMherLRnAXaPAQAAyG6stGcBdo8BAADIboT2LMDuMQAAANmN8ZgswO4xAAAA2Y3QniUuBPULozEEdwAAgOxBaM8SXIwKAACQvZhpzxJcjAoAAJC9CO1ZgotRAQAAshfjMVmCi1EBAACyFyvtAAAAgM2x0p4luBAVAAAge7HSniW4EBUAACB7EdqzBBeiAgAAZC/GY7IEF6ICAABkL0J7FuGuqAAAANmJ0J5FuBgVAAAgOzHTnkW4GBUAACA7EdqzCBejAgAAZCfGY7IIF6MCAABkJ0J7luFiVAAAgOxDaM8yXIwKAACQfZhpzzJcjAoAAJB9UrLSvn37du3fv19DhgzR5s2bJUnRaFQVFRU6deqUhg0bptWrV8vr9cqyLO3atUsHDhxQXl6eysvLNWbMGElSfX29XnrpJUnSwoULNXv2bEnS0aNHtW3bNnV1dWnq1KlatmyZDMPo8xzZrOdi1J73yMWoAAAA2SElK+2zZ8/WunXreh2rqanR5MmTVVlZqcmTJ6umpkaSdODAAZ08eVKVlZVavny5duzYIakn5O/evVsbNmzQhg0btHv3bkWjUUnSs88+qwcffFCVlZU6efKkmpubr3iObHbhYtSHH25jNAYAACBLpCS0T5w48ZIV7qamJs2aNUuSNGvWLDU1NUmSgsGgZs6cKcMwNGHCBJ07d06RSETNzc2aMmWKvF6vvF6vpkyZoubmZkUiEXV0dGjChAkyDEMzZ85Mfq2+zgEAAABkkrRdiHrmzBn5fD5Jks/n09mzZyVJ4XBYRUVFyef5/X6Fw2GFw2H5/f7k8cLCwssev/D8K53jcmpra1VbWytJ2rhxY68aUsXtdn/k8+7bZ+jv/s6tri4pN3eQfvazbpWWWtepQlwP16PPsD/67Az02RnoszPYvc+22z3Gsi4NmIZhXPa5hmFc9vnXoqysTGVlZcnHra2t1+XrXo2ioqKPfN5XXvGqq2uQ4nFDXV2WXnmlQ+PGRa9ThbgerkefYX/02RnoszPQZ2dIV59HjRrVr+elbfeYIUOGKBKJSJIikYgGDx4sqWel/OI/sFAoJJ/Pp8LCQoVCoeTxcDgsn88nv9/f63goFFJhYeEVz5HNuCsqAABA9klbaDdNU3v27JEk7dmzR9OmTUseb2hokGVZOnTokPLz8+Xz+VRSUqKDBw8qGo0qGo3q4MGDKikpkc/n08CBA3Xo0CFZlqWGhgaZpnnFc2SzCxei3nNPu/7v/21PdzkAAAC4Dgzres2XXMGWLVv01ltvqa2tTUOGDNFdd92ladOmqaKiQq2trSoqKtKaNWuSWz7u3LlTBw8eVG5ursrLyzV27FhJUl1dnV5++WVJPVs+zpkzR5L0zjvvaPv27erq6lJJSYnuv/9+GYahtra2y56jP44fP35j/jCu4Hr9WKb3DZYsdpGxGX7M6gz02RnoszPQZ2ew+3hMSkJ7Jsrk0F5V5dWmTT1z7S6XpYcfbtPKlcy12wX/8XcG+uwM9NkZ6LMz2D20c0fULMRcOwAAQHax3e4x+OhMM6bHHz+jn/50oObN62A0BgAAIMMR2rNQMOjR+vVDFIsZev31XP2f/9NNcAcAAMhgjMdkocbGPMVihuJxQ7GYocbGvHSXBAAAgI+A0J6FmGkHAADILozHZKELe7Xv3p0vic2BAAAAMh0r7VnsRz8aqH/7twItXuxXMOhJdzkAAAC4RoT2LMVcOwAAQPYgtGcp5toBAACyBzPtWYq5dgAAgOzBSnuWY64dAAAg8xHasxhz7QAAANmB0J7FAoFOuVyWDKNntp25dgAAgMxEaM96xgd+BQAAQKYhtGexxsY8xeOSZRmKx8V4DAAAQIYitGcxtn0EAADIDmz5mMXY9hEAACA7sNLuAGz7CAAAkNkI7VmObR8BAAAyH6E9y12Ya8/J6dn60eeLp7skAAAAXCVCe5YzzZgef/yMcnKkRMLQ+vVDGJEBAADIMIR2B4hEXLKsntDOiAwAAEDmIbQ7AHdGBQAAyGyEdsfgzqgAAACZitDuANwZFQAAILMR2h2AHWQAAAAyG6HdAdhBBgAAILMR2h2CHWQAAAAyF6HdIdhBBgAAIHMR2h2FHWQAAAAyEaHdIdhBBgAAIHMR2h2CHWQAAAAyF6HdIdhBBgAAIHMR2h2EHWQAAAAyE6HdQRiRAQAAyEyEdgdhRAYAACAzEdodJhJxKZHoCe1dXYzIAAAAZAJCu8P4fHElEpJkKZEQIzIAAAAZgNDuMJGISzk5kmQoJ6fnMQAAAOyN0O4wgUCncnN7LkbNyeFiVAAAgExAaHcYLkYFAADIPIR2B+JiVAAAgMxCaHcgLkYFAADILIR2B7r4YlTDkN54IzfdJQEAAOAK3Oku4Ktf/aoGDBignJwcuVwubdy4UdFoVBUVFTp16pSGDRum1atXy+v1yrIs7dq1SwcOHFBeXp7Ky8s1ZswYSVJ9fb1eeuklSdLChQs1e/ZsSdLRo0e1bds2dXV1aerUqVq2bJkMw0jX27WFQKBTbrdXXV2SZUnV1flatKhdphlLd2kAAAC4DFustK9fv16bNm3Sxo0bJUk1NTWaPHmyKisrNXnyZNXU1EiSDhw4oJMnT6qyslLLly/Xjh07JEnRaFS7d+/Whg0btGHDBu3evVvRaFSS9Oyzz+rBBx9UZWWlTp48qebm5vS8SRsxzZgWL25Xz/cuhmIxaffu/HSXBQAAgD7YIrR/UFNTk2bNmiVJmjVrlpqamiRJwWBQM2fOlGEYmjBhgs6dO6dIJKLm5mZNmTJFXq9XXq9XU6ZMUXNzsyKRiDo6OjRhwgQZhqGZM2cmv5bTLVrUIY/HkmQlV9vZRQYAAMCe0j4eI0lPPPGEJOmv/uqvVFZWpjNnzsjn80mSfD6fzp49K0kKh8MqKipKvs7v9yscDiscDsvv9yePFxYWXvb4hedfTm1trWprayVJGzdu7HWeVHG73Sk77513Sl/+sqVnnzUkGerutvSb3/h0552JlJzfyVLZZ6QPfXYG+uwM9NkZ7N7ntIf2b3zjGyosLNSZM2f0zW9+U6NGjerzuZZlXXKsr/l0wzAu+/y+lJWVqaysLPm4tbW136+9XoqKilJ63nHjBkoaqgu7yOTmtqm1tSNl53eqVPcZ6UGfnYE+OwN9doZ09flK2fdiaR+PKSwslCQNGTJE06ZN05EjRzRkyBBFIhFJUiQS0eDBgyX1rJRf/IcZCoXk8/lUWFioUCiUPB4Oh+Xz+eT3+3sdD4VCyfOBXWQAAAAyRVpD+/nz59XR0ZH8/W9+8xvdfPPNMk1Te/bskSTt2bNH06ZNkySZpqmGhgZZlqVDhw4pPz9fPp9PJSUlOnjwoKLRqKLRqA4ePKiSkhL5fD4NHDhQhw4dkmVZamhokGmaaXu/dtOziwxz7QAAAHaX1vGYM2fO6Mknn5QkxeNxffrTn1ZJSYnGjh2riooK1dXVqaioSGvWrJEkTZ06Vfv379dDDz2k3NxclZeXS5K8Xq+++MUv6tFHH5UkLVq0SF6vV5L0wAMPaPv27erq6lJJSYmmTp2ahndqTxd2kfnXfy3Qhbn2xsY8tn4EAACwGcO6msFvBzl+/HjKz5mOWarnnx+oRx4Zmnz87W+f1pIlzLXfSMxGOgN9dgb67Az02RmYaYetMdcOAABgf4R2h2OuHQAAwP4I7Q7H3VEBAADsj9COS+6O+uKLrLYDAADYCaEdMs2Ybr/9/J8fGeruZrUdAADATgjtkCQNG5bo9fjUqcvfaRYAAACpR2iHpJ4RGbdbknp2AP3FLwbq+ecHprUmAAAA9CC0Q1LPiMzdd5/78yND8bj02GNDmW0HAACwAUI7knpW23suSL0Q3Bsb89JdFgAAgOMR2pFkmjEtXx7986OenWTOnk1rSQAAABChHR8weLCSe7ZL0jPPDGJEBgAAIM0I7eglEOhUTk7vERm2fwQAAEgvQjt6Mc2Y/uqvOnsdY/tHAACA9CK04xIrVkTZ/hEAAMBGCO24xOW2f1y3ju0fAQAA0oXQjstatKhDLlfv2fbvfteb7rIAAAAcidCOy7rcbPvPfz6A1XYAAIA0ILSjTytWRJWTI11YbU8kWG0HAABIB0I7+mSaMd1xx/lex159dQAXpQIAAKQYoR1XtGJFVC6XdGG13bK4KBUAACDVCO24ItOMacOG03++SyoXpQIAAKQDoR0fasmSDn32s4zJAAAApAuhHf3ywYtSLUtau3YowR0AACAFCO3ol0svSmW+HQAAIFUI7ei3FSuicrulntX29+fbN2wYnN7CAAAAshyhHf1mmjH9+MetGj++u9fx11/P1RNPcGEqAADAjUJox1UxzZiefLL3bjKStH37III7AADADUJox1UzzZhWrGj786PewX3hwkJm3AEAAK4zQjuuyWOPRVVefmlwf/31PP3t3xaxqwwAAMB1RGjHNbt8cDeUSLAdJAAAwPVEaMdHcmlwly5sB/nII0MZlwEAALgOCO34yB57LKpvf/vii1Oli8dl/uZvivSVr/gI7wAAANeI0I7rYsmSDm3cePqiu6a+Py4jST/72QAtWFDEDjMAAADXgNCO62bJkg69/HKrpk/v+vORi1fde0Zmtm8fpOnThzHvDgAAcBUI7biuTDOml14KfWDOvffIzLFjbj3yyFCZ5nDNn+8nwAMAAHwIQjtuiMcei+onP2nVnXee//ORS0dmTpxwqbk5V488MlSTJg3X9OnDmX0HAAC4DHe6C0D2Ms2Ydu6MKBj0aMOGwXr99Vx9cNW9h6XTp106fVo6dsyln/1sgIYNi2vYsIRycy3dfXe7lizpSMM7AAAAsAdCO264CyMzwaBHjz46RG+95dH74V3qHeAlydKpUy6dOuWSJDU352rLlkHyehPyeKSzZ3tW64uL45owIaZFizpkmrEUvRsAAIDUI7QjZUwzpl/8olXPPz9QL76Yr9Onc/Tee271DvDS5UL8iRMuSa5eR48dc+n113P1r/9aoI9/PKbu7p4wP3hwIhnsBw9OKBZTr7A/eDAr+AAAILMYlmV9MDFB0vHjx1N+zqKiIrW2tqb8vOkUDHr03e969cYbbkWjhk6fdl3mWR8M8R907X+F/+Iv4nK5embt+xP2+/r91Tz33Dm34vHEVb8uFbVlyusyobaL+2y32uz855ZptRUWGgqHE7asLVNfZ8faXC6XCgq6bVmbnf/cMqm23FxL/+//ubRgQYtSbdSoUf16HqG9D4T29Hj++YHasaNAHR056u6WTp68XIi/mPERzsZffQAA8L5vf/t0yn8KT2j/iAjt9nBhlKary+j1nXNnp5Gceb92HyXwAwCA7GJp1qxO/du/hVN61v6GdmbaYWtLlnT0+R3vhdGad95xXfHHYB/8Mdn7K/h8vwoAAN43b559r3VzxEp7c3Ozdu3apUQioblz52rBggUf+hpW2rPbxSv4qZ7XY6Y9u98TM+3Oqo2Zdme8J2bas7+2TJhpz/qV9kQioZ07d+prX/ua/H6/Hn30UZmmqZtuuindpSGNrrSCf6PxzZkz0GdnoM/OQJ+doafP6a6ib1l/R9QjR45o5MiRGjFihNxut2bMmKGmpqZ0lwUAAAD0W9aH9nA4LL/fn3zs9/sVDqf2AgMAAADgo8j68ZjLjewbhnHJsdraWtXW1kqSNm7cqKKiohte2we53e60nBepRZ+dgT47A312BvrsDHbvc9aHdr/fr1AolHwcCoXk8/kueV5ZWZnKysqSj9Mxu8bMnDPQZ2egz85An52BPjtDuvrc3wtRs348ZuzYsTpx4oRaWlrU3d2tvXv3yjTNdJcFAAAA9FvWr7S7XC7df//9euKJJ5RIJDRnzhyNHj063WUBAAAA/Zb1oV2Sbr31Vt16663pLgMAAAC4Jlk/HgMAAABkOkI7AAAAYHOEdgAAAMDmCO0AAACAzRHaAQAAAJsjtAMAAAA2R2gHAAAAbI7QDgAAANgcoR0AAACwOcOyLCvdRQAAAADoGyvtNrJ27dp0l4AUoM/OQJ+dgT47A312Brv3mdAOAAAA2ByhHQAAALA51z/90z/9U7qLwPvGjBmT7hKQAvTZGeizM9BnZ6DPzmDnPnMhKgAAAGBzjMcAAAAANkdoBwAAAGzOne4C0KO5uVm7du1SIpHQ3LlztWDBgnSXhGvU2tqqbdu26fTp0zIMQ2VlZZo3b56i0agqKip06tQpDRs2TKtXr5bX65VlWdq1a5cOHDigvLw8lZeX23qmDu9LJBJau3atCgsLtXbtWrW0tGjLli2KRqO65ZZbtHLlSrndbsViMW3dulVHjx7VoEGDtGrVKg0fPjzd5aMfzp07p6efflp/+tOfZBiGVqxYoVGjRvFZzjL/8R//obq6OhmGodGjR6u8vFynT5/m85zhtm/frv3792vIkCHavHmzJF3T/4vr6+v10ksvSZIWLlyo2bNnp+X9sNJuA4lEQjt37tS6detUUVGh1157TceOHUt3WbhGLpdL9913nyoqKvTEE0/o1Vdf1bFjx1RTU6PJkyersrJSkydPVk1NjSTpwIEDOnnypCorK7V8+XLt2LEjze8A/fXTn/5UxcXFycfPP/+85s+fr8rKShUUFKiurk6SVFdXp4KCAlVVVWn+/Pl64YUX0lUyrtKuXbtUUlKiLVu2aNOmTSouLuaznGXC4bBeeeUVbdy4UZs3b1YikdDevXv5PGeB2bNna926db2OXe3nNxqNavfu3dqwYYM2bNig3bt3KxqNpvy9SIR2Wzhy5IhGjhypESNGyO12a8aMGWpqakp3WbhGPp8v+d35wIEDVVxcrHA4rKamJs2aNUuSNGvWrGSPg8GgZs6cKcMwNGHCBJ07d06RSCRt9aN/QqGQ9u/fr7lz50qSLMvSm2++qdLSUkk9/7O4uMcXVmZKS0v1xhtviD0A7K+9vV2/+93vdPvtt0uS3G63CgoK+CxnoUQioa6uLsXjcXV1dWno0KF8nrPAxIkT5fV6ex272s9vc3OzpkyZIq/XK6/XqylTpqi5uTnl70ViPMYWwuGw/H5/8rHf79fhw4fTWBGul5aWFr377rsaN26czpw5I5/PJ6kn2J89e1ZST/+LioqSr/H7/QqHw8nnwp6+973vacmSJero6JAktbW1KT8/Xy6XS5JUWFiocDgsqfdn3OVyKT8/X21tbRo8eHB6ike/tLS0aPDgwdq+fbv+8Ic/aMyYMVq6dCmf5SxTWFiov/7rv9aKFSuUm5urT37ykxozZgyf5yx1tZ/fD2a0i/8upBor7TZwue/QDcNIQyW4ns6fP6/Nmzdr6dKlys/P7/N59D/z/Pd//7eGDBnS73llepyZ4vG43n33Xd1xxx36zne+o7y8vOSP0i+HPmemaDSqpqYmbdu2Tc8884zOnz9/xZVU+pydrqav6eo3K+024Pf7FQqFko9DoRArMxmuu7tbmzdv1mc+8xlNnz5dkjRkyBBFIhH5fD5FIpHkqozf71dra2vytfTf/t5++20Fg0EdOHBAXV1d6ujo0Pe+9z21t7crHo/L5XIpHA6rsLBQ0vufcb/fr3g8rvb29kt+ZAv78fv98vv9Gj9+vKSeUYiamho+y1nmt7/9rYYPH57s4/Tp0/X222/zec5SV/v5LSws1FtvvZU8Hg6HNXHixJTXLbHSbgtjx47ViRMn1NLSou7ubu3du1emaaa7LFwjy7L09NNPq7i4WJ///OeTx03T1J49eyRJe/bs0bRp05LHGxoaZFmWDh06pPz8fP5Hb3P33HOPnn76aW3btk2rVq3SJz7xCT300EOaNGmS9u3bJ6lnt4ELn+PbbrtN9fX1kqR9+/Zp0qRJrMxlgKFDh8rv9+v48eOSesLdTTfdxGc5yxQVFenw4cPq7OyUZVnJPvN5zk5X+/ktKSnRwYMHFY1GFY1GdfDgQZWUlKSldu6IahP79+/Xc889p0QioTlz5mjhwoXpLgnX6Pe//72+/vWv6+abb07+h/zuu+/W+PHjVVFRodbWVhUVFWnNmjX/v737C2lyDeA4/nXTdylOZ5lJGgQR3kSsC6EgUVsXEd7UxSijTOcotAu7EUGKSKqLIPozal1sGRWLEgpWd60kyAiCLNCIgqQLYSTT5tJtre1cHM44nU7neA6n47Lf52rv8z7v+z7vA+/48fC8z5tdZsrn8/H8+XMMw6Cjo4NVq1bN813IXI2MjBAMBunp6SEcDn+1RFxBQQHJZBKPx8Pbt28pLi6mq6uLZcuWzXfTZQ7Gxsbwer2kUikqKiro6Oggk8noWV5gbty4wdDQEGazmZUrV7J//34ikYie5x/c6dOnGR0dZXp6mtLSUpxOJ7W1tf/4+b1//z63bt0Cfl3ysbGxcV7uR6FdRERERCTHaXqMiIiIiEiOU2gXEREREclxCu0iIiIiIjlOoV1EREREJMcptIuIiIiI5DiFdhER+S52795NOBye72aIiCwICu0iIgtUZ2cnL168YHBwkEOHDn3Xax05coRQKPRF2ZUrV7R+tYjIf0ShXURE/tLnz5/nuwkiIj89fVxJRGSB6uzspKmpiatXr5JKpTAMA7PZTH9/P58+fSIQCPD48WNSqRS1tbXs3bsXwzAYGRnh3LlzbNmyhbt377J27VpaW1vxeDy8fv2adDpNTU0NbrebJUuWEAgEuH37Nvn5+ZhMJhoaGnC5XDidTs6ePUtlZSUzMzP4/X6ePXuGxWLB4XCwbds2TCYTg4ODhEIhVq9ezYMHDygqKqK9vZ1169YBv35CfmBggGg0itVqZceOHdTV1c1z74qI/L/y57sBIiLy/VRVVeF2uwmFQvT19WXLr127Rjgc5uTJk5jNZs6cOcPAwADNzc0ATE1NEYvFOH/+PJlMhkQiQUNDAwcPHiSdTnPhwgV8Ph/d3d3s3LmTV69eUVdXh8Ph+NN2+P1+ZmZm8Hg8TE9Pc+zYMcrKyti0aRMAb968ob6+Hp/Px7179/B6vXi9XhKJBJcuXeLEiRMsX76cyclJYrHY9+84EZEco+kxIiI/mUwmQygUoqWlheLiYgoLC9m+fTuPHj3K1snLy8PpdFJQUIBhGFitVtavX4/FYsnWf/ny5Zyul06nGRoaorm5mcLCQioqKmhqauLhw4fZOuXl5WzevBmTyUR9fT2Tk5N8+PAh25Z3796RTCYpKytjxYoV/22HiIj8ADTSLiLyk4lGoyQSCXp6erJlmUyGdDqd3S4pKcEwjOx2IpHg8uXLDA8P8/HjRwBmZ2dJp9OYTH89/hONRkmlUpSXl2fLli5dSiQSyW7bbLbsb4vFAkA8Hsdms9HV1UUwGMTr9VJTU8OePXuoqqr6l3cvIvJjUmgXEfnJWK1WDMPg1KlTLF68+E/r5OXlfbEdDAYZHx/n+PHj2Gw2xsbG6O7u5rfXov5Y//dKSkowm81MTExQXV0NwMTExDev/Ud2ux273U4ymeT69etcvHiRo0ePzulYEZGFQtNjREQWOJvNRiQSIZVKAWAymXA4HLG9oSYAAAEiSURBVPT392enoEQiEYaHh795jng8jmEYFBUVEYvFuHnz5hf7S0tLv7kmu8lkYsOGDQQCAWZnZ3n//j137tyZ08ukU1NTPH36lHg8Tn5+PosWLfrbkX0RkYVI/3wiIgvcmjVrqK6uxu1243K5ANi1axeVlZX09vbS0tJCX18f4+Pj3zzH1q1bSSaTuFwuent7sdvtX+1/8uQJra2t+P3+r45va2vDYrFw4MABDh8+zMaNG2lsbPzbtmcyGYLBIPv27aOtrY3R0VHa29v/YQ+IiPz4tOSjiIiIiEiO00i7iIiIiEiOU2gXEREREclxCu0iIiIiIjlOoV1EREREJMcptIuIiIiI5DiFdhERERGRHKfQLiIiIiKS4xTaRURERERy3C9IwBiArUvQdgAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 864x576 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig,ax = plt.subplots(figsize=(12,8))\n",
"\n",
"ax.set_ylabel('J(Theta)')\n",
"ax.set_xlabel('Iterations')\n",
"_=ax.plot(range(n_iter),cost_history,'b.')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3> After around 150 iterations the cost is flat so the remaining iterations are not needed or will not result in any further optimization. Let us zoom in till iteration 200 and see the curve"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-19T05:22:30.561842Z",
"start_time": "2018-08-19T05:22:30.371532Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAHVCAYAAABMuKcFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3WtsVOedx/HfmfHgYgac8ZjLwhKJq3YT4bXpoGDaAAEXoRCtEGFDabfdkEak6xYUULJb0t1tVymIXUrMmouiTRDKpVWdIqCvupEcL7iCofEEDC3ZhhASKRSI4xkHbC72eHz2heupgRnfZ+aZOd/Pq/hkmHP+jME/nv9zsWzbtgUAAABjuTL9AAAAAOgbgQ0AAMBwBDYAAADDEdgAAAAMR2ADAAAwHIENAADAcAQ2AAAAwxHYAAAADEdgAwAAMByBDQAAwHB5mX6AVLh8+XJK37+4uFjNzc0pvYfJqN+59Tu5don6qd+59Tu5dim19U+ePHlAr2OEDQAAwHAENgAAAMMR2AAAAAxHYAMAADAcgQ0AAMBwBDYAAADDEdgAAAAMR2ADAAAwXL8b5zY3N2vv3r364osvZFmWKioq9Oijj+qtt97SO++8o3HjxkmS1q5dq7lz50qSDh8+rLq6OrlcLq1bt06lpaWSpMbGRh04cEBdXV1aunSpVq5cKUlqamrSrl271NbWpmnTpmnDhg3Ky8tTNBrVnj17dPHiRY0dO1bPPvusJkyYkKrfCwAAACP1G9jcbre+9a1vafr06bp165Z+8IMfqKSkRJK0YsUK/e3f/u0dr7906ZJOnDihl156SS0tLXrxxRf1X//1X5Kk/fv361/+5V/k9/u1ZcsWBQIB/eVf/qXefPNNrVixQl/5ylf03//936qrq9OyZctUV1enMWPGaPfu3Tp+/Lh+9rOfadOmTSn4bQAAADBXvy1Rn8+n6dOnS5JGjx6tKVOmKBKJJH19Q0ODFixYII/HowkTJmjSpEm6cOGCLly4oEmTJmnixInKy8vTggUL1NDQINu2de7cOc2fP1+StHjxYjU0NEiSQqGQFi9eLEmaP3++fv/738u27eHWDAAAkFUGNYetqalJH3/8sWbOnClJevvtt/Xcc89p3759amtrkyRFIhH5/f74rykqKlIkErnnut/vVyQSUWtrqwoKCuR2u+94/d3v5Xa7VVBQoNbW1mGUCwAAkH0GfPj77du3tXPnTj355JMqKCjQsmXLtHr1aklSTU2NXn/9dVVWViYdAUt03bKsPu850F9TW1ur2tpaSdL27dtVXFzcbz3DkZeXl/J7mIz6nVu/k2uXqJ/6nVu/k2uXzKh/QIGts7NTO3fu1MMPP6yHHnpIknTffffF///SpUv1H//xH5K6R87C4XD8/0UiERUVFUnSHdfD4bB8Pp/Gjh2rmzdvKhaLye123/H6nvfy+/2KxWK6efOmvF7vPc9XUVGhioqK+NfNzc0D/g0YiuLi4pTfw2TU79z6nVy7RP3U79z6nVy7lNr6J0+ePKDX9dsStW1bL7/8sqZMmaLHHnssfr2lpSX+3++++66mTp0qSQoEAjpx4oSi0aiampp05coVzZw5UzNmzNCVK1fU1NSkzs5OnThxQoFAQJZl6cEHH9TJkyclSUePHlUgEJAkffnLX9bRo0clSSdPntSDDz7Y76gcAABArul3hO2DDz5QfX297r//fj3//POSurfwOH78uD755BNZlqXx48dr/fr1kqSpU6eqvLxcmzdvlsvl0ne+8x25XN258KmnntLWrVvV1dWlRx55JB7yvvnNb2rXrl36xS9+oWnTpmnJkiWSpCVLlmjPnj3asGGDvF6vnn322ZT8JgAAAJjMsnNw2eXly5dT9t6hkEdnz/pUUtKiQCCasvuYjKFx59bv5Nol6qd+59bv5NolM1qiA150gO6wtmaNX9GoJY/Hr5qasGNDGwAASB+OphqEYDBf0ailWMxSNGopGMzP9CMBAAAHILANQnl5uzweW263LY/HVnl5e6YfCQAAOAAt0UEIBKKqqQk7fg4bAABILwLbIAUCUS1f3qXmZsIaAABID1qiAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsAEAABiOwAYAAGA4AhsAAIDhCGwAAACGI7ABAAAYjsAGAABgOAIbAACA4QhswxAKebR7t1ehkCfTjwIAAHJYXqYfIFuFQh6tWeNXNGrJ4/GqpiasQCCa6ccCAAA5iBG2IQoG8xWNWorFLEWjloLB/Ew/EgAAyFEEtiEqL2+Xx2PL7bbl8dgqL2/P9CMBAIAcRUt0iAKBqGpqwgoG81Ve3k47FAAApAyBbRgCgShBDQAApBwtUQAAAMMR2AAAAAxHYAMAADAcgQ0AAMBwBDYAAADDEdgAAAAMR2ADAAAwHIENAADAcAQ2AAAAwxHYAAAADEdgAwAAMByBbQSEQh7t3u1VKOTJ9KMAAIAcxOHvwxQKebRmjV/RqCWPx6uamjAHwgMAgBHFCNswBYP5ikYtxWKWolFLwWB+ph8JAADkGALbMJWXt8vjseV22/J4bJWXt2f6kQAAQI6hJTpMgUBUNTVhBYP5Ki9vpx0KAABGHIFtBAQCUYIaAABIGVqiAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsAEAABiOwAYAAGA4AhsAAIDhCGwAAACGI7ABAAAYjsA2wkIhj3bv9ioU8mT6UQAAQI7gLNERFAp5tGaNX9GoJY/Hq5qaMGeMAgCAYWOEbQQFg/mKRi3FYpaiUUvBYH6mHwkAAOQAAtsIKi9vl8djy+225fHYKi9vz/QjAQCAHEBLdAQFAlHV1IQVDOarvLyddigAABgRBLYRFghECWoAAGBE0RIFAAAwHIENAADAcAQ2AAAAwxHYAAAADEdgAwAAMByBDQAAwHAENgAAAMMR2AAAAAxHYAMAADAcgQ0AAMBwBLYUCoU82r3bq1DIk+lHAQAAWYyzRFMkFPJozRq/olFLHo9XNTVhzhgFAABDwghbigSD+YpGLcVilqJRS8FgfqYfCQAAZCkCW4qUl7fL47HldtvyeGyVl7dn+pEAAECWoiWaIoFAVDU1YQWD+Sovb6cdCgAAhozAlkKBQJSgBgAAho2WKAAAgOH6HWFrbm7W3r179cUXX8iyLFVUVOjRRx9VW1ubqqqq9Pnnn2v8+PHatGmTvF6vbNvWgQMHdPr0aeXn56uyslLTp0+XJB09elSHDh2SJK1atUqLFy+WJF28eFF79+5VR0eHysrKtG7dOlmWlfQeAAAATtLvCJvb7da3vvUtVVVVaevWrXr77bd16dIlHTlyRHPmzFF1dbXmzJmjI0eOSJJOnz6tq1evqrq6WuvXr9err74qSWpra9PBgwe1bds2bdu2TQcPHlRbW5sk6ZVXXtEzzzyj6upqXb16VY2NjZKU9B4AAABO0m9g8/l88RGy0aNHa8qUKYpEImpoaNCiRYskSYsWLVJDQ4MkKRQKaeHChbIsS7Nnz9aNGzfU0tKixsZGlZSUyOv1yuv1qqSkRI2NjWppadGtW7c0e/ZsWZalhQsXxt8r2T0AAACcZFCLDpqamvTxxx9r5syZunbtmnw+n6TuUHf9+nVJUiQSUXFxcfzX+P1+RSIRRSIR+f3++PWioqKE13teLynpPe5WW1ur2tpaSdL27dvvuH8q5OXlpfweJqN+59bv5Nol6qd+59bv5NolM+ofcGC7ffu2du7cqSeffFIFBQVJX2fb9j3XLMtK+FrLshK+frAqKipUUVER/7q5uXnY79mX4uLilN/DZNTv3PqdXLtE/dTv3PqdXLuU2vonT548oNcNaJVoZ2endu7cqYcfflgPPfSQJKmwsFAtLS2SpJaWFo0bN05S9whZ76LC4bB8Pp+KiooUDofj1yORiHw+n/x+/x3Xw+GwioqK+rwHAACAk/Qb2Gzb1ssvv6wpU6bosccei18PBAI6duyYJOnYsWOaN29e/Hp9fb1s29b58+dVUFAgn8+n0tJSnTlzRm1tbWpra9OZM2dUWloqn8+n0aNH6/z587JtW/X19QoEAn3eAwAAwEn6bYl+8MEHqq+v1/3336/nn39ekrR27VqtXLlSVVVVqqurU3FxsTZv3ixJKisr06lTp7Rx40aNGjVKlZWVkiSv16vHH39cW7ZskSStXr06vkXH008/rX379qmjo0OlpaUqKyuTpKT3yDahkIcTDwAAwJBZ9khMIjPM5cuXU/r+g+llh0IerVnjVzRqyeOxVVMTzvrQxlwG59bv5Nol6qd+59bv5NqlLJrDhqELBvMVjVqKxSxFo5aCwfxMPxIAAMgyBLYUKy9vl8djy+225fHYKi9vz/QjAQCALMPh7ykWCERVUxNmDhsAABgyAlsaBAJRghoAABgyWqIAAACGI7ABAAAYjsAGAABgOAIbAACA4QhsAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsKVZKOTR7t1ehUKeTD8KAADIEhxNlUahkEdr1vgVjVryeLyqqQlzZBUAAOgXI2xpFAzmKxq1FItZikYtBYP5mX4kAACQBQhsaVRe3i6Px5bbbcvjsVVe3p7pRwIAAFmAlmgaBQJR1dSEFQzmq7y8nXYoAAAYEAJbmgUCUYIaAAAYFFqiAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsAEAABiOwAYAAGA4AluGcAg8AAAYKDbOzQAOgQcAAIPBCFsGcAg8AAAYDAJbBnAIPAAAGAxaohnAIfAAAGAwCGwZwiHwAABgoGiJAgAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsAEAABiOwAYAAGA4AhsAAIDhCGwG4CB4AADQFzbOzTAOggcAAP1hhC3DOAgeAAD0h8CWYRwEDwAA+kNLNMM4CB4AAPSHwGYADoIHAAB9oSUKAABgOAIbAACA4QhsAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsBmEQ+ABAEAibJxrCA6BBwAAyTDCZggOgQcAAMkQ2AzBIfAAACAZWqKG4BB4AACQDIHNIBwCDwAAEqElCgAAYDgCGwAAgOEIbAAAAIYjsAEAABiOwAYAAGA4ApuhOKYKAAD0YFsPA3FMFQAA6I0RNgNxTBUAAOiNwGYgjqkCAAC90RI1EMdUAQCA3ghshuKYKgAA0IOWKAAAgOEIbAAAAIYjsAEAABiOwAYAAGC4fhcd7Nu3T6dOnVJhYaF27twpSXrrrbf0zjvvaNy4cZKktWvXau7cuZKkw4cPq66uTi6XS+vWrVNpaakkqbGxUQcOHFBXV5eWLl2qlStXSpKampq0a9cutbW1adq0adqwYYPy8vIUjUa1Z88eXbx4UWPHjtWzzz6rCRMmpOQ3AQAAwGT9jrAtXrxYL7zwwj3XV6xYoR07dmjHjh3xsHbp0iWdOHFCL730kn74wx9q//796urqUldXl/bv368XXnhBVVVVOn78uC5duiRJevPNN7VixQpVV1drzJgxqqurkyTV1dVpzJgx2r17t1asWKGf/exnI1k3AABA1ug3sD3wwAPyer0DerOGhgYtWLBAHo9HEyZM0KRJk3ThwgVduHBBkyZN0sSJE5WXl6cFCxaooaFBtm3r3Llzmj9/vqTucNjQ0CBJCoVCWrx4sSRp/vz5+v3vfy/btodYZvbiTFEAADDkfdjefvtt1dfXa/r06fr2t78tr9erSCSiWbNmxV9TVFSkSCQiSfL7/fHrfr9fH374oVpbW1VQUCC3233P6yORSPzXuN1uFRQUqLW1Nd6G7a22tla1tbWSpO3bt6u4uHioZQ1IXl5eyu8hSSdPWvr61/PU0SGNGjVW//M/nZo/P/OhNV31m8rJ9Tu5don6qd+59Tu5dsmM+ocU2JYtW6bVq1dLkmpqavT666+rsrIy6QhYouuWZfV5j8H8moqKClVUVMS/bm5u7vO9h6u4uDjl95CkX//aq46OsYrFLHV02Pr1r29p5sy2lN+3P+mq31ROrt/JtUvUT/3Ord/JtUuprX/y5MkDet2QVoned999crlccrlcWrp0qT766CNJ3SNn4XA4/rpIJKKioqJ7rofDYfl8Po0dO1Y3b95ULBa74/V3v1csFtPNmzcH3JrNFZwpCgAApCEGtpaWlvh/v/vuu5o6daokKRAI6MSJE4pGo2pqatKVK1c0c+ZMzZgxQ1euXFFTU5M6Ozt14sQJBQIBWZalBx98UCdPnpQkHT16VIFAQJL05S9/WUePHpUknTx5Ug8++GC/o3K5pudM0eefb1VNTZijqgAAcKh+W6K7du3S+++/r9bWVn33u9/VE088oXPnzumTTz6RZVkaP3681q9fL0maOnWqysvLtXnzZrlcLn3nO9+Ry9WdCZ966ilt3bpVXV1deuSRR+Ih75vf/KZ27dqlX/ziF5o2bZqWLFkiSVqyZIn27NmjDRs2yOv16tlnn03V74HROFMUAABYdg4uvbx8+XJK359ePvU7tX4n1y5RP/U7t34n1y5l8Rw2AAAApA+BDQAAwHAENgAAAMMR2LIIpx4AAOBMQz7pAOkVCnm0Zo1f0aglj8fLNh8AADgII2xZIhjMVzRqKRazFI1aCgbzM/1IAAAgTQhsWYJTDwAAcC5aolmi59SDYDBf5eXttEMBAHAQAlsW4dQDAACciZYoAACA4QhsAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbFmII6oAAHAWtvXIMhxRBQCA8zDClmU4ogoAAOchsGUZjqgCAMB5aIlmGY6oAgDAeQhsWYgjqgAAcBZaogAAAIYjsAEAABiOwAYAAGA4AluWYxNdAAByH4sOshib6AIA4AyMsGUxNtEFAMAZCGxZjE10AQBwBlqiWYxNdAEAcAYCW5ZjE10AAHIfLVEAAADDEdgAAAAMR2ADAAAwHIENAADAcAS2HMKpBwAA5CZWieYITj0AACB3McKWIzj1AACA3EVgyxGcegAAQO6iJZojOPUAAIDcRWDLIZx6AABAbqIlCgAAYDgCGwAAgOEIbAAAAIYjsOUgNtAFACC3sOggx7CBLgAAuYcRthzDBroAAOQeAluOYQNdAAByDy3RHMMGugAA5B4CWw5iA10AAHILLVEAAADDEdgAAAAMR2DLcezJBgBA9mMOWw5jTzYAAHIDI2w5jD3ZAADIDQS2HMaebAAA5AZaojmMPdkAAMgNBLYcx55sAABkP1qiAAAAhiOwAQAAGI7A5hDsxwYAQPZiDpsDsB8bAADZjRE2B2A/NgAAshuBzQHYjw0AgOxGS9QB2I8NAIDsRmBzCPZjAwAge9ESBQAAMByBDQAAwHAENgdiTzYAALILc9gchj3ZAADIPoywOQx7sgEAkH0IbA7DnmwAAGQfWqIOw55sAABkHwKbA7EnGwAA2aXfwLZv3z6dOnVKhYWF2rlzpySpra1NVVVV+vzzzzV+/Hht2rRJXq9Xtm3rwIEDOn36tPLz81VZWanp06dLko4ePapDhw5JklatWqXFixdLki5evKi9e/eqo6NDZWVlWrdunSzLSnoPAAAAp+l3DtvixYv1wgsv3HHtyJEjmjNnjqqrqzVnzhwdOXJEknT69GldvXpV1dXVWr9+vV599VVJ3QHv4MGD2rZtm7Zt26aDBw+qra1NkvTKK6/omWeeUXV1ta5evarGxsY+7wEAAOA0/Qa2Bx544J6RrYaGBi1atEiStGjRIjU0NEiSQqGQFi5cKMuyNHv2bN24cUMtLS1qbGxUSUmJvF6vvF6vSkpK1NjYqJaWFt26dUuzZ8+WZVlauHBh/L2S3QMjh/3YAADIDkOaw3bt2jX5fD5Jks/n0/Xr1yVJkUhExcXF8df5/X5FIhFFIhH5/f749aKiooTXe17f1z0wMtiPDQCA7DGiiw5s277nmmVZCV9rWVbC1w9FbW2tamtrJUnbt2+/IzSmQl5eXsrvkWpnz7ri+7F1f+3T8uVdA/q1uVD/cDi5fifXLlE/9Tu3fifXLplR/5ACW2FhoVpaWuTz+dTS0qJx48ZJ6h4ha25ujr8uHA7L5/OpqKhI77//fvx6JBLRAw88IL/fr3A4fMfri4qK+rxHIhUVFaqoqIh/3fsZUqG4uDjl90i1khKPPJ7u0U2Px1ZJSYuamwc2wpYL9Q+Hk+t3cu0S9VO/c+t3cu1SauufPHnygF43pI1zA4GAjh07Jkk6duyY5s2bF79eX18v27Z1/vx5FRQUyOfzqbS0VGfOnFFbW5va2tp05swZlZaWyufzafTo0Tp//rxs21Z9fb0CgUCf98DI6NmP7fnnW2mHAgBguH5H2Hbt2qX3339fra2t+u53v6snnnhCK1euVFVVlerq6lRcXKzNmzdLksrKynTq1Clt3LhRo0aNUmVlpSTJ6/Xq8ccf15YtWyRJq1evji9kePrpp7Vv3z51dHSotLRUZWVlkpT0Hhg57McGAEB2sOyRmkhmkMuXL6f0/Rkapn6n1u/k2iXqp37n1u/k2qUsboki97DFBwAA5uJoKrDFBwAAhmOEDQoG8+NbfESjloLB/Ew/EgAA6IXABpWXt8vjseV22/J4bJWXt2f6kQAAQC+0RBHf4iMYzFd5eTvtUAAADENggyS2+AAAwGS0RHEHVosCAGAeRtgQx2pRAADMxAgb4lgtCgCAmQhsiGO1KAAAZqIlijhWiwIAYCYCG+7AalEAAMxDSxQAAMBwBDYkxRYfAACYgZYoEmKLDwAAzMEIGxJiiw8AAMxBYENCbPEBAIA5aIkiIbb4AADAHAQ2JMUWHwAAmIGWKPrFalEAADKLETb0KdFq0eXLM/1UAAA4CyNs6BOrRQEAyDwCG/rEalEAADKPlij6xGpRAAAyj8CGfrFaFACAzCKwYVBCIY/OnnWppMRDiAMAIE0IbBiwO1eM+jlfFACANGHRAQaMFaMAAGQGgQ0DxopRAAAyg5YoBqxnxejZsz6VlLTQDgUAIE0IbBiUQCCq5cu71NwcVSjkYbsPAADSgMCGIUl0ZBWhDQCA1GAOG4aEBQgAAKQPgQ1DwgIEAADSh5YohoQjqwAASB8CG4aMI6sAAEgPWqIYtlDIo927vQqFPJl+FAAAchIjbBgWVosCAJB6jLBhWFgtCgBA6hHYMCysFgUAIPVoiWJYWC0KAEDqEdgwbL1Xi3JcFQAAI4/AhhHDAgQAAFKDOWwYMSxAAAAgNQhsGDEsQAAAIDVoiWLEsAABAIDUYIQNIyoQiGrDhjZJ4vQDAABGCCNsGHEsPgAAYGQxwoYRx+IDAABGFoENI47FBwAAjCxaohhxdy8+kLrns7EQAQCAoSGwISV6Tj9gPhsAAMNHSxQpxXw2AACGj8CGlGI+GwAAw0dLFCnFZroAAAwfI2xIOTbTBQBgeBhhQ1qw+AAAgKFjhA1pweIDAACGjsCGtGDxAQAAQ0dLFGnBZroAAAwdgQ1pw2a6AAAMDS1RpB3z2QAAGBwCG9KO+WwAAAwOLVGkXe/5bD5fLD7CRlsUAIDECGzIiJ5wxlw2AAD6R0sUGcNcNgAABobAhoxhLhsAAANDSxQZw95sAAAMDIENGcXebAAA9I+WKIzAfDYAAJIjsMEIzGcDACA5WqIwAvPZAABIjsAGYzCfDQCAxIYV2L73ve/pS1/6klwul9xut7Zv3662tjZVVVXp888/1/jx47Vp0yZ5vV7Ztq0DBw7o9OnTys/PV2VlpaZPny5JOnr0qA4dOiRJWrVqlRYvXixJunjxovbu3auOjg6VlZVp3bp1sixreBXDeL3ns/V8TWADADjZsOew/ehHP9KOHTu0fft2SdKRI0c0Z84cVVdXa86cOTpy5Igk6fTp07p69aqqq6u1fv16vfrqq5KktrY2HTx4UNu2bdO2bdt08OBBtbW1SZJeeeUVPfPMM6qurtbVq1fV2Ng43MdFFug9n83ttvXHP7oUCnky/VgAAGTMiC86aGho0KJFiyRJixYtUkNDgyQpFApp4cKFsixLs2fP1o0bN9TS0qLGxkaVlJTI6/XK6/WqpKREjY2Namlp0a1btzR79mxZlqWFCxfG3wu5rWc+2ze+cVOSpZ//fIzWrPET2gAAjjXsOWxbt26VJH3ta19TRUWFrl27Jp/PJ0ny+Xy6fv26JCkSiai4uDj+6/x+vyKRiCKRiPx+f/x6UVFRwus9r0+ktrZWtbW1kqTt27ffcZ9UyMvLS/k9TJaO+pcvl86edSkWU7w1evasT8uXd6X0vgPh5M/fybVL1E/9zq3fybVLZtQ/rMD24osvqqioSNeuXdNPfvITTZ48Oelrbdu+51qy+WiWZSV8fTIVFRWqqKiIf93c3DzgXzsUxcXFKb+HydJVf0mJRx5Pd2j3eGyVlLSouTnzc9mc/Pk7uXaJ+qnfufU7uXYptfX3lZ16G1ZLtKioSJJUWFioefPm6cKFCyosLFRLS4skqaWlRePGjZPUPULWu9hwOCyfz6eioiKFw+H49UgkIp/PJ7/ff8f1cDgcvx+coac1+vzzraqp6f5e2L3bS2sUAOA4Qw5st2/f1q1bt+L/ffbsWd1///0KBAI6duyYJOnYsWOaN2+eJCkQCKi+vl62bev8+fMqKCiQz+dTaWmpzpw5o7a2NrW1tenMmTMqLS2Vz+fT6NGjdf78edm2rfr6egUCgREoGdkkEIhqw4buRShr1vi1Y8dY5rMBABxnyC3Ra9eu6ac//akkKRaL6atf/apKS0s1Y8YMVVVVqa6uTsXFxdq8ebMkqaysTKdOndLGjRs1atQoVVZWSpK8Xq8ef/xxbdmyRZK0evVqeb1eSdLTTz+tffv2qaOjQ6WlpSorKxtWschebPUBAHAyyx7MZLEscfny5ZS+P7389NffezNdt9vWmjU3tXr1rYyENid//k6uXaJ+6ndu/U6uXcqBOWxAurDVBwDAyQhsyBqBQFRTpsTiW31Eo5aCwfxMPxYAAClHYENW6X0Kgsdjy+eLsXIUAJDzOPwdWaWnNRoM5svni+lHPyrkkHgAQM5jhA1Zp2erj5YWd3zlKO1RAEAuI7Aha3FIPADAKQhsyFqsHAUAOAWBDVmNlaMAACcgsCHrsXIIy9A4AAAP2UlEQVQUAJDrWCWKrMfKUQBArmOEDTkh0crRjg5LL700lpE2AEDWI7Ahp/S0R10uW11d0m9+k89CBABA1iOwIaf0tEcffrhdLpfU1cVCBABA9iOwIecEAlFt3tyqUaNYiAAAyA0sOkBOYiECACCXMMKGnMVCBABAriCwIeexEAEAkO0IbMh5LEQAAGQ7AhscgYUIAIBsxqIDOAYLEQAA2YoRNjhKooUI0ailgwcLGG0DABiLETY4UvdCBK8kye22VVNToFhMjLYBAIzECBscqac9+vzzrVqz5qZiMbHtBwDAWAQ2OFZPe3T16lts+wEAMBqBDY6XbNsP5rUBAEzBHDZAf97247e/HaVolHltAACzMMIG/Anz2gAApiKwAb0wrw0AYCICG5BAX/Pa/vM/XQQ3AEBaMYcNSKLveW1+5rUBANKGETagD8xrAwCYgBE2oB+BQFSBQFShkEe//GWBbFvxeW2//e0o/fu/X1NLi1vl5e2MuAEAUoIRNmCAekbblizpis9r6+iw9MMf3qcdO8ayKAEAkDIENmAQAoGo/vVfuzRqlC23+8+rSGmTAgBSiZYoMEjz59uqqQkrGMyXzxfTj35UqI4O2qQAgNQhsAFD0DOvTZL+6q869dJLY/Wb3+T/qU0q/fCH98m2OSUBADAyaIkCw9Sz/UeiNilnkgIARgIjbMAI6FmQ0LtNevfebW63V2vW3NTq1bcYcQMADAqBDRghd7dJg8F8/fGPLv3852MUi1mKxaQ33xyjX/6ygDYpAGBQCGxACty9d1tXl2Tbkm1bikalgwcLFAzGWJQAABgQAhuQQj2t0oMHC/7UGrVpkwIABo3ABqRY92jbNa1efbPPNilbgQAAkiGwAWnSV5u091YgjLoBAO7Gth5AmvW0Sf/+729q1CglPDHhzTfHaM0av958czRbggAAGGEDMuHuNmnPViDt7YlH3diAFwCcjcAGZNDdW4H0XpxgWba6uix1dVmybemll8Zq8+ZWQhsAOBCBDTBEslG3ZOeU+nwxFikAgEMQ2ADDDOSc0q6u7hDnckmjRnlZYQoAOY7ABhis55zS3/52lKJR3dEmleyEh80T3gAg9xDYAMMlOqe0p03qcvWsMP3zXDe2BwGA3ENgA7JAonNKe+aw9T5svvcIHJvyAkDuILABWaZ3eOvRO8T1tT1I71E3SQoG8wlxAJAFCGxADhjo9iA9o241NQWSLM4zBYAsQWADcsxANuWN/imX2Xbi1ilbhgCAWQhsQI7qa9TN7bYlWYpG7Xtap31tGbJ8eWZrAgCnIrABDnD3qFt5ebskJW2dJtoyxO326h/+wdZjj3Wfa8r8NwBIHwIb4CB3L1jo62SFu7cMicWkV1+19NprfvXMf+u97xttVABIHQIb4HAD2TIk2fy3nn3fkrVRCXEAMDIIbADikm0Zkmj+W39t1IGEOInWKgAMBIENQJ96z387e9ankpIWSRpQG7WvEJeX51WirUV63psQBwB/RmADMCCBQFTLl3epuTka/1pK3kbtL8Ql2lqk9/5wyebHSbrjfgQ7AE5AYAMwLP2dvJAsxOXl3bu1SH/z43pG5To7abcCcBYCG4ARN5AQl2hrkf7mx/UOdMNptyYLdr3/m5AHwCQENgBpkSjEdV+/d3+4ZPPjekblOjvtIbdbe4/O3R3s+gt5HR0ulZR47rhGWxZAOhDYAGTcvfvDJZ4fl2gO21DarYmCXVfXn/+775Dnv+faQNqydz93f9cIgQB6I7ABMFbyUbmht1t7j871BLtkx3X1F/IG05ZNNrrX14jfQFq5BETAGQhsALLeUNqtyULOQEPeYNqyvUfv+gp+g23l9tfeHemAePZs8pZwqq6l+r0JqcgWlm3bdqYfYqRdvnw5pe9fXFys5ubmlN7DZNTv3PqdUnso5EkYFjo6xt6zD92dbVmrVxjqGb1Tr+Cne/5/omvRaPfJEpIly+r+K7on2A3k2lB+jWTLsiSPJ/lzDaWW4VxL5f163jvR/n93f75/Dqw+jRrVamT4TPX97v7ez+ZaBnu/8vJ2LV9emLK/+yZPnjyg1xHYhsApP7SSoX7n1u/k2qW+6+8JeSPxw+bPo3yDDyypDohDDYPpDp9DCanpCIi5FHZzoZb+7te9J6Stt9+OaebMzxP90R+2gQY2WqIAMAIGMt9uIPPxuq8NvJU7sgFx4HP9+msTj8S1VN4v2VzFobSvB3otle+d7vvlUi3930+qr7c0c6YyihG2IWCUgfqdWr+Ta5dyu/5kbeC7W4KJWsLZ2ApLNIrJqFTm39u0+5k0wkZgG4Jc/kt7IKjfufU7uXaJ+nOt/kQhlTlszGFjDlsaEdhSi/qdW7+Ta5eon/qdW7+Ta5dSW3/OzGFrbGzUgQMH1NXVpaVLl2rlypWZfiQAAIC0cmX6AfrS1dWl/fv364UXXlBVVZWOHz+uS5cuZfqxAAAA0srowHbhwgVNmjRJEydOVF5enhYsWKCGhoZMPxYAAEBaGd0SjUQi8vv98a/9fr8+/PDDe15XW1ur2tpaSdL27dtVXFyc0ufKy8tL+T1MRv3Ord/JtUvUT/3Ord/JtUtm1G90YEu0HsKyrHuuVVRUqKKiIv51qidGMvmS+p1av5Nrl6if+p1bv5Nrl8xYdGB0S9Tv9yscDse/DofD8vl8GXwiAACA9DM6sM2YMUNXrlxRU1OTOjs7deLECQUCgUw/FgAAQFoZ3RJ1u9166qmntHXrVnV1demRRx7R1KlTM/1YAAAAaWV0YJOkuXPnau7cuZl+DAAAgIwxuiUKAAAAAhsAAIDxCGwAAACGI7ABAAAYjsAGAABgOAIbAACA4QhsAAAAhrPsRAd2AgAAwBiMsA3BD37wg0w/QkZRv3Prd3LtEvVTv3Prd3Ltkhn1E9gAAAAMR2ADAAAwnPvHP/7xjzP9ENlo+vTpmX6EjKJ+59bv5Nol6qd+59bv5NqlzNfPogMAAADD0RIFAAAwHIENAADAcHmZfoBs09jYqAMHDqirq0tLly7VypUrM/1IKdPc3Ky9e/fqiy++kGVZqqio0KOPPqq33npL77zzjsaNGydJWrt2rebOnZvhp02N733ve/rSl74kl8slt9ut7du3q62tTVVVVfr88881fvx4bdq0SV6vN9OPOuIuX76sqqqq+NdNTU164okndOPGjZz9/Pft26dTp06psLBQO3fulKSkn7dt2zpw4IBOnz6t/Px8VVZWZnyOy3Alqv+NN97Qe++9p7y8PE2cOFGVlZUaM2aMmpqatGnTJk2ePFmSNGvWLK1fvz6Tjz8siWrv6++6w4cPq66uTi6XS+vWrVNpaWnGnn0kJKq/qqpKly9fliTdvHlTBQUF2rFjR8599sl+1hn3Z9/GgMViMfv73/++ffXqVTsajdrPPfec/emnn2b6sVImEonYH330kW3btn3z5k1748aN9qeffmrX1NTYv/rVrzL8dOlRWVlpX7t27Y5rb7zxhn348GHbtm378OHD9htvvJGJR0urWCxmP/3003ZTU1NOf/7nzp2zP/roI3vz5s3xa8k+7/fee8/eunWr3dXVZX/wwQf2li1bMvLMIylR/Y2NjXZnZ6dt292/Fz31f/bZZ3e8Ltslqj3Z9/qnn35qP/fcc3ZHR4f92Wef2d///vftWCyWzscdcYnq7+21116zf/nLX9q2nXuffbKfdab92aclOggXLlzQpEmTNHHiROXl5WnBggVqaGjI9GOljM/ni/+rYfTo0ZoyZYoikUiGnyrzGhoatGjRIknSokWLcvp7oMfvfvc7TZo0SePHj8/0o6TUAw88cM9oabLPOxQKaeHChbIsS7Nnz9aNGzfU0tKS9mceSYnq/5u/+Ru53W5J0uzZs3P274BEtSfT0NCgBQsWyOPxaMKECZo0aZIuXLiQ4idMrb7qt21bwWBQX/nKV9L8VOmR7GedaX/2aYkOQiQSkd/vj3/t9/v14YcfZvCJ0qepqUkff/yxZs6cqT/84Q96++23VV9fr+nTp+vb3/52TrYEe2zdulWS9LWvfU0VFRW6du2afD6fpO4/6NevX8/k46XF8ePH7/jL2kmff7LPOxKJqLi4OP46v9+vSCQSf20uqqur04IFC+JfNzU16Z/+6Z80evRoff3rX9df//VfZ/DpUiPR93okEtGsWbPirykqKsrZICtJ//d//6fCwkL9xV/8Rfxarn72vX/WmfZnn8A2CHaCHVAsy8rAk6TX7du3tXPnTj355JMqKCjQsmXLtHr1aklSTU2NXn/9dVVWVmb4KVPjxRdfVFFRka5du6af/OQn8TkbTtLZ2an33ntP3/jGNyTJUZ9/X5z298GhQ4fkdrv18MMPS+r+AbZv3z6NHTtWFy9e1I4dO7Rz504VFBRk+ElHTrLv9USffS67+x9sufrZ3/2zLplM/dmnJToIfr9f4XA4/nU4HM7pf01L3T+sd+7cqYcfflgPPfSQJOm+++6Ty+WSy+XS0qVL9dFHH2X4KVOnqKhIklRYWKh58+bpwoULKiwsjA9/t7S0xCck56rTp09r2rRpuu+++yQ56/OXlPTz9vv9am5ujr8ul/8+OHr0qN577z1t3Lgx/oPJ4/Fo7Nixkro3FJ04caKuXLmSycccccm+1+/+WRCJROJ/V+SaWCymd999946R1Vz87BP9rDPtzz6BbRBmzJihK1euqKmpSZ2dnTpx4oQCgUCmHytlbNvWyy+/rClTpuixxx6LX+/dq3/33Xc1derUTDxeyt2+fVu3bt2K//fZs2d1//33KxAI6NixY5KkY8eOad68eZl8zJS7+1/XTvn8eyT7vAOBgOrr62Xbts6fP6+CgoKcDGyNjY361a9+pX/+539Wfn5+/Pr169fV1dUlSfrss8905coVTZw4MVOPmRLJvtcDgYBOnDihaDSqpqYmXblyRTNnzszUY6bU7373O02ePPmO6UC59tkn+1ln2p99TjoYpFOnTum1115TV1eXHnnkEa1atSrTj5Qyf/jDH/Rv//Zvuv/+++P/ql67dq2OHz+uTz75RJZlafz48Vq/fn1O/qD67LPP9NOf/lRS978yv/rVr2rVqlVqbW1VVVWVmpubVVxcrM2bN+fsHK729nb94z/+o/bs2RNvEezevTtnP/9du3bp/fffV2trqwoLC/XEE09o3rx5CT9v27a1f/9+nTlzRqNGjVJlZaVmzJiR6RKGJVH9hw8fVmdnZ/x7vGcLh5MnT+qtt96S2+2Wy+XS3/3d32X1P2AT1X7u3Lmk3+uHDh3S//7v/8rlcunJJ59UWVlZhisYnkT1L1myRHv37tWsWbO0bNmy+Gtz7bNP9rNu1qxZRv3ZJ7ABAAAYjpYoAACA4QhsAAAAhiOwAQAAGI7ABgAAYDgCGwAAgOEIbAAAAIYjsAEAABju/wEFb67P9oyr6QAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 720x576 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"\n",
"fig,ax = plt.subplots(figsize=(10,8))\n",
"_=ax.plot(range(200),cost_history[:200],'b.')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<b>It is worth while to note that the cost drops faster initially and then the gain in cost reduction is not as much"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### It would be great to see the effect of different learning rates and iterations together"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Let us build a function which can show the effects together and also show how gradient decent actually is working"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T04:06:54.948007Z",
"start_time": "2018-08-25T04:06:54.916435Z"
},
"scrolled": false
},
"outputs": [],
"source": [
"\n",
"def plot_GD(n_iter,lr,ax,ax1=None):\n",
" \"\"\"\n",
" n_iter = no of iterations\n",
" lr = Learning Rate\n",
" ax = Axis to plot the Gradient Descent\n",
" ax1 = Axis to plot cost_history vs Iterations plot\n",
"\n",
" \"\"\"\n",
" _ = ax.plot(X,y,'b.')\n",
" theta = np.random.randn(2,1)\n",
"\n",
" tr =0.1\n",
" cost_history = np.zeros(n_iter)\n",
" for i in range(n_iter):\n",
" pred_prev = X_b.dot(theta)\n",
" theta,h,_ = gradient_descent(X_b,y,theta,lr,1)\n",
" pred = X_b.dot(theta)\n",
"\n",
" cost_history[i] = h[0]\n",
"\n",
" if ((i % 25 == 0) ):\n",
" _ = ax.plot(X,pred,'r-',alpha=tr)\n",
" if tr < 0.8:\n",
" tr = tr+0.2\n",
" if not ax1== None:\n",
" _ = ax1.plot(range(n_iter),cost_history,'b.') "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot the graphs for different iterations and learning rates combination"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2018-08-25T04:07:09.977910Z",
"start_time": "2018-08-25T04:07:00.484794Z"
}
},
"outputs": [
{
"data": {
@JayRenData
Copy link

I've output 'theta_history', 'cost_history' and 'x', 'y', but can not get the same 'cost' as the output by the program above

@betidav
Copy link

betidav commented Dec 7, 2018

@JayRenData
I think the reason you can't get it is because of the integer divisions in the code.
if you replace m = len(y) by float(len(y)) in line 8
and do the same in line 9 the everything should work fine.

@Rouxkein
Copy link

I need your help: f(x)=x*x+2x+5
Generate some data with: theta0=5,theta1=2,theta3=1
i don't know how to make it !

@sagarmainkar
Copy link
Author

I need your help: f(x)=x*x+2x+5
Generate some data with: theta0=5,theta1=2,theta3=1
i don't know how to make it !

It is quite simple refer to this in above code
X = 2 * np.random.rand(100,1)
y = 4 +3 * X+np.random.randn(100,1)

So you can do
X = can be any number randomly generated ( np.random.rand(100,1))
Y = 5+ XX + 2X

There is no need for theta3

@bhargava1996
Copy link

The result is not matching with the initial values in the mini batch gradient descent.
You gave 4,3 but the values coming out are 3.6,3 for you. So i think this is the best local gradient descent you arrived.

But I arrived at 4.0 and 2.9 with your code(psted and tested) and my code as well.

Despite these things I learnt many things here.
Thank you very much.

@IvonneEO
Copy link

IvonneEO commented Mar 2, 2021

I need your help: f(x,y) = xx + yy

@sivi299
Copy link

sivi299 commented Aug 3, 2021

Hi,
There seems to be a flaw in the cost function

cost = (1/2*m) * np.sum(np.square(predictions-y))

Shouldn't it be

cost = 1/(2*m) * np.sum(np.square(predictions-y))

Nice walkthrough

@kenwyee
Copy link

kenwyee commented Aug 4, 2021 •
edited
Loading

I was just about to make the same observation as sivi299 regarding the cost function.
In this case, since m is fixed from iteration to iteration when doing the gradient descent, I don't think it matters when it comes to optimizing the theta variable. As written, it's proportional to the mean-squared error, but it should optimize towards the same theta all the same.

The relative magnitudes of the cost function history curves differ between the gradient_descent and minibatch_gradient_descent due to different batch sizes when the cal_cost function is called, but since each algorithm uses the same number of points from iteration to iteration internally it should be OK.

@paocarvajal1912
Copy link

Hi, this is fantastic material; thanks so much.
I think there is a typo in equation (8). Shouldn't the X's subindex be j? Meaning X_j instead of X_0?
Regards,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment