{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Beta calibration\n", "\n", "## Introduction\n", "\n", "Logistic calibration is designed to correct for a specific kind of distortion where classifiers tend to score on too narrow a scale. However, many classifiers including Naive Bayes and standard Adaboost suffer from the opposite distortion where scores tend too much to the extremes. \n", "\n", "In this tutorial, we will motivate Beta calibration and our betacal Python package.\n", "\n", "## Probability estimation with Adaboost\n", "\n", "First, let's train an Adaboost model with 200 decision stumps to estimate class probabilities for the well-known spam dataset. The dataset will be divided into a training set (50%), a test set (25%) and a calibration set (25%). The classifier will be trained on the training set and we'll estimate class probabilities for the test set.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEhlJREFUeJzt3X+s3Xddx/HnqytFQdZMcbfaDqhOxmbUjUjR4I+jzEkx\ntgt/zIrBjcVoMhWMibqamLWJSdV//BEzEyJiNWAtEmxFzMqcV4NhdtMBk5Za0NZypXfCBIJEbdnb\nP8537Ky0u9977jn33PXzfCTf9PP9nM/3nPf99t7X+dzPOd9zU1VIki5/62ZdgCRpdRj4ktQIA1+S\nGmHgS1IjDHxJaoSBL0mNWDLwk7w0ySNJ/qn797NJ3pTkqiRHkpxIcl+SjSPH7E5yMsnxJLdM90uQ\nJPWR5bwPP8k64BPAK4GfAT5dVb+R5JeAq6rq7iQ3AG8HXgFsAe4Hvql8w78kzdRyl3RuBj5eVWeA\nncD+rn8/cGvX3gEcqKrzVXUKOAlsm0CtkqQVWG7g/wjwjq49V1WLAFV1Fri6698MnBk5ZqHrkyTN\nUO/AT/IchrP3d3ZdFy7RuGQjSWvY+mWM3Q78Y1V9qttfTDJXVYtJNgGPdf0LwDUjx23p+p4miU8Q\nkjSGqso4xy1nSedHgT8Z2T8M3NG1bwcOjfTvSrIhyVbgWuDoxe6wqtyquOeee2Zew1rZPBeeC8/F\nM28r0WuGn+R5DF+w/cmR7l8HDia5EzgN3NaF+LEkB4FjwDngrlpplZKkFesV+FX1BeBrL+h7nOGT\nwMXG7wP2rbg6SdLEeKXtGjAYDGZdwprhuXiK5+IpnovJWNaFVxN94MSVHklapiTUKrxoK0l6FjPw\nJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+S\nGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUiF6Bn2RjkncmOZ7kI0lemeSqJEeSnEhy\nX5KNI+N3JznZjb9leuVLkvrqO8P/beC9VXU98G3AR4G7gfur6jrgAWA3QJIbgNuA64HtwL1JMunC\nJUnLs2TgJ7kS+O6qehtAVZ2vqs8CO4H93bD9wK1dewdwoBt3CjgJbJt04ZKk5VnfY8xW4FNJ3sZw\ndv8w8HPAXFUtAlTV2SRXd+M3Ax8YOX6h6/vyB1//3HHrnogrrljPgw++n5tuummmdUjSaugT+OuB\nlwM/XVUPJ/lNhss5dcG4C/eX9MUv/sLI3vcA37vcu1iR5z//dZw5c8bAl7Rmzc/PMz8/P5H76hP4\nnwDOVNXD3f67GAb+YpK5qlpMsgl4rLt9Abhm5PgtXd9F/Oo4NU/QFTN+fEl6ZoPBgMFg8KX9vXv3\njn1fS67hd8s2Z5K8tOt6NfAR4DBwR9d3O3Coax8GdiXZkGQrcC1wdOwKJUkT0WeGD/Am4O1JngP8\nK/BGhtPjg0nuBE4zfGcOVXUsyUHgGHAOuKuqlr3cI0marF6BX1UfAl5xkZtuvsT4fcC+FdQlSZow\nr7SVpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMM\nfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqRG9Aj/JqSQf\nSvJIkqNd31VJjiQ5keS+JBtHxu9OcjLJ8SS3TKt4SVJ/fWf4TwCDqrqpqrZ1fXcD91fVdcADwG6A\nJDcAtwHXA9uBe5NksmVLkparb+DnImN3Avu79n7g1q69AzhQVeer6hRwEtiGJGmm+gZ+Ae9L8lCS\nn+j65qpqEaCqzgJXd/2bgTMjxy50fZKkGVrfc9yrquqTSb4WOJLkBMMngVEX7vewZ6Q96DZJ0pPm\n5+eZn5+fyH31Cvyq+mT3738m+XOGSzSLSeaqajHJJuCxbvgCcM3I4Vu6vovYM17VktSIwWDAYDD4\n0v7evXvHvq8ll3SSPC/JV3Xt5wO3AI8Ch4E7umG3A4e69mFgV5INSbYC1wJHx65QkjQRfWb4c8C7\nk1Q3/u1VdSTJw8DBJHcCpxm+M4eqOpbkIHAMOAfcVVVjLPdIkiZpycCvqn8DbrxI/+PAzZc4Zh+w\nb8XVSZImxittJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJek\nRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWpE\n78BPsi7JPyU53O1fleRIkhNJ7kuycWTs7iQnkxxPcss0CpckLc9yZvhvBo6N7N8N3F9V1wEPALsB\nktwA3AZcD2wH7k2SyZQrSRpXr8BPsgV4LfD7I907gf1dez9wa9feARyoqvNVdQo4CWybSLWSpLH1\nneH/JvALQI30zVXVIkBVnQWu7vo3A2dGxi10fZKkGVq/1IAkPwQsVtUHkwyeYWg9w22XsGekPeg2\nSdKT5ufnmZ+fn8h9LRn4wKuAHUleC3wl8IIkfwycTTJXVYtJNgGPdeMXgGtGjt/S9V3EnjHLlqQ2\nDAYDBoPBl/b37t079n0tuaRTVb9cVS+qqm8AdgEPVNUbgL8A7uiG3Q4c6tqHgV1JNiTZClwLHB27\nQknSRPSZ4V/KrwEHk9wJnGb4zhyq6liSgwzf0XMOuKuqxljukSRN0rICv6r+Fvjbrv04cPMlxu0D\n9q24OknSxHilrSQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS\n1AgDX5IaYeBLUiMMfElqxEo+D1+SmrJp00tYXDw96zLGZuBLUk/DsJ/133PK2Ee6pCNJjTDwJakR\nBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiOWDPwkz03yD0keSfJoknu6/quSHElyIsl9STaOHLM7yckk\nx5PcMs0vQJLUz5KBX1X/C3xfVd0E3AhsT7INuBu4v6quAx4AdgMkuQG4Dbge2A7cm2T8KwUkSRPR\na0mnqr7QNZ/L8OrcAnYC+7v+/cCtXXsHcKCqzlfVKeAksG1SBUuSxtMr8JOsS/IIcBZ4X1U9BMxV\n1SJAVZ0Fru6GbwbOjBy+0PVJkmao12fpVNUTwE1JrgTeneSb+fIPlBjjAyb2jLQH3SZJesp8t63c\nsj48rao+l2QeeA2wmGSuqhaTbAIe64YtANeMHLal67uIPcssV5JaM+Dpk+G9Y99Tn3fpvPDJd+Ak\n+UrgB4DjwGHgjm7Y7cChrn0Y2JVkQ5KtwLXA0bErlCRNRJ8Z/tcB+5OsY/gE8adV9d4kDwIHk9wJ\nnGb4zhyq6liSg8Ax4BxwV1XN+vNEJal5SwZ+VT0KvPwi/Y8DN1/imH3AvhVXJ0maGK+0laRGGPiS\n1AgDX5IaYeBLUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mN\nMPAlqREGviQ1wsCXpEYY+JLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRSwZ+ki1JHkjykSSPJnlT\n139VkiNJTiS5L8nGkWN2JzmZ5HiSW6b5BUiS+ukzwz8P/HxVfTPwncBPJ3kZcDdwf1VdBzwA7AZI\ncgNwG3A9sB24N0mmUbwkqb8lA7+qzlbVB7v254HjwBZgJ7C/G7YfuLVr7wAOVNX5qjoFnAS2Tbhu\nSdIyLWsNP8lLgBuBB4G5qlqE4ZMCcHU3bDNwZuSwha5PkjRD6/sOTPJVwJ8Bb66qzyepC4ZcuN/D\nnpH2oNskSU+Z77aV6xX4SdYzDPs/rqpDXfdikrmqWkyyCXis618Arhk5fEvXdxF7xihZkloy4OmT\n4b1j31PfJZ0/AI5V1W+P9B0G7ujatwOHRvp3JdmQZCtwLXB07AolSROx5Aw/yauAHwMeTfIIw6Wb\nXwZ+HTiY5E7gNMN35lBVx5IcBI4B54C7qmqM5R5J0iQtGfhV9ffAFZe4+eZLHLMP2LeCuiRJE+aV\ntpLUCANfkhph4EtSIwx8SWqEgS9JjTDwJakRBr4kNcLAl6RGGPiS1AgDX5IaYeBLUiMMfElqhIEv\nSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAlqREGviQ1YsnAT/LWJItJ\nPjzSd1WSI0lOJLkvycaR23YnOZnkeJJbplW4JGl5+szw3wb84AV9dwP3V9V1wAPAboAkNwC3AdcD\n24F7k2Ry5UqSxrVk4FfV+4H/uqB7J7C/a+8Hbu3aO4ADVXW+qk4BJ4FtkylVkrQS467hX11ViwBV\ndRa4uuvfDJwZGbfQ9UmSZmz9hO6nxjtsz0h70G2SpKfMd9vKjRv4i0nmqmoxySbgsa5/AbhmZNyW\nru8S9oz58JLUigFPnwzvHfue+i7ppNuedBi4o2vfDhwa6d+VZEOSrcC1wNGxq5MkTcySM/wk72D4\n9PI1Sf4duAf4NeCdSe4ETjN8Zw5VdSzJQeAYcA64q6rGXO6RJE3SkoFfVa+/xE03X2L8PmDfSoqS\nJE2eV9pKUiMMfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGmHgS1IjDHxJaoSBL0mNMPAl\nqREGviQ1wsCXpEZM6k8cStLUbNr0EhYXT8+6jGe9zOrvkySpsf8U7oRceeUO1q17iM985uxM65ib\nezFnz56aaQ3SWpaEWefF0FqoI1RVlh735Zqf4Q/Dfrb/gYuLX9F9Q8+WTzzS5a35wF8b/pdZP+kA\nLC7O/klHa4/LKZcPA1/SMxqG/awnJE5GJsF36UhSI5zha8RzZ/5agq8jPMWlFE2aga8Rs38twdcR\nnrI2llLA5ZTLx9SWdJK8JslHk/xLkl+a1uNIkvqZygw/yTrgd4FXA/8BPJTkUFV9dBqPp8vJc2a+\nrASwbt3zeOKJL8y6DGmiprWksw04WVWnAZIcAHYCBr6WcI61sIzxxBNr4wIbaZKmtaSzGTgzsv+J\nrk+SNCMzfdH2yit/eJYPz//939GZPr4kraZpBf4C8KKR/S1d39N87nPvmdLDL9da+NV5LdQAa6OO\ntVADrI061kINsDbqWAs1wNqpY/mm8uFpSa4ATjB80faTwFHgR6vq+MQfTJLUy1Rm+FX1xSQ/Axxh\n+DrBWw17SZqtmX08siRpdU39s3T6XICV5HeSnEzywSQ3TrumWVnqXCR5fZIPddv7k3zLLOpcDX0v\nzEvyiiTnkrxuNetbTT1/RgZJHknyz0n+ZrVrXC09fkauTHK4y4pHk9wxgzKnLslbkywm+fAzjFl+\nblbV1DaGTygfA14MPAf4IPCyC8ZsB/6ya78SeHCaNc1q63kuvgPY2LVf0/K5GBn318B7gNfNuu4Z\nfl9sBD4CbO72Xzjrumd4LnYD+548D8CngfWzrn0K5+K7gBuBD1/i9rFyc9oz/C9dgFVV54AnL8Aa\ntRP4I4Cq+gdgY5K5Kdc1C0uei6p6sKo+2+0+yOV77UKf7wuAnwX+DHhsNYtbZX3OxeuBd1XVAkBV\nfWqVa1wtfc5FAS/o2i8APl1V51exxlVRVe8H/usZhoyVm9MO/D4XYF04ZuEiYy4Hy70Y7SeAv5pq\nRbOz5LlI8vXArVX1ezyb3we3tD7fFy8FvjrJ3yR5KMkbVq261dXnXPwucEOS/wA+BLx5lWpba8bK\nTT8tcw1K8n3AGxn+Wteq3wJG13Av59Bfynrg5cD3A88HPpDkA1X1sdmWNRM/CDxSVd+f5BuB9yX5\n1qr6/KwLezaYduD3uQBrAbhmiTGXg14XoyX5VuAtwGuq6pl+pXs263Muvh04kOEnqb0Q2J7kXFUd\nXqUaV0ufc/EJ4FNV9T/A/yT5O+DbGK53X076nIs3AvsAqurjSf4NeBnw8KpUuHaMlZvTXtJ5CLg2\nyYuTbAB2ARf+wB4GfhwgyXcAn6mqxSnXNQtLnoskLwLeBbyhqj4+gxpXy5Lnoqq+odu2MlzHv+sy\nDHvo9zNyCPiuJFckeR7DF+kux+ta+pyL08DNAN2a9UuBf13VKldPuPRvtmPl5lRn+HWJC7CS/NTw\n5npLVb03yWuTfAz4b4bP4JedPucC+BXgq4F7u5ntuaraNruqp6PnuXjaIate5Crp+TPy0ST3AR8G\nvgi8paqOzbDsqej5ffGrwB+OvF3xF6vq8RmVPDVJ3gEMgK9J8u/APcAGVpibXnglSY3wj5hLUiMM\nfElqhIEvSY0w8CWpEQa+JDXCwJekRhj4ktQIA1+SGvH/HLG4BAE48ZwAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.datasets import load_iris\n", "from sklearn.naive_bayes import GaussianNB\n", "from sklearn.model_selection import train_test_split\n", "from adaboost import AdaBoostClassifier\n", "\n", "np.random.seed(42)\n", "\n", "data = np.genfromtxt('spambase.data', delimiter=',')\n", "target = data[:,-1]\n", "data = data[:,0:-1]\n", "\n", "x_train, x_test, y_train, y_test = train_test_split(data, target, test_size=0.5, stratify=target)\n", "x_cal, x_test, y_cal, y_test = train_test_split(x_test, y_test, test_size=0.5, stratify=y_test)\n", "\n", "ada = AdaBoostClassifier(n_estimators=200)\n", "ada = ada.fit(x_train, y_train)\n", "probas = ada.predict_proba(x_test)\n", "\n", "plt.clf()\n", "plt.hist(probas[:,1])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calibrating the scores\n", "\n", "We can clearly see from the histogram that the probabilities produced by the model tend to assume extreme values. Therefore, it might be useful to apply calibration techniques to try and fix these distortions. Two calibration methods have been widely used in machine learning literature: logistic calibration and isotonic regression. The first one is a parametric method that assumes an underlying distribution of the scores composed of two Gaussians of equal variance, (one for the positives and another for the negatives). The second method is a non-parametric approach, therefore it doesn't make any assumption about the distribution of the scores, however, it needs lots of data to produce a good model. Let's see the effect of applying these methods to the previously trained classifier's outputs." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEQCAYAAACwSgOGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXZ+PHvE1B2sgFCQCEE96KEQMUNAwFcW5UY8DUp\nuEDi8rN9ld02FZpWRFxebatCwLqEVnaVQpVAjIVCWINKqyyZASQBsZBF9iXP749nskGWSWZyzszk\n/lzXXJk558w59xySuTnnWW6ltUYIIYQIsjsAIYQQvkESghBCCEASghBCCBdJCEIIIQCbEoJSKrqW\ndfFKqTil1AQrYxJCiKbO8oSglIoDFtawLhrQWuvVQJFSqo+lwQkhRBNmeUJwfdnn1bB6JFDkeu4A\nhlgSlBBCCJ9rQwgBjlR6HW5XIEII0dQ0tzuA+lJKyUg6IYSoJ621qmsbX7tCKATCXM9DgMPVbaS1\nRmvN888/X/68KT/kPMg5kPMg56G2c+Auu64QqmQqpVSw1roYWADEAFlATyCzujdPnToVgOzsbLKz\ns4mNjW3MWIUQwq+UfTeWfVe6y/KEoJSKB2KUUsO11ktci1cB/bXWuUqpGFdPpEKt9bbq9lH2IadO\nnSrJQAghzhMbG0tsbGz5d+W0adPcep/lCUFrvRhYfN6y/pWez3F3X5IMDDkPcg7KyHkw5Dw07Byo\n+txf8gVKKe1vMQshhLucTiepqank5+fTtWtX0tLSiIyM9GifSim0G43KAZMQevTowd69e22IqOnp\n3r07e/bssTsMIQKO0+lk6NCh5OVVDNWKiooiMzPTo6TQ5BKC6wPbEFHTI+daiMaRlJTEvHnzLlie\nmJhIRkZGre/VWqNU9d/57iYEX+t2KoQQTVZ+fn61ywsKCup879P/eJql3yz16PiSEIQQwkd07dq1\n2uURERG1vq/gxwLmffVXLj54K7Nnw69+BUOGwOFqR3LVTG4Z+bnc3FySk5PZtGmTW8vrs9+EhAR2\n7959wbqmeq6FaGzutCGUloLDAbm5sG0bfPUVfNFiPD8eOwuf/l+V/X3xBQwc6P4tI7+bukJU1bNn\nT2bMmOH28vqo6X6kEKJxREZGkpmZSWpqKgUFBXTpEsHYsdPZuPFS3nwTNm+GrVuhpKTSm1odgV++\nQ4t3vuTavvCTn8C115rHddfVMwC7h1fX92FCvlBNyx0Oh05MTNSxsbE6MTFROxyOarerjaf7yMzM\n1FFRUTosLEyPGDFCFxUVle83JiZGp6Sk6NDQUD1s2DC9detWHRUVpYOCgvTMmTO11lovWrRIp6Sk\n6ISEBK2U0v369dNOp1NrrfXWrVt1TExM+f6GDh2qJ02apK+++mrdr1+/8hgWLlxYHkNKSkr58lmz\nZunQ0FAdFBR0wX579epV7eep6VwLITxTUqJ1ZqbW06ZpfdddWnfooDVc+OjSRes779T6uee0TvjT\nND38vUf02bM179f1N1v396s7G/nSoz4JweFw6KioKA2UP6Kiour1he7pPoqKinRoaKjOysrSxcXF\n+vHHH9cJCQnl+1ZK6SVLluji4uLyL+ySkhK9atUq7bo9phctWqSVUnrOnDm6uLhYp6SklCeBrVu3\nln/xl+3viSee0PPmzStfnpeXp0NDQ/W2bdu00+nUvXr10osXL9Zaax0UFKS3bdumi4uLdUJCgp48\neXL5fiUhCNG4vv9e60WLtH76aa379NE6KOjCL/9OnbS++26tp07V+u9/1/rAgYr3Hz11VHd8qaP+\n9odvaz2OuwkhoG8ZpaamVrkXB5CXl0dqamqdXbi8tY8FCxYwdOhQBg0aBMD06dPp2bNn+frQ0FDu\nv/9+AIYMGUJoaCjt2rUjLi4OpRQlrmvDmJgYHnvsMQDefvttmjVrVr6uMqUUb775Jrm5ueXLFi9e\nTEpKCtdffz0ACxdW1CcqLCykffv2AISFhVFUVIQQonEcPgzZ2fD555CVBd98U3V98+YQEwM33QQD\nBphH9+5Q093b9K3pDOw+kCs7XOmV+AI6IXjShctb+8jLy2PhwoWsWrUKMFdklb/Iw8LCyp+HhIQQ\nHl59CYjKSQTMvUaHw1HndmUx9OvXr/x1nz4Vhej+8Ic/sHr16vLjVvd+IUTDnD4N69bBypXmseXS\n0dDV1dFjMKg4aN264tGqFRxVsBJYeRhYXvv+95fsJ2t0ltfiDeiE0NAuXN7cR1RUFAkJCcyfP798\nWUNG+Z5/leJwOOjZs+cFy6sTEhJSpbdQbm5ueTLJysri888/p127dqSnp7N169Z6xyaEqFBQACtW\nmEdmJhw9WmnlkE/o+80/uLlfe264wTQAX3RRw4/V9uK2XBZ8mccxlwnohJCWlkZOTs4FXbjS0tIs\n28eIESOYPHkyq1evJiYmhkmTJrFnzx4+++wzgFq7b1Zet3XrVubMmUNCQgITJ06kX79+5bd6anpP\nmZSUFPr168fIkSMJDg4mISGByZMno7UmLCyMdu3aUVRUxKxZs4iKinLrcwkhDK1N189PPjGPzZur\nrr/2Wrj9drh5cAmjvjzD5o9u8NkefAGdEM7vwhUREVHviaI83UdwcDALFy4kOTm5vI9x5Xv4tf1i\nVF43dOhQMjMzSU5Opl+/fixYsKDO91T+DDNmzCAuLo7i4mJSUlIYM2YMYNoTwsLCiIqK4qWXXiIh\nIYGsrCxCQ0Pd+nxCNEWlpbBxIyxeDEuWmHEBZVq1MoPC7r4b7rwTLnP9B/7r7/fSfW93n00GIAPT\n/MLixYtZsGBBldtOdgrkcy1ETbQ2YwA+/BDmz4fvvqtY16kT/PzncO+9EBdnksL5lu1Yxlub32JF\n4grrgnaRgWlCCOEFeXkwbx5kZMCuXRXLu3WD+HgYPhxuvhmaNat9P3uL99I9uHvjBushSQhCCHGe\nkhJYsAD+8hfTS6jMJZdAQgI8+CDceCME1WM2uL1Fe+kR0sPrsXqTJAQ/EB8fT3x8vN1hCBHQtIa1\nayE9HRYtghMnzPI2bcxVQFISDB5sxgo0xN7ivcRExHgv4EYgCUEI0aQdOQLvvw+zZ1cdKHbbbfDo\no+a2UJs2nh9HbhkJIYSP+vJL+NOfTPtA2dVA584mCTz6KHi7B/aeoj10D5GEIIQQPuHcOTNW4LXX\nYM2aiuVDhsDjj5ueQp4MFKvJiTMnKD5ZTOe2nb2/cy+ShCCECHjHjsE778D//V/FmIF27eCRR+DJ\nJ+FK70wFVKN9xfu4NPhSgpRv1ySThCCECFiHD5vbQm+8YdoKACIj4X//1ySDdu2sicMf2g9ASmg2\nutzcXHr16uXxPvr371/ndkGuPnDubi9EoDp4EMaNM6OEp041yeCGG8zI4l274Je/tC4ZgKv9wA8S\nglwhWMDToeruVj8rO443qqUJ4Y8KCmDGDNNj6ORJs+yOO2Dy5LJSkvbE5Q9jEECuECy3aNEievXq\nRbNmzRg5cmSVqbBnz55NWFgYl19+Oenp6eVXFg6Hg4kTJ5Zvl5KSQlhYGOHh4cycOROAYcOGobUm\nPDz8gu3LjhkeHs7jjz9u0ScVwjrffw/PPAM9e5rbQydPwv33w5Yt8I9/mC6kdk4htLd4r8/3MAIC\nu2JaxTrvPeqrcuWxvLw8rZQqr56WkJBQXs6yclWz4uJiHRMTU/6+ylXRFi5cqHv16qVLSkp0bm6u\nDgoK0sXFxVprU/3s/O1rq5bWULWdayGsdOSI1lOmaN26dcXf6AMPaP3ll3ZHVtUt79yis53Zth0f\nNyumyRWChcoqlw0aNIj27dszY8aM8plPZ8+eXV7VrH379kyZMqXafZTdFtq9ezd9+vSpUvGstmNe\nf/319OjRg4ULF0oRHOH3Tp6EV14xYwWmT4fjx+GeeyA3FxYubEBx+Ua2t8g/rhCaRELw5jWCJw4f\nPlyl3kBkZGR5yUqHw1FlXU1f2vHx8UyaNImEhATCw8OZNWtWrcfMy8urst8+ffpUqZgmhD8pLTUD\nya68EsaPh8JCiI2F9eth2TLwxV/tM+fOcPDoQbq2q77Yli9pEgnBV4SHh1epXFZUVERISAhgEkDl\ndTVVQnM6ncTFxbF79262bNnCrFmzyMoyJfR0NRmrumppixcv9srnEcJK69ebWsNJSbBvn6k2tny5\nqU08YIDd0dVsf8l+urTrwkXNGmHEm5dJQrDQAw88QHp6OllZWRQVFZGcnMzIkSMBGDlyJOnp6eTm\n5lJUVMSLL75Y7T4WLVpEQkICTqeT0tJSAIqLi4HqezOlpKSU79fhcJCQkEBhYWEjfUIhvC8/Hx56\nyCSDDRvM9BJz58K2bXDXXd5tLHY6nSQlJTFo0CCSkpJwOp0e79NfxiAATaNR2U6VG5W11nrx4sU6\nKipKBwUF6ZEjR5Y3CGutdXp6ug4NDdW9evXS6enp5Q3DlRuJtdZ66NChOigoSIeFhekpU6aUL09I\nSNBBQUE6Nze3yvZl+w0KCtJPPPGEx5/JV8+1CCynTmk9Y4bWbdqYG7YtW2r9619rXVLSOMdzOBw6\nKipKA+WPqKgo7XA4PNrvu7nv6qQlSV6KsmFws1FZKqb5CKfTSVFREdHR0YBpDJ49e3Z57WVf4u/n\nWvi+zz+HJ56AHTvM6/vvh1dfhR49Gu+YSUlJzJs374LliYmJZGRkNHi/07Kncab0DL8f/HtPwvOI\nuxXT5JaRjygqKiqveQwwa9YsEhISbI5KCGv98AOMHm3qDuzYAVdcAZ9+auoWN2YyAMjPz692eUFB\ngUf73VvsH4PSQBKCz4iOjua5554jMjKS8PBwevXqxZgxY+wOSwhLaA3vvgtXXWVqE7RsCb//PXz9\nNdx+uzUxdO1afS+giIgIj/brT20IcstI1Juca+FNe/ZAcjJkZprXw4bBm296vx5BXZxOJ0OHDq3S\nwy8qKorMzEwiIyMbvN9eb/RiReIKrgi/whthNoi7t4wsTwhKqXigCOirtZ5Zy/pIrfWcatZLQrCZ\nnGvhDaWl8Oc/w5QpZnrq8HAzPXVion3TTDidTlJTUykoKCAiIoK0tDSPkkGpLqXVH1pRPLmYls1b\nejHS+vHJhKCUisZ80S9RSo0FNmmtt523Hq11rlIqDjhceb1rG0kINpNzLTzldJrpp7/4wrx+8EF4\n/XXo1MneuM73yrpX2FSwqcHvP33uNOu+W8fB8Qe9GFX9uZsQrJ7tdCSw0vXcAQwBtp23zQxgGNBT\na73awtiEEI1MazMT6bhx5qqgUyeYNQvuu8/uyC70wZcfMGvLLKbFTvNoxuJxN47zYlSNy+qEEAIc\nqfQ6vPJK15WBQyl1BJAWVSECyPffm1rFK1aY1yNGmFtGHTrYG1d1vjz4Jc+ufJbPR3/OTzr9xO5w\nLONTvYyUUsFAIfACkK6U6mFrQF7gjWI1ZYVv7IxBCE8sWwa9e5tkEBoKH34I8+f7ZjIoOllE/IJ4\nXr/j9SaVDMD6K4RCIMz1PAQ4fN76ZGC61rpEKeUAHgBePn8nU6dOLX8eGxtLbGxsY8TqFd4oVmNV\ngR0hvO3ECXN76K23zOu4ONO9tFs3W8Oq0bnSc4xaOoo7e93JQ70fsjucBsvOziY7O7v+b3RnOLO3\nHkA0MMb1fALQx/U82PVzfNlz1+sx1eyjtqHZPmfr1q06Jiam/HVycrIODQ3VYWFh+qWXXipfvnDh\nwvIpLUaMGFE+pcXQoUO1UkqHhYXVup3D4dAxMTH6pZdeKp/+Ijc394IYyt4fFhZWXouhvnz1XAvf\n8p//aN27t5l24uKLtX7lFa3PnbM7qpqdPXdWj146Wg9+b7A+dfaU3eF4FW5OXWHHXERjgLjKX/aY\n3kZlzycAw6tLBtpPE0JdxW1qK5yjdUXhm9q2czgcWimlX375Za211ikpKXrYsGFVYnA4HF4pluOr\n51r4htJSrefO1bpVK/MNc/nlWm/dandUtTtXek4//NHDOvbdWH301FG7w/E6dxOC5TWVdTVjC7TW\n/Ss9v2Bsgjeoad7p2Kyfb3h3y8rFbaKjo8uL28yaNau8cA7AjBkziImJ4e23367y/soFdqrbLjQ0\nlHHjTI+GlJQUkpOTK+LWmkWLFpUXywHKi/MI4S3HjsFTT8F775nXo0bBn/5kbUH7ykpOldTZRVqj\nGffZOByFDlY8tII2F7exKDrfY3lCsIsnX+TeEh8fT2FhYfkU1FOmTGH8+PHVFs4pm9Oosrq2CwsL\nu+A9lTkcDmJiYspfS6Ec4U3ffgsPPAD//je0amXaDUaPti+eZTuWEb8gnlYXtapz2wHdBrD8oeVN\nOhlAE0oIvqCsuM2YMWPYs2cPQ4YMoW/fvrUWzqnM3e1qUl2xHIfDQXx8fAM/kRDGokVmoNnRo6aa\n2aJFpoCNnVY5VvG7Qb9j8i2T7Q3Ej/hUt9NAd35xG6UURUVF1RbOGTFixAXvr63ADlRfMa2MUoqU\nlBRmz54txXKE15w9CxMnQkKCSQYjR8KmTfYnA4A1+9Zw62W32h2GX5GEYKEJEyYQFhZGr1696N+/\nPwkJCQwfPpzIyEgWLlxIcnIy4eHhBAUFVekmGh8fT7NmzardrnJltbq6p/bo0YMZM2YQFxfH5Zdf\nzrBhw2RGVdFg//0v3HEHzJwJzZrBa6/B3/5mX3tBZSWnSth5eCf9IvrZHYpfkdlORb3JuRZffgn3\n3gt795rpJxYsgNtuszuqCp/u/pQX175I9sPZdofiE6RAjhCiUSxebOob790L/fvDli2+lQwA1uyV\n20UNIQlBCOEWrWHqVNOT6Phx+MUv4J//9M1Rx2v2reHW7pIQ6kt6GQkh6nTihOlFNH8+BAXBSy/B\ns8/aV7egNqfOnmLrga3c2O1Gu0PxO5IQhBC1OnjQTE+9YYNpMP7wQ7jrLrujqtmmgk1c1eEq2rXw\ngdZtPyMJQQhRo+3b4e67Yd8+6N69YtZSXybtBw0XMAmhe/fuHs8KKtzTvbt/FAwXnlm9GoYPh5IS\nGDAAPvoILrnE7qjqtmbfGsb0le7UDREw3U6FEN7z3nswZowZePbAA/D++2Y6Cl93rvQc4S+Fs+vp\nXXRs09HucHyGdDsVQtSb1pCWBg8/bJLB+PGmIdkfkgHA14e+pku7LpIMGihgbhkJITxz7pyZqXTW\nLNN76I9/NK+torVm15FdnC092+B9LP1mqbQfeEASghCCEyfgoYdMO0GLFmYKivvvtzaGDfkbGPL+\nEC4Lvsyj/bx6+6teiqjpkYQgRBNXVAQ/+xmsXQshIaYn0S23WB/HpvxNJPZOZNbPZll/cAFIQhCi\nSTtwwExQ99VXZsTxp5/CtdfaE8vWg1sZ0HWAPQcXgDQqC9FkORzmSuCrr0wNg3/9y75kALD1wFb6\ndulrXwBCEoIQTdH27XDzzSYp9OsHa9bAZZ7duvfIybMn2XV4F70v8fFRbwFOEoIQTcymTWZ20oMH\nYfBgyMqCjjb30tx+aDvd23VnzMNjGDRoEElJSTidTnuDaoKkDUGIJuSLL+Cee0x1s5//3IwxaNnS\n7qhg5dcryd+cz7fzvi1flpOTQ2ZmJpGRkTZG1rTIFYIQTcSnn5oG5KNHTRfTRYt8IxkA/OWzv/Dj\nzh+rLMvLyyM1NdWmiJomSQhCNAEff2yuCE6ehORkMxXFRRfZHVWF79X3cODC5QUFBdYH04RJQhAi\nwC1YYOYjOnMGfvUrePttUwPZV5w5d4bjbY/DwQvXRUREWB9QEyYJQYgAlpEB//M/Zl6iSZPgtdd8\nr6jNt//9lu4h3Ym6NKrK8qioKNLS0myKqmmShCBEgHr3XRg1CkpLTenL6dN9LxmAGX/w08t+SmZm\nJomJiQwaNIjExERpULaB9DISIgC9846Zvlpr+MMf4Lnn7I6oZrkHc+nbuS+RkZFkZGTYHU6TJlcI\nQgSY9HR47DGTDF580beTAcgIZV8iVwhCBJA5c0wvIoCZM009A19WqkvZdnAb0V2i7Q5FIAlBiIDx\nzjswdqx5/vLLMG6cvfG4Y/eR3YS3DiesVZjdoQgkIQgREMpKXoK5MrAqGRw6dohTZ081+P1Zziyi\nO8vVga+QhCCEn8vIgEceMW0GM2ZYd5vo0LFDdHu1G5e0vcSj/fx24G+9FJHwlCQEIfzY/PkwerRJ\nBi+8ABMnWnfsPUV7uO6S69icvNm6g4pGJb2MhPBTS5dCYmLFOIMpU6w9/v6S/XRr383ag4pGJQlB\nCD+0fDmMHAnnzplE8Fsb7rp8V/wdl7a/1PoDi0Zj+S0jpVQ8UAT01VrPrGZ9NNATQGu92OLwhPB5\nq1ZBfLyZm+jZZ83AMztGIMsVQuCx9ArB9WWvtdargSKlVJ9qNpviSgSRNawXoslaswbuvRdOnYIn\nnzTdS+2ajuK7ku8kIQQYq28ZjcRcHQA4gCGVV7quHjYCaK1f1lpvszY8IXzXxo1w991w/LjpVfTH\nP9o7N9H+kv1cGiy3jAKJ1QkhBDhS6XX4eev7A+FKqWil1ATrwhLCt339tSlu8+OP8OCDZnqKIJtb\nAOUKIfD4YrfTw1rrXKXUEKVUfHXtCFOnTi1/HhsbS2xsrIXhCWGtXbtg6FAoLDRFbt5/3/56BudK\nz3HgxwN0bdfV3kBEtbKzs8nOzq73+5TW2vvR1HQwpaYDmVrrLNftoUit9cuV1k8A8rTWS5RSY4Ge\nWusp5+1DWxmzEHbatw9uvdX8HDIEli3zjbKXB348QJ9Zffh+/Pd2hyLcoJRCa13nDUarLzoX4OpB\n5Pq5CkApFexatqjS+hBgk6XRCeFDvv/eJIF9++Cmm+Cjj3wjGYC5XSRdTgOPpQlBa50LoJSKAwor\nNRqvcq13YnofxQNhWuslVsYnhK8oKoLbbze3i/r0MeMO2rSxO6oK0uU0MFnehqC1nlPNsv7VrJcx\nCKJJOnbM9Cb68ku44gr47DMICbE7qqpkUFpgkpHKQviQU6dg+HBYtw4uvRQyM6FTJ7ujupBcIQQm\nSQhC+Ihz5yApCVauhI4dzYjkyy6zO6rqSZfTwCQJQQgfoDU8/jgsWgTt25vbRFdcYXdUNZNBaYFJ\nEoIQPmDKFFP+smVL+PvfIdrHa8bIFUJgkoQghM1mzjSFbZo3N1cIt95qd0S1k0FpgUsSghA2mju3\noqjNu++a3kW+7tCxQ4S2CqVF8xZ2hyK8TBKCEDZZuhSSk83zN94wxW78gQxKC1ySEISwQXY2/M//\nmGpnzz8PTz9td0Tuky6ngUsSghAW27rVTFJ36hQ89RSMGuUkKSmJQYMGkZSUhNPptDvEWsmgtMDl\ni7OdiibE6XSSmppKfn4+Xbt2JS0tjcjISLvDajS7dlVMYz1yJDzzjJNhw4aSl5dXvk1OTg6ZmZk+\nex7kCiFwSUIQtnE6nQwd6l9fhp4oKIBhw+CHH8zP99+HRx9NrfL5AfLy8khNTSUjI8Oy2OqTmL8r\n+Y4+naWYYSCShCBsk5rqG1+GVigqMlcGe/bAT38KixfDxRdDfn5+tdsXFBRYFlt9E7MMSgtc0oYg\nbOMLX4ZWOHECfvYzU/XsqqvMzKVt25p1XbtW35c/IiLCsvhqS8zVkUFpgUsSgrCNL3wZNrazZ01b\nwdq10K2bmZKiQ4eK9WlpaURFRVV5T1RUFGlpaZbFWJ/ELIPSAluDEoJSarxS6i2l1BilVHtvByWa\nBl/4MmxMWptxBsuWQViYSQbnT1YXGRlJZmYmiYmJDBo0iMTERMvbUOqTmGVQWmBzq4SmUmowcERr\nvU0p9WKlVX2BOCBZaz23kWI8PxYpoRlAyhozCwoKiIiICKheRpMnmykpWrc2M5feeKPdEVWvujaE\nqKioahPTxvyNPLn8STYnb7Y6TOEBd0to1pkQXMmgZ1nhmvML3yulQoDJwEYrKpxJQhD+4NVXYdw4\nMz/RJ5/AnXfaHdGF9hbtZcfhHQAcPHiQ9957j8OHDxMeHs7o0aPp3LnzBe9Z/916cg/m8tGDH1kd\nrvCAuwnBnV5GMUBmTSu11kXAZKXU2HrEJ0TA+uADkwwA/vIX30wGAGOWjaH4ZDEhLU05tma3NqMT\nphpPhiMDHNW/76HeD1kVorBYnQlBaz1TKTVWKVWktd4DOJRSm4CJwBatdYlr08ONGKcQfmHFCnjk\nEfP81VdNwRtfVKpL2Zi/kd1P76Zjm452hyN8hFuNylrrdFcyABgJLACeAPYopTYppT4Deiql2gEo\npYY3RrBC+LL16+GBB0zls0mT4Jln7I6oZt/+91s6tO4gyUBU0ZCBaXlAptZ6JoBSKhoYAgwDnlNK\nlV0pNHp7ghC+4t//NlNXnzhhrhCmT7c7otrl7M/hhq432B2G8DH17naqtU4HQpVSPVyvc7XWM7XW\nw7TWYZgrCN+enUsIL9q3D26/HQoLzaR1s2eDqrP5zl4b9m9gQLcBdochfEyDxiG4ksCeGtZtBSZ5\nEpQQ/uK//zXzEuXnm0pnH35oehb5ug35G+QKQVygUUYqa61zG2O/QviSH3+Eu+6CHTvguutM99JW\nreyOqm7HTh9j15FdMkGduIBMXSFEA5w6BcOHw6ZNEBkJn34KISF2R+WezQWb6d2pt4w2FheQhCBE\nPZ07B6NGmdHHl1wCK1dCly52R+U+uV0kaiIJQYh60NqUu1ywANq3N1cGvXrZHVX9bMjfwA3dJCGI\nC0lCEKIenn8e3noLWrQwbQZ9/PA2vHQ5FTWRhCCEm15/HdLSoFkzc4Vw2212R1R/+0v2c/rcaXqG\n9rQ7FOGDJCEI4YYPPoD//V/zfO5cM97AH23Yb9oPlK8PlBC2kIQgRB2WLauYn+iVV2D0aHvj8YQ0\nKIvaSEIQohbZ2ZCQYHoWTZkCzz5rd0SeydmfIw3KokZuFcjxJVIPwfeNWjqKf+z+h91heOzsWSgu\nMj2LWraqqIPsz46ePsqBcQfKp7wWTYM36yEIUS85+3P4+MGPuTzscrtDabBdu+BnPwN9BO67H956\n0zQm+7sWzVvQvoVUvRXVszwhKKXigSKgb9mMqTVsN6G29cJ3HTx6kGs6XuO3/wvdswdG3ANH8k1x\nm/nvwMUX2x2VEI3P0jYE11TZWmu9GihSSlXbi1spFYeZUlv4mWOnj3Gm9AzBLYLtDqVBDh6EoUMr\nJqtbtEjcSm5wAAAUk0lEQVSSgWg6rG5UHom5OgBToE++9APMgaMH6Ny2s192azxyxCSD3buhb1/T\nu6h1a7ujEsI6VieEEOBIpdfh52+glIp2XUH43zeK4ODRg3Rp60cT+7iUzVy6fTtcdZWZkiLYPy9y\nhGgwX+x2Gmp3AKLhDvx4gC7t/CshHD9uGpA3bIAePSAzEzpKZUnRBFndqFwIhLmehwCHK690XR1k\nuV7W2Ld06tSp5c9jY2OJjY31apCi4Q4cPeBXVwinTkF8PHzxBUREmBlMu3WzOyohPJOdnU12dna9\n32fpOARXo3KM1nqOUmoCpjbzNqVUsNa62NUDSWNuJaUAY7TW287bh4xD8GHPrX6O1he15jcDf2N3\nKHU6exZGjoQlS6BDB/jnP+Hqq+2OSgjvc3ccgqW3jMoqqbl6ERVW+rJf5Vq/WGu9xLVM7uD6IX+5\nQjh3zkxBsWSJKWyTmSnJQAjLxyForedUs6z/ea/TgXTLghJe4w9tCKWlkJwMf/2rGX08d24BL788\nkfz8fLp27UpaWhqRkZF2hymE5WSksvCqg0cP0rltZ7vDqFFZgZt33jH1j+fMOcDEiQPJy8sr3yYn\nJ4fMzExJCqLJ8cVeRsKP+fItI61h3Dh4801T4Objj2HZsglVkgFAXl4eqampNkUphH0kIQivOVt6\nliMnjtCpTSe7Q7mA1jBpErz2Glx0kRmBbEYk51e7fUFBgcURCmE/SQjCaw4dO0R4q3CaBfnWLHBa\nw29+AzNnQvPmsHAh3HOPWde1a9dq3xMREWFhhEL4BkkIwmt8tUF52jR44QUzW+mHH8K991asS0tL\nIyoqqsr2UVFRpKWlWRylEPaTRmXhNb7YfjB1qkkIQUGmV1F8fNX1kZGRZGZmkpqaSkFBAREREdLL\nSDRZkhCE1/haD6PKyWDePBgxovrtIiMjycjIsDQ2IXyR3DISXnPgR9+4QtAann++ajJ48EG7oxLC\n90lCEF5z4Kj9bQhlDci/+50kAyHqSxKC8Bq7bxmVdS0ta0D+618lGQhRH9KGILzGzkZlreGZZ+D1\n103X0g8/vLABWQhRO0kIwmvs6nZaWgpPPQVvv10x6OznP7c8DCH8niQE4RVaa1tuGZ09C48+Ch98\nYKajWLLEVD4TQtSfJAThFUUni2jRvAWtL7KuCPHp05CYaK4I2rSBTz6BwYMtO7wQAUcSgvAKq9sP\njh+HhARYscLUPl6xAm66ybLDCxGQJCEIr7DydlFxsamBvGYNhIfDypXQt68lhxYioElCEF5hVYPy\nDz/A7bdDbi507SqVzoTwJhmHILzCiltGe/fCrbeaZNCrF6xdK8lACG+ShCC8orFvGW3fbtoIduyA\n664zt4t69Gi0wwnRJElCEF7RmFcIa9eaK4OCAhg4EL74Ajr7zhx6QgQMSQjCKxqrDeHjj01ls6Ii\nuP9++OwzCAnx+mGEEEhCEF7SGLeM/vxnGD4cTp6E5GRT6axlS68eQghRiSQE4RXevGVUWgqTJ8P/\n+3/meVqamZaimW9V5hQi4Ei3U1EjrTUvrHmB4lPFdW53/MxxwlqFeXzMkyfNVBR/+5uZpC49HR5+\n2OPdCiHcoLTWdsdQL0op7W8x+6udh3dy619uZdyN4+rcNqJdBEnXJXl0vB9+MO0E//oXtG1rpqS4\n/XaPdimEAJRSaK1VXdvJFYKoUc7+HAb1GMTEmyc2+rF27DCT0jkc0K0bLF9uupcKIawjbQiiRjn7\ncxjQbUCjH2fVKhgwwCSDvn1hwwZJBkLYQRKCqNH6/esbPSH8+c9wxx2mW+l998E//wkREY16SCFE\nDSQhiGodO32MnYd3Et05ulH2f+YMPPmk6Ul07hw89xwsXmymsRZC2EPaEES1Nhds5rpLrqNF8xZe\n3/ehQ2bq6n/+0xS1mTvX1DUQQthLEoKoVs7+HAZ09f7tos2bTU+i/fvNraElS+CGG7x+GCFEA8gt\nI1Gtxmg/ePdduOUWkwxuugm2bJFkIIQvkYQgLqC19moPo7KpJx55BE6dgpQU+PxzmaBOCF8jt4zE\nBfYW7yVIBXFZ8GWe72svxMebq4EWLUyvosce80KQQgivkyuEJszpdJKUlMSgQYNISkrC6XQCFeMP\nlKpzYGOtPvkEoqNNMoiMhPXrJRkI4cssv0JQSsUDRUBfrfXMataPdT2N0lpPtjS4JsTpdDJ06FDy\n8vLKl+Xk5JCZmenx7aLTp83kdK+9Zl7fcw+8/z6EhnoatRCiMVl6haCUiga01no1UKSU6nPe+jgg\nU2udDvRUSg22Mr6mJDU1tUoyAMjLyyM1NdWjBmWHwxSzee01Mzndyy+bKwVJBkL4PqtvGY3EXB0A\nOIAh563vWWmZw/VaNIL8/Pxql3934Du2H9pOTJeYeu8zIwP69IGNG6F7d1Pmctw48PDOkxDCIlbf\nMgoBjlR6HV55pevKoExf4EMrgmoqjp85jqPQgaPQwdFrjsLJC7f54Sc/cFWHq2hzsftDhouL4amn\nYN488/qBB2D2bLkqEMLf+GQvI9etpS1a6212x+LPlu9czqwts/iu5Dv2l+zn6Omj9AjpQVRoFFfe\nciW7Du+iuLii1kFwcDA33HwD9153r9vHyM6G0aNh3z5o3RreeMPUM5CrAiH8j9UJoRAoq6ISAhyu\nYbs4rfWUmnYyderU8uexsbHExsZ6KbzA8tvs3/KL637BrZfdSrf23ejYpiNBquIuoXOAk9TUVAoK\nCoiIiCAtLY3IyEi39n3yJPz61/Dqq+Z1TIy5Qrjyysb4JEKI+sjOziY7O7ve77O0QI7rf/4xWus5\nSqkJmAbkbUqpYK11sWubsWW3jpRSca4G6Mr7kAI5bjjw4wGuefMaDo0/xEXNLvLqvjdsMIPMvvnG\nlLX8zW9McrjIu4cRQniJuwVyLG1U1lrnQnlvosJKt4RWVVr+olJqt1LqMCDf/A20YtcKhkUN82oy\nOHkSJk0y00588425Gli3DqZOlWQgRCCwvA1Baz2nmmX9XT9Xc15Ds2iY5buWc99V93ltf2vWwNix\nprJZUBBMmADTpkGrVl47hBDCZjJSOQCdOnuK1c7V3NnrTo/3VVRk5h4aONAkg6uvNlcFL70kyUCI\nQCMJwSY1TRvhDWv3reXqDlfTsU3HBu9Da5g/3ySA2bPNLaHf/hZyc2WGUiEClU92Ow10tU0b4W4v\nn9os37Wcuy6/q8Hv37nTjCtYtcq8vukmSE+Ha67xODQhhA+TKwQb1DZthDcs37Wcuy+/u97vO3rU\n9Bbq3dskg9BQc3WwZo0kAyGaArlCsEFN00YUFBR4vO/dR3ZTcqqE6C7u10LWGv76V5g4EcpCePRR\nePFF6Njwu05CCD8jCcEGXbt2rXZ5RESEx/tevnM5d/W6q8oAtNrk5Jj5htatM6/79TOjjW+80eNQ\nhBB+RhKCDdLS0sjJyTG3jToCwdClSxeGPD6ET3d/6tG+5/97PuNvGl/ndg4HTJkCCxaY15dcAtOn\nm2koguRGohBNkqUjlb0hUEYqO51Onvz9k6zuvJoOZzpw+eWX08oL/ThDWoYw5+dzaHtx22rXHzoE\nf/gDvPUWnDkDLVvCs8+aAWft23t8eCGED3J3pLJcIdikbae2/Kf3f/jw9g8ZfvXwRj9eSQm88op5\nHDtmlv3iFyY5XHppox9eCOEHJCHY4GzpWR5c/CAP/eShRk8GP/4If/yjSQRHXBOP33OPSQTXXdeo\nhxZC+BlJCDaYvGoyzYOa8/vBv2+0Y5SUwJtvmoplh11zyg4cCC+8ADff3GiHFUL4MUkIFtpfsp/x\nK8eTezCXdY+uo1lQM68f4/BheP11c1VQ5KpNd/PN8LvfwaBBUqdACFEzSQhedODHA5w+d/qC5RrN\n/O3zmbluJk/2f5J37n2H1he19uqx9+41dYznzKloIxg40ExNPWSIJAIhRN0kIXjoXOk5PtnxCa/l\nvMb2Q9tp16JdtdtFd45mw5gNRIVFefX4W7aY9oEFC+DcObPsjjvMiONbbvHqoYQQAU4Sggc+3f0p\nT614io6tO/LMgGeIvyae5kGNf0rPnoWlS82toX/9yyxr1gwSE2H8eFPoXggh6ksSQgN98OUHTMic\nwLzh84jrGWfJMQ8cMLeEZs+G/fvNsuBgGDMGfvlLuOwyS8IQQgQoSQgN8Mq6V3hj4xt8Pvpzru54\ndaMeq7QUsrJMEli61FwdAFxxBfzqVzBqFLStfgyaEELUS5NKCCvzVjLv63l4MtL5yIkjOAodrH1k\nLZcGN96Iru++g/feg7lzYc8es6xZMxg+HJ54AgYPlikmhBDe1SQSwsGjB3nms2fYsH8DE26aQJuL\n2zR4XwrF3VfcTVirMC9GaBw7BkuWmESQlWVmIQXo3h0ee8wUtu/WzeuHFUIIIEATwsx/zWRf8T4A\nzpSeYfE3ixkTPYa5P5/r9e6enjp9GlauhL/9DT76CI4fN8tbtID77jOJIC5OrgaEEI0v4Ca3KzlV\nQsQrEUyPm16+bHDkYK7tdK0V4bnlzBlYvRoWLjRJoGxKCTDVyUaPhhEjICTEvhiFEIGjyU5utzF/\nI3279OXpG562O5Qqjh83VwJLl8KyZVBYWLHuJz+Bhx6CBx8EL1TQFEKIBgm4hJCzP4cB3QbYHQZg\nqo8tXw5//ztkZsKJExXrrr0WEhLMQ8pTCiF8QUAmhMeiH7Pl2GfOwPr18Nln8I9/QG5u1fU//Snc\nf79pG7jqKltCFEKIGgVUQtBak7M/h9k/m23R8eDbb01B+lWr4PPPzXTTZVq1MvMI3XMP3H031FA5\nUwghfEJAJYTdR3bT5uI2RLTzvDZxdbSGHTvgiy8gO9s8Dh6sus3VV8Ptt5v5hAYONElBCCH8QUAl\nhPX713NjN+9Vhz950tz2WbcO1q41j//+t+o2l1xiuoUOGWJ+yvQRQgh/FVAJwZMG5dJS2L0bNm2C\njRthwwaTDE6fN5t1587mf/6DBsFtt5m2AJlaWggRCAIqIazfv55R14+qc7uzZ2HnTti2DbZurXgU\nF1fdTinTG+jGG81U0rfcAj17SgIQQgSmgEkIx04fY+fhnUR3ji5fpjV8/z1s3w5ff21+fvWV+Xny\n5IX76NzZ9ASq/AgOtvBDCCGEjQIiIZw9Cx9v3sylF1/Hn15vwbffwjffwH/+U3UAWGU9esD110Pf\nvhWPiMZpixZCCL/glwmhsNDUCN61yzycTjhzw3poM4Dxn1XdNiTE3Pbp3duMCO7dG4KD9zJz5q/J\nz89n586u/OIXaUREyBBhIUTT5pdzGR07pmlz3oSlLR++j14nHuK2DiO48koz+veaa8xtoMr3/J1O\nJ0OHDiUvL698WVRUFJmZmUTKvBFCiADk7lxGfpkQtNb88Y/mFs8VV0DPnpqeb3Vm09hNXBZce7/P\npKQk5s2bd8HyxMREMjIyGitsIYSwTcBPbvd0pbnrnIV7aB7UnEvb112wJj8/v9rlBQUF3gpNCCH8\nkuUJQSkVDxQBfbXWM+u7vjrNg5rzwuAXUG70B+1aw/wREdKiLIRo4iy9ZaSUigYitdZLlFJjgU1a\n623urndtU2s9hLpIG4IQoqlx95aR1XW4RmL+9w/gAIbUc73HIiMjyczMJDExkUGDBpGYmCjJQAgh\nsD4hhACV6oMRXs/1VWRnZzcoiMjISDIyMsjKyiIjI8Pvk0FDz0MgkXNgyHkw5Dw07Bz4daVe+Uc3\n5DzIOSgj58GQ89Cwc2B1o3IhEOZ6HgIcrud6AKZOnQqYD5ydnU1sbKy34xRCCL9V9t1Y9l3pLqsT\nwgIgBsgCegKZAEqpYK11cU3rz1f2IadOnSrJQAghzhMbG0tsbGz5d+W0adPcep/lA9OUUmMAJ6Y3\n0RzXsk1a6/41rT/v/f41kk4IIXxAQI5UFkII0Tj8ulFZCCGE90hCEH5HKRWvlIpTSk2oY7ta1wsR\nSFwDe2ta59bfjN8khLo+kLsf2J+5cQ7Guh4vWh2bVVy/9FprvRooUkr1qWG7OBphYKMvceP3Idq1\nTbzVsVmlHt8LY6yOzUqu3/eFNaxz628G/CQh1PWB6vOB/ZUb5yAOyNRapwM9lVKD7YjTAo0+mt0f\nuPk7P0VrvRiIbKJ/E9GAw7XeGYjnoIzrM+bVsNrtvxm/SAj4wJQXPqCuz9iz0jKH63UgqnM0u1Iq\n2vUHEsjVr2v9fXBdFWwE0Fq/fP6cYAHCnb/7Ga6fPQP0HLjD7Rkg/CUheHXKCz9V62fUWqdX6qbb\nF9hsVWA+KNTuACxQ1+98fyDcddsoUG+j1vU3kQs4lFJHqGGQq6jKXxKCcJPrMnlLAP9vqNbR7K6r\ngyzXy6bep/qw60ux7IqhSVFKBWN+X14A0pVSPWwNyD5uzQAB/pMQvDLlhZ9z9zPGaa2nWBOSLRZQ\ncTusJ7AKyv/4wbSfDHdNnx4ewPeN6/p9OIy5jQLmtko/i+KyUl3nIBmYrrV+GRgLPGBhbHaocou0\n0t9EtX8z1fGXhFDXl4DbH9iP1XUOUEqNdf3ylzUyB5xK/+ONAworXQmtcq1frLVe4loWXM0uAkVd\nvw+LKq0PATZZGp016joHGteXpOt3ouj8HQQK1xVgjFJqeKXFZX8TNf3NXLgffxmp7OmUF4GgtnPg\n+sdegPlfUyiQUOnWiQhAbv5NFAL9AvWq0Y1zMAHT+yYsUL8XvMlvEoIQQojG5S+3jIQQQjQySQhC\nCCEASQhCCCFcJCEIIYQAJCEIIYRwkYQghBACkIQghBDCRRKCEEIIQBKCEEIIF0kIQgghAGhudwBC\n+DvXZGojMLNthgOrtNZOe6MSov7kCkEIz6UD810zah4GmlztAREYZHI7ITyklNqNmW56ftlUw0L4\nI7lCEMJzD+Caj18pFYh1B0QTIQlBCA8opeK11tu01iO01uFAaBMu1Sj8nCQEIRrI1Zg8+bzFeVrr\nPTaEI4THpA1BCA+4ShYWYarUhQILtNYl9kYlRMNIQhBCCAHILSMhhBAukhCEEEIAkhCEEEK4SEIQ\nQggBSEIQQgjhIglBCCEEIAlBCCGEiyQEIYQQgCQEIYQQLv8fcMWtge5ADDMAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from sklearn.isotonic import IsotonicRegression\n", "from sklearn.linear_model import LogisticRegression\n", "from calmap import plot_calibration_map\n", "\n", "cal_probas = ada.predict_proba(x_cal)[:, 1]\n", "\n", "lr = LogisticRegression(C=99999999999)\n", "lr.fit(cal_probas.reshape(-1, 1), y_cal)\n", "\n", "iso = IsotonicRegression()\n", "iso.fit(cal_probas, y_cal)\n", "\n", "linspace = np.linspace(0, 1, 100)\n", "pr = [lr.predict_proba(linspace.reshape(-1, 1))[:, 1], iso.predict(linspace)]\n", "idx = cal_probas.argsort()\n", "scores = cal_probas[idx]\n", "y_c_2 = y_cal[idx]\n", "methods_text = ['logistic', 'isotonic']\n", "fig_map = plot_calibration_map(pr, [scores, y_c_2, linspace], methods_text, alpha=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calibration map\n", "\n", "In the calibration map shown above, we can see that:\n", "\n", "* Logistic calibration fit the scores in the first and last bins well, but was either underconfident or overconfident in the rest of the map, due to the limitations of its shape\n", "* As a non-parametric method, Isotonic regression managed to fit the scores better than Logistic calibration by finding an inverted sigmoid shape\n", "\n", "## Fitting a beta calibration model with the betacal package\n", "\n", "Our beta calibration is a parametric method which is able to circumvent logistic calibration's shape limitations by fitting three parameters, i.e. two for shape (a and b) and one for location (m). These parameters can be fitted by training a logistic regression model with two features extracted from the classifier's probability outputs. Beta calibration is adequate for calibrating probabilities, because the beta distribution has support [0, 1], while logistic calibration assumes the scores are distributed in two gaussians, which have infinite support.\n", "\n", "To make it easy for practitioners to fit a beta calibration model, we have provided a Python package called __betacal__. The package can be installed via pip (_pip install betacal_). Below, we show how to use the package to fit the three-parameter version of beta calibration. For the other two versions, the practitioner can set parameters=\"ab\" (fit shape parameters a and b and fix location parameter m = 0.5) or parameters=\"am\" (fit shape parameter a, setting a = b, and fit location parameter m)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEQCAYAAACwSgOGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVPX6wPHPAZW0BIHUBFNZ1PZUxMybhSK2bypwu1jd\ncsHqdq8trkVqZGnLbbtWpu20CdpmK6CU9stSwUzLSkAtcUVAzY3l+/vjOzMMMsAMDDPD8Lxfr3nF\nnHPmzHcmOQ/nuzyPoZRCCCGE8HF3A4QQQngGCQhCCCEACQhCCCFMJCAIIYQA3BQQDMPoX8++0YZh\nxBqGMcWVbRJCiNbO5QHBMIxYIL2Off0BpZTKBkoNw+jn0sYJIUQr5vKAYLrY59exOxEoNf1cAIxw\nSaOEEEJ43BhCJ+CA1fNgdzVECCFamzbuboCjDMOQlXRCCOEgpZTR0DGedodQAgSZfu4EFNs6SCmF\nUopZs2ZZfm7ND/ke5DuQ70G+hxqP779n1tChluf2ctcdQo1IZRhGgFKqDFgCRAErgHAg09aLZ8+e\nDUBOTg45OTnExMQ0Z1uFEKLlOHKEnDFjyDntNMu10l4uDwiGYYwGogzDGKWUWmbanAVEK6XyDMOI\nMs1EKlFKbbB1DvOHnD17tgQDIYSwNn06MUOHEtO7t+VaOWfOHLte6vKAoJRaCiw9aVu01c+L7T2X\nBANNvgf5Dszke9Ba7feQlQUffAAbNxLz448Ov9xwpH/JExiGoVpam4UQwl6FhYWkpKSwc+dOQkND\nSU1NJSwsrOEXlpbCBRfA4sUwcmSNXYZhoOwYVPaagNCrVy+2b9/uhha1Pj179mTbtm3uboYQXqew\nsJC4uDjy86uXakVERJCZmVl/UFAKEhPh9NPhhRdq7W51AcH0gd3QotZHvmshmsfYsWN5++23a21P\nSkoiLS2t7hc+8gi7313Egc+WcU7PqFq77Q0InjbtVAghWq2dO3fa3F5UVFT3i5Yto/zFBfS/cgdR\naZdQeqyU/fvh669h0ybH3l8CghBCeIjQ0FCb20NCQmxuP/rdBsrHJTNxTBS7O0Lwjn9yVq9OdO4M\nMTHw4ouOvb8EhBYuLy+P6Ohou7c7ct7IyMimNE0I4aDU1FQiIiJqbIuIiCA1NZXSUlixAp54Av7x\nD7g0soi9Q65n7PFZvB7wJVT5sDN9Cnv2VL9282bH3r/Fpa4QNYWHhzN//ny7tzvCMBrschRCOFFY\nWBiZmZnMnPkwv/3WAcMYTEjIjYwceRpbt1YfF0Ap33AFL3IHS2J+A98K2PgPKAkHoH17OPtsuPBC\nBxvg9iXWDj50k2ura3tBQYFKSkpSMTExKikpSRUUFNg8rj5NPUdmZqaKiIhQQUFBKiEhQZWWllrO\nGxUVpZKTk1VgYKAaOXKkys3NVREREcrHx0c98cQTSimlMjIyVHJysoqPj1eGYaiBAweqwsJCpZRS\nubm5KioqynK+uLg4NW3aNHX22WergQMHWtqQnp5uaUNycrJl+8KFC1VgYKDy8fGpdd7IyEibn6eu\n71oI0Ti7dyu1dKlS996r1ODBSrVtq5SeOlT70Y5jagUx6lnuVnTYo4wH2ytmo+56eKP64AOlfv9d\nqYqKmuc3/c42fH215yBPejgSEAoKClRERIQCLI+IiAiHLuhNPUdpaakKDAxUK1asUGVlZWrSpEkq\nPj7ecm7DMNSyZctUWVmZ5YJ98OBBlZWVpUwzqlRGRoYyDEMtXrxYlZWVqeTkZEsQyM3NtVz4zee7\n44471Ntvv23Znp+frwIDA9WGDRtUYWGhioyMVEuXLlVKKeXj46M2bNigysrKVHx8vJo+fbrlvBIQ\nhGgeO3cqlZam1IQJSvXtW/fF3/rRpo1SA/pVqh/CEtTW/mPUd6sr1JQvHlDMRl3zzjX1vp+9AcGr\nu4xSUlJqzOcFyM/PJyUlpf4pXE48x5IlS4iLi2PYsGEAPPbYY4SHh1v2BwYGcuONNwIwYsQIAgMD\n6dixI7GxsRiGwcGDBwGIiopi3LhxALz00kv4+vpa9lkzDIMXXniBvLw8y7alS5eSnJzMhab7x/T0\n6vpEJSUl+Pv7AxAUFERpaSlCCOcqK4MVKxRvfP01a3/eS41JQ22Ac2u/5owzoHdviIzUjx5nKgbP\nfwX/gp18s3AGFe3SWbR6AQAzLpnhlHZ6dUBo1BQuJ58jPz+f9PR0srKyAH1HZn0hDwoKsvzcqVMn\ngoNtl4CwDiKg+xoLCgoaPM7choEDB1qe9+tXXYhu7ty5ZGdnW97X1uuFEI6pqoLcXPj8c/jiC/j+\ne6g8Mwf+ORz+Zt85dpseqw4DefD443BgGwy4BQ59crPluEt7XsqQM4c4pd1eHRAcncLVHOeIiIgg\nPj6e999/37KtMat8T75LKSgoIDw8vNZ2Wzp16sRWqxGpvLw8SzBZsWIFK1eupGPHjixatIjc3FyH\n2yaEgMOHITMTPv4YPvsM9u496YAzTLmF9vfBZ9+FBAdDly7QuTMEBYGvb93nHp3xM4P//JPZj8Rw\nxWntLNvb+bZj+iXTnfYZvDogpKamsmbNmlrLwFNTU112joSEBKZPn052djZRUVFMmzaNbdu28eWX\nXwKYx0Vsst6Xm5vL4sWLiY+PZ+rUqQwcONDS1VPXa8ySk5MZOHAgiYmJBAQEEB8fz/Tp01FKERQU\nRMeOHSktLWXhwoW1prwJIeq2b58OAMuWQXY2HD9u+zjDgM59trMXmDBwHM8mTKV9ezvf5L//hbwN\n8MOvvNa1q7OabpNXr0MwT+FKSkpi2LBhJCUlNZwTxMnnCAgIID09neTkZIKDg9m+fXuNPvz6pnZa\n74uLiyMzM5PAwEDy8vJYsmRJg6+x/gzz588nNjaW3r17M3LkSMaPH8+ECRMsQSEuLo7HH3+crKws\nVqxYYddnE6I12rtXL/gaPlz3848fr+8ITg4Gp58ON98M77yjX/O3q3SutREDe9kfDJ55BhYs0NGm\nmYMBSC6jFmHp0qUsWbKkRreTO3nzdy2ELWVlsHQpvPeevjZXVdk+7vzz4brr4NprIToafKz+5I56\nOYrcXbmsGbeGi7pf1PCbPvusfuTkQI8eTWq/vbmMvLrLSAghGqu8XA8Kp6XpbiFb3UGGAUOGwKhR\ncMMNUN+cjG2l2wDo2alnw2/+/PM6GKxc2eRg4AgJCEIIYWXzZnjtNXjrLRsDwyaXXKKzTY8eDd26\nNXzOwycOc+DoAfx8/ehyapf6D37qKd1NtGIF9LQjeDiRdBkJh8l3LbzNkSOwZAm8/DJ8953tY/r3\nh7FjISEBund37Pyb927mvBfPo09wH3791691Hzh3Lrzxhu6XOvNMx96kHtJlJIQQDfj1V11P5o03\n9DjByUJC4JZbdCA418biMXttL9MDyj0D6viLXyl46CE9UPH11/bddjQDCQhCiFalqgo+/VR302dm\n1t7fti1cfz3cfruuRFnf+gB7bS+tJyBUVcF99+kuopwcvTjBTSQgCCFahb/+0ncCTz9NjcyhZpGR\nMHEi3Hqr86/JdQ4oV1TAhAn6ViUnBwIDnfvGDpKAIITwanv3wnPP6a6hkpKa+3x89BTRu+6C2Nia\n00Sdydxl1KtTr+qNx4/rwgbmJc6nnto8b+4ACQhCCK+0bRs8+SS88gocO1ZzX6dO+g/zO++EXr2a\nvy21xhAOHdJzVQMC9JxWP7/mb4QdvHqlsidwRuUxe6uf+Zj+vGlqtTQhWrL8fBg3TmcKXbCgZjAI\nD9djB3/8oZPFuSIYgNUYQqeesGePrm8ZEQHvv+8xwQDkDsElmlp5zN7qZ+b3cUa1NCFamvx8ePhh\nePttqKysuW/AAJg+Xf9R7oxBYkccqzjGrsO78DV8Cdl7FK4crnNaPPSQXtnmQeQOwcUyMjKIjIzE\n19eXxMTEGqmwX375ZYKCgujduzeLFi2y3FkUFBQwdepUy3HJyckEBQURHBzME088AcDIkSNRShEc\nHFzrePN7BgcHM2nSJBd9UiFcY8cOPRjcty+8+WbNYBATo7vn162D+HjXBwOAP8r+AOCqsi60iRkO\n998Ps2Z5XDAAvLtiWvU+5z4cYV15LD8/XxmGYameFh8fbylnaV3VrKysTEVFRVleZ10VLT09XUVG\nRqqDBw+qvLw85ePjo8rKypRSuvrZycfXVy2tser7roVwlX37lJo8Wal27Wr/jo4YodQ337i7hVpm\nfqa66h+oAx3bKvXhh25pA3ZWTJM7BBcyVy4bNmwY/v7+zJ8/35L59OWXX7ZUNfP392fGDNsVkMzd\nQlu3bqVfv341Kp7V954XXnghvXr1Ij09XYrgiBbtyBF49FHdBf/MM3DiRPW+mBhYvVrfFQwd6rYm\n1nDKq2+y+GN4/oEReoGDB5OA4ELFxcU16g2EhYVZSlYWFBTU2FfXRXv06NFMmzaN+Ph4goODWbhw\nYb3vmZ+fX+O8/fr1q1ExTYiWoqpK5xfq0wceeACsK8hedBFkZelccH+zsyJZs6uqghkzOPv1Txl6\nG1QMGtjwa9ysVQQEZ3caNVZwcHCNymWlpaV06tQJ0AHAel9dldAKCwuJjY1l69atrF+/noULF1rq\nFygbjbNVLW3p0qWN/xBCuMH//R8MHqzTSFhXte3bVxen+e47vY7AYxw9Cn//O6xeTcrcWPKDT1qD\n4KFaRUDwFGPGjGHRokWsWLGC0tJSJk6cSGJiIgCJiYksWrSIvLw8SktLmTdvns1zZGRkEB8fT2Fh\nIVWmpOxlpiQstmYzJScnW85bUFBAfHw8JSevzhHCQ+3apSfk/O1vsHZt9fauXeGll2DTJrjxRueN\nzxYWFjJ27FiGDRvG2LFjKSwsdPwke/fq6jlt20JWFpvVHqCePEaexJ6BBk960IhBZXeyHlRWSqml\nS5eqiIgI5ePjoxITEy0DwkoptWjRIhUYGKgiIyPVokWLLAPD1oPESikVFxenfHx8VFBQkJoxY4Zl\ne3x8vPLx8VF5eXk1jjef18fHR91xxx1N/kye+l0L73HihFJPPaVUx44178/9/JSaMUOpgwed/54F\nBQUqIiJCAZZHRESEKigosP8kGzcq1auXUikpSlVVKaWU6vl0T8Vs1O/Fvzu/0XbCzkFlt1/gHX20\ntIBgr4KCApWbm2t5npGRoUaOHOnGFtWtpX/XwrN9+61S551Xu7N29GilHLk2OyopKalGMDA/kpKS\n7DvBJ58o1bmzUm+/bdlUXlmufOf4KmajjpUfa6aWN8zegCBdRh6itLSU2NhYS/fPwoULiY+Pd3Or\nhHCdkhJITtbdQ5s2VW8/6yw9aygjAxwoh+6wndaDE1aKiorqf6FSOkdGcjJ88onOT2Q+58GdVKpK\nQjqG4NfGc1Yk10VWKnuI/v37M3PmTMLCwjAMg8TERMaPH+/uZgnhEkuX6gRze/ZUb+vQAebMgX//\nG9q1a/42hIaG2tweEhJS94uOHdOr4jZt0iPbJ5W7bLAOgoeROwQPcv/993PgwAGKi4t54YUX3N0c\nIZrdrl26DOWYMTWDwbXXwi+/6EW9rggGAKmpqTWmaANERESQmppq+wVFRXDZZTpr6erVNmsf18hh\n1AK4/A7BMIzRQCkwQCn1RD37w5RSi13dPiFE81NK5xy6+24wLcUBdIWy//1PF6x3dWaHsLAwMjMz\nSUlJoaioiJCQEKakTOF4x+Ns2b+lxrGnrNtA6O2TKb3tJoonT4QjO+BI7XPm7soFWs4dgktrKhuG\n0R99oV9mGMYEYK1SasNJ+1FK5RmGEQsUW+83HaNstVnq/LqOfNeiKfbsgUmT4MMPa26fMEFnIDUt\nzXG7VdtXcdU7V3H4xOEa28eth0ezYdz1sLyvfed64aoXuCP6jmZopX08taZyIvCV6ecCYASw4aRj\n5gMjgXClVLYL2yaEaGbLlumx1/37q7eFhcHixXrqvqfYdWgXCRkJHD5xmB4BPWjfpj1tKxQPZOwh\neutRbr03lMKu7bAnHgR3CObavtc2e5udwdUBoRNwwOp5sPVO051BgWEYBwAZURXCSxw6BJMnw6uv\n1tx+xx36ruC009zTLlvKK8tJyEhg9+HdxPSKIfPmTNrs2qMHOrqeDZ+9yef15A9ryTxqUNkwjACg\nBHgUWGQYRi+3NsgJnFGsxqeJdf2kYI5wpzVroH//msGge3f48ktd1tKTggHAtKxprN6xmpCOIbw3\n+j3afLMaoqP1SPeyZeClwQBcf4dQAgSZfu4EFJ+0fyLwmFLqoGEYBcAY4MmTTzJ79mzLzzExMcTE\nxDRHW53CGcVqXFVgRwhnqqqC+fMhJaVmjYK//10HAjfXk7fpzR/f5Ok1T9PGpw3pY5bQdWEaPPGE\nLrQwcqS7m2e3nJwccnJyHH+hPavXnPUA+gPjTT9PAfqZfg4w/fd+88+m5+NtnKO+lXgeJzc3V0VF\nRVmeT5w4UQUGBqqgoCD1+OOPW7anp6dbUlokJCRYUlrExcUpwzBUUFBQvccVFBSoqKgo9fjjj1vS\nX+Tl5dVqg/n1QUFBlloMjvLU71p4jl27dE0C65XG/v41FvF6nLc3vq185vgoZqNeypqv1I03KhUd\nrdS2be5uWpNh50pll94hKD1GEGWaQVSiqmcQZQHRSqknDcOYYhhGPhCknDTt1JjjvPlrapbjs2vM\nf+FnZGSwYsUKtm/fTn5+PlFRUSQnJ7N//34SEhLIzs4mKiqK8ePHM3XqVF566SW++uorfH19KS4u\npqCgoM7jAHJzc7nppps4cOAAkyZNYtq0aXz55ZeWNhQWFjJx4kRWrlxJQEAAcXFxLFu2jFGjRjnt\n+xEiKwuSknSON7MhQ+Cdd6Cnh86+fPend7n5g5upUlW8FDqJ5EmL4PLL4d13ParmcXNz+ToEWxd5\npVS01c+11iZ4C+viNv3797cUt1m4cKGlcA7A/PnziYqKslzozawL7Ng6LjAwkPvuuw/QWU4nTpxo\nea1SioyMDEuxHMBSnEcIZ6ishLlzYfbs6jTxhgEzZ+ptbVx8tflh5w+8t+m9BqdIHyk/wuK8xVRV\nVfHRX9dy3YwMePbZGikoWotWkbqiMX/VN4fRo0dTUlJiSUE9Y8YM7r//fpuFc8w5jaw1dFxQUFCt\n11grKCggKirK8lwK5Qhn2bcPxo6Fr76q3ta1q1585q46Bbd/dDub922269hTj8O3a8/nwl0F8M03\ncPbZzdw6z9QqAoKnMBe3GT9+PNu2bWPEiBEMGDCg3sI51uw9ri62iuUUFBQwevToRn4iIXQB+1Gj\n4I8/qrfFxOjeljPOcE+b9v21j837NtO+TXvmDp9b77HB+bsYNetdTr00Cj5aoJMotVISEFwoIyOD\n999/n/T0dJRSGIZBaWkpY8aMITIykoSEBAYMGMDEiRNJSEio9Xpbx5kL7IDtimlmhmGQnJzMgAED\nSExMJCAggPj4eKZPn94sn1W0Dq++CnfeqdP5mM2cqZPSubqLyNrqHasBGNx9MPdcfI/tg5TSK+Jm\nvqazld56qwtb6Jk8ah2Ct5syZQpBQUFERkYSHR1NfHw8o0aNIiwsjPT0dCZOnEhwcDA+Pj41pomO\nHj0aX19fm8dZV1ZraHpqr169mD9/PrGxsfTu3ZuRI0dKRlXRKCdO6EVl48ZVB4OAAFi+XI8juDMY\nAKzasQqAoT2G2j7g4EG46SadOGnVKgkGJi7NZeQMksvI/eS7bt327tWLdletqt523nnwwQcQGem+\ndlmLXhTNuqJ1ZN6cyYjwETV3/vCDDgZxcfD009C+vXsa6UL25jKSOwQhhN02bNCLdq2DQWKiXo3s\nKcHg8InD5O3Kw9fwZXD3wdU7qqp0noxrr9X/femlVhEMHCFjCEIIuyxbpgveHzGleTYMeOwxmDrV\n9amq6/PdH99RqSqJDonmtHamvBi7duluoaNHYe1am7ULhNwhCCEaoBTMm6cL2ZiDgb+/rhY5bZpn\nBQOwMX7w8cc6mdKQIbBypQSDesgdghCiTidO6HTVr79evS0iQgcDT52qbw4IMV0G6SlQn3+ua3T+\n7W9ubpnnk4AghLCppESvL7DOkXbppbrrKDi4zpe51fGK46z5cw0DiuDKvz8IAwfpgY+AAHc3rUWQ\ngCCEqGXbNrjqKl3X2Oy22/Q4rKtqHDfG+j9/YPLKY9z/vS9tXn5YzyYSdpMxBCFEDevWwUUX1QwG\njz0Gr7zi2cGAwkLOvP4WRubDE88mSjBoBAkIzWzp0qU2Vx3bo6mFcYRw1Oefw2WXVWcqbddOp6CY\nPt3zBo8tlNLRatAgvjy/PbG3wPkDr3J3q1ok6TJygcYWuGlqYRwhHPH66zB+fHUxm8BA+OgjGFrH\nYl9nW/PnGp77/jkqqirsfk1A6TEmvryezvv+4vnpF7Hw+Leochja00WN9jISEFxAKUVCQgIZGRlE\nRUWxZMkSwsLCAMjKymLSpEmUlJQwYsQIFi1ahL+/PyNHjkQpRXBwMMXFxbz88stMnz6dsrIyBgwY\nQHp6Or169XLvBxNewTytdObM6m09e8IXX8BZZ7muHQ+ueJDswmy7jx/1M6R+Cq/1h9k3w4nDOtVq\n3+C+9AiQqaWNIakrmtnSpUuJj49n0aJFxMfHM3XqVNatW8e6desoLS0lPDycpUuXEhUVxbRp0ygu\nLmbJkiWA7jKqqqoCwNfXl9zcXMLCwhg/fjwRERE89thjbvlMnvpdC8dVVcF998Ezz1Rvu/BC+Owz\nCAlxXTuUUgQ9HkTpsVJeue6V6gVlNrQrO0z/R18laFMB3z96Jwcu7FNj/8XdL+bMgDObu8ktir2p\nK1rPHYKzul8acSGMiopi3LhxALz00kv4+vpy8OBB0tPTiYuLsxS8eeyxxwgPD7dqcnWbzcV0QNc9\nKC0tbcqnEILycrj9dkhLq942bJjOSeTqWZrbSrdReqyULqd24bZ+t9XdXbp8OUy6V8+HXfIVI1px\nqurm0HoCghv/orW+yIMubFNQUEB+fj7p6elkZWUB+q+kgwcP2jzH3Llzyc7OJtg0AfzkcwrhiKNH\nISFBX1/NRo/WBW3cUTEyd1cuAAO6DbAdDEpLYfJkXbwmLU0XXBBOJ9NYXCA/P7/G88LCQsLDw4mI\niCA+Pp7i4mKKi4s5cOBAjQI2ZuZazCtXruTLL79kzJgxrmq68EIHD8IVV9QMBhMmwPvvu698sCUg\nnDGg9s5PP4Xzz9eFazZulGDQjCQguEBeXh7Lli2jtLSU5ORk4uLi8Pf3JyEhgaysLLKzsy37kpOT\na72+pKSEoKAgOnbsSGlpKQsXLuTAgQNu+CSipSsu1iUtv/mmetuMGbBwIfj6uq9debvzAFiVvoph\nw4YxduxYtm/YAP/8J9x9N7zxBrzwApxW99iCcAKlVIt66CbXVtd2d8vIyFCTJk1SI0eOVIZhqMsv\nv1yVlZVZ9mdnZ6uIiAjl4+NTa198fLzy8fFRSikVFxenAgMD1cCBA1V2drYKCgpS2dnZLv88Snnu\ndy3qV1Sk1LnnKqX7T/Xj8cfd3SqlqqqqVPC8YMVsFIEoQF0Hapevryq7+WalDh1ydxNbPNPvbIPX\nV5llJBwm33XLs2OHvjMw90gaBrz4ok5c525Fh4oI/W8oHIPT58FzwEBgPHBmUhJp1qPeolGkQI4Q\nAoDCQr362BwMfH31uKwnBAMwjR8o+Pu38BOwE7gQ+AYoKipyb+NamdYzy0iIVui332D4cNi5Uz9v\n2xaWLIEbbnBvu6xt/XEln7wDPXfBdcBaq30hrlwMIeQOQQhv9csv+s7AHAz8/HQqCo8JBlVV8MIL\n3Dbuf3zfHa4c1LlGMIiIiCA1NdVtzWuNZAxBOEy+a8+3ebO+MzAnqevQQRcOi411b7ssfv5Zz3UF\nRgwtJLv9Lr667iveeOoNioqKCAkJITU11ZLiRTSNvWMIEhCEw+S79mybNulgsG+ffn7qqToVxaWX\nurddABw/rnNpL1gAc+aw/5YxdH6qKx3aduDg9IP4+rhx7qsXa3WpK3r27CnZQV2kZ8+e7m6CqMNP\nP+lgsH+/ft6xo05p7RHVI7/+Wo9kn3UW5OVB9+7k5WcCcGHXCyUYeACvCQjbtm1zdxOEcCvznYF1\nMPjyS7j4Yve2iwMHYOpU3ZjnnoMbb7Tssk5ZIdxPBpWF8ALmMQNzMPD3h8xMNwcDpeCtt+Ccc6B9\ne91Iq2AA1SuUJSB4Bq+5QxCitfrll5pjBh07wldf6TKYzSn/QD7//e6/HKs4Vmtf151l3PzS/9Hh\nrxO8ef8QtvU+AivvqXVcVoFO7CgBwTN4zaCyEK3Rb7/pqaW7d+vnruwmuvPTO3lx3Ys1tvmVw4zV\ncNcPMPdSeH4QVDYwNBDgF8DeKXtp5+vJBZtbtlY3qCxEa5Ofr+8MzMHgtNN0lTNXdRMVlhYC8J+L\n/sP5Xc4n9LvNXDz/bQ70OZPMpYmc1zWIhXacJzo0WoKBh5CAIEQLtH17zRXIHTroqaVDhriuDX8e\n/BOA8V2u5Lz5r8LatfDiqwRcfTWyeqBlkkFlIVqYnTt1MNixQz8/5RT45BMY6uK68rtL/uCe/4Nz\n4v4BffroaU5XX+3aRgincvkdgmEYo4FSYIBS6gkb+/sD4QBKqaUubp4QHm3vXhgxAgoK9PN27eDD\nD3WAcKUj2V+w4tky9nY0ML79Vq8tEC2eS+8QTBd7pZTKBkoNw+hn47AZpkAQVsd+IVqlAwcgLg62\nbNHP27SBjAy4/HIXNmL3brj5ZtreejtzLoPku8MwJBh4DVd3GSWi7w4ACoAR1jtNdw8/ACilnlRK\nbXBt84TwTOaylxs36uc+PvDOO3DttS5qQHk5PP20LmUZGsqqz15i6blwZqceLmqAcAVXB4ROgHXt\nx+CT9kcDwYZh9DcMY4rrmiWE5zpyRF/415pSgRoGvP46xMe7qAE5OdC/v86BsXo1zJvH9spiALr7\nd3dRI4QreOIso2KlVJ5hGCMMwxhtaxxh9uzZlp9jYmKIkaLbwksdPw6jR9esgfzii3DzzS548z/+\ngPvvhzVr4L//hVGjdDQC/jj4BwBn+p/pgoYIR+Xk5JCTk+Pw61wdEEqAINPPnYDik/YXo7uSQHct\nDQTqDQg7krKjAAAgAElEQVRCeKuKCkhK0msLzJ56ygWVzo4d0wHgqafgrrvgtdf0vFYr5imncofg\nmU7+Q3nOnDl2vc7VXUZLMM0gMv03C8AwjADTtgyr/Z2oWTxJiFajqkqXC1hq9efQrFlw773N+KZK\n6aIJ554LP/yg+6gefrhWMAC5Q/BWLr1DMHUFRRmGEQuUWA0aZwHRSqlCwzBKTYPLQbampQrh7ZTS\nF/7XX6/eds89OiA0m19+gcmTdTfRSy/p6Uz1kDsE7yS5jITwMHPmgHWv6LhxsGiRpfveuUpK9Bu+\n/TbMnAn/+pcuvNyATvM6UXa8jH1T9nF6h9OboWHCmezNZSQrlYXwIM89VzMYjBkDCxc2QzCorNQn\nPussOHpUl7S85x67gsGh44coO17GKW1OIbj9yRMFRUvmibOMhGiV3nwT/vOf6ucjR0JaGvg6u5DY\nihW6eygwUKdG7efY+k/r7iKpUuhdJCAI4QE+/hhuv736+cUXw7Jl4OfnxDfZuhWmTIEff4Qnnqgx\njdQRMn7gvaTLSAg3y8mBhATdiwN6MfCnn8KppzrpDcrKdCAYPBgGDdLdQ6NHN7ofSgKC95KAIIQb\n5ebCddfpBWgAERG6Fycw0Aknr6jQq9j69tWDx5s2wYwZOj1qE8iUU+8lXUZCuMlvv+n8RIcO6efd\nuuk6yN26OeHkX34J990HXbrolW0OjhPUR+4QvJcEBCHcYOdOPdXfXAe5UyddBzmsqZVlNm3S3UP5\n+fDkkzoJkpMHfuUOwXtJl5EQLnbggJ5BZC5wY652dt55TTjp7t0wcaIujHDFFTowXHddsyxekDsE\n7yUBQQgX+usvuOYaPa4LuqbBggW7WbBgLMOGDWPs2LEUFhbaf8IjR+CRR3S6iY4d4ddf9dzVds1X\no/iPMtMdQoDcIXgb6TISblVYWEhKSgo7d+4kNDSU1NRUwprcb+KZyst1yurvvtPPDQOefHIvjzxy\nCfn5+Zbj1qxZQ2ZmZv3fQ2WlXqTw4IN6juoPP+gR6WYmi9K8mwQE4TaFhYXExcU5fjFsgaqq4Lbb\ndEkBs2efhe+/v7fG5wfIz88nJSWFtLQ02yfLzNTjBB06wJIlOiA0kb2BWRaleTcJCMJtUlJSHL8Y\ntkBK6Qk/b79dvS0lBe6+G5Yt22nzNUVFRbU3/vRT9YDxvHmNXlh2MkcCs4wfeDcZQxBus3OnAxfD\nFmzePHjmmernyck6nxxAaGiozdeEhIRUP/nzT317MWIEXH01bN7cpIVlJ6svMJ9MZhh5NwkIwm3s\nuhi2cIsX6ySiZqNHw4IF1dfy1NRUIk7q+4+IiCA1NRVKS/VCsgsvhJAQvXDh7rudPmDsSGCWOwTv\n1qiAYBjG/YZhvGgYxnjDMPyd3SjROtR7MfQCH35Ys7rZ8OG628g6WV1YWBiZmZkkJSUxbNgwkpKS\nyFy+nLAPP4Q+fWDPHp17aO5cCAio/SZO4EhgtswwkjsE76SUavABDAf6mX6eZ/X4CqgExtlzHmc8\ndJOFtygoKFBJSUlq2LBhKikpSRUUFLi7SU7x9ddK+fkppUcQlBowQKmysgZeVFmpVFqaUr16KXXN\nNUr99JNL2lpQUKAiIiIUYHlERETY/H9xRdoVitmoj7d87JK2CecwXTcbvL42WCDHMIzhQLhSarHp\neY3C94ZhdAKmAz8opZY5M1jV0R7VUJuFcKcff4TLLtM55QAiI+Hbb3UWCZuU0jOHpk3T6U3nz9cn\naEYVVRVM+GQCv+7/FYDjx4+zY8cOTpw4Qbt27ejRowd+NlKtbtyzkb/K/yJ3Yi79u/Vv1jYK57G3\nQI49s4yigMy6diqlSoHphmFMcKB9QnilggK9UNgcDM44Q6ekqDMYrFsH06fr0pWPPuq0mUMN+XbH\nt7y+4fWaG9ubHsD+ffvrfK2/nz+RQZHN1jbhPg0GBKXUE4ZhTDAMo1QptQ0oMAxjLTAVWK+UOmg6\ntLgZ2ymEx9uzBy6/XGeRAN3l/+WXdeQn+v13eOABfevw0EO6GIId1cqc5fud3wMQf048kwdPdui1\n4YHhdPTr2BzNEm5m1zoEpdQiq6eJwBLgDmCEYRj5wAEg0zCMjkqpQ4ZhjHJF95EQnuLgQbjySl2D\nBnSG6U8+gQsuOOnAoiJ4+GHIyIB774XXXnNi4QP7mQPC1b2vZsiZQ1z+/sIzNWaWUT6QrpRKUEoF\nAROBLGAksN0wjN+B+U5soxAe7dgxuOEGyMvTz3184L33YOhQq4PMU0jPP78659DMmW4JBgDf/6kD\nwkXdL3LL+wvP5HBAMN0tBBqG0cv0PE8p9YRSaqQpQCQCDmTnEqLlqqyEpCRYubJ626JFcP31pidH\njuhB4t69da7rDRt0+cpg9+UB+vPgn+w8tJMAvwD6BPdxWzuE52lU6gqlVF49+3INw5jW+CYJ0TIo\nBXfeqWsfm82bZ6qNXF4Or7wCqakwZAisWgVnneW2tloz3x0MCh2EjyFrU0W1ZsllVF/AEMJbPPgg\nvPxy9fN77oGp91fBO+/pgeLwcL06LTrafY20wTx+MLj7YDe3RHgaSW4nRCM884yeJWo2NknxZMyn\nGAMe0CPKL7+slyZ7IHNAuChUxg9ETQ0uTPM0sjBNuFtaGtx8c/XzaRfl8KgxE59DB3Wxmuuvd8la\ngsaoqKogYF4AR8qPsPf+vXQ+tbO7myRcwJkL04QQJp99phOPAkTzAwsCHiBqbwE+qQ/D3/9eM1GR\nB9q0dxNHyo8QHhguwUDUIgFBCDutXq2zlfat2EQqKQxps5bTZqXg8y/XLiprCvOAsowfCFtkioEQ\ndti4Ee6+ciuLjo0lm1h+DryEyi2/c+o9yS0mGACs2bkGkPEDYZsEBCEasH31H2wcPJHMw4P5lb5c\nfPpWEr6/j5CI9u5umsMsC9IkIAgbpMtIiLrs2cPhBx7D/7W3+KNqIn34jUr/IL7O1OvMWpqyY2Vs\n2b+Fdr7t6HdGP3c3R3ggCQjCqZRSfPzrx5bKWi2RX9lh+qdlc86H3/Lu6YN48IL72dvWnzZt3+Vf\nd8G3J+DbH9zdSscVlhaiUPQ/oz9+bWqnthZCAoJwqnVF67jh/Rvc3YxG8T8Gk9fA3d9Dxjlwwzj4\nMyAHyAGgAnjmd+B397XRGS7ufrG7myA8lAQE4VS/H9BXyz7BfRgRNsLNrbGP39FyLl3+E7Ef/sjm\nqB488+xA3lgXwJ+/mg4wdH37Pi2wm+hkHdp24J6L73F3M4SHcnlAMAxjNFAKDFBKPVHPcVPq2y88\n0+7DuhjAlZFX8swVz7i5NQ04ehRefBEefxyGDYM1rzEw8iweT4A/P6g+7H//g7vucl8zhXAVl84y\nMgyjP7q2ZzZQahiGzZEtwzBigZbx56WoYdehXQCccdoZbm5JPY4f11f5yEi9uCAzE959l6o+ZzFu\nHHxgFQwefliCgWg9XD3tNBF9dwBQgFz0vc6uwzogdDutm5tbYkN5uc4x1KcPfPGFrmCzbBmcfz5K\nweTJ8Oab1Yffd59OYCdEa+HqgNAJXV3NrFZSeMMw+pvuIDwzGYyolyUgdPSggFBRAa+/Dn376kpl\n778Py5fDgAGWQx58EJ5/vvolEybosgUempJIiGbhiYPKge5ugGg88xiCR9whVFbq0mVz5kC3bjoo\nXHpprcMefbRm5tLERD20IMFAtDauDgglQJDp505AsfVO093BCtPTOlOazp492/JzTEwMMTExTm2k\naDyPGEOoqtJ3ArNnQ6dO+uo+fLjNK/xzz+la92bXXANvveXxOeqEqFdOTg45OTkOv86l6a9Ng8pR\nSqnFhmFMATKVUhsMwwhQSpWZZiApdFdSMjBeKbXhpHNI+msPdaziGO3ntqeNTxuOP3jc9dW4lNIj\nwrNmQfv2+s7giivq/FP/lVdg/Pjq58OHw6ef6nIGQngTj0x/rZTKMwwjyjSLqMTqYp8FRCullgIY\nhjEBCHBl20TT7Tm8B9B3By4NBkrpMYFZs/Tzxx6Dq6+ut88nLU2PE5gNGQIffSTBQLRuLh9DUEot\ntrEt+qTni4BFLmuUcArzgLLLuouU0rOFHnoITpzQXUQ33NBg539GBtx6q345wHnnHSc09G6uvfZ3\nQkNDSU1NJSwsrPnbL4SH8cRBZdFCmccPmn1AWSm9duChh+DQId01NGoU+DR8V/LJJ3DTTXqYAaBv\n3xMcPvw30tPXW45Zs2YNmZmZEhREqyPpr4XTNPsMI6UgOxuGDoV//1svHNi4EcaMsSsYfP65PrSi\nQj8/6yw499zJbNu2vsZx+fn5pKSkNMcnEMKjSUAQTtOsXUZffw0xMTBpEiQnw+bNDpWs/OoruPFG\n3bMEEB4OWVlw4MAvNo8vKipyUsOFaDmky0g4jaXLyJmL0lav1oPF27dDSgokJUEbx/7Zrlih694f\nP66f9+oFK1dCaCiEhobafE1ISEgTGy5EyyN3CMJpdv/lxC6jNWtg5Ei4+WYdBH75RY8EOxgMcnLg\n2mvh2DH9vEcPHQx69NDPU1NTiYiIqPGaiIgIUlNTm/4ZhGhh5A5BOI1TFqX98IO+I/j5Z51P4tZb\noV27Rp0qJweuukonNQXo3l3fLfTqVX1MWFgYmZmZpKSkUFRUREhIiMwyEq2WBAThNE3KY7R+vQ4E\nP/6olw5/+CH4Nb6q18nBICREB4OTbgYAHRTS0tIa/V5CeAvpMhJOUaWqLAvTup7a1f4X5uXpDv7r\nr9erirdu1QPHTQgGK1bUDgY5OS2zDrIQriQBQTjF/iP7qVSVBLUPsq9e78aNMHq0XlEcG6sDwb/+\n1aRAAHo20dVXSzAQojEkIAinsHtR2qZNEB+vB4yHDNGB4N//dkrOiM8+g+uuqx5ADg2VYCCEIyQg\nCKewLEqra/xgyxa9RDg2FgYNgvx8XYGmQwenvP/HH+usFeappT166KULEgyEsJ8EBOEUdS5K+/13\nPXX00kvhwgt1IJgyBU491Wnv/f77uvepvFw/79VLBwNbA8hCiLpJQBBOUavLqKAAbrtNdwv16aO7\nhqZPh9NOc+r7vvYa/OMf1ekoIiLgm29qTi0VQthHpp0KpzB3GfU55KfzSn/wgR4k/v13XaSmGSxY\noN/C7OyzdToKWWQsRONIQBBOcXx7Pi8sh5uffgbu+jf89hsEBTX8wkZQSpe8fPDB6m39+ukZRp07\nN8tbCtEqSJeRaJpdu+Df/2b+/V9y0A/WrUiDuXObNRhMmVIzGFx0kV57IMFAiKaRgCAaZ+9ePUvo\n3HOhTRsun9Gd6XFweo+zmu0tKyt1b9RTT1VvGz5cl0YIDGy2txWi1ZCAIBxTXAwzZugO++PH9bqC\n//6XTcY+oPmqpR07ppcvvPJK9bYbbtA1kDt2bJa3FKLVkYAg7FNWpnMN9e0LBw7olBP/+x+EhHDo\n+CH+Kv+L9m3a4+/n3yxvfcUVepza7NZbIT1daiAL4UwSEET9Dh/WI7i9e8OOHTob6cKF1fmjqbko\nzWignrGjdu2Cyy7T6wrM7r0XXn3V4UzYQogGSEAQth09Cv/9L0RG6m6hVav0pP/w8FqHNleltC1b\n9DKGH3+s3vb443oMwY6KmUIIB8nfWKKmEydg8WI9U+iii/TE/vPOq/clducxcsDq1TovUUmJfu7r\nq5v1z3867S2EECeRgCC0igp46y2YM0cPGH/8MURFcaLyBBXlR+p96Y6yHYDzAkJGBowdW52XqEMH\nWLJEZzEVQjQfCQitXVWVvtrOmgXdukFaGlxyCQAf/PIBiRmJlFeV23WqpnYZKQVPPglTp1Zv69JF\nzyQaOLBJpxZC2EECQmulFCxfrld4nXKKnjE0YgRYDQq/t/k9yqvKaefbDl/Dt97TdTqlE1dEXtHo\n5pSX6zQUL79cva13b/jiC5vDFkKIZiABoTVauRJmztQziB55RHfW25gdtObPNQDkTszl3C7nNltz\nSkshMVGnnjC75BI9zfT005vtbYUQJ5GA0JqsXavrFefn67GCm27So7U2FB0qYkfZDvz9/Dm789nN\n1qTff4drr4Vff63elpSkF6A1sXiaEMJBMnmvNfjlF10w4MYb9X+3bNGjtnUEA6i+OxgUOggfo3n+\nmWRn64lM1sFg1iw9ti3BQAjXk4DgzXbsgNtv1yu7Bg/Wf44nJ0Pbtg2+1BwQBocOdnqzlNKpqy+/\nvHpa6SmnwDvvwOzZNnuvhBAuIF1G3mj/fr26+I03YNIknYrawZoE5oBw8ZkXO7Vpx47BnXfqNW5m\n3brBRx9BdLRT30oI4SC5Q/Amhw/Dww/rfEPHj8PmzXqBmYPBoLyynHVF6wC4KPQipzVv5059s2Id\nDAYO1EMbEgyEcD8JCN6gvBxeeEHP09yyRecbWrAAzmjcuoCNezZytOIovYN6E9wh2ClN/PpriIrS\nTTO75RZd7jI01ClvIYRoIgkILZlSelHZOefoPpfPPtMd8U2sLm8ZP+je9PEDpeCJJyA2Fvbs0dt8\nfeGZZ+D116F9+ya/hRDCSWQMoaXKydFLeisr4aWX9BXXSdbsNI0fdG/a+EFpqR7Ttk5b3bkzvP8+\nDBvWpFMLIZqB3CG0NJs3wzXX6CvtPffoDvhGBoPCwkLGjh3LsGHDGDt2LIWFhYBz7hDWroUBA2oG\ngyFDdBkFCQZCeCaX3yEYhjEaKAUGKKWesLF/gunHCKXUdJc2zpMVFelJ+h99pFcZL13apMn6hYWF\nxMXFkZ+fb9m2Zs0a3vvkPbYe2Er7Nu05v+v5Dp9XKXjuOV33uNwqBdJ//qNTV7dr1+gmCyGamUvv\nEAzD6A8opVQ2UGoYRr+T9scCmUqpRUC4YRjDXdk+j3T4sJ6cf/75unDwr7/C5MlNXrmVkpJSIxgA\n5Ofnc/8z9wMQHRpNGx/H/l7Yv1+XtZw8uToY+PvrymbPPCPBQAhP5+ouo0T03QFAATDipP3hVtsK\nTM9bp8pKnb+hb1+9jmD9ev0ntpOqye/cudPm9m3l2wDHF6RlZcEFF+is2WZRUZCbC2PGNLaVQghX\ncnWXUSfggNXzGnMaTXcGZgOA91zRKI+Tna3rRPr76074QYMadZrDJw7z6/5f+fPgn/xx8A92lO2g\noKSA/JJ8fhn6C9i45u/w07UN7F2Qdvy4Tpj65JM1t//nPzB/vqSgEKIl8chZRqaupfVKqQ3ubotL\n/for3H8//PyzvhsYNarReRwOHT9E7+d7s+evPbYP8DU9TqJQdDutG5f1vKzB9/jxR50SadOm6m1d\nuujppFde2ahmCyHcyNUBoQQIMv3cCSiu47hYpdSMuk4ye/Zsy88xMTHExMQ4qXluUlKiVxi/9RZM\nm6ZLhjXxT+uv8r9iz197CGofxMXdL6a7f3d6BPQgPDCc8MBwegb0ZPfO3cydO5ddu3bRrVs3Hnjg\nAXr27MmpbU+lrW/d+Y4qKvQdwUMP1Rw4vuIKHQy6dm1S04UQTZSTk0NOTo7jL1RKuewB9AfGm36e\nAvQz/RxgdcwEq59jbZxDeY3ycqVeeEGpLl2USk5Was8ep536tg9vU8xGPfrNo047p1JKbdqkVHS0\nUno+kX60b6/UggVKVVU59a2EEE5ium42eI126aCyUioPLLOJSlR1l1CW1fZ5hmFsNQyjGFCubJ9L\n5eToUdclSyAzUy8u69LFKaeuUlV8vvVzAK7u45xCxBUV8Nhjem3B2rXV2wcNgg0bdMI6yVIqRMtm\n6ODRchiGoVpam2vYvl2PE6xdq/tdRo92+pV0fdF6Bi4aSHf/7uyYvAOjiedftw4mTNAXfrN27XSX\n0bRp0MYjR6KEEGaGYaCUavBCICuVXeXYMV2ucsAAOO88XbRmzJhm+bP6098/BeCqyKuaFAwOH9aT\nnS66qGYwiI7W00kfeECCgRDeRAKCKyxfDueeq6+i69fDrFkU7t5tM22EM5gDQmO7i5TSs13POQee\nfhqqqvT2U07RU0n/7//0xxFCeBf5+645bdumJ+T/8otOT3355UDdaSMyMzMJCwtr0lvu/Wsva3eu\nxc/Xj9gwx3McFRTA3XfrxKnWRozQwxxNTKQqhPBgcofQHE6c0BXLoqJ0/8pPP1mCAdSdNiIlJaXJ\nb/3F1i9QKGJ6xXBqu1Ptft1ff+kFZuecUzMYnH66Lrz21VcSDITwdnKH4Gxffw133AHh4Xo01sZf\n/HWljSgqKmry21u6i3rb112klC6hMG2armhmZhi6/PLcuRAUVPfrhRDeQwKCsxQX69lDWVnw7LNw\n4411DhiH1lEiLCQkpElNKK8s58utXwJwVe+rGjx+9Wq4776aVcxA39QsWCBlLYVobWTaaVMpBe++\nq6+sCQl6JlHHjvW+xDKGsDcfrgACwM/Pj7PPPhu/JqxQPlZxjB/3/Ejf4L5s+deWOo/77TeYMQOW\nLau5vWtXmDdPl7b0kc5EIbyGvdNO5Q6hKbZvh0mTdF/LRx/ZnYQuLCyM5V8s55LFl1DcXmfvOM5x\nNux3Tuqmv5/3d5vb//xTZ8h49VWdTNXMz0+PfT/wgM6nJ4RonSQgNEZVlZ5yM2uWrlo2ZQq0rTv3\njy1P//I0xe2L6RHQg7dufIt2vs4pFuDn68cFXS+osW3vXl3X+H//08shrP3jH3qcoFcvp7y9EKIF\nk4DgqK1bYdw4ndXtm2/g7LMdPsXi3MW8nPsyfr5+LEtYRlRIVDM0VBesMQeCI0dq7hs+XHcPyTiB\nEMJMeortVVUFzz8PgwfrsmCrVjUqGGTmZ3LXZ3cBsPCahc0SDHbt0jctvXrpLNrWwWDgQJ06KTtb\ngoEQoia5Q7DHtm26qP3Ro3qZbp8+Dp+i6FARUzKn8M5P7wBwV/Rd3NrvVqc2s7BQp0d65RVduMba\nBRfoSpw33CBJ6IQQtklAqI9S8OabejrplCl6JpFv7aoyVaqK5b8tZ/lvyymvKq+1v6Kqgg+3fMjh\nE4c5pc0pTP/bdGYOnem0Zq5fr7uG0tOr00yYnXeeDgQ33igzh4QQ9ZOAUJcDB/TKrC1bdP/KBRfU\nOuSvE3/x2obXePb7Z9l6YGuDp7y+7/U8ffnThAU2LT0F6FlCn3yilzzYqoMRHa1nDV17rQQCIYR9\nJCDYsnKlnowfH6+rmJ1ySq1DCkoKuDztcksg6BnQk+SoZM447Qybp+wd3JtLelzS5KaVlMBrr+mB\nYlv58EaMgKlT9X+la0gI4QgJCNYqKvRE/cWLdS3IkSNtHrZh9wauSLuCPX/t4dzO5zI7ZjY3nHUD\nbXya7+tct07nx3v33dpTR319ITFR92z1799sTRBCeLlWFRAqqyrZUbYDZaMQm+/OXXSZeA+qXTv2\nrfiAyq6doaSg1nFb9m/hpqU3cfD4QYaHDeeDxA/w92ue1VylpTrP0OLFkJdXe39QEEycqKuVnXlm\nszRBCNGKtJrUFV9v+5pJn05iy/7aKR1i8yFtGTx3Ecy7BJQdfe7x58Tz1o1v4dem8akmbKmshBUr\ndIbRpUtr3w0A9Oung0BSEnTo4NS3F0J4IXtTV3hlQDhSfoSj5UcBOHziMLO/ns3rG14HILh9sOUv\neqNKcUd2GTevOsg9N3dmTe/29rw/Ceck8MjwR/D1qT3jqDGU0hmy33kH0tJqZh01O+UUPaRx5526\ngpmMDwgh7NVqA8Kq7auIfTO21vRPP18/Zg6dybS/TdN/1R88qAeO9+7V8zXryEDanLZuhSVLdCDY\nvNn2MRdeqOsZ/+MfEBjo2vYJIbxDq01u98GWDyivKqdD2w6c0kbPDrq4+8U8ffnT9A7urQ8qKIDr\nroNLLtFX5HbOySNkj19/1V1B6ek16xRb69xZB4BbbtGDxHI3IIRwBa8LCN/9+R0AHyZ+SFxEXO0D\nvvlGT8l54AG4665mv9pWVekZQh9+qOsUb6kjK3X79nD99XDTTXDllQ7nyhNCiCbzqoBwvOI4ubty\nMTAYFGojFfVbb+nVxm+/DXE2goWTlJXptWzLl8Onn+peKVvatYMrrtBlFK6/Hk47rdmaJIQQDfKq\ngLBh9wZOVJ7gnM7nEHBKQM2dTz6pk9Pl5OjCwU5UWQm5uTpp3Bdf6HRH1vUGrLVvr8srjxmjVxFL\n/QEhhKfwqoBg7i66uPvF1RurqnQeoi++gG+/he7dm/w+SsHPP+vYkp2tFzaXltZ9fOfOcPXVOrFc\nXJxMFRVCeCavCghr/lwDwODug/WGykpdu2DrVp2uupHV4isq4McfdQ3iVav0MMS+ffW/JipKdwdd\ne63OKyT5hIQQns57A0JVFYwfDzt2wFdfOfRn+f798P338N13+vHDD3D4cP2v6dYNYmN1d9DIkdCl\nS1M+iRBCuJ7XBIRdh3axvWw7/n7+nBN8ls7pUFioR3XrCQalpTotxPr1sHatfthKGney4GC47DKI\nidGJ5M46S6aHCiFaNq8JCOa7g0Eh0fjc9S/47Tf47DM49VRA9/vv2AEbN+r5/z/+qP+bn2/f+bt3\nh6FD9dKFoUPh3HOlG0gI4V28JiCYB5TvW604sWYtqx7O4afFp/Hzz7Bpk34cOmTfudq10wvCBg+G\niy/WjzPPlDsAIYR3a7EBobQUfv+9+vH6kTVcuwPOz/iR8IoN7Lyxo13nadtWVxUbMEAPBA8aBOef\n79LFy0II4RFaZEC45Ra9xszCp5zzxv3A4o/hGt932Vlhe2ppUJC+2IeFlbFly/uUl6+nd+/jPPro\nLMLCml7FTAghWrIWGRDOOKkoWWDnVXycfpzJl3Rl7VdxBAbqtWfWj/PP16/btq2QuLg48k2DB3ow\neTWZmZkSFIQQrVqLDAi9e4OfH0RG6p8Td88j6wQcGHU5e9Pg9NPr7u9PSUmxBAOz/Px8UlJSSEtL\nc0HrhRDCM7XIgHDrrXq9mY8PsG8fh8O/5pzxMGPAYDp3rv+1O20VGwCKioqc31AhhGhBXD5x0jCM\n0YZhxBqGMaUx+0EP+FqmfD71FF9f1JU/OlmtUK5HaB11D0JCQuxovRBCeC+XFsgxDKM/EKaUWmYY\nxieKz7AAAASsSURBVARgrVJqg737TcdUF8jZvx/69IENGyjt4k/Hdh0brGJWWFhzDAEgIiJCxhCE\nEF7L3gI5rr5DSATMaeAKgBEO7q/pqad0bYMePeh0Sie7SlqGhYWRmZlJUlISw4YNIykpSYKBEELg\n+oDQCThg9TzYwf3V9u8nZ8ECmDHD4UaEhYWRlpbGihUrSEtLa/HBICcnx91NcDv5DjT5HjT5Hhr3\nHbTc5AvPP09Onz7Qo4e7W+J28o9fvgMz+R40+R4a9x24epZRCWDOQd0JKHZwPwCzZ8+G8nJy/PzI\nyckhJiamGZoqhBAtU05ODjk5Ofpa6QBXB4QlQBSwAggHMgEMwwhQSpXVtf9k5g85e/ZsCQZCCHGS\nmJgYYmJiLNfKOXPm2PU6l84yAjAMYzxQiJ5NtNi0ba1SKrqu/Se93rUNFkIIL2DPLCOXBwQhhBCe\nqeUOKgshhHAqCQiixbFnNbvpuHr3C+FNTAt769pn1+9MiwkIzkh50dLZ8R1MMD3mubptrmL6R6+U\nUtlAqWEY/eo4LpaGFja2cHb8e+hvOma0q9vmKg5cF8a7um2uZPr3nl7HPrt+Z6CFBISGPpAjH7il\nsuM7iAUylVKLgHDDMIa7o50u4Nhqdi9l57/5GUqppUBYK/2d6A8UmPYXeuN3YGb6jHUVBLb7d6ZF\nBAScnfKiZWroM4ZbbSswPfdGDa5mNwyjv+kXxJuLntb778F0V/ADgFLqyZNzgnkJe37v55v+G+6l\n34E97M4A0VICgvNSXrRc9X5GpdQiq2m6A4B1rmqYBwp0dwNcoKF/89FAsKnbyFu7URv6ncgDCgzD\nOEAdi1xFTS0lIAg7mW6T13vxX0P1rmY33R2sMD1t7XOqi00XRfMdQ6tiGEYA+t/Lo8AiwzB6ubVB\n7mNXBghoOQHBKSkvWjh7P2OsUsrxjH8txxKqu8PCgSyw/PKDHj8ZZUqfHuzF/cYN/XsoRnejgO5W\nGeiidrlSQ9/BROAxpdSTwARgjAvb5g41ukitfids/s7Y0lICQkMXAbs/cAvW0HeAYRgTTP/4zYPM\nXsfqL95YoMTqTijLtH+pUmqZaVuAjVN4i4b+PWRY7e8ErHVp61yjoe9AYbpImv5NlJ58Am9hugOM\nMgxjlNVm8+9EXb8ztc/TUlYqNzXlhTeo7zsw/c9egv6rKRCIt+o6EV7Izt+JEmCgt9412vEdTEHP\nvgny1uuCM7WYgCCEEKJ5tZQuIyGEEM1MAoIQQghAAoIQQggTCQhCCCEACQhCCCFMJCAIIYQAJCAI\nIYQwkYAghBACkIAghBDCRAKCEEIIANq4uwFCtHSmZGoJ6GybwUCWUqrQva0SwnFyhyBE0y0C3jdl\n1CwGWl3tAeEdJLmdEE1kGMZWdLrp982phoVoieQOQYimG4MpH79hGN5Yd0C0EhIQhGgCwzBGK6U2\nKKUSlFLBQGArLtUoWjgJCEI0kmkwefpJm/OVUtvc0BwhmkzGEIRoAlPJwlJ0lbpAYIlS6qB7WyVE\n40hAEEIIAUiXkRBCCBMJCEIIIQAJCEIIIUwkIAghhAAkIAghhDCRgCCEEAKQgCCEEMJEAoIQQghA\nAoIQQgiT/wcT6E002EswcwAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from betacal import BetaCalibration\n", "\n", "# Fit three-parameter beta calibration\n", "bc = BetaCalibration(parameters=\"abm\")\n", "bc.fit(cal_probas.reshape(-1, 1), y_cal)\n", "\n", "pr = [lr.predict_proba(linspace.reshape(-1, 1))[:, 1], iso.predict(linspace), bc.predict(linspace)]\n", "methods_text = ['logistic', 'isotonic', 'beta']\n", "fig_map = plot_calibration_map(pr, [scores, y_c_2, linspace], methods_text, alpha=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Beta calibration was able to find the inverted sigmoid shape, as isotonic regression did, resulting in a much better fit for the scores than logistic calibration. Now let's see what happens when a classifier outputs probabilities that tend to be concentrated around mean values.\n", "\n", "## Probability estimation with logistic regression\n", "\n", "Below, we train a logistic regression using the same spam dataset. We can see that its probability outputs are concentrated around 0.5. This distortion is what logistic calibration works on best." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD/CAYAAADllv3BAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADj5JREFUeJzt3c9uHNeZxuH3HQuw4gkgWhqD2gQWaXgvysy+YXIwezOJ\nbkBUZi6AQLwyV4kG1gV4Il1ARp5oPYAsotdjGyZm64h0EBiREPNPAmRhjIJvFn0olTrN7upms6v4\n8fcAhKpOFas+lrpeHp6q6nZECACQxz80XQAAYLoIdgBIhmAHgGQIdgBIhmAHgGQujFrB9kqZ/OeI\n+EVpW5N0KOlGRHx8XBsAYPaG9thLqP8kIh5LumH7uu0lSVHaDmwv9bUd2r5++qUDAAYZGuwR8Tgi\n/q3MLkTEtqSb6vXMJWlX0mpf205pAwA0oNYYu+0NST8vs3OS9iuLr0i6NKANANCAWsFexsz/1fal\nU64HAHBCQy+eVsbOt9UbYrkt6UDS5bLKnKTvJEVf296AbfHeBQAwgYjwOOuP6rGv6tXAfiLpPyUt\nlrZFSZ8d0zaoOL4i9NFHHzVeQ1u+OBYcC47F8K9JjLrd8T8k/cz2O71cjoeSZHu53DFzEL3e/MA2\nAMDsDQ32iPiLpPsD2mu1AQBmjydPG9DpdJouoTU6nY6uXr0m241+Xb16relDweuigmNxMp50DGfs\nHdkxq33hbLGt3vX3RquYeDwTOE22FVO+eAoAOGMIdgBIhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQI\ndgBIhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQIdgBI\nhmAHgGQIdgBIhmAHgGQIdgBIhmAHgGQIdgBIZmSw214vX3cqbXeOllXa1myv2N44nVIBAHUMDXbb\nK5IeRcQ9SYu23y+Lbtv+WtKTst6SpIiIx5IObV8/zaIBAMcb1WNflLRapnfKvCTdioh3I2KrzN+U\ndFhZb1UAgEZcGLaw9NSP3JD0mzK9WHrzNyLiY0lzkvYr616ZapUAgNqGBvuRMtTyZURsS1JE3C3t\nqyXgAQAtUfeumJWI+FB6cTH1g9K+L2lB0oGky6VtTtLeVKsEANQ2ssdue73SQ1+R9Ll64+iS9I6k\nTyR9IWlZ0pZ64/CPBm1rc3PzxXSn01Gn05m8cgBIqNvtqtvtnmgbjojjF/aC/IF6PfI3Jf00IrbK\nbY77khYqoX9L0m5puz9gWzFsXzi/bEtq+rVh8fpEG9lWRHis75nVi5lgx3EIduB4kwQ7T54CQDIE\nOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAk\nQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7ADQDIEOwAkQ7AD\nQDIEOwAkc6HpAoB2eF22G61gfv5tPX36TaM1IAdHxPAV7PUy+U5E/KK0rUk6lHQjIj4+rq1vOzFq\nXzifeoHa9GujHTVwjqCfbUXEWL2OoT122yuSHkXEN7Yf2H5f0oGkiIjHthdsL5XVj9oWbV+PiO3J\nfgwAwEmMGmNflLRapnfK/E31euaStFuWV9t2Kt8DAJixoT32iLhXmb0h6TeSliXtV9qvSLo0oA0A\n0IBad8WU4ZYvGV4BgPare1fMSkR8WKYPJF0u03OSvlPvqlO1bW/QRjY3N19MdzoddTqd8aoFgOS6\n3a663e6JtlHrrpijIZlyMXVP0nJE3Le9IelRWfWVtv7ePXfF4DjcFfOyBs4R9JvkrpihQzElyO/Y\n/p3tPfXufNmuLDuIiO1BbZP9CACAkxrZY5/ajuix4xj02F/WwDmCflPvsQMAzh6CHQCSIdgBIBmC\nHQCSIdgBIBmCHQCSIdgBIBmCHQCSIdgBIBmCHQCSIdgBIBmCHQCSIdgBIBmCHQCSIdgBIBmCHQCS\nIdgBIJm6H2aNhK5evaZnz37fdBkApoyPxjvH2vGRdFJbPpauDTVwjqAfH40HACDYASAbgh0AkiHY\nASAZgh0AkiHYASAZgh0AkiHYASCZWsFue6lv/k75d73StmZ7xfbGdEsEAIxjZLDbXpH0aV/zbdtf\nS3pS1lmSFBHxWNKh7etTrxQAUMvIYC9h/aSv+VZEvBsRW2X+pqTDMr0jaXV6JQIAxjHpGPti37DL\nnKT9yvIrJysLADCpid7dMSLuSpLt1TJUAwBoibF77LbXbX9QZvclLUg6kHS5tM1J2ptOeQCAcdXt\nsVffMvJz9cbRJekdSZ9I+kLSsqQtSYuSHg3ayObm5ovpTqejTqczVrEAkF2321W32z3RNka+H7vt\nNUm/lrQeEQ9L27pKb70yLHNL0m5puz9gO7wfe8vwfuztq4FzBP0meT92PmjjHCPY21cD5wj68UEb\nAACCHQCyIdgBIBmCHQCSIdgBIBmCHQCSIdgBIBmCHQCSmehNwACchtfLQ2PNmp9/W0+fftN0GTgB\nnjw9x3jylBoG4wnYNuHJUwAAwQ4A2RDsAJAMwQ4AyRDsAJAMwQ4AyRDsAJAMwQ4AyRDsAJAMwQ4A\nyRDsAJAMwQ4AyRDsAJAMwQ4AyRDsAJAMwQ4AyRDsAJAMwQ4AydQKdttLffNrtldsbwxrAwDM3shg\nt70i6dPK/JKkiIjHkg5sL/W1Hdq+fmoVAwCGGhnsJayfVJpuSjos07uSVvvadkobAKABdcfYq5+Q\nPSdpvzJ/RdKlAW0AgAZcaLqA8+rq1Wt69uz3TZcBIKG6wR6V6QNJl8v0nKTvyvJq295UqkusF+ox\ncr3T5dGrADhz6gZ7NQEeSHpP0pakRUmPSvvygLZXbG5uvpjudDrqdDpjFQsA2XW7XXW73RNtwxHD\ne4221yT9WtJ6RDwsbbfUu3C6EBH3j2vr206M2td5Ylvt6LE3XYPUjjqo4SWLc7U9bCsixvrzemSw\nTwvB/iqCvaoNdVDDSwR7m0wS7Dx5CgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzB\nDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJ\nEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJEOwAkAzBDgDJTBTstu+Uf9crbWu2V2xvTKs4AMD4\nJu2x37b9taQnkmR7SVJExGNJh7avT6tAAMB4Jg32WxHxbkRslfmbkg7L9I6k1RNXBgCYyKTBvtg3\n7DInab+y/MrJygIATOrCJN8UEXclyfaq7ZW63/ftt9/q+fPnk+xyat544w299dZbjdYAtNvrst1o\nBfPzb+vp028areEsGzvYywXTvYh4qF4vfUHSgaTLZZU5SXuDvvdHP3pbFy78oyTptdcu6rXXLk5S\n84n87W/7+uMf/6C5ubmZ7xs4G76XFI1W8OxZs79YmtTtdtXtdk+0DUeM9x9YLozuRMRfbH8i6ZOy\naDki7pfhmUcRsd33fXHp0r/oz3/+7xMVfFI/+MG8dnf/V/Pz843W0esRNXvySG2oQWpHHdTwUhvq\nsMbNpqxsKyLG+k03do89IrZtr9vel/S7owC3vVyGZQ76Qx0AMDtj99gn3lGLeuw//OHr+tOf/tBo\nHT1N90ja0DOT2lEHNbzUhjouqjck1Kw2jPXPpMeeQS/Um37hnt8xRGC05sf5pbM71s9bCgBAMuey\nxw4A9TR/6+ckCHYAOFYbhoTG/8XCUAwAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0Ay\nBDsAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0AyBDsAJEOwA0AyBDsA\nJEOwA0AyBDsAJEOwA0AyF6a1Idtrkg4l3YiIj6e1XQDAeKbSY7e9JCki4rGkQ9vXp7FdAMD4pjUU\nc1O93rok7UhandJ2k+o2XUCLdJsuAEhnWsE+J2m/Mn9lSttNqtt0AS3SbboAIB0ungJAMtO6eHog\n6XKZnpO0N2ilv/61q4sX/2lKu5zU/zW8fwA4XdMK9geS3pO0JWlR0qNBKz1//r2eP/9+Srs8KTdd\ngKihqg11UMNLbaijDTVI7amjvqkEe0R8Zfs92yuSDiJie8A6Z+/oAMAZ5IhougacE3WfdbC9wbMQ\nOK9sL0XEV8csq3UOncrFU9trtldsb0yyPJMax2K9fN2ZdW2zVPdZh/JXX/rbZWu8LpbKOmuzrm3W\nxsiLW7OubdbK6//TY5bVfl5o6sE+aufn6WGmGsdiRdKjiLgnadH2+03UOSM861DUPAc+jIjfSlo4\n5+fIkqSdsnw387GQpPJzPjlmce1z6DR67KN2fp5O8FE/62KlbafMZzXyWYfyJ+hjncWrVeMZ+roo\nvfT/kaSIuDvomlUidfLg38u/i8mPxSi1nxc6jWAftfPz9DDT0J81Iu5FxP0ye0PSF7MqrKXebLqA\nGRl1DvxY0pUyHJN9uHLUOfKVpB3b+zrmNmr8PR5QaoHy5+aXyXsjQ591KL31rTLLFX1p7+gC2nkY\nZz+O7UvqvXZ+Keme7WuNFtSsWs8LSacT7KN2Xru4BOr+rCsR8eFsSmrMA70calqU9Jn04sSVetcY\nPrC9rl5vNfNY6qjXxZ56wxJSb5hieUZ1NWHUsbgt6VcRcVfSuqSfzLC2prwyFFk5RwaeQ4OcRrCP\nOoFrF5fAqGMh2+vlRXt0MTWlSu+z/1mHz8ry30bEw9J2acAmMhn1uvivyvI5SZ/PtLrZGnUsQiXo\nyuvjsH8DmZS/zt6z/UGl+egcOe4c+vvtnMZ97OW2pF1JC0djyLY/j4gfH7c8q2HHovwHPVCv1/Km\npJ9WhiOQWM1z5EDScva/5mociw317hS5nD0vpoUHlAAgGS6eAkAyBDsAJEOwA0AyBDsAJEOwA0Ay\nBDsAJEOwA0AyBDsAJPP/ZgdFaV71jYIAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "lrc = LogisticRegression(C=0.001)\n", "lrc = lrc.fit(x_train, y_train)\n", "probas_lrc = lrc.predict_proba(x_test)\n", "\n", "plt.clf()\n", "plt.hist(probas_lrc[:,1])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's train the calibration models." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEQCAYAAACwSgOGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xlc1HX+wPHXZzhVVBg8EjUFvErNA622XwcuYm3bqYGV\ndqvY1naX2uYurZaZtdXuViq1XdqBYtd2KKLUdmqK3VnCYCqaCgwoyv35/fEdBpABZmCYgeH97PF9\nOPO95j0T833P5/h+PkprjRBCCGHydgBCCCHaB0kIQgghAEkIQgghbCQhCCGEALyUEJRS45rYNk0p\nFaeUuteTMQkhRGfn8YSglIoD1jSybRygtdYZgFUpNdajwQkhRCfm8YRgu9hnN7J5OmC1Pc4BJnsk\nKCGEEO2uDSEUKKjzPNxbgQghRGfj7+0AXKWUkjvphBDCRVpr1dw+7S0hFAJm2+NQIN/RTjV3Vycn\nJ5OcnOyRwNoz+RzkM6ghn4PB2c+hpAQOHYL8fCgogMJC49/iYigqMpYjR2qXo0eNY0pK4NgxYzl+\nHCoq2v49uS6Ziy9O5p13QKlmcwHgvYRQLzqlVE+tdRGQCsQAm4AoIN3RwTX/ozMzM8nMzCQ2NrYt\nYxVCdDBVVZCTA7m5sHcv7Nlj/HvggLHs3w8HDxoXc8/SdOUY3SghhKN0o4RulNjXdeE4XTlGV44R\nTCldOE4wpQ2WIMoIooxAyu3/1l2yKOLvVPHTT+DK7wOPJwSl1DQgRik1VWu9zrZ6IzBRa52llIqx\n9UQq1FrvcHSOmoSQnJwsyUCITkpr+O03+PZb+Okn2LnTWLKzjUSweHGbvCo9KKYXh+nFYcwUEEYh\nZgro419IuJ8Vs8lKqLLSg2J66CK6VxfTrfoIXauPEFxVQoUpiFL/EMr8u1Hu343ygG6UB3Slwr8r\nFYHdqAjoQmVAFyoDu1IVEEx1QDCVgT2Mx/5BVAcEUR0YTEVAEGX+gRQFBqH9A6n2D0QHBqEDAqn2\nC2D4ZyuYfWcyF18MDz74oFPvzuMJQWudBqSdsG5incfPOXsuSQYG+RzkM6jhq5+D1sYv/C+/hC1b\nYPt2+OYbo7rHsVinzhsYCL17w0nhFQztlkdU4F5ONu3lJL2f3pX7CSvbT49jv9Gt5CDBRw4SWHQI\nHRRMdVgvdHg4KjwcUy8zpvAwlDkMQvtB6CnQs6ex9OhhLN27G0tICH7+/gS764NpwqDMi3D1z0F1\ntNFOlVK6o8UshHCN1vDzz5CZCR99ZCx5ea6dIyICIiNh4EAYOEAzPGQf0XoX/UuzCbdm0/2wBf+8\n3ajdu43M0rcvDBgA/fsbB/frZyx9+xpLnz5G9ggKapP33JaUUk41KvtMQhg8eDC7d+/2QkSdz6BB\ng8jNzfV2GMLHHD8OmzbB++/De++BM1/nkBAYNQpGjoThw2HEsGpOCcphYNF3BP3yHfzwg1Gf9PPP\nxi/0IUMgOtpYIiNh0CAYPNhIAH5+bf4evaXTJQTbG/ZCRJ2PfNbCXUpLYf16eOMNeOcdo/dOY0JC\nYOJEOP10Yxl7WjWDy3Zi2volbNsGWVnw9ddgNsPo0bWZYsQIGDbMqMLppCQhiDYjn7Vora+/hpQU\nWLXK6NrpSPfuEBtbu4wZUYbfti1G/dH//mc0JpjNcMYZEBMD48fD2LEQFubBd9I+aK15Z+c7VOkq\npp4ytcF2SQiizchnLVqiogJSU+Gpp2DrVsf7DB8Ol1wCF14I/3eWJuDHb4wixIYNRovyiBFw3nlw\n7rlw5plGvX4n93P+z9z+4e18uOtD+nTrw85bdxIaHFpvH2cTQnsbukK4KCsri4kTJzq93pXzDhky\npDWhCQEYN3Q98YRRbT9zZsNkEBUF999vlBp+/LqcR+PWE/vGzQQM7g/TphmNCbfdBvv2GQc/9piR\nNTp5MiitLGXBxgWMemYUH+76kJ5BPXngnAcICQxp8Tnb253KwkVRUVEsXbrU6fWucPbuRiEcKSuD\nZ5+Fhx6Cw4frbwsMNK71s2fDef9XiSlzE/zjVXj7bTj1VLjsMqOL0bBhXondmywWCwsXLmTfvn30\n79+fRYsWERkZWW+f4rJiLnntEj7a/REAN469kSWTl9CnWyuTpNa6Qy1GyA01tj4nJ0fPmDFDx8bG\n6hkzZuicnByH+zWltedIT0/X0dHR2mw268TERG21Wu3njYmJ0UlJSTosLExPmTJFb9++XUdHR2uT\nyaSXLVumtdZ67dq1OikpSSckJGillJ4wYYK2WCxaa623b9+uY2Ji7OeLj4/X8+bN06eccoqeMGGC\nPYY1a9bYY0hKSrKvX7FihQ4LC9Mmk6nBeYcMGeLw/TT2WQuhtdZVVVq/9JLWJ5+stdGBtHbp3Vvr\nRYu0PnRIa/3jj1rfdZfWfftqffrpWj/5pNZ5ed4O36tycnJ0dHS0BuxLdHR0vWvOoZJDesLKCZpk\ndL/H+unP93ze7Hlt39lmr68+3YZgsViIj48nO7t2tO3o6GjS09MbZNzGtPYcRUVFREZGkpaWRkxM\nDPPmzSM/P5/U1FQsFgvR0dGkpaURFxfH+PHjKSwsJDc3ly1bthAfH091dTVpaWkkJCSQkpJCQkIC\n9913H1999RVfffUVWVlZzJkzh61bt9rPN3fuXM4++2yeeOIJtm7dSk5ODhMmTGDz5s307NmT+Ph4\nli5dytSpU/Hz82P79u1ERkYya9YsoqOjWbJkCVlZWSQmJvLLL7849VkLAcZdw3Pnwmef1V9/8smw\nYAFcd3UFXf67BlasMLqC3nCDsQwd6p2AXbCnaA/b9m9r09d44okn+PjjjxusP/fcc7nzzjupqq5i\n4eaF/Hj4R6LCoki/Jp2osKhmz+tsG4JPVxktXLiw3oUcIDs7m4ULF7Jq1SqPnCM1NZX4+HgmTZoE\nwJIlS4iKqv0fGBYWxuWXXw7A5MmTCQsLo3v37sTFxaGUori4GICYmBhuuukmAJYvX46fn599W11K\nKZ555hmysrLs69LS0khKSmLMmDEArFlTOz9RYWEhPXr0AMBsNmO1WhHCVSUl8OCD8I9/GOMI1ejd\nG/7yF5h7pZWgl1bCyH8ZF//bbjPaAQICvBe0C6ylVk5/7nQOHD3Qti8UAVzZcPXHfMzHb9QmilF9\nRrFh5gb6de/n1pf36YSwb98+h+vzXLjlsbXnyM7OZs2aNWzcuBEwqujqXsjNZrP9cWhoKOHhjqeA\nqJtEACIjI8nJyWl2v5oYJkyYYH8+dmztRHQPPfQQGRkZ9td1dLwQTdm2Da6+2vjBXyMgAO65BxYk\nFdB95eNwyrNG16G33za6h7ZTjdXf/yXjLxw4eoCosChG9xndZq+/bds29u7d22D9gAEDiImJMR73\nGMDfJ/0dcxdzg/1ay6cTQv/+/R2uj4iI8Ng5oqOjSUhI4I033rCva8ldvieWUnJycoiKimqw3pHQ\n0FB27dplf56VlWVPJps2bWLz5s10796dlJQUtm/f7nJsonOqqjI6/DzwAFRW1q4/91xY8WgRI97/\nB8Q8DVOnGjeNDRrkvWCd4Kh6+IsvvuCx1x7j2a+exd/kz1vT32J037ZLCJYzGqmiftj5au7W8Olu\np4sWLSI6OrreuujoaBYtWuSxcyQmJrJx40YyMjKwWq0kJSWRlJRk395UXXzdbdu3b+e5556jqKiI\npKQkJkyYYK/qaeyYGklJSaSkpNgTQUJCAoWFhRQUFGA2m+nevTtWq5UVK1ZQUFDQ4HghTpSfDxdc\nAPPn1yaDkBB4fmUVmVetYMSlw43uolu2wMqVTiUDi8XCzJkzmTRpEjNnzsRisbTxu6jPYfWwJZsb\n192IRnPXmXe1aTIAo+Sfnp7OjBkzmDRpEjNmzHCpzbPVnGl5bk8LLexlNGnSpFb3MmrpOTIyMuw9\nh84//3xdVFRkP2/dnjzz5s2z9yzSWmuTyaSLior02rVr9ZQpU3RiYqJWSumJEyfW6w1U05uo7vnq\nrtda65SUFHtvoptvvtm+Pj4+XoeFhekJEybojIwMbTabdUZGhvQyEo369luto6Lq9x464wyt967a\nrPXo0Vqfe67W27e7dE5nete0tdjY2HqvD2jOQJOMHvTEIH207KjHYnE3pJeR70hLSyM1NbVetZM3\n+fJnLRpXras5758z+OTgf43LpU2fKs2yjDIm7apiwYVBvD3KD1y8h6W0tJTKisoG6/0D/AkO9sRg\n0Y3EEACY4N2r3uWiYRd5JI62IL2MhBBuNf3xp/mk5HUIrF132Y/w7/fhrREw8mY4ElwKLZlO0gQ4\nGFW6kkqOlh9tachuieGyqMs6dDJwhSQEIUSTtIY7Fv/E2rL7jF/Ma97g1LJz2HzqffT64XNK33ya\na88+i2tb8RqzZs0iNTW1wfrExESee87pObNaLXd3LosXLWb/gf30O6kfCxcuZNSwUR57fW+TKiPh\nMvmsOw+t4c57KnjqyFnQ/yvYcR3XZt/Of45fhd/Zv4N//ctoTW4ld9xEKhonVUZCiBbbU7SHkvJj\nLFoMr/7wApz9FVhP5vGfx3Lnb1NQ/3zKuPnATWp61yxcuJC8vDwiIiIcjuEj2paUEITL5LP2bS9k\nvcCN79xYb51/JaSmXsKlpl8wvfN2hxhqQtSSEoIQokU25W4yHhzpB2XdCT9exQcfaGLGVmF69fNO\nPfOYr5OEIISo55MfbWNQrHmDwb8O4NOuU+h78zRMSx/26XmHhY/fqSyEcM369ZrcYiMhjDxUxZbA\nc+i1+A78HlsqyaATkITQxtwx85izs5+ZTCaX9heirp07IeH6w9DFypnZ3fioYjrdnl1G4J23eDs0\n4SFSZeQBrZ15zNnZz2pexx2zpYnOpbDQGI36SOBO/m83rFtbhnohla4zL/R2aMKDpITgYWvXrmXI\nkCH4+fkxffr0ekNhr1y5ErPZzNChQ0lJSbGXLHJycrjvvvvs+yUlJWE2mwkPD2fZsmUATJkyBa01\n4eHhDfavec3w8HDmzp3roXcqOorKSpg+3Ri++vTgD1n3Bjw591zMkgw6nU6REJRy79JSOTk5JCYm\nkpKSQmFhIVpr+4U7JyeH+fPns3nzZrZt28aKFSvqlSxqHq9du5ZNmzaxe/duMjIymD9/PsXFxWzY\nsAGlFPn5+fX2z8nJYc6cOaSlpbFt2zYyMjJYt25dy9+E8Dl//zukp8NYsngn9wluuAyCLz3P22EJ\nL+gUCaG9qJm5bNKkSfTo0YOlS5faZy9buXKlfVazHj16sGDBAofnqLnQ79q1i7Fjx9ab8ayp1xwz\nZgyDBw9mzZo1MgmOsNu8GRYvhlP4gQ/4A4svG8X7w2BYeOeb3F5IQvCo/Pz8enMrREZG2qeszMnJ\nqbetsYv2tGnTmDdvHgkJCYSHh7NixYomXzM7O7veeceOHVtvxjTReR08CDNmQB99gPf4Iy+e8iib\nzysBJCF0Vp0iIdQfub31S0uFh4fXm7nMarUSGhoKGAmg7rbGZkKzWCzExcWxa9cue9XSpk2bbO+z\nYXCOZktLS0tr+ZsQPqG6Gq6/Hqz7j/EOl5Da5Vq+HPEFP/z2AwABRzrGXMfCvTpFQmgvrrjiClJS\nUti0aRNWq5U5c+Ywffp0AKZPn26f1cxqtfLII484PMfatWtJSEjAYrFQXV0NQFFREeC4N1Njs6WJ\nzu2f/4T1H1Sxmhn8xAie6vkrb21+Fm3SUAyXX3i5x2csE94nCcGDIiMjWbNmDXPmzCE8PByTyWS/\n8I8bN46lS5cSFxfHxIkTmTt3rr30UNe9996L2WxmyJAhTJw4kcTERC6//HLAqE7y8/OrlxgiIyPt\n5x06dChTpkxh1qxZnnnDol365RdYsAAeYT5hFPL48MvZf+BFCLftkG+UUBcuXOjNMIUXyOB27YTF\nYsFqtTJu3DjAaAxeuXIl69ev93JkDXX0z7ozq66GSZOg98drWca9XHfqV6heV/LxxxvhdOBC4Cvg\nvzBp0iR7daTo2Jwd3E5KCO2E1WolLi7OXv2zYsUKEhISvByV8DXLl8OBj3fyLDcz3bSWJ18JZ+DA\nvsbGOiUEgIiICK/EKLxHEkI7MW7cOO6//34iIyMJDw9nyJAhUrUj3Co3F5LvLSGNadzPw8TPj2H8\neFi0aJHRE61OQoiOjmbRokXeDFd4gVQZCZfJZ93xaA0XnK+5Jv0aKghg2Yj/sD1LUTN/vcViYfTz\noykJKOHi3It56q9PyeQ0PqTdzoeglJoGWIHxWutlTWyP1Fp7bjJVIXzYunXQO301Y9nBGWwh44Xa\nZADQb2A/jgUcw0/5kfZcGgF+0u20M/JolZFSahygtdYZgFUpNdbB9hzbdsuJ24UQrjt2DB677Vee\n4E5msoobbunKmWfW32dXwS40mqiwKEkGnZin2xCmY/z6B8gBJjvYp2aYziit9Q6PRCWED1u6pJqH\n867jce5mX6+xOGoa+DnfmANB7lDu3DydEEKBgjrPw+tu1FpnATlKqQLsfR2EEC2VkwPHlzxJABUs\n414efhjCwhruJwlBQDvrZaSU6gkUAg8DKUqpwV4NyA3cMVlNzcQ33oxBdEz/uOl77q1awrW8zPgJ\nftx4o+P9JCEI8HyjciFgtj0OpWEpYA6wRGtdrJTKAa4AHjvxJMnJyfbHsbGxxMbGtkWsbuGOyWo8\nNcGO8L5Xvn6F+RnzqayubPW5Kko17/5s5a+Tg7CM/R1WM0Q84XjfwuPGcCaSEHxDZmYmmZmZLh/n\n6RJCKlAzjGcUsBHsJQMADSgArfU6atsb6klOTrYv7TkZgHOT20DjE+fUnfimqf0sFgsTJkxg2bJl\n9kl2duzY0SAGmSynfXv5m5fJO5LHwZKDrV4SvziEMlWw4qyjEHKQwvLG962orqBPtz6M7zfe2x+B\ncIPY2Nh610lnebSEoLXOUkrFKKXigMI6jcYbgYla68eUUvcqpbIBs7u6naoHW/cLuy79N9f73zua\n3CY7O5uYmBiSkpI4fPgwiYmJZGRkEBMTw6xZs7jvvvtYvnw5GzZswM/Pj/z8fPsEO472A9i+fTtX\nXXUVBQUFzJ07l3nz5tmHvlBKYbFYmDNnDps3b6Znz57Ex8ezbt06pk6d6rbPR7TO/iP7Aci4NoOR\nvUe2+DwbXvyNC96bxKQuaQT++xQ+/wIi+jV9TGhwKEH+QS1+TdHxefw+BEcXea31xDqPG9yb4Cvq\nTm4zbtw4++Q2K1assE+cA7B06VJiYmLsF/oadSfYcbRfWFgYd999N2CURObMmWM/VmvN2rVr7ZPl\nAPbJeUT7sf+okRBG9h5J35C+LTpHRQWEz/8zKSTx/fFY7vszjBvqziiFr/J4QvCGlvyqbwvTpk2j\nsLDQPgT1ggULuOeeexxOnFMzplFdze1nNpsbHFNXTk4OMTEx9ucyUU77UlZZRsHxAvyUH7279W7x\neTbc/h7DS7YzjZcIDYX5890YpPBp7aqXka87cXKb5cuXs2nTpiYnzqnL2f0aI5PltG8Hjh4AoG9I\nX0yqZV/No/llnLrydm7haUrpwv33O+5mKoQjkhA86MTJbZRSWK1WhxPnJCYmNji+qQl2wPGMaTWU\nUiQlJbFy5UqZLKedqqku6hfSTGV/E76Y+S++qzqFDZzPgAFw663uik50BpIQPOjEyW0SEhKYOnWq\nw4lz6nYTrZn4pqkJdqD57qmDBw+WyXLasZoG5YjuLRt2+mjOQcatf4R7bD21k5OhSxd3RSc6Axnt\nVLhMPuu28czWZ7jl/VuYM34OKy5e4fLxWWfM5aMtwdzJk0RFwc6d4N8pWglFc9rtaKdCCMdqSgj9\nurteZXTsy28ZsHUdf+cnAP7yF0kGwnXyJyNEO9HiNgStOXjNXTym/0ohZgYNgmuuaYMAhc+TNgQh\n2gl7QnCxhFD6XgZVObtZQRIA998PATKCtWgBSQhCtBP2KiNXSghaU3jLX/hL1d+pJICBA+G669oo\nQOHzJCEI0U60pIRQkfYOBXmlpGJ0U54/H4Jk9AnRQtKGIEQ7UFVdxcGSgygUfbs5OWRFdTVH73iA\n+ZUPozHRrx+NDm8thDOkhCBEO3Cw5CDVuppeXXs5PYVl9auv82tBN/7LRQDcfjv15kkWwlWSENpY\nWlqaw7uOndHaiXFEx2CxWLjpjpsAqCiswGKxNH9QRQXH5/2NO48/DChCQiApqW3jFL5Prjge0NIJ\nblo7MY5o/ywWC/Hx8Xzwvw8AsO6xEh8f33xSWLWKnccGspnfAzB7NrgwrJUQDklC8ACtNYmJiZhM\nJiZOnFjvy75x40b7hDVNTYyzcuVKzGYzfn5+TJw4kdzcXG+8FeFmCxcuJDs7G7rbVhyB7OxsFi5c\n2PhBVVWUJj/C3VZjHz8/uOOOto9V+D5JCB6wdu1azj//fKxWKzExMSQkJADGaKWJiYmkpKRgsVgw\nm832sYU2bNgAGENeA9x8881s3ryZwsJCIiMjWbHC9aENRPuzb98+40FNQjhq/JOXl9f4QWlp/FoS\nTiaxAEyfDief3GYhik6k8/Qyclf1SwvG8ImJieGmm4w64uXLl+Pn50dxcTFr1qwhPj7ePuHNkiVL\niIqKsh9Xt8qoZjIdMOY9sFodzi4qOpj+/fsbD0JsK44Y/0RENDLAndaUJz/MPQWLsc02y733tmmI\nohPpPAnBi4Ox1b3IgzGxTU5ODtnZ2axZs4aNGzcCRtVSTZXRiR566CEyMjLsVUgnnlN0TIsWLeKL\nL74gu3u2seIoREdHs2jRIscHvPcehw/Du/qPAMTFgcxzJNxFqow8IDs7u95zi8VCVFQU0dHRJCQk\nkJ+fT35+PgUFBfUmsKlRMxfz5s2bWb9+PVdccYWnQhdtLDIykvT0dMIHG4k+/nfxpKenExkZ2XBn\nrala9BAPHLufmtLB7bd7MFjh8yQheEBWVhbr1q3DarWSlJREfHw8PXr0IDExkY0bN5KRkWHfluSg\n72BhYSFms5nu3btjtVpZsWIFBQUFXngnoi1ERkbStU9XAJYvW+44GQBkZnJkdyEvlUyzHQcXXuip\nKEVnIAnBA+bMmcOKFSswm83s3r2b1NRUAHr27MmaNWtISkoiPDyc3bt315v4vmZinNmzZ6O1xmw2\nEx8fz6OPPsrGjRvZtGmTt96ScCOttX36zKbGMdKPPso//O+jGj/AmA3Nz88jIYpOQibIES6Tz9q9\nDh87TO9lvekZ1BPr/EY6C3z/PWXnTqZHQS7lBNG1K+zbJ/ceCOc4O0GOlBCE8DKnJsZ58knSTrqF\ncoyR6665RpKBcL/O08tIiHaqZpTTRudSPniQqjVp3FH8s33Vrbd6IjLR2UgJQQgva3YehGefZVtk\nAod0LwAmTYJRozwVnehMpIQghJc1OXVmaSn62We5u3KzfZWUDkRbkRKCEF7WZBvC6tUciIjhk/xT\nAIiIgEsu8WR0ojPxmRLCoEGDZHRQDxk0aJC3Q/ApjZYQtIYnnuCJwCftq2bNAn+f+daK9sZn/rRk\n9E/RUTU6deZHH1FWpln2fRwAJpOREIRoK1JlJISXNdqo/PTTvD/oT9QMU/HHP8LAgR4OTnQqPlNC\nEMLbDpYcZMLKCewt3uvScRrjJr96JYR9+9AZGdzJ8/ZVMiOaaGtSQhDCTbbs28Ke4j1oF/8DiIuM\no3tg99qTrVzJLzFXsbvQGPL85JPhggu88a5EZyIlBCHcpPB4IQBXjbqKVVNXuXSsQtV2iigvh5QU\nHu6fbt8+e7aMWyTaniQEIdyksNRICOYuZkyqFYXvN9/k2MBhvLRlJGAkghtvdEeEQjRNqoyEcJOa\nEkJYcFjrTvTMM7wZcYv96R//aNx/IERb83gJQSk1DbAC47XWyxxsHwdEAWit0zwcnhAtVlNCCOvS\nioTw3XfoXbuYX3WZfZV0NRWe4tESgu1ir7XWGYBVKeVo8r8FtkQQ2ch2Idole0JoTQkhJYWfz76R\nvb8FANCvH/zhD+6ITojmebrKaDpG6QAgB5hcd6Ot9LAFQGv9mNZ6h2fDE6Ll7FVGLS0hlJbC6tU8\nVnCTfdV118mdycJzPJ0QQoG6cz+Gn7B9IhCulBqnlLrXc2EJ0XqtLiGkpVE2KoYXNg+2r5LGZOFJ\n7fG3R77WOkspNVkpNc1RO0JycrL9cWxsLLGxsR4MTwjHWl1CSEnhgwG3UlVlPD33XBg61E3BiU4l\nMzOTzMxMl4/zdEIoBMy2x6FA/gnb8zGqksCoWpoANJkQhGgvWlVC+Pln9I8/8kBe7VCmN93UxP5C\nNOHEH8oPPvigU8d5usooFVsPItu/GwGUUj1t69bW2R4KbPVodEK0QqtKCCkp7Iu7ju9/CQSgRw+4\n4gp3RidE8zyaELTWWQBKqTigsE6j8UbbdgtG76NpgFlrvc6T8QnRUmWVZRyvPI6/yZ9uAd1cO7i8\nHF5+mWcravuXXnkldO3q5iCFaIbH2xC01s85WDfRwXa5B0F0GHWri1yel+Ptt6kafir//HCYfdUN\nN7gzOiGcI3cqC+EGraou+s9/+PzUmzh61Hg6YgSccYYbgxPCSe2xl5EQHY611Li9JjQ41LUD9+2D\nL7/k4eO1BeLrrweZ/E94g5QQhHCDFvcweuUVjpx/BR98ZDQYmExwzTXujk4I50hCEMINWlRlpDW8\n8AJrQmobDM4/XwayE94jCUEIN2hRCeHzz9FKsXjjmfZV11/v5sCEcIEkBCHcoEVDX7/4IpZzr8eS\nazQYhIbCJZc0c4wQbUgSghBu4PLQ18eOwdq1/LuotsHgqqsgOLgtohPCOZIQhHADl6uM1q2jcsIZ\npLzf375KqouEt0lCEMINXG5UfvFFPht+Q717DyZObPoQIdqa3IcghBu4VELYsweysnikurbB4Lrr\n5N4D4X1SQhDCDVwqIaxezdE/XMGHmUaDgVIwc2ZbRieEcyQhCOEGTpcQtIaXX+bNkGvR2lgVFwcD\nBrRxgEI4QRKCEG7gdAlh2zZ0WRkPZ55lX3XddW0ZmRDOk4QgRCtVVFVQUlGCn/Kje2D3pnd+5RXy\n4q7lp51Gg0FICFx+uQeCFMIJLWpUVkrdA0QD24BUrXWxW6MSogOpqS4KDQ5teujrigp47TVWnv+F\nfdUVV0D9+y55AAAZPUlEQVQ3F6dPEKKtOFVCUEr9Xik11vb4EaAXUAQkAoVKKZnsT3RaNdVFzY50\n+uGHVA8Zxr/fj7Kvkuoi0Z40W0JQSv0eiNJab7Kt2lp34nulVCgwXyk1VWY4E51RzdDXzbYfvPwy\nX4+5loLPjaeDBsG557ZxcEK4wJkSQgzwVWMbtdZWrfV8INxtUQnRgTjVw6iwENLTeWx3gn3VNdcY\nw10L0V40++eotV4GTFRKDbatylFKbVVKTVJK9aiza34bxCdEu+dUD6O1ayk7ZzKp6bX7XHttW0cm\nhGuc+n2itU7RWufank4HUoGbgVxbclgPRCmlugMopaa2RbBCtEdOlRBWrSL9pGuorDSe/u53MHSo\nB4ITwgUt6WWUDaTbSg4opcYBk4EpwP1KqZqSgrQniE6h2aGvd++G77/noaN/sK+SxmTRHrlcg6m1\nTgHCaqqQtNZZWutlWuspWmszRgnC4tYohWjHmh36+tVXKZicwBfbAwEIDITERE9FJ4TzWtSkZUsC\nuY1s2w7Ma01QQnQkTZYQtIZXXuF1v9rBii65BMJcnHpZCE9okz4OWuustjivEO1RkyWEHTvQx4/z\n0GYZqkK0f9LpTYhWarJRedUqcs6aSd5+4w7m3r3h/PM9GZ0QzpOEIEQrNdrttKoKXnuNZ4tm2FfN\nnAkBAZ6MTgjnSUIQopUaLSFs2kTlSf15OmOEfZVUF4n2TBKCEK3UaAlh1Sq2Dp1BaanxdMwYYxGi\nvZKEIEQrVFZXcqT8CApFj6A6N+6XlMDbb7PEcqV91fXXez4+IVwhCUGIVqgZ2K5ncE9Mqs7X6Z13\nKDntTN7dehIA/v5w9dXeiFAI50lCEKIVGr0HYfVqPjDX3ntw4YXQp48nIxPCdS2aIEcIYXA49PWh\nQ+hPPuGBkNftq6QxWXQEUkIQohUc9jB64w1+i/kjO/eFABAeDhdd5I3ohHCNJAQhWsFhD6PVq3m5\nura66KqrjPGLhGjvPJ4QlFLTlFJxSql7m9mvye1CtAcNSgi7dlGdncPfP4+373Pjjd6ITAjXeTQh\n2IbK1lrrDMBaM0+zg/3iMIbUFqJda9CovHo134+aTkmZ0Tw3diyMG+et6IRwjadLCNMBq+1xDnLR\nFx1cvYHttIbVq3nit9qhKqR0IDoSTyeEUKCgzvMG8zArpcbZShDKY1EJ0UIHjh4AoHfX3rBlC2Xl\n8MIPpwNGu4HceyA6kvbYqCwjxYsOI9eaC0BkWCS88gqb+l9DzW+ZSy81ehgJ0VF4+j6EQsBsexwK\n5NfdaCsdbLI91Y2dJDk52f44NjaW2NhYtwYphLMsVmNywMFdI9BvvMFfq7+0b5PqIuEtmZmZZGZm\nunyc0rrR667b2RqVY7TWz9l6EaVrrXcopXpqrYuUUtMwEkE4kATM0lrvOOEc2pMxC9GY8qpyghcH\no5Si7LS1FC14nF4/fQJA//7GVMp+fl4OUghAKYXWutlqeI9WGdXMpGbrRVRY52K/0bY9TWu9zrau\npydjE8JVe4r2oNEM6DEA/1df4/WAa+zbrr9ekoHoeDxaQnAHKSGI9iIjJ4PJr0zmD73P4r/zv6P3\n0VwKtNEEtmsXREd7OUAhbNplCUEIX1LTfjDtB83OAXH2ZBAXJ8lAdEySEIRooZoeRr//ZD9PHKqt\nLpo920sBCdFKMtqpEC2Ua83lZCv0zj7MS0cvBKBXL7jsMi8HJkQLSQlBiBayWC1c8zWkR/yecoIA\nY5jroCAvByZEC0kJQYgWyi20cO3XcF3l9fZ1Ul0kOjIpIQjRAmWVZQz6YT/VCr44Ykx2cO65MHy4\nlwMTohUkIQjRAr8W/cp1X8Mrp4SCDgBgzhwvByVEK0lCEKIFdh/4iYTv4aWTRgAQFgbTpnk5KCFa\nSRKCEC3xzjtsi4B9FUZCuOEGCA72ckxCtJIkBCFa4KQ1m3hpDGCNBODmm70bjxDuIAlBCFfl5THw\nxz28eQpgHcwFF8CQId4OSojWk4QghIsqXljFm4NCORYIWAfzpz95OyIh3EMSghCu0Jpj/36elHFV\nAPTvFsmFF3o5JiHcRBKCEK749FMKrfDZ8AKo8udPMyNkmGvhMyQhCOGCw0uf52nTJcYsmcUnM3uW\nZAPhOyQhCOGs4mK6rH+Tl/uMB+CkoMH07u3lmIRwIxnLSAgnFS5/g48qJ3GwVxEAZwyP9HJEQriX\nlBCEcNKRp57nOX0ThBoT40wYMti7AQnhZpIQhHDCsa3f479/Dx9yAYTmAjA4dLBXYxLC3aTKSIhm\naK3ZePef+C7qJKqG34Ff1CdUAZGhUmUkfIskBCGa8dnPmfxu28fcOQswb6cK8FN+DA0f6u3QhHAr\nSQhCNOPnZS9ypB/kHDmH4B2JLF4MMQNH0adbH2+HJoRbSUIQohkj313Pw5OAHy/nz+fcyt3neDsi\nIdqGNCoL0YRtL35Dv5JC/jsMTPmj+fOfvR2REG1HSghCNOHAX5/h7fFQ5QdXnXcaAwd6OyIh2o7S\nWns7BpcopXRHi1l0TN98eoQB5wxk9J1F5Jn6sHP2bwwb5u2ohHCdUgqttWpuP6kyEqIRX962ms3d\nTiWvB/RhtCQD4fMkIQjhwM87NRO3L+fZwcMBiB9zmpcjEqLtSUIQwoE37/ofXTjOpuHHAIgbOdrL\nEQnR9iQhCHGCnBwY9sGTPMkd6L7fATC6ryQE4fskIQhxgmfuyeEc/TEvm6ZD752YlIlTe5/q7bCE\naHOSEISo48cfYcCb/+J5buJYn19BVTHEPISuAV29HZoQbU7uQxCijiULinmSlxjD15wWn8k3wOg+\nUl0kOgcpIQhhk5UF5rf/wwamsJeBjIn/BoDT+koPI9E5SAlBCJu/PVDFk/yTq3iNqVPhoPoWkBKC\n6Dw8nhCUUtMAKzBea73MwfbZtofRWuv5Hg1OdFqffAIB77/Fb/RlqzqD//wdpqy3JQTpYSQ6CY9W\nGSmlxgFaa50BWJVSY0/YHgeka61TgCil1O89GZ/onKqr4c47NAtYwlLmcfXVcFJkPnlH8uga0JWo\nsChvhyiER3i6DWE6RukAIAeYfML2qDrrcmzPhWhTq1eDedsGgillfeAlPPQQfHvQKB2M6jMKk5Km\nNtE5eLrKKBQoqPM8vO5GW8mgxnjgdU8EJTqvkhKYPx9e5WGWDD+F4CuHMuktzdHyowCc1kcalEXn\n0S4blW1VS9u01ju8HYvwbY8+CpF5nzCAPbwx7UuqVBlWa+3284ec773ghPAwTyeEQsBsexwK5Dey\nX5zWekFjJ0lOTrY/jo2NJTY21k3hic5kzx5YtgzW8jBLT5pCVeAKYvrFkJqQCkC3gG70Denr5SiF\ncF1mZiaZmZkuH+fR+RBsv/xjtNbPKaXuxWhA3qGU6qm1LrLtM7um6kgpFWdrgK57DpkPQbjF9Onw\nS+p23uESJlx/Hr8NfpUlcUuYf7Z0bhO+pV3Oh6C1zgJ7b6LCOlVCG+usf0QptUsplQ/IlV+0ifff\nh9RUeIDFPK7uIL//mwB89vxnWCwWL0cnhHfIjGmi0ykpgZEjoffurbzJ5YwecR3WKx+GQ8DTEB0d\nTXp6OpGRkd4OVQi3aJclBCHag7/9DXbvhiUsYIn/fVgj/2Fs+Mn4Jzs7m4ULF3ovQCG8RBKC6FSy\nsuDJJyGOjZzMr3wUXQwjSo2NP9Xul5eX550AhfAiSQii06iogFmzoKpKs4QFvDFyEQPO+hR6AsVA\nnRwQERHhrTCF8BpJCKLTePBB2L4dprIOf1XFlWkJDLko2tj4E/YuDNHR0SxatMhrcQrhLdKoLDqF\nTz6B884DU3UF3zKa72c9ybSUCxj5zEh+OPQDv9/7e3S2JiIigkWLFkmDsvApzjYqS0IQPq+4GMaM\ngdxcuIMnmBH2AeMPrWfv0T0MenIQPYN6cujeQwT4BXg7VCHahLMJoV0OXSGEO912m5EM+nKAB9RD\nlK/7FJOf4uf8nwEYe9JYSQZCIG0Iwse99JKxACxlHgcvuol+scMByLXmAjA4dLB3ghOinZESgvBZ\n27fD3LnG47P4lIu7ZGBe/aN9uyQEIeqTEoLwSfn5MHUqlJaCiSqeC76Vrk8vg+7d7ftIQhCiPkkI\nwudUVcFVVxl3IwPcG/QvBo3uSfD1V9bbTxKCEPVJlZHwKVrD3XdDerrxfBg7eTBwMUGvfgGqficL\nSQhC1CcJQfiUf/wDnnrKeGyiivT+1xM0PxmGDKm3X1llGXlH8vBTfgzoMcDzgQrRDklCED7j1Vfh\nnntqn7846nEG9uoCf/pTg333FO9BoxnQYwD+JvkaCAGSEISPyMiA66+vfX7N+O+Z+esy1LtbwdSw\nqUyqi4RoSBqVRYe3aRNcfLExeB3A+BHH+E/Z1aglS2DwYIfHSEIQoiEpIYgObeNGIxmU2kaw7h+h\n+fjUufh3GwM33dTocZIQhGhIEoLosDZsgEsvrZMM+sP22cvplrYDvmjYq6guSQhCNCQJQXRIq1fD\njTdCebnxfMAA+OyJL+nzp7/Bp59C165NHi8JQYiGpA1BdChaw6JFMHNmbTI4+WT45PW9DLwrAVJS\nYOjQZs+zu8i4a00SghC1pIQgOoyyMmNsohdfrF136qnw4asFDLz6fGNY00svbfY85VXl7CveJ/cg\nCHECSQiiQ8jJgenT4auvatdNngxrXj5OaMIl8Ic/1L8JoQl7iuQeBCEckSoj0e6tWwfjx9dPBjfe\nCO+/U0no3CshMhIefdTp80n7gRCOSUIQ7VZxsVFFNG0aFBUZ6wIC4Mkn4bmnywiYkQiVlfCf/zi8\n+awxkhCEcEzKy6Jdev99SEqCvXtr1w0aBKmpcPrIErjkcujZE95808gSLpCEIIRjUkIQ7crevTBj\nBvzxj/WTweWXQ1YWnD7MClOmGP1MX3sNAgNdfo3colxAEoIQJ5KEINqFo0fhr3+FYcOMQepq9O4N\nr78OaWkQduBHOPNMOOMMeO458G9ZAVdKCEI4JlVGwquOHYMVK4w24QMH6m+7+mpjKOtevTAyws03\nw9KlcMMNrXpNSQhCOCYJQXiF1Wokgscfh0OH6m8bN86Y1yA2FuPmg/sWGo0HH3wAMTGtel25B0GI\nxklCEB71zTfw9NOwapVROqgrIgIeegiuvdbWaejLL43+pUOHGn1Oe/Vq9evLPQhCNE6+EaLNHTpk\n/MBftcoYc+5EAwfCggVGTVBwMHDkCDz4oHHAU09BYmKTA9W5QqqLhGicJATRJg4cgHffhbfeMkYl\nraxsuM/o0XD77XDNNbbOQuXl8O+VsHgxnH8+fPut0arcCrutu/nnl//k872fo9EcPnYYkIQghCOS\nELzEYrGwcOFC9u3bR//+/Vm0aBGRkZHeDqvFysuNGp6NG2H9etiyxRiI7kT+/jB1Ktx6K5x9tu2H\nf1kZvPiaMWrdsGHw4YcwdqxLr6+1Zvv+7eQfzzfiqSrn1W9fJfX7VKp0VYP9J0ZMbMnbFMKnKe3o\nW9uOKaV0R4v5RBaLhfj4eLKzs+3roqOjSU9P7zBJobDQSACffw6ffWb8W1LS+P6/+51REkhIqNMU\ncPAgPPssLF8OY8bAvHkwaVKL4pmXPo9HP2s4fIWf8mP6qOncMPYGugV0A6BrQFdO63sayk3VUEK0\nd0optNbN/sFLQvCCmTNnsnr16gbrZ8yYwapVq7wQUeMqKmDXLvjhB/j+e9ixA7Zvh927mz7OZIJz\nzjEGH730UoiKsm0oKYF33jEmNPjkE4ovvJDFR46w9ejRFpeUnt/+PLPenYW/yZ/YwbEojL/7MX3H\n8Ocz/szJPU9uwTsXwnc4mxA8XmWklJoGWIHxWutlrm73Bfv27XO4Pi8vz8ORQHU15OfDr78aF/lf\nfwWLxUgCu3YZo4w6qv93JCoK4uKMUUjj4iA83LYhNxeWf2h0G83MNIoLM2aQu2QJky+/vF5J6Ysv\nvnCppJSZm8nc9+YC8MyFzzA7Zrbzb14IUY9HE4JSahygtdYZSqkopdRYrfUOZ7f7iv79+ztcHxER\n0arzam1Ux1uttUtBgXHBr1l++82oqfntN9i/H/Lyaiend0VgIJx2mnFtP+ss499Bg4Djx42ixOt1\n6pNKSoxG4unT4fnnoVcv9hXv47p7rya7azaMrj1vNtnMfHQmt/zplmZjKK8q5671d1FZXcldZ94l\nyUCIVvJolZFS6hFgg9Z6k1IqDhintX7M2e22fVpdZeRqg67WxlJdXbtUVdX+W/dxZaXmm0M7eM+y\nlqJSK9W243Q19sdHj5TwyaefcaykFGP0EBNduoQwbvwEAgO6GOe0nw+qKo3HFZXG88pK4yJeWWH8\nW1EB5RVQUW6c391CQiDMDOYwMIdDRM9KIqtL6HX4CL32F9P7QBG984qIyM0n7PBRDkaEsntYHywj\n+pJzykkc7B+GNhml1cLSQj7f+7m9+6c7XDTsIt6a/hZ+Jj+3nVMIX9Jeq4xCgYI6z8Nd3A4YvRKf\neQZKSzMJDIylrH8G1V3323u11M0XJ66rrq6mrKwMrc8zVhQoVl+2iYCAALQ22S/+GuMi7pLu++C0\nVdD3u+b3HVn/6XHgs4pv4cRf6yagufHbLMApjW82VUNAFfhXQ2CVsQRVQXAldKmALpXQtQJCyqFb\nOXQvh9BS6Flq/NsrF3ofg94l0O+osV9ed9jTA3aZ4SMz7IqA78bCz+FQ6ZcP5AM/wm8Yywl6BPWg\nW0E39ufsb7Bt0OBBnPW7s5p507Z9ew7i/nPu538f/4/Y2FinjvFlmZmZ8jkgnwO07DPokN1Oi4th\n5v5HKeURbsMMJXkQcNzbYRlNmT8A2gTlIVDl+EquaLyEo3TtdoVG2dcbj411tsdaY0KzrKqM+aYA\nTGj8tMaPamPR1fjZMmGlnx/VJhOV/v5UBvhzrKqCI1VlHFfVlJrgmAmO+kFQLzMDRo6g7KSulIcE\nUxbShdLQEIrCQvitZwhf9upBaWhIvRvFImzLuU58RsH+wUzsP5GRvUfy6+5fHfe2WuZ6byu5ABjk\nczDI59AxEkIhYLY9DsX4GenKdgA+/TSZLI6hGUoat9Fl2NfQa2ebBFxXzTVQKdtS5zEKVFUQQQcm\n0eVALH5dA/H3Bz+/2sU/APxtjwMCFf7+xrrAAGNI/5olMEgREKgIDITgLorAIEVQEHTpqgjuoujS\n1Vi6hpjoFqKoePUx9D3z6dLDRFAXE8q/7ov6g8nUoJBx+aRJZGZmNniPk8aNYdNbm9r4kzRERkaS\nnp7OwoULycvLIyIiosPfjyFEe5CZmUlmZibJyckuHefphJAKxACbgCggHUAp1VNrXdTY9hO9/34y\nJSXw+OPJ3HPPDJSagXGe+hdtR+uSkuawdm0qtkoh+zJ9+nReeOF5TKbaY0wm7M/bs9CPutFruMPa\ntUa1VcO2qyIjI9tdV1shOrrY2FhiY2PtCeHBBx906jiP34eglJqFUesdqbV+zrZuq9Z6YmPbTzi+\nY9+EIIQQXuCTN6YJIYRoGzJjmhBCCEASguiAlFLTlFJxSql7m9mvye1C+BLbjb2NbXPqO9NhEkJz\nb8jZN9yROfEZzLYtj3g6Nk+pezc7YFVKORwW1XZj42SPBudhTvw9jLPtM83TsXmKC9eFWZ6OzZNs\nf+9rGtnm1HcGOkhCaO4NufKGOyonPoM4IF1rnQJEKaV+7404PWA6xlhXADn4+EW/MU7+zS/QWqcB\nkZ30OzEOyLFtt/jiZ1DD9h6zG9ns9HemQyQEmn9DneEi0dx7jKqzLsf23Bc1eze7Umqc7QvSzjsM\nt0qTfw+2UsEWAK31Y744JhjOfe+X2v6N8tHPwBlOjQABHSchuGXIiw6uyfeotU6p0013PPCVpwJr\nh8K8HYAHNPc3PxEIt1Ub+Wo1anPfiSwgRylVQCM3uYr6OkpCEE6yFZO3+fCvoSbvZreVDmpute7s\nfarzbRfFmhJDp6KU6onx9/IwkKKUGuzVgLzHqREgoOMkBLcMedHBOfse47TWCzwTklekUlsdFgVs\nBPuXH4z2k6lKqdkYv5B9td64ub+HfIxqFDCqVSZ4KC5Pau4zmAMssY2YPBu4woOxeUO9KtI63wmH\n3xlHOkpCaO4i4PQb7sCa+wxQSs2uGS7c1sjsc+r84o0DCuuUhDbatqdprdfZ1vV0cApf0dzfw9o6\n20OBrR6NzjOa+ww0touk7W/CeuIJfIWtBBijlJpaZ3XNd6Kx70zD83SUO5VbO+SFL2jqM7D9z07F\n+NUUBiTUqToRPsjJ70QhMMFXS41OfAb3YvS+MfvqdcGdOkxCEEII0bY6SpWREEKINiYJQQghBCAJ\nQQghhI0kBCGEEIAkBCGEEDaSEIQQQgCSEIQQQthIQhBCCAFIQhBCCGEjCUEIIQQA/t4OQIiOzjaY\nWiLGaJvhwEattcW7UQnhOikhCNF6KcAbthE184FON/eA8A0yuJ0QraSU2oUx3PQbNUMNC9ERSQlB\niNa7Att4/EopX5x3QHQSkhCEaAWl1DSt9Q6tdaLWOhwI68RTNYoOThKCEC1ka0yef8LqbK11rhfC\nEaLVpA1BiFawTVloxZilLgxI1VoXezcqIVpGEoIQQghAqoyEEELYSEIQQggBSEIQQghhIwlBCCEE\nIAlBCCGEjSQEIYQQgCQEIYQQNpIQhBBCAJIQhBBC2Pw//LFM88Nitm0AAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cal_probas_lrc = lrc.predict_proba(x_cal)[:, 1]\n", "\n", "lr = LogisticRegression(C=99999999999)\n", "lr.fit(cal_probas_lrc.reshape(-1, 1), y_cal)\n", "\n", "iso = IsotonicRegression()\n", "iso.fit(cal_probas_lrc, y_cal)\n", "\n", "bc = BetaCalibration(parameters=\"abm\")\n", "bc.fit(cal_probas_lrc.reshape(-1, 1), y_cal)\n", "\n", "pr = [lr.predict_proba(linspace.reshape(-1, 1))[:, 1], iso.predict(linspace), bc.predict(linspace)]\n", "methods_text = ['logistic', 'isotonic', 'beta']\n", "idx = cal_probas_lrc.argsort()\n", "scores = cal_probas_lrc[idx]\n", "y_c_2 = y_cal[idx]\n", "fig_map = plot_calibration_map(pr, [scores, y_c_2, linspace], methods_text, alpha=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice how all three methods find sigmoid shapes. This is because the sigmoid shape is part of the beta calibration family, which also contains an identity calibration map, which just keeps the scores with their original values (this is not possible with logistic calibration). In both maps, isotonic regression did a great job fitting the scores, but as a non-parametric method, it needs a lot of data to train (as is the case with the spam dataset), which is not a problem for beta calibration. Therefore, given that beta calibration is able to fit the classifier's output probabilities better than or at least equal to logistic calibration, we argue that a practicioner who wants to use a parametric calibration method should choose beta calibration." ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.13" } }, "nbformat": 4, "nbformat_minor": 0 }