The Linear Regression Equation<\/h2>\n\n\n\n
Before diving into the coding details, first, let’s know a bit more about simple linear regression.<\/p>\n\n\n\n
In simple linear regression, we have only one feature variable and one target variable. Let the feature variable be x<\/em><\/strong> and the output variable be y<\/em><\/strong>. Our job is to find the value of a new y<\/em> when we have the value of a new x<\/em>. <\/p>\n\n\n\n The equation for the above operation is,<\/p>\n\n\n $$ where \\(\\beta_0\\) is the intercept parameter and \\(\\beta_1\\) is the slope parameter. <\/p>\n\n\n\n So, we will have to build a linear model by using the features x<\/em><\/strong> and target y<\/em><\/strong> that should be a straight line plot on a graph. For that, we have to optimize the parameters \\(\\beta_0\\) and \\(\\beta_1\\) using a cost function (also known as objective function<\/em>).<\/p>\n\n\n\n So, fire up your favorite IDE or Jupyter Notebook and follow along.<\/p>\n\n\n\n As this post is meant to be an introductory one, so we will be using Scikit-Learn’s Let’s get our data from Scikit-Learn first.<\/p>\n\n\n\n Now, we have 1 feature and 200 sample values in total. The above function defines the scatter plot for the feature values and their corresponding targets. It will also plot our linear model when we give it the slope and intercept parameters. It takes four arguments, the data points We do not have any optimal values for We can randomly initialize the values for Now, to visualize the initial linear model and the data points, let’s call the The above code gives us the following plot as output:<\/p>\n\n\n\n We can clearly see that the linear regression line is not at all a good fit, at least for now. Let’s move ahead and try to get better results.<\/p>\n\n\n\n In almost all types of learning, to make the algorithm learn from data, we need to minimize some sort of cost function. This cost function is otherwise known as an objective function. In the case of linear regression, we are going to optimize the Mean Square Error (MSE). The following is the mathematical expression of MSE.<\/p>\n\n\n $$ By using the above function we will optimize the slope and intercept parameters. In some books and articles, you may encounter the word ‘weight’<\/em> for the slope (\\(m\\)) parameter and ‘bias’<\/em> for the intercept parameter (\\(m\\) ). This is a common practice in machine learning terms.<\/p>\n\n\n\n So, what do we do to minimize the cost function? We use the Gradient Descent<\/a><\/strong> method. As we have to optimize both the parameters, we will have to use partial derivative for that. Also, the cost functions will be dependent on two parameters now.<\/p>\n\n\n $$ To get into the mathematical aspects, you see refer to this amazing article<\/a><\/strong>.<\/p>\n\n\n\n We are now ready to define the code for implementing the cost function. The following code block does so.<\/p>\n\n\n\n Okay, to optimize the parameters, we need to update them, and use those updated values in each iteration. In the next function, we will update the weights. But first, we calculate the partial derivatives. Then we need to update Learning rate is a very important parameter. We do not want it to be too high, as it may miss the optimal solution because of very large steps. We also do not want it to be too low. In that case, the steps will be more and the convergence may take a lot of time to happen.<\/p>\n\n\n\n Finally, we define the function to train the algorithm. In addition to the five arguments that In the end, we just call the We get four plots for the linear model while training process. The following four are those plots.<\/p>\n\n\n\n By the end of 200 epochs our model seems to do pretty well. You can always try to play around with different learning rates and epochs. This will give you better insights on how these affect the linear regression model.<\/p>\n\n\n\n In this article, you learned how to implement simple linear regression from scratch. Obviously, you do not need to write such code while doing practical industry-oriented machine learning. There are amazing libraries like Scikit-Learn<\/a><\/strong> which help us to carry these out with much ease.<\/p>\n\n\n\n Subscribe to the website to get timely article information in your inbox. You can also follow me Twitter<\/a><\/strong> and LinkedIn<\/a><\/strong> to get regular updates about my articles.<\/p>\n","protected":false},"excerpt":{"rendered":" Learn how to implement Linear Regression for Machine Learning from scratch using Python.<\/p>\n","protected":false},"author":1,"featured_media":2025,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47,17],"tags":[50,24,78],"class_list":["post-1996","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","category-machine-learning","tag-data-science","tag-machine-learning","tag-statistical-methods-and-analysis"],"yoast_head":"\n
\ny = \\beta_0 + \\beta_1x
\n$$<\/p>\n\n\n\nUsing Scikit-Learn Make Regression<\/h2>\n\n\n\n
make_regression<\/code> module to generate random regression data.<\/p>\n\n\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nfrom sklearn.datasets import make_regression<\/pre>\n\n\n\n
x, y = make_regression(n_samples=200, n_features=1, noise=20, bias=0)<\/pre>\n\n\n\n
x<\/code> holds all the feature values and y<\/code> holds all the target values.<\/p>\n\n\n\ndef plot_fig(x, y, b0, b1, name):\n plt.figure()\n\n plot_x = np.linspace(min(x), max(x), 200)\n plot_y = b0 + b1 * plot_x\n\n plt.scatter(x, y)\n\n plt.plot(plot_x, plot_y, color='red')\n\n plt.savefig(name)<\/pre>\n\n\n\n
x<\/code>, the targets y<\/code>, and both of the simple linear regression parameters.<\/p>\n\n\n\nInitial Plotting<\/h2>\n\n\n\n
b0<\/code> and b1<\/code> till now. We need to optimize both these values so that we will have the best fit for the data in hand.<\/p>\n\n\n\nb0<\/code> and b1<\/code> so that we can visualize the initial linear model before any optimization. Later we will be optimizing the values to get the best linear model that we can. The code snippet below initializes the parameters b0<\/code> and b1<\/code> with random values.<\/p>\n\n\n\nb0= np.random.rand()\nb1 = np.random.rand()<\/pre>\n\n\n\n
plot_fig()<\/code> function.<\/p>\n\n\n\nplot_fig(x, y, b0, b1)<\/pre>\n\n\n\n
![\"Image](\"https:\/\/debuggercafe.com\/wp-content\/uploads\/2019\/08\/initial-plot.png\")
Define the Cost Function<\/h2>\n\n\n\n
\nMSE = \\frac{1}{N} \\sum_{i=1}^{n} (y_i – (m x_i + b))^2
\n$$<\/p>\n\n\n\n
\nf(m,b) = \\frac{1}{N} \\sum_{i=1}^{n} (y_i \u2013 (mx_i + b))^2
\n$$<\/p>\n\n\n\n# cost function\ndef cost_function(x, y, b1, b0):\n num_values = len(x)\n total_error = 0.0\n for i in range(num_values):\n total_error += (y[i] - (b1*x[i] + b0))**2\n return total_error \/ num_values<\/pre>\n\n\n\n
Update the Weights<\/h2>\n\n\n\n
b0<\/code> and b1<\/code>. We will do so by subtracting the partial derivative values from b0<\/code> and b1<\/code>.<\/p>\n\n\n\ndef update_weights(x, y, b1, b0, learning_rate):\n b1_deriv = 0\n b0_deriv = 0\n num_values = len(x)\n\n for i in range(num_values):\n # Calculate partial derivatives\n b1_deriv += -2*x[i] * (y[i] - (b1*x[i] + b0))\n\n b0_deriv += -2*(y[i] - (b1*x[i] + b0))\n\n # update the weights\n b1 -= (b1_deriv \/ num_values) * learning_rate\n b0 -= (b0_deriv \/ num_values) * learning_rate\n\n return b1, b0<\/pre>\n\n\n\n
update_weights()<\/code> function accepts five arguments. We pass the data points x<\/code> and targets y<\/code> as usual along with b0<\/code> and b1<\/code>. The fifth argument is the learning rate. Learning rate specifies how quickly our algorithm will converge to an optimal solution (find a solution).<\/p>\n\n\n\nThe Training Function<\/h2>\n\n\n\n
update_weights()<\/code> function accepts, it takes one more argument. That is the number of iterations. The number of iterations defines how many times our algorithms will be trained on the data fully. For a low learning rate, number iterations will be high and for a high learning rate, it will be less.<\/p>\n\n\n\ndef train(x, y, b1, b0, learning_rate, iterations):\n costs = []\n\n for i in range(iterations):\n b1, b0 = update_weights(x, y, b1, b0, learning_rate)\n\n # calculate the cost\n cost = cost_function(x, y, b1, b0)\n costs.append(cost)\n\n # plot the linear model\n if i % 50 == 0:\n plot_fig(x, y, b0, b1, name='linear-epoch-'+str(i))\n\n return b1, b0, costs<\/pre>\n\n\n\n
train()<\/code> function. We will run the algorithm for 200 epochs and plot the linear model every 50 epochs to get a good look at the learning process. Also, the learning rate is going to be 0.01.<\/p>\n\n\n\ntrain(x, y, b1, b0, 0.01, 200)<\/pre>\n\n\n\n
![\"Image](\"https:\/\/debuggercafe.com\/wp-content\/uploads\/2019\/08\/linear-epoch-0.png\")
![\"Image](\"https:\/\/debuggercafe.com\/wp-content\/uploads\/2019\/08\/linear-epoch-50.png\")
![\"Image](\"https:\/\/debuggercafe.com\/wp-content\/uploads\/2019\/08\/linear-epoch-100.png\")
![\"Image](\"https:\/\/debuggercafe.com\/wp-content\/uploads\/2019\/08\/linear-epoch-150.png\")
Summary and Conclusion<\/h2>\n\n\n\n