Linear Regression with ODE's

Aug 20, 2025

Failed to render LaTeX expression — no expression found

Linear regression is one of the simplest and most widely used machine learning algorithms. It helps us find the best-fitting line that explains the relationship between the independent variable x and the dependent variable y.

What is Linear Regression?

The model assumes that the dependent variable yy can be expressed as a linear function of the independent variable x:

y = mx + c

where:

m = slope of the line (how much y changes with x)
c = intercept (value of y when x=0)

When we have n data points (x1, y1), (x2, y2), …, (xn, yn), we want to find m and c that minimize the difference between the actual data and our predicted line.

Error Function (Least Squares Method)

For each point, the error (residual) is:

\(e_i = y_i - (m x_i + c) \)

We want to minimize the sum of squared errors (SSE):

\(E(m, c) = \sum_{i=1}^{n} \left(y_i - (m x_i + c)\right)^2 \)

Why squared error instead of absolute error?

Squaring makes all errors positive.
It gives more weight to larger errors.
It is differentiable, making calculus-based optimization possible.

Derivation of m and c

To minimize E(m,c), we take partial derivatives to m and c, and set them to zero.

Step 1: Differentiate with respect to c

\(\frac{\partial E}{\partial c} = -2 \sum_{i=1}^{n} \big(y_i - (m x_i + c)\big) = 0 \)

\(\sum_{i=1}^{n} y_i = m \sum_{i=1}^{n} x_i + n c \)

\(c = \frac{\sum_{i=1}^{n} y_i - m \sum_{i=1}^{n} x_i}{n} \)

Step 2: Differentiate with respect to m

\(\frac{\partial E}{\partial m} = -2 \sum_{i=1}^{n} x_i \big(y_i - (m x_i + c)\big) = 0 \)

\(\sum_{i=1}^{n} x_i y_i = m \sum_{i=1}^{n} x_i^2 + c \sum_{i=1}^{n} x_i \)

\(m = \frac{n \sum_{i=1}^{n} x_i y_i - \Big(\sum_{i=1}^{n} x_i\Big)\Big(\sum_{i=1}^{n} y_i\Big)}{n \sum_{i=1}^{n} x_i^2 - \Big(\sum_{i=1}^{n} x_i\Big)^2} \)

Example Calculation

Suppose we have the dataset:

\((x_1, y_1) = (1, 2), \quad (x_2, y_2) = (2, 3), \quad (x_3, y_3) = (3, 5), \quad (x_4, y_4) = (4, 4) \)

Step 1: Compute required sums

\(\begin{aligned} \sum_{i=1}^{4} x_i &= 1 + 2 + 3 + 4 = 10,\\ \sum_{i=1}^{4} y_i &= 2 + 3 + 5 + 4 = 14,\\ \sum_{i=1}^{4} x_i^2 &= 1^2 + 2^2 + 3^2 + 4^2 = 30,\\ \sum_{i=1}^{4} x_i y_i &= 1\cdot 2 + 2\cdot 3 + 3\cdot 5 + 4\cdot 4 = 39. \end{aligned} \)

Step 2: Compute slope mm

\(\begin{aligned} m &= \frac{n\sum_{i=1}^{n} x_i y_i - \left(\sum_{i=1}^{n} x_i\right)\left(\sum_{i=1}^{n} y_i\right)} {n\sum_{i=1}^{n} x_i^2 - \left(\sum_{i=1}^{n} x_i\right)^2} \\ &= \frac{4(39) - (10)(14)}{4(30) - (10)^2} \\ &= \frac{156 - 140}{120 - 100} \\ &= \frac{16}{20} \\ &= 0.8. \end{aligned} \)

Step 3: Compute intercept cc

\(\begin{aligned} c &= \frac{\sum_{i=1}^{n} y_i - m \sum_{i=1}^{n} x_i}{n} = \frac{14 - (0.8)(10)}{4} = \frac{14 - 8}{4} = \frac{6}{4} = 1.5,\\[6pt] \text{Final line: }\quad y &= 0.8\,x + 1.5. \end{aligned} \)

Applications of Linear Regression

Predicting house prices based on size.
Estimating sales from advertising spend.
Understanding relationships in economics and biology.

Balasai’s Substack

Discussion about this post

Ready for more?