# 线性回归

## 1. 简单线性回归

只有一个未知数x，两个![\Theta](https://private.codecogs.com/gif.latex?%5CTheta)参数的，称为简单线性回归，一条直线

![img](https://img-blog.csdnimg.cn/20190218160316334.png)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FHW1yaJXk8XnWUXtTYsiO%2Ffile.gif?alt=media)

1.1表示形式

![img](https://img-blog.csdnimg.cn/20190218160417382.png)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FN3YBPDiLf339Aa2Z4Hce%2Ffile.gif?alt=media)

1.2 定义损失

![img](https://img-blog.csdnimg.cn/20190218160501540.png)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FhESn1VIQO4PJ3WKlt9nN%2Ffile.gif?alt=media)

1.3 求参，极大似然

![img](https://img-blog.csdnimg.cn/20190218160817842.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FGDk86S7U4aknyAkG8pMh%2Ffile.gif?alt=media)

## 2.多元线性回归

2.1形式

![img](https://img-blog.csdnimg.cn/2019021816111388.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2F38HSvhDSxaCC1ScYjNIe%2Ffile.gif?alt=media)

2.2误差

![img](https://img-blog.csdnimg.cn/20190218161244600.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FeDJDWgnLYtnDGORZN0e7%2Ffile.gif?alt=media)

2.3求参

![img](https://img-blog.csdnimg.cn/20190218161323304.png)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FFBJ6jEMEId3sNnf79nXN%2Ffile.gif?alt=media)

2.4问题

![X^{T}X](https://private.codecogs.com/gif.latex?X%5E%7BT%7DX)通常不是nxn矩阵，既，数据量`行数n`，自变量及偏置`列p+1`，通常`n!=p+1`，也就是说矩阵![X^{T}X](https://private.codecogs.com/gif.latex?X%5E%7BT%7DX)不存在逆；

`解决办法：`

`1)加入单位矩阵`，让其变正定，其中![\lambda](https://private.codecogs.com/gif.latex?%5Clambda)要足够大使得括号内矩阵可逆。

![img](https://img-blog.csdnimg.cn/20190218161823206.png)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2F4TD4uCCTE8W433vbRyRe%2Ffile.gif?alt=media)

`2）可以通过一些其它随机优化器算法`寻找参数。

## 3 实现/应用

### 3.1 `np.linalg`、`scipy优化器`实现

np.linalg.lstsq

```python
import matplotlib.pyplot as plt

x=Aarray([[ 0.,  1.],       [ 1.,  1.],       [ 2.,  1.],       [ 3.,  1.]])
y = np.array([-1, 0.2, 0.9, 2.1])
m, c = np.linalg.lstsq(x, y)[0]


plt.plot(x, y, 'o', label='Original data', markersize=10)
plt.plot(x, m*x + c, 'r', label='Fitted line')
plt.legend()
plt.show()
```

![img](https://img-blog.csdnimg.cn/20190209115428751.png)

scipy的最小二乘法优化器：from scipy.optimize import leastsq

```python
import numpy as np
from scipy.optimize import leastsq
import matplotlib.pyplot as plt

plt.rcParams['font.sans-serif'] = ['SimHei']  # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False  # 用来正常显示负号


# np.random.seed(0)


def obj_func(x):
    # 创建目标函数，后面假设不知道，需要拟合出来接近这个目标函数
    # f = 1 / (1 + np.exp(-x))
    # f = np.tan(x)
    f = np.sin(x)
    return f


size = 100
x = np.linspace(-5, 5, size)
y_ = obj_func(x)
# 根据目标函数，产生随机的噪声，近似接近目标函数obj_func的数据x,y
y = np.random.normal(0, 1, (size)) / 100 + y_


def scatter(x, y, c, label):
    plt.scatter(x, y, label=label, c=c)


def plot(x, y, c, label):
    plt.plot(x, y, label=label, c=c)


# 创建拟合函数，参数为p的多项式函数，去拟合obj_func
def fit_func(p, x):
    # poly1d([2, 1, 1]) 等同于2*x^2  + 1* x^1 +1*x^0 = 2x^2 + x +1
    f = np.poly1d(p)
    return f(x)


# 创建损失函数，Scipy优化器依照损失最小化取优化，寻找最佳参数，这里既为多项式函数的系数，和偏置。
def coust_func(p, x, y):
    f = fit_func(p, x)
    # 定义为残差最小
    coust = (f - y)
    return coust


# 还可以定义正则化损失
regularization = 0.0001


def coust_func_regularization(p, x, y):
    f = fit_func(p, x)
    reg = np.sqrt(0.5 * regularization * np.square(p))
    coust = (f - y)
    coust = np.append(coust, reg)
    return coust


# 适配函数
def fiting_func(x, y, coust_func, M=1):
    init_p = np.random.rand(M + 1)
    # scipy最小二乘函数，传入:(损失函数，损失函数初始参数,(数据x,y))
    leastq = leastsq(coust_func, init_p, args=(x, y))
    return leastq


# -------绘制真实函数-------
plot(x, y_, c='red', label='real function')

# 多项式拟合中的最高次
p = 5
# 无正则化损失函数
leastq = fiting_func(x, y, coust_func, p)
# 正则化损失函数
leastq_reg = fiting_func(x, y, coust_func_regularization, p)

pred_x = np.linspace(-5, 5, 10)
# 预测
predY = fit_func(leastq[0], pred_x)
predY_reg = fit_func(leastq_reg[0], pred_x)

# -------拟合函数-------
plot(pred_x, predY, c='blue', label='fit function')
# -------正则拟合函数-------
plot(pred_x, predY_reg, c='yellow', label='fit reg function')
plt.legend()
plt.show()
```

![img](https://img-blog.csdnimg.cn/20190218163221278.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)

### 3.2 sklearn库的实现

1）LinearRegression：<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html，普通的线性回归，它的损失函数也是最简单的，如下：>

![J(\mathbf\theta) = \frac{1}{2}(\mathbf{X\theta} - \mathbf{Y})^T(\mathbf{X\theta} - \mathbf{Y})](https://private.codecogs.com/gif.latex?J%28%5Cmathbf%5Ctheta%29%20%3D%20%5Cfrac%7B1%7D%7B2%7D%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29%5ET%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29)

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.linear_model import LinearRegression

'''
https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression
使用场景：一般来说，只要我们觉得数据有线性关系，LinearRegression类是我们的首先。如果发现拟合或者预测的不好，再考虑用其他的线性回归库。
'''
fig = plt.figure()
plt.rcParams['font.sans-serif'] = ['SimHei']  # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False  # 用来正常显示负号
ax = Axes3D(fig)

# 构造X数据
X = np.random.randint(0, 7, (20, 2))
# 构造y数据， y = 1 * x_0 + 2 * x_1 + 3，后面打印参数会发现，是一致的
y = np.dot(X, np.array([1, 2])) + 3
# 绘制原始数据
ax.scatter(X[:, 0], X[:, 1], y, marker='o')

# 参数打印
reg = LinearRegression().fit(X, y)
print('分数：', reg.score(X, y))
print('参数：', reg.coef_)
print('截距：', reg.intercept_)

# 测试数据生成
test_x0 = np.linspace(0, 5, 10)
test_x1 = np.linspace(0, 5, 10)
test_X = np.array([test_x0, test_x1]).T

pred_y = reg.predict(test_X)
print('预测：', pred_y)

# 生成预测图形
ax.plot(test_X[:, 0], test_X[:, 1], pred_y, c='r')
plt.show()
```

![img](https://img-blog.csdnimg.cn/20190218172415851.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FK3hmfiYkNN9YSvm8xqRv%2Ffile.gif?alt=media)

2）Ridge/RidgeCV岭回归/`L2正则化`：<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV>

使用场景：一般来说，只要我们觉得`数据有线性关系`，用LinearRegression类拟合的`不是特别好`，需要`正则化`，可以考虑用RidgeCV类。

其思想是对损失函数`加入正则化`：

![J(\mathbf\theta) = \frac{1}{2}(\mathbf{X\theta} - \mathbf{Y})^T(\mathbf{X\theta} - \mathbf{Y}) + \frac{1}{2}\alpha||\theta||\_2^2](https://private.codecogs.com/gif.latex?J%28%5Cmathbf%5Ctheta%29%20%3D%20%5Cfrac%7B1%7D%7B2%7D%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29%5ET%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29%20\&plus;%20%5Cfrac%7B1%7D%7B2%7D%5Calpha%7C%7C%5Ctheta%7C%7C_2%5E2)

使得系数`求解变为`：

![\mathbf{\theta = (X^TX + \alpha E)^{-1}X^TY}](https://private.codecogs.com/gif.latex?%5Cmathbf%7B%5Ctheta%20%3D%20%28X%5ETX%20\&plus;%20%5Calpha%20E%29%5E%7B-1%7DX%5ETY%7D)

其中额外参数a，在sklearn中，Ridge类并没有用到交叉验证之类的验证方法，`RidgeCV类的损失函数和损失函数的优化方法完全与Ridge类相同`，区别在于验证方法:alphas=\[0.1, 1.0, 10.0,...]，`RidgeCV可以传入多个参数，自动帮你选择最佳的`。

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.linear_model import RidgeCV


fig = plt.figure()
plt.rcParams['font.sans-serif'] = ['SimHei']  # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False  # 用来正常显示负号
ax = Axes3D(fig)

# 构造X数据
X = np.random.randint(0, 7, (20, 2))
# 构造y数据， y = 1 * x_0 + 2 * x_1 + 3，后面打印参数会发现，是一致的
y = np.dot(X, np.array([1, 2])) + 3
# 绘制原始数据
ax.scatter(X[:, 0], X[:, 1], y, marker='o')

# 参数打印
reg = RidgeCV(alphas=[0.1, 1.0, 10.0]).fit(X, y)
print('分数：', reg.score(X, y))
print('参数：', reg.coef_)
print('截距：', reg.intercept_)

# 测试数据生成
test_x0 = np.linspace(0, 5, 10)
test_x1 = np.linspace(0, 5, 10)
test_X = np.array([test_x0, test_x1]).T

pred_y = reg.predict(test_X)
print('预测：', pred_y)

# 生成预测图形
ax.plot(test_X[:, 0], test_X[:, 1], pred_y, c='r')
plt.show()

#
分数： 0.9999988863966677
参数： [0.99947475 1.9975713 ]
截距： 3.009245913853947
预测： [ 3.00924591  4.6742715   6.33929708  8.00432266  9.66934824 11.33437383
 12.99939941 14.66442499 16.32945058 17.99447616]
```

![img](https://img-blog.csdnimg.cn/20190218173930150.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYW5nNDI1Nzc2MDI0,size_16,color_FFFFFF,t_70)![点击并拖拽以移动](https://firebasestorage.googleapis.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LlRDjw7ExCWOBrbokF1%2Fuploads%2FgVqZoCWIkoGBBD9033bW%2Ffile.gif?alt=media)

3）Lasso/LassoCV/`L1正则化`:<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV>

如果输入特征的`维度很高`，而且是`稀疏线性关系`的话，`RidgeCV类就不合适`了。 这时应该主要考虑下面Lasso回归类家族。

![J(\mathbf\theta) = \frac{1}{2m}(\mathbf{X\theta} - \mathbf{Y})^T(\mathbf{X\theta} - \mathbf{Y}) + \alpha||\theta||\_1](https://private.codecogs.com/gif.latex?J%28%5Cmathbf%5Ctheta%29%20%3D%20%5Cfrac%7B1%7D%7B2m%7D%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29%5ET%28%5Cmathbf%7BX%5Ctheta%7D%20-%20%5Cmathbf%7BY%7D%29%20\&plus;%20%5Calpha%7C%7C%5Ctheta%7C%7C_1)

Lasso回归可以`使得一些特征的系数变小`，甚至还是一些绝对值较小的系数直接变为0。`增强模型的泛化能力`。

**使用场景：**&#x8981;在一堆特征里面找出主要的特征，那么Lasso回归更是首选了。但是Lasso类需要自己对α调优，所以不是Lasso回归的首选，一般用到的是下一节要讲的LassoCV类，可以自动帮你选择，或者你提供一组alphas数据让它选择。

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.linear_model import LassoCV

fig = plt.figure()
plt.rcParams['font.sans-serif'] = ['SimHei']  # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False  # 用来正常显示负号
ax = Axes3D(fig)

# 构造X数据
X = np.random.randint(0, 7, (20, 2))
# 构造y数据， y = 1 * x_0 + 2 * x_1 + 3，后面打印参数会发现，是一致的
y = np.dot(X, np.array([1, 2])) + 3
# 绘制原始数据
ax.scatter(X[:, 0], X[:, 1], y, marker='o')

# 参数打印
reg = LassoCV(alphas=[0.1, 1.0, 10.0]).fit(X, y)
print('分数：', reg.score(X, y))
print('参数：', reg.coef_)
print('截距：', reg.intercept_)

# 测试数据生成
test_x0 = np.linspace(0, 5, 10)
test_x1 = np.linspace(0, 5, 10)
test_X = np.array([test_x0, test_x1]).T

pred_y = reg.predict(test_X)
print('预测：', pred_y)

# 生成预测图形
ax.plot(test_X[:, 0], test_X[:, 1], pred_y, c='r')
plt.show()
#
分数： 0.9998181323039398
参数： [0.97476331 1.98150169]
截距： 3.109000578571484
预测： [ 3.10900058  4.75137002  6.39373947  8.03610891  9.67847836 11.3208478
 12.96321725 14.60558669 16.24795614 17.89032558]
```

还有许多其它类型的回归介绍：<https://www.cnblogs.com/pinard/p/6026343.html>

官方文档关于线性模型的使用说明：<https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model>

其使用方法和上面一致，这一点在sklearn库中很好。


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://im-qianuxn.gitbook.io/pytorch/ji-suan-ji/ml/linearregression.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.