向量 / 矩阵范数定义

一范数：元素绝对值之和
$${\Vert \begin{array}{c}X\end{array}\Vert }_1=\sum _{i=1}^n|x_i|$$
二范数：元素平方和的平方根
$${\Vert \begin{array}{c}X\end{array}\Vert }_2=\sqrt{\sum _{i=1}^n|x_i^2|}$$
MAE：Mean Absolute Error，平均绝对误差
$$MAE(x,y)=\frac{1}{n}\sum _{i=1}^n|y_i-x_i|$$
MSE：Mean Square Error，均方误差
$$MSE(x,y)=\frac{1}{n}\sum _{i=1}^n(y_i-x_i{)}^2$$
RMSE：Root Mean Square Error，均方根误差
$$RMSE(x,y)=\sqrt{MSE(x,y)}=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i-x_i{)}^2}$$

PyTorch范数API

torch.norm(input, p='fro', dim=None, keepdim=False, out=None, dtype=None)

p：范数类型，默认为Frobenius

范数计算

>>> import torch
>>> x = torch.tensor([1, 2, 3], dtype=torch.float32)
>>> x
tensor([1., 2., 3.])
>>> torch.norm(x, 1) # 一范数
tensor(6.)
>>> torch.norm(x, 2) # 二范数
tensor(3.7417)

PyTorch损失函数

torch.nn.L1Loss(size_average=None, reduce=None, reduction='mean')
torch.nn.MSELoss(size_average=None, reduce=None, reduction='mean')

reduction为默认参数'mean'时，计算结果会除以n（元素个数）；设置为'sum'时，计算结果等于范数（不除n）

官方文档：

reduction (string*,* optional) – Specifies the reduction to apply to the output: 'none' | 'mean' | 'sum'. 'none': no reduction will be applied, 'mean': the sum of the output will be divided by the number of elements in the output, 'sum': the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: 'mean'

损失函数和范数计算结果对比

>>> import torch
>>> x = torch.tensor([1, 2], dtype=torch.float32)
>>> y = torch.tensor([3, 4], dtype=torch.float32)

>>> MAE = torch.nn.L1Loss() # MAE = (abs(3 - 1) + abs(4 - 2)) / 2
>>> MAE(x, y)
tensor(2.)

>>> l1_norm = torch.nn.L1Loss(reduction='sum') # (y - x)的一范数
>>> l1_norm(x, y)
tensor(4.)
>>> torch.norm((y - x), 1) # (y - x)的一范数
tensor(4.)

>>> MSE = torch.nn.MSELoss() # MSE = ((3 - 1) ** 2 + (4 - 2) ** 2) / 2, 默认 reduction='mean'
>>> MSE(x, y)
tensor(4.)
>>> torch.sqrt(MSE(x, y)) # RMSE
tensor(2.)

>>> MSE_sum = torch.nn.MSELoss(reduction='sum') # MSE * n
>>> l2_norm_square = MSE_sum # (y - x)的二范数的平方
>>> l2_norm_square(x, y)
tensor(8.)
>>> torch.sqrt(l2_norm_square(x, y)) # (y - x)的二范数, sqrt(MSE * n)
tensor(2.8284)
>>> torch.norm((y - x), 2) # (y - x)的二范数
tensor(2.8284)