Dropout

torch.nn.Dropout(p=0.5, inplace=False)

• p – probability of an element to be zeroed. Default: 0.5
• inplace – If set to True, will do this operation in-place. Default: False

训练过程中以概率P随机的将参数置0，其中P为置0的概率，例如P=1表示将网络参数全部置0

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.

注意: Pytorch文档中给出了一点，输出的参数会以$\frac{1}{1-p}$进行一个缩放

Furthermore, the outputs are scaled by a factor of $\frac{1}{1-p}$​ during training. This means that during evaluation the module simply computes an identity function.

下面例子展示出在dropout之后，参数变为了原来的$\frac{1}{1-p}=2$倍

input = torch.tensor([[1, 2, 3],
                      [4, 5, 6],
                      [7, 8, 9]], dtype=torch.float64)
input = torch.unsqueeze(input, 0)
m = nn.Dropout(p = 0.5)
output = m(input)

print("input: ", input)
print("output: ", output)
print("input: ", input)
'''
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
output:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
'''

当我们把nn.Dropout的inplace=True时，计算的结果就会替换掉原来的输入input，如下：

input = torch.tensor([[1, 2, 3],
                      [4, 5, 6],
                      [7, 8, 9]], dtype=torch.float64)
input = torch.unsqueeze(input, 0)
m = nn.Dropout(p = 0.5, inplace=True)
output = m(input)

print("input: ", input)
print("output: ", output)
print("input: ", input)
'''
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
output:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
input:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
'''

训练与测试的不同

在训练和测试的时候，nn.Dropout的表现是不同的，在训练时nn.Dropout会以概率p随机的丢弃一些神经元，但是在测试时，所有神经元都不会被丢弃，如下

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
        
model = Model(p=0.5) # functional dropout
# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model ', model(inputs))

# switching to eval mode
model.eval()
# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model ', model(inputs))
# show model summary
print('Print summary:')
print(model)
'''
Normal (train) model:
Model  tensor([0.0000, 1.3517, 0.0000, 0.2766, 0.3060, 1.6334, 0.0000, 0.9740, 0.9118,
        0.0000])

Evaluation mode:
Model  tensor([0.9284, 0.6758, 0.3947, 0.1383, 0.1530, 0.8167, 0.2038, 0.4870, 0.4559,
        0.2730])
Print summary:
Model(
  (drop_layer): Dropout(p=0.5, inplace=False)
)
'''