Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network

Block Diagram

Abstract

An activation function has crucial role in a deep neural network. A simple rectified linear unit (ReLU) are widely used for the activation function. In this paper, a weighted sigmoid gate unit (WiG) is proposed as the activation function. The proposed WiG consists of a multiplication of inputs and the weighted sigmoid gate. It is shown that the WiG includes the ReLU and same activation functions as a special case. Many activation functions have been proposed to overcome the performance of the ReLU. In the literature, the performance is mainly evaluated with an object recognition task. The proposed WiG is evaluated with the object recognition task and the image restoration task. Then, the expeirmental comparisons demonstrate the proposed WiG overcomes the existing activation functions including the ReLU.

Evaluations

Comparison of the Validation Accuracy

	cifar-10	cifar-100
ReLU	0.927	0.653
selu	0.899	0.572
ELU	0.903	0.550
softplus	0.908	0.598
Leaky ReLU	0.918	0.673
SiL	0.919	0.638
PReLU	0.935	0.678
Swish	0.935	0.689
WiG (Pro.)	0.949	0.742

Comparison of the Denoising Results

Average PSNR

Noise Level	5	10	15	20	25	30
ReLU	37.13	33.86	32.02	30.76	29.77	28.96
selu	36.89	33.68	31.88	30.60	29.60	28.79
ELU	37.07	33.63	31.86	30.60	29.65	28.86
softplus	36.41	33.42	31.67	30.45	29.50	28.69
Leaky ReLU	37.07	33.81	31.97	30.68	29.69	28.89
Swish	37.13	33.70	31.91	30.69	29.72	28.93
WiG (Pro.)	37.29	34.00	32.16	30.88	29.90	29.10

Average SSIM

Noise Level	5	10	15	20	25	30
ReLU	0.9383	0.8971	0.8646	0.8378	0.8142	0.7932
selu	0.9355	0.8929	0.8607	0.8328	0.8084	0.7855
ELU	0.9362	0.8910	0.8601	0.8331	0.8094	0.7888
softplus	0.9337	0.8878	0.8547	0.8274	0.8041	0.7822
Leaky ReLU	0.9380	0.8959	0.8628	0.8350	0.8104	0.7888
Swish	0.9377	0.8928	0.8621	0.8370	0.8137	0.7928
WiG (Pro.)	0.9390	0.8993	0.8679	0.8412	0.8188	0.7981

Publication

Masayuki Tanaka, Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network, Pattern Recognition Letters, Vol. 135, pp. 354-359, Jul. 2020. [PRL]

Masayuki Tanaka, Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network, arXiv preprint arXiv:1810.01829, 2018. [arXiv]

WiG: Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network