Low-area architecture design of multi-mode activation functions with controllable maximum absolute error for neural network applications

In the development of the neural network (NN), the activation function has become more and more important. The selection of the activation function indirectly affects the convergence speed and accuracy. This study proposes the multi-mode activation function design (MMAFD) based on the least square method (LSM) with a controllable maximum absolute error (MAE) to support multiple activation functions. MMAFD selects the activation function to maintain the accuracy for different deep learning applications. MMAFD is implemented by TSMC 90 nm CMOS technology. In MMAFD, the power consumption is 0.98 mW, the operational frequency is 250 MHz, and the area is 0.416mm2. MMAFD is also verified by Xilinx Spartan-6 XC6SLX45 development board. Compared to the related works verified in the FPGA boards, the LUTs and slices registers are reduced by up to 62.96 % and 73.90 %.