Convolutional neural network (CNN) has been widely adopted in many tasks. Its inference process is usually applied on edge devices where the computing resources and power consumption are limited. At present, the performance of general processors cannot meet the requirement for CNN models with high computation complexity and large number of pa-rameters. Field-programmable gate array (FPGA)-based custom computing architecture is a promising solution to further enhance the CNN inference performance. The software/hardware co-design can effectively reduce the computing overhead, and improve the inference performance while ensuring accuracy. In this paper, the mainstream methods of CNN structure design, hardware-oriented model compression and FPGA-based custom architecture design are summarized, and the improvement of CNN inference performance is demonstrated through an example. Challenges and possible research directions in the future are concluded to foster research efforts in this domain.