With the increasing popularity of artificial intelligence, deep learning has been applied to various fields, especially in computer vision. Since artificial intelligence is migrating from cloud to edge, deep learning nowadays should be edge-oriented and adaptive to complex environments. Aiming at these goals, this paper proposes an ICONet (illumination condition optimized network). Based on OTSU segmentation algorithm and fuzzy c-means clustering algorithm, the illumination condition classification subnet increases the environmental adaptivity of our network. The reduced time complexity and optimized size of our convolutional neural network (CNN) model enables the implementation of ICONet on edge devices. In the field of fatigue driving, we test the performance of ICONet on YawDD and self-collected datasets. Our network achieves a general accuracy of 98.56% and our models are about 590 kilobytes. Compared to other proposed networks, the ICONet shows significant success and superiority. Applying ICONet to fatigue driving detection is helpful to solve the symmetry of the needs of edge-oriented detection under complex illumination condition environments and the scarcity of related approaches.