The hickory nut has rich nutritional and high economic value, but its inner wall structure is complex, making it inconvenient to separate the kernel from the hickory nut fragments for consumption. In this study, we apply a deep convolutional neural network algorithm to hickory nut fragments after shell breaking, to accurately sort kernels, shells, and unseparated bodies. Region segmentation is used to eliminate interference when locating multiple small targets of different sizes, and a custom dataset of 15,000 target images after shell breaking is created. Then, three different loss function optimization methods are evaluated for model optimization. Results revealed that the model only minimizes the average cross-entropy loss, and that using dropout at the first fully connected layer with a dropout rate 0.1 provides the best shell, kernel, and unseparated body identification accuracy of 97.14%, 99.66%, and 96.48%, respectively. Notably, the kernel and unseparated body recognition rates are considerably improved over other methods. Finally, a total of 450 hickory nut fragments were divided into 30 batches for a sorting experiment. The resulting shell, kernel, and unseparated body identification accuracy is 94.00%, 98.67%, and 95.33%, respectively. These results demonstrate that the proposed method provides excellent classification of hickory nut fragments according to the material. This approach can be easily applied to any small target sorting application.