Automatic classification of fish is essential for developing marine ecology, behavior analysis, aquaculture management, and health monitoring. However, the existing underwater categorization detection technology is far from keeping up with the escalating demands. In this study, Deepwater-SE based on the Squeeze-and-Excitation mechanism (SE) was proposed to classify fish with an unrestricted and real natural environment. To improve the efficiency of the network, the Depthwise Separable Convolution and the SE-ResNet were added, and at the same time, the detailed analysis module and the feature extraction module were designed before the backbone network specifically. A total of 27,370 photos of Fish4Knowledge(F4K) datasets were used for training and testing. The experimental results demonstrated that the proposed model had the highest mean accuracy of 99.58% among nine comparison models, including LDA+SVM, Raw-Pixel SVM, Raw-Pixel Softmax, Raw-Pixel Nearest Neighbour, Deep fish-Softmax-Aug, VLFeat Dense-SIFT, Alex-FT-Soft, Deep Fish-SVM, Deep-CNN. In addition, the proposed method also led to better results by providing excellent rates of Precision, Recall, and F1, respectively. All the results indicated that Deepwater-SE could categorize fish properly and effectively while showing great robustness and accuracy in large datasets.