Analyzing the Performance of Machine Learning Models on Aircraft Classification Tasks across Various Conditions

Abstract

As more and more aircraft are used, using an efficient and accurate tool to identify aircraft types will help manage air safety. As a basic framework for machine learning, a Convolutional neural network is widely used to make image recognition. This paper presents a method to classify different types of military and civilian aircraft using neural networks and explore the performance differences by changing batch size and learning rate. Five models are selected to achieve classification for ten types of aircraft images from the Fine-Grained Visual Classification of Aircraft (FGVC)-Aircraft dataset. Except for the existing models like Visual Geometry Group 16-layer Network (VGG16), Residual Network with 152 Layers (Resnet152), Densely Connected Convolutional Networks (DenseNet), and Efficient Neural Network (EfficientNet), a customized Convolutional Neural Network (CNN) is also designed to complete the image classification task. The parameters used to demonstrate performance are latency, throughput, model convergence speed, accuracy, f1-score, and confusion matrix. By analyzing the results, the customized CNN performs best among these five models, having both good accuracy and latency. VGG16 and DenseNet perform well although each has certain deficiencies in terms of latency and accuracy respectively. The performance of EfficientNet and Resnet152 is not good, and their accuracy is relatively lower compared to other models. Therefore, the best model can be obtained for solving this task and make predictions on the performance of each model when classifying pictures of more types of aircraft.