EMS702P–Statistical Thinking and Applied Machine Learning

Case Study: Artificial Intelligence in Air Traffic Management (ATM)

Data-Centric Engineering in Airport Airside Operations

16/10/2024

Student Pack – 2024/2025

1.   Problem description

In order to optimise airport airside operations, accurate taxi time prediction has played an indispensable role. It is not only important to create more robust schedules and identify choke points between gate and runway for practitioners, but also helps the government analysts to estimate the optimal airport capacity and evaluate the regulation impacts.

This case study utilises taxiing data from Manchester International Airport (MAN) ranking among the 2nd busiest airports in the UK.

In order to ensure taxi time prediction accuracy, one should comprehensively consider relevant features that may affect taxi time. In this case study, the data comes with up to 25 features, aiming to provide a sufficient set of features for the taxi time prediction. These relevant features are divided into three categories, including

(a) aircraft and airport operational factors, (b) airport congestion level and (c) aircraft average speed.

You will need to complete 4 tasks in this case study:

(i)    Collecting/Selecting and pre-processing data using the programs downloaded from QM+.

(ii)   Applying feature  engineering technologies,  in  particular  Principal  Component Analysis  (PCA),  for feature extraction using the dataset that has been collected/selected by you.

(iii)  Applying supervised learning, including the Neural Network (NN), Linear Regression (LR) and Adaptive Neuro-Fuzzy Inference Systems (ANFIS) to predict the taxi time.

(iv)  Discussing the pros and cons of the  different machine learning tools from the aspects of prediction accuracy, generalisation capability and model transparency.

You may also find Appendix C: Steps to Success helpful to complete the above tasks.

2. Description of tasks

2.1 Data collection/selection and processing

The students will work together in groups. Each group should collect data from https://opensky-network.org/ for MAN or select data over a period of time from the provided dataset. In order to achieve appropriate data sets for machine learning studies, the following rules need to be followed:

Each data set should be a minimum of 10 times the number of useful features data points (you need to decide the time to start recording/retrieving your data and for how long), but do not exceed 1000 data points (otherwise it will consume lots of your time).

Data sets collected/selected by groups should be different (consider different time periods).

The collected/selected data sets should be pre-processed, so that they are suitable for machine learning based modelling. Data processing is conducted using the programs downloaded from QM+. Details of the data processing programs are shown in Appendix A.

Appropriately dividing the data set into sub-sets including training, validation and testing.

2.2 Feature extraction and selection

The available features in the collected/selected data are explained in Appendix B.

The feature extraction is conducted by using PCA. PCA is a linear dimensionality reduction technique that can be utilised for extracting useful information from a high-dimensional space by projecting it into a lower dimensional sub-space. You need to do research on how to use PCA as a tool to select features (e.g., a paper included in W3.3 slide provides a potential way). You need to decide how many features you will use and explain the reason of the choice.

2.3 NN & LR & ANFIS

NN and LR are classical machine learning models and the foundation of many other advanced machine learning approaches. ANFIS represent a hybrid intelligent system. In this case study, you need to apply these three models to solve the taxi time prediction problem.

For NN, refer to the following rules:

Applying Back Propagation (BP) NN for prediction.

Choose the number of the hidden layer nodes and explain the reason.

For LR, refer to the following rules:

Applying the polynomial basis function for LR modelling.

Decide the maximum order of the polynomial function and explain the reason.

For ANFIS, refer to the following rules:

Applying the Clustering for constructing the initial Fuzzy Rule-based System.

Decide the learning algorithms to further train Fuzzy Rule-based System and explain the reason.

You will also need to decide the training, validation or testing data sets used for the machine learning models.

Statistical tests of the obtained regression models are needed.

2.4 Comparison

Compare NN, LR and ANFIS from the aspects of prediction accuracy, generalisation capability and model transparency.

The prediction accuracy is quantified by the Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Relative Error (MRE), etc.. Statistical tests and/or interval estimation are needed as the means to report the model performance/skill.

2.5 Conclusion

Read the following three reports that are provided on QM+. You will gain an understanding of challenges associated with the safe use of AI, the importance of data to AI, methods for determining different sub-datasets for training and verification purposes, and appreciate the urgent need to develop new technologies, processes, tools, and guidance for assuring the safety of systems based on these technologies. Draw conclusions of your work and results in this coursework, using these understandings of AI in life critical engineering.

•    Read  the  report  “The  FLY AI  Report”  produced  by  EUROCONTROL.  This  report  provides  an overview of the many ways that AI is already applied in the aviation sector and ATM and assesses its potential to transform. the sector.

•    Read the report “AFE 87 – Machine Learning” produced by Aerospace Vehicle Systems Institute. This report provides  an  overview  of different Machine Learning paradigms  and how  these  emerging technologies present new challenges to existing certification processes in the aviation sector.

•    Read the article “Aircraft taxi time prediction: Comparisons and insights” to get a better idea of how different Machine Learning algorithms have been used and the pros and cons of different methods.

3.   Assessment

The first deliverable of this case study is a 5-minute pre-recorded group presentation (15% of the whole module). The second deliverable of this case study is a 10-page group report (25% of the whole module). The main parts and the suggested weightage of each part to be included in the presentation/report are indicated in the table below. Both deliverables will be marked against these main parts.

Main parts	Guidance	Weightage (100%)
Introduction	Details of individual contributions towards the case study and report. Description of the problem and how the predictive model can be used to improve ATM at present and in future operations concepts.	5%
Data Preparation	Description of your data set and justification of your choice of training, validation and testing sub-datasets. Description of the steps taken during the PCA analysis and interpreting results, including 1. Pre-processing data for the PCA, 2. Calculating the principal components, 3. Choosing the appropriate number of principal components, 4. Interpreting the principal components and feature importance, 5. Plotting selected principal components, 6. Assessing the information loss.	30%
NN & LR&ANFIS	Description of steps taken to construct the NN, LR and ANFIS models and interpreting results, including 1. Choice of model structures, 2.    Which sub-dataset(s) is(are) used for training, testing and validation, 3. Quantify and visualise the performance of the predictive model on all sub-datasets, 4. Conducting the statistical tests of the regression models, 5. Assessing transparency and generalisation capability of the model.	35%
Comparison	• Discussing the pros and cons of the NN, LR and ANFIS approaches in taxi   time prediction    from   the    aspects   of   prediction    accuracy, generalisation capability and model transparency. • Quantifying and visualising the comparison results in the report.	20%

Conclusions

• Drawing  conclusions of the  study,  as well  as further crucial  steps  for machine learning approaches to be applied in real operational environment at airports.

10%

The pre-recorded presentation should be no more than 5 minutes consisting of 5~10 slides. The presentation should be clear and concise, including all the main parts mentioned above. Submission via QM+ for this deliverable include:

o Pre-recorded presentation (5 minutes).

o Slides (5~10 slides).

The group report should have the generic/standard structure and describe the problem, the way in which it was tackled, the analysis carried out and the results obtained. The report should be clear and legible and conform to the following requirements:

Font size 11 is the minimum font that is acceptable and page margins should be at least 2 cm in all directions.

Maximum 10 pages of A4 inclusive of the title, authors, details of individual contributions, main body of the report and Lists of references.

You need to include a list of references based on a standard method (such as Harvard or IEEE style.: you can information about Harvard/IEEE style. in the internet)). All the references need to be cited within the text.

Report exceeding the page limit, or not adhering to the specified format, will be penalised.

Submission should be made in a single PDF file through the QM+ submission point.

Machine Learning codes and dataset need to be ready to run for checking and assessment, packed and uploaded to the QM+.

Codes and dataset that are required to be submitted via QM+ include:

o Collected/Selected dataset: your dataset named as features.csv that will be used as the input to Feature Reduction.

o Feature Reduction: reduce_gX.py (Example: reduce_g1.py for Group 1 submission). This file of yours will read the file ‘features.csv’ (see Appendix A). Then, it will use feature reduction techniques to produce a dataset of the reduced features that is ready for Machine Learning.

o Machine Learning: XX_gX.py (Example: NN_g1.py for Neural Network of Group 1 submission) for NN and LR, and ANFIS_gX.m for ANFIS. These files of yours will read the dataset of the reduced features and apply the respective machine learning methods (NN, LR, ANFIS) to train, test and validate the elicited models for taxi time prediction.