Machine Learning Part 2

Refer code @ https://github.com/dinesh028/SparkDS/blob/master/src/indore/dinesh/sachdev/FlightDelayDriver.scala

This post talks about predicting flight delays. Since, there is growing interest in predicting flight delays beforehand in order to optimize operations and improve customer satisfaction.

We will use Historic Flight status data with Random Forest Classifier algorithm to find common patterns in late departures in order to predict flight delays and share the reasons for those delays. And, we will be using Apache Spark for same.

As mentioned in Part 1 - Classification is kind of supervised learning which requires Features (if questions) and Labels (outcome) in advance to build the model.

What are we trying to predict (Label)?

• Whether a flight will be delayed or not. (TRUE or FALSE)

What are the “if questions” or properties that you can use to make predictions?

• What is the originating airport?
• What is the destination airport?
• What is the scheduled time of departure?
• What is the scheduled time of arrival?
• What is the day of the week?
• What is the airline carrier?

Decision Trees-
Decision trees create a model that predicts the label (or class) by evaluating a set of rules that follow an if-then-else pattern. The if-then-else feature questions are the nodes, and the answers “true” or “false” are the branches in the tree to the child nodes. A decision tree model estimates the minimum number of true/false questions needed to assess the probability of making a correct decision.

For example -

1. if departure time is less than 10:15
1. True - if origination is from ORD, SFO
1. TRUE - if it is Monday or Sunday
1. True - delayed = 1
2. False - delayed = 0
2. FALSE - if destination is St. Louis
1. True - delayed = 1
2. False - delayed = 0

Random Forest is a popular ensemble learning method for classification and regression. The algorithm builds a model consisting of multiple decision trees, based on different subsets of data at the training stage. Predictions are made by combining the output from all of the trees, which reduces the variance and improves the predictive accuracy.

Machine Learning Workflow - 
Its an iterative process, which involves -

1. Data discovery and model creation-
1. Analysis of historical data
2. Identifying new data sources, which traditional analytics or databases are not using, due to the format, size, or structure.
3. Collecting, correlating, and analyzing data across multiple data sources.
4. Knowing and applying the right kind of machine learning algorithms to get value out of the data.
5. Training, testing, and evaluating the results of machine learning algorithms to build a model.
2. Using the model in production to make predictions.
3. Data discovery and updating the model with new data (#1 again)

Lets build a decision Tree with label - 

• Delayed = 0
• Delayed = 1 if delay > 40 minutes

Features - 

day of the week, scheduled departure time, scheduled arrival time, carrier, scheduled elapsed time, origin, destination, distance

In order for the features to be used by a machine learning algorithm, they must be transformed and put into feature vectors, which are vectors of numbers representing the value for each feature.

Transformer: A transformer is an algorithm that transforms one DataFrame into another DataFrame. We will use transformers to get a DataFrame with a features vector column.

Estimator: An estimator is an algorithm that can be fit on a DataFrame to produce a transformer. We will use a an estimator to train a model, which can transform input data to get predictions.

Pipeline: A pipeline chains multiple transformers and estimators together to specify an ML workflow.

Once pipeline is defined then we train the model. We would like to determine which parameter values of the Random Forest Classifier produce the best model. A common technique for model selection is k-fold cross validation, where the data is randomly split into k partitions. Each partition is used once as the testing data set, while the rest are used for training. Models are then generated using the training sets and evaluated with the testing sets, resulting in k model performance measurements. The model parameters leading to the highest performance metric produce the best model.

Once Model is created then we need to do predictions and evaluate the model using test data set.

Once Model is developed then we can save it for future use in Production. This saves both the feature extraction stage and the random forest model chosen by model tuning.

Refer code @ https://github.com/dinesh028/SparkDS/blob/master/src/indore/dinesh/sachdev/FlightDelayDriver.scala

Part 3 - https://techdevins.blogspot.com/2019/12/machine-learning-part-3.html