[IEEE SoutheastCon 2016 - Norfolk, VA, USA (2016.3.30-2016.4.3)] SoutheastCon 2016 - An empirical analysis of feature engineering for predictive modeling

Machine learning models, such as neural networks, decision trees, random forests, and gradient boosting machines, accept a feature vector, and provide a prediction. These models learn in a supervised fashion where we provide feature vectors mapped to the expected output. It is common practice to engineer new features from the provided feature set. Such engineered features will either augment or replace portions of the existing feature vector. These engineered features are essentially calculated fields based on the values of the other features. Engineering such features is primarily a manual, timeconsuming task. Additionally, each type of model will respond differently to different kinds of engineered features. This paper reports empirical research to demonstrate what kinds of engineered features are best suited to various machine learning model types. We provide this recommendation by generating several datasets that we designed to benefit from a particular type of engineered feature. The experiment demonstrates to what degree the machine learning model can synthesize the needed feature on its own. If a model can synthesize a planned feature, it is not necessary to provide that feature. The research demonstrated that the studied models do indeed perform differently with various types of engineered features. I Introduction II Background and Prior Work III Experiment Design and Methodology III-A Counts III-B Differences and Ratios III-C Distance Between Quadratic Roots III-D Distance Formula III-E Logarithms and Power Functions III-F Max of Inputs III-G Polynomials III-H Rational Differences and Polynomials IV Results Analysis IV-A Neural Network Results IV-B Support Vector Machine Results IV-C Random Forest Results IV-D Gradient Boosted Machine V Conclusion & Further Research References Annotation Annual Region 3 Trifecta of Student competitions, Technical papers and volunteer training and administration