Feature engineering: transforming raw data into useful model inputs
Feature engineering transforms raw data into predictive model features. It includes techniques like normalization, one-hot encoding, and feature crosses. The process is iterative and requires domain knowledge to select relevant attributes.
What is feature engineering?
Feature engineering is the process of transforming raw data into relevant information for machine learning models, as defined in IBM's feature engineering overview. This process creates predictive model features, which are input variables used to generate predictions.
The quality of features directly affects model performance. Data scientists often spend a large portion of time on data preparation and feature creation to ensure high-quality models.
Feature engineering is iterative and depends on the specific model type and problem domain. It requires careful analysis of the data and an understanding of the business context.
The feature engineering process
The steps required to engineer features include data extraction, data cleansing, feature creation, and feature storage, as outlined in AWS's feature engineering overview. Data extraction involves gathering data from various sources, which can be challenging due to different formats and volumes.
After data is cleaned, the next step is feature creation, which includes data labeling to provide context for the model. Exploratory data analysis with visualizations helps confirm data correctness and discover patterns.
Feature storage allows teams to organize and reuse features across training and inference. This step is important for maintaining consistency and enabling collaboration.
Normalization for numerical data
Normalization transforms features to be on a similar scale, which helps models converge more quickly and infer better predictions, as explained in Google's normalization guide. Without normalization, features with wide ranges can dominate the model.
Common normalization methods include linear scaling, Z-score scaling, and log scaling. Linear scaling maps values to a standard range, often 0 to 1. Z-score scaling centers data by the mean and scales by the standard deviation.
Clipping is another technique to limit extreme outliers. The choice of normalization method depends on the data distribution and the presence of outliers.
Encoding categorical data
Categorical data must be converted to numerical form for machine learning models. One-hot encoding converts each category into a vector with a single 1.0 and all other elements 0.0, as described in Google's guide to one-hot encoding. This approach works well for low-dimensional categorical features.
Feature crosses combine two or more categorical or bucketed features to capture interactions, as discussed in Google's feature crosses guide. They are analogous to polynomial transforms but for categorical data.
When using high-dimensional categorical features, one-hot encoding can become inefficient. Sparse representation stores only the position of the 1.0 to save memory.
Standardization with scikit-learn
The sklearn.preprocessing package provides tools for scaling and transforming data. The StandardScaler utility centers features by removing the mean and scales them to unit variance, as documented in scikit-learn's preprocessing guide.
Scaling features to a range, such as 0 to 1, can be done with MinMaxScaler. These transformers can be integrated into pipelines to apply consistent transformations during training and prediction.
Standardizing is often a prerequisite for many estimators, such as linear models and support vector machines, to ensure that features with larger variances do not dominate the objective function.
| Technique | Purpose | Use case |
|---|---|---|
| Linear scaling | Map values to a fixed range, typically 0 to 1 | Scaling a feature like age |
| Z-score scaling | Center by mean and scale by standard deviation | Standardizing test scores |
| One-hot encoding | Convert categorical categories to binary vectors | Encoding car color |
| Feature crosses | Combine features to capture interactions | Crossing leaf edges and arrangement |
Frequently asked questions
What is the difference between feature engineering and feature selection?
Feature engineering creates new variables from raw data, while feature selection picks a subset of existing variables to reduce dimensionality.
Why is normalization important?
Normalization places features on a similar scale, which helps gradient descent converge faster and prevents features with wide ranges from dominating.
How does one-hot encoding work?
One-hot encoding represents each category as a vector with a single 1.0 and all other elements 0.0, allowing models to treat categories as distinct numeric inputs.
What are feature crosses used for?
Feature crosses combine two or more features to capture interactions and nonlinearities, enabling linear models to handle more complex patterns.
How does scikit-learn support preprocessing?
scikit-learn provides transformers like StandardScaler and MinMaxScaler to standardize and scale features, and these can be used in pipelines.