Chapter 9 presents support vector regression (SVR), a relatively newer supervised learning algorithm for predictive regression modeling, which – like random forests for regression – also may outperform the least-squares-based methods. Discussed is ε-insensitive loss used by SVR, the ε-tube concept, as well as algorithms for linear and nonlinear SVRs.

Kernel methods provide an alternative family of non-linear methods to neural networks, with support vector machine being the best known among kernel methods. Almost all linear statistical methods have been non-linearly generalized by the kernel approach, including ridge regression, linear discriminant analysis, principal component analysis, canonical correlation analysis, and so on. The kernel method has also been extended to probabilisitic models, for example Gaussian processes.

This chapter deals with the topic of feature expansion and imputation with a particular emphasis on computable features. While poor domain modelling may result in too many features being added to the model, there are times when plenty of value can be gained by looking into generating features from existing ones. The excess features can then be removed using feature selection techniques (discussed in the next chapter). Computable Features will be particularly useful if we know the underlining ML model is unable to do certain operations over the features, like multiplying them (e.g., if the ML involves a simple, linear modelling). Another type of feature expansion involves calculating a best effort approximation of values missing in the data (Feature Imputation). The most straightforward expansion for features happens when the raw data contains multiple items of information under a single column (Decomposing Complex Features). The chapter concludes by borrowing ideas from a technique used in SVMs called the kernel trick. The type of projections that practitioners have found useful can lend themselves to be applied directly without the use of kernels.