8 - Distributional Hypothesis and Representation Learning

Summary

All the algorithms we covered so far rely on handcrafted features that must be designed and implemented by the machine learning developer. This is problematic for two reasons. First, designing such features can be a complicated endeavor. Second, most words in any language tend to be very infrequent. In our context, this means that most words are very sparse, and our text classification algorithm trained on word-occurrence features may generalize poorly. For example, if the training data for a review classification dataset contains the word great but not the word fantastic, a learning algorithm trained on these data will not be able to properly handle reviews containing the latter word, even though there is a clear semantic similarity between the two. In this chapter, we will begin to addresses this limitation. In particular, we will discuss methods that learn numerical representations of words that capture some semantic knowledge. Under these representations, similar words such as great and fantastic will have similar forms, which will improve the generalization capability of our machine learning algorithms.