Spark SQL is a module in Spark for structured data processing, which improves upon RDDs. The chapter explains how imposing structure on the data helps Spark perform further optimizations. We talk about the transition from RDDs to DataFrames and Datasets, including a brief description of the Catalyst and Tungsten projects. In Python, we don’t have Datasets, and we focus on DataFrames. With a learn-by-example approach, we see how to create DataFrames, inferring the schema automatically or manually, and operate with them. We show how these operations usually feel more natural for SQL developers, but we can interact with this API following an object-oriented programming style or SQL. Like we did with RDDs, we showcase various examples to demonstrate the functioning of different operations with DataFrames. Starting from standard transformations such as select or filter, we move to more peculiar operations like Column transformations and how to perform efficient aggregations using Spark functions. As advanced content, we include implementing user-defined functions for DataFrames, as well as an introduction to pandas-on-Spark, a powerful API for those programmers more used to pandas.

This chapter provides an overview of the theory of controlled sensing, and its application to the sequential design of data-acquisition and decision-making processes. Based on the theory, it provides an overview of the applications to the quickest detection and localization of anomalies in power systems. This application is motivated by the fact that agile localization of anomalous events plays a pivotal role in enhancing the overall reliability of the grid and avoiding cascading failures. This is especially of paramount significance in the large-scale grids due to their geographical expansions and the large volume of data generated. Built on the theory of controlled sensing, the chapter discusses a stochastic graphical framework for localizing the anomalies with the minimum amount of data. This framework capitalizes on the strong correlation structures observed among the measurements collected from different buses. This framework, at its core, collects the measurements sequentially and progressively updates its decision about the location of the anomaly.