Sparse coding theories suggest that the visual brain is optimized to encode natural visual stimuli to minimize metabolic cost. It is thought that images that do not have the same statistical properties as natural images are unable to be coded efficiently and result in visual discomfort. Conversely, artworks are thought to be even more efficiently processed compared to natural images and so are esthetically pleasing. This project investigated visual discomfort in uncomfortable images, natural scenes, and artworks using a combination of low-level image statistical analysis, mathematical modeling, and EEG measures. Results showed that the model response predicted discomfort judgments. Moreover, low-level image statistics including edge predictability predict discomfort judgments, whereas contrast information predicts the steady-state visually evoked potential responses. In conclusion, this study demonstrates that discomfort judgments for a wide set of images can be influenced by contrast and edge information, and can be predicted by our models of low-level vision, whilst neural responses are more defined by contrast-based metrics, when contrast is allowed to vary.