Listening comprehension in a second language (L2) is a complex and particularly challenging task for learners. Because of this, L2 learners and instructors alike employ different learning supports as assistance. Captions in multimedia instruction readily provide support and thus have been an ever-increasing focus of many studies. However, captions must eventually be removed, as the goal of language learning is participation in the target language where captions are not typically available. Consequently, this creates a dilemma particularly for language instructors as to the usage of captioning supports, as early removal may cause frustration, while late removal may create learning interference. Accordingly, the goal of the current study was to propose and employ a testing instrument, the Caption Reliance Test (CRT), which evaluates individual learners’ reliance on captioning in second language learning environments; giving a clear indication of the learners’ reliance on captioning, mirroring their support needs. Thus, the CRT was constructed comprised of an auditory track, accompanied by congruent textual captions, as well as particular incongruent textual words, to provide a means for testing. It was subsequently employed in an empirical study involving English as a Foreign Language (EFL) high school students. The results exhibited individual variances in the degree of reliance and, more importantly, exposed a negative correlation between caption reliance and L2 achievement. In other words, learners’ reliance on captions varies individually and lower-level achievers rely on captions for listening comprehension more than their high-level counterparts, indicating that learners at various comprehension levels require different degrees of caption support. Thus, through employment of the CRT, instructors are able to evaluate the degree to which learners rely on the caption supports and thus make informed decisions regarding learners’ requirements and utilization of captions as a multimedia learning support.