To investigate the atmospheres of ultracool brown dwarfs with temperatures covering the range of transiting and directly imaged planets, we have monitored a sample of 76 L, T and Y brown dwarfs for infrared photometric variability. This survey was conducted in the J-band using both the SOFI camera on the 3.5-m NTT and the SWIRC camera on the 6.5-m MMT. Each target was observed for a period ranging from 2.0 hours to 6.0 hours, covering a significant fraction of the expected rotation period. Breakup of the iron and silicate clouds into a patchy cloud layer has been suggested as an explanation for the variability of several objects identified at the L/T transition, and a similar process with sulfide clouds may be manifest in T/Y transition objects; our data provides the first test of these patchy cloud scenarios across the entire brown dwarf spectral range.