Motivated by problems from compressed sensing, we determine the threshold behaviour of a random $n\times d \pm 1$ matrix $M_{n,d}$ with respect to the property ‘every $s$ columns are linearly independent’. In particular, we show that for every $0\lt \delta \lt 1$ and $s=(1-\delta )n$ , if $d\leq n^{1+1/2(1-\delta )-o(1)}$ then with high probability every $s$ columns of $M_{n,d}$ are linearly independent, and if $d\geq n^{1+1/2(1-\delta )+o(1)}$ then with high probability there are some $s$ linearly dependent columns.