A compact finite difference scheme is derived for a time fractional differential equation subject to Neumann boundary conditions. The proposed scheme is second-order accurate in time and fourth-order accurate in space. In addition, a high order alternating direction implicit (ADI) scheme is also constructed for the two-dimensional case. The stability and convergence of the schemes are analysed using their matrix forms.