In this study low temperature micro-photoluminescence technology was employed to investigate effects of the irradiation and nitrogen concentration on nitrogen-vacancy (NV) luminescence, with the photochromic and vibronic properties of the NV defects. Results showed that the NV luminescence was weakened due to recombination of self-interstitials created by electron irradiation in diamond and the vacancies within the structure of NV centers. For very pure diamond, the vacancies migrated the long distance to get trapped by N atoms only after sufficient high temperature annealing. As with the increase in nitrogen content, the migration distance of vacancies got smaller. The nitrogen also favored the formation of negatively charged NV centers with the donating electrons. Under the high-energy ultraviolet laser excitation, the photochromic property of the NV− center was also observed, though it was not stable. Besides, the NV centers showed very strong broad sidebands, and the vibrations involved one phonon with energy of ~42 meV and another with ~67 meV energy.