The results of a high-resolution Raman scattering study of a diamond crystal with a high content of single substitutional nitrogen impurities (550 ppm) in the temperature range from 50 to 673 K are presented and compared with the data for defect-free diamond. It is established that the increase of the nitrogen concentration in diamond leads to the temperature-independent increase of the Raman line width. Analysis of the experimental data allows us to conclude that this broadening should be attributed to the defect-induced shortening of the Raman phonon lifetime. We believe that this mechanism is responsible for the increase of the Raman line width caused by most point-like defects in diamond. No pronounced effects of the nitrogen defects on the Raman line position and phonon anharmonicity are observed.