Single crystal diamonds grown by chemical vapor deposition (CVD) before and after different post-growth treatments were studied using optical spectroscopy. The most pronounced changes in color were observed after irradiation with subsequent annealing at 800 °C whereas a weakening of gray tint took place after the high-pressure high-temperature annealing. Low-pressure high-temperature annealing in microwave plasmas of different composition and by electric arc discharge did not produce a noticeable effect on the diamond properties. Signals from nitrogen-vacancy complexes were detected in the absorption and luminescence spectra of CVD diamonds, but the pink tint in the irradiated and annealed diamonds is due to Si–V complexes, not N–V. The peaks at 150 and 262 K associated with boron were observed in thermoluminescence (TL) curve of original CVD diamonds. Combined irradiation and annealing allowed us to create deep traps of charge carriers responsible for TL peaks in the 430 to 470 K range, which made CVD diamonds promising for TL dosimetry.