We propose a vectorial perturbation theory for axially symmetric, generally nonspherical whispering gallery resonators made of isotropic and anisotropic optical materials. It is based on analysis of the leading terms in the coupled equations for independent light-field components, as derived from Maxwell's equations, and true boundary conditions. Strong localization of the whispering gallery modes (WGMs) near the resonator rim, controlled by the azimuth modal number m, is the main prerequisite for our analysis. The theory gives high-precision expressions for the WGM frequencies and modal functions, including the evanescent effects. One of important applications of the theory is analysis of anticrossings of the WGM resonances in anisotropic resonators detected in experiments. Simple relations for the frequency avoidance gaps during the anticrossings are derived and compared with experimental data obtained in lithium-niobate-based WGM resonators. We show also that the vectorial effects substantially restrict the field of applicability of the scalar WGM models.