A search is presented for anomalous quartic gauge boson couplings in vector-boson scattering. The data for the analysis correspond to 20.2 fb-1 of s=8 TeV pp collisions and were collected in 1972 by the ATLAS experiment at the Large Hadron Collider. The search looks for the production of WW or WZ boson pairs accompanied by a high-mass dijet system, with one W decaying leptonically and a W or Z decaying hadronically. The hadronically decaying W/Z is reconstructed as either two small-radius jets or one large-radius jet using jet substructure techniques. Constraints on the anomalous quartic gauge boson coupling parameters α4 and α5 are set by fitting the transverse mass of the diboson system, and the resulting 95% confidence intervals are -0.024<α4<0.030 and -0.028<α5<0.033.