We simulate the motion of hydrogen and deuterium molecules in the magnetic system of a setup intended for obtaining nuclear-spin-polarized molecules. Spatial separation of molecules with different magnetic moment projections by the spin filtration method in a nonuniform magnetic field is performed using superconducting sextupole magnets. Calculations are carried out for a magnetic field induction of 3.7 T at the poles and a nozzle temperature of 7 K. Simulation show that the ratio of polarized flux of hydrogen molecules to the detector to the total flux from the source nozzle is 2.3 × 10–6, nuclear polarization being close to 100%. Calculations performed for deuterium reveal that this ratio is 7 times smaller due to the smallness of the magnetic moment relative to hydrogen molecules. Trajectories of molecules in the magnetic system and their spatial distribution are represented graphically. Mathematical aspects of the algorithm of the computer program developed for this purpose are considered.