To simulate the dynamics of fluid with polydisperse particles on macroscale level, one has to solve hydrodynamic equations with several relaxation terms, representing momentum transfer from fluid to particles and vice versa. For small particles, velocity relaxation time (stopping time) can be much shorter than dynamical time of fluid that makes this problem stiff and thus computationally expensive. We present a new fast method for computing several stiff drag terms in two-phase polydisperse medium with Smoothed Particle Hydrodynamics (SPH). In our implementation, fluid and every fraction of dispersed phase are simulated with different sets of particles. The method is based on (1) linear interpolation of velocity values in drag terms, (2) implicit approximation of drag terms that conserves momentum with machine precision, and (3) solution of system of N linear algebraic equations with O(N2) arithmetic operation instead of O(N3). We studied the properties of the proposed method on one-dimensional problems with known solutions. We found that we can obtain acceptable accuracy of the results with numerical resolution independent of short stopping time values. All simulation results discussed in the paper are obtained with open source software.