Abstract: The vibration isolation system with cubic stiffness and Bouc-Wen type hysteresis shows complex nonlinear dynamic characteristics. An undamped response model is established based on an anhysteretic restoring force. An approximate analytical solution is derived by the harmonic balance method (HBM) and the Taylor expansion. An analytical/numerical method is proposed to calculate the damped response of the vibration isolation system, based on the HBM and Levenberg-Marquardt algorithm. For the multi-value non-smooth function terms, the harmonic term coefficients are obtained by applying the fast Fourier Transform for the calculated time-domain response. The proposed methods are applied to a nonlinear vibration isolation system with horizontal wire ropes. It is verified that the vibration isolation system has a softening-hardening stiffness with cubic stiffness and Bouc-Wen type hysteresis. Both hysteretic damping and linear damping can effectively suppress the resonance, while with the hysteretic damping, better vibration isolation performance is shown at high frequencies.