This paper investigates periodic bifurcation solutions of a mechanical system which involves avan der Pol type damping and a hysteretic damper representing restoring force. This systemhas recently been studied based on the singularity theory for bifurcations of smooth functions.However, the results do not actually take into account the property of nonsmoothness involvedin the system. In particular, the transition varieties due to constraint boundaries were ignored,resulting in failure in _nding some important bifurcation solutions. To reveal all possible bifurcationpatterns for such systems, a new method is developed in this paper. With this method,a continuous, piecewise smooth bifurcation problem can be transformed into several subbifurcationproblems with either single-sided or double-sided constraints. Further, the constrainedbifurcation problems are converted to unconstrained problems and then singularity theory isemployed to _nd transition varieties. Explicit formulas are applied to reconsider the mechanicalsystem. Numerical simulations are carried out to verify analytical predictions. Moreover, symbolicnotation for a sequence of bifurcations is introduced to easily show the characteristics ofbifurcations, and also the comparison of di_erent bifurcations. The method developed in thispaper can be easily extended to study bifurcation problems with other types of nonsmoothness.