From the viewpoint of nonlinear dynamics, the stability and bifurcation of the rotor dynamical system supported in gas bearings are investigated. First, the dynamical model of gas bearing-Jeffcott rotor system is given, and the finite element method is used to approach the unsteady Reynolds equation in order to obtain gas film forces. Then, the method for stability analysis of the unbalance response of the rotor system is developed in combination with the Newmark-based direct integral method and Floquet theory. Finally, a numerical example is presented, and the complex behaviors of the nonlinear dynamical system are simulated numerically, including the trajectory of the journal and phase portrait. In particular, the stabilities of the system’s equilibrium position and unbalance responses are studied via the orbit diagram, phase space, Poincaré mapping, bifurcation diagram, and power spectrum. The results show that the numerical method for solving the unsteady Reynolds equation is efficient, and there exist a rich variety of nonlinear phenomena in the system. The half-speed whirl encountered in practice is the result from Hopf bifurcation of equilibrium, and the numerical method presented is available for the stability and bifurcation analysis of the complicated gas film-rotor dynamic system.