In this paper routes to chaotic oscillation in power systems and relationships between chaos and various power system instability modes are deeply studied. Firstly, using a three-bus simple system, three routes may cause power systems to chaos are illustrated and discussed. They are route of cascading period doubling bifurcation (PDB), route of torus bifurcation (TB) and route of directly initiated by a large disturbance of energy. The route of cascading PDB is a typical route to chaos, where PDB is caused by a real Floquet multiplier moving counter to the real axis and going out of the unit circle from point (-1,0) in the complex plane. TB is caused by a couple of conjugate Floquet multipliers going out of the unit circle with a nonzero imaginary part in the complex plane. Chaos caused by TB has some interesting features, such as self-organizing phenomenon, coexistence of divergent and chaotic subspaces, etc. The last route, which is directly initiated by a large disturbance of energy, is reported and studied for the first time so far as we know. Secondly, relationships between chaos and various instability modes are discussed. Also using a simple system, we illustrate the scenes that chaos leads power system to voltage collapse, angle divergence, or voltage collapse with angle divergence simultaneously when the stability conditions of the chaotic oscillation are broken. It tells us that chaos is very likely to be an intergrade existing in the transient stage after a large disturbance. In order to prevent the appearance of instability incidents effectively, it is necessary to keep up on the study of chaotic phenomena in power systems. All these studies are helpful to deeply understand the mechanism of various instability modes and to find effective anti-chaos strategies.