The performance of Gas Electron Multipliers (GEMs) in gaseous He, Ne, He+H2 and Ne+H2 was studied at temperatures in the range of 3-293K. This paper reports on previously published measurements and additional studies on the effects of the purity of the gases in which the GEM performance is evaluated. In He, at temperatures between 77 and 293 K, triple-GEM structures operate at rather high gains, exceeding 1000. There is an indication that this high gain is achieved through the Penning effect as a result of impurities in the gas. At lower temperatures the gainvoltage characteristics are significantly modified probably due to the freeze-out of these impurities. Double-GEM and single-GEM structures can operate down to 3K at gains reaching only several tens at a gas density of about 0.5g/l; at higher densities the maximum gain drops further. In Ne, the maximum gain also drops at cryogenic temperatures. The gain drop in Ne at low temperatures can be re-established in Penning mixtures of Ne+H2: very high gains, exceeding 104, have been obtained in these mixtures at 30-77K, at a density of 9.2g/l which corresponds to saturated Ne vapor density at 27K. The addition of small amounts of H2 in He also re-establishes large GEM gains above 30K but no gain was observed in He+H2 at 4K and a density of 1.7g/l (corresponding to roughly one-tenth of the saturated vapor density). These studies are, in part, being pursued in the development of two-phase He and Ne detectors for solar neutrino detection.