In conventional wisdom of video modeling, the background is often treated as the primary target and foreground is derived using the technique of background subtraction. Based on the observation that foreground and background are two sides of the same coin, we propose to treat them as peer unknown variables and formulate a joint estimation problem, called Hierarchical modeling and Alternating Optimization (HMAO). The motivation behind our hierarchical extensions of background and foreground models is to better incorporate a priori knowledge about the disparity between background and foreground. For background, we decompose it into temporally low-frequency and high-frequency components for the purpose of better characterizing the class of video with dynamic background; for foreground, we construct a Markov random field prior at a spatially low resolution as the pivot to facilitate the noise-resilient refinement at higher resolutions. Built on hierarchical extensions of both models, we show how to successively refine their joint estimates under a unified framework known as alternating direction multipliers method. Experimental results have shown that our approach produces more discriminative background and demonstrates better robustness to noise than other competing methods. When compared against current state-of-the-art techniques, HMAO achieves at least comparable and often superior performance in terms of F-measure scores, especially for video containing dynamic and complex background.