This paper presents a real-time background estimation and maintenance based people tracking technique in an indoor and an outdoor environments for visual surveillance system. In order to detect foreground objects, first, background scene model is statistically learned using the redundancy of the pixel intensity values during learning stage, even the background is not completely stationary. A background maintenance model is also proposed for preventing some kind of falsies, such as, illumination changes, or physical changes. And then for people detection, candidate foreground regions are detected using thresholding, noise cleaning and their boundaries extracted using morphological filters. From these, a body posture is estimated depending on skeleton of the regions. Finally, the trajectory of the people in motion is implemented for analyzing the people actions tracked in the video sequences. Experimental results demonstrate robustness and real-time performance of the algorithm.