Eye-gaze tracking is the process of measuring the position of user's gaze. It is widely being employed in Human Computer Interaction (HCI) research area as an alternative of traditional input devices such as mouse and keyboard. In this paper, a real time vision based gaze direction detection system which can recognizes gazes in four different directions (left, right, up and bottom of the screen) and performs required user actions in related directions is introduced. In proposed system, face region is detected using Adaboost machine learning algorithm and Haar-like features, eye region is detected using Support Vector Machines (SVM) and grayscale image features. Gaze directions are classified and recognized using SVM and grayscale image features. An Artificial Neural Network (ANN) based system is implemented for the performance evaluation of proposed system. The proposed system shows 97.2% recognition accuracy of gazes in four different directions, which is effective and consistent with results from research on eye-gaze direction detection.