Automatic detection of the pectoral muscle in mammograms is widely used in computer-aided diagnostic (CAD) systems for breast cancer. The pectoral muscle region has some prominent features such as the upper corner position, high density, and triangular shape. But, these features may be distorted due to the masses, artifacts, skin folds, and overlapping tissues, and other reasons. Despite recent developments in CAD technology, accurate detection of distorted pectoral muscle images remains a challenging task. In this study, we proposed an automatic method that uses a divided topographic representation to detect distorted pectoral muscle boundaries. After the preprocessing stage, firstly an isocontour map is generated and then divided into horizontal blocks. The contours of the pectoral muscle boundary in the blocks often reveal specific patterns in terms of location, geometric and topological features. We developed a new segmentation algorithm, rule-based contour detection (RBCD), to detect these specific patterned isocontours. The method applied to two datasets consisting of 84 and 201 mammogram images from MIAS and Inbreast databases respectively. Besides, some distorted pectoral muscle samples selected from these datasets were used to further analyze the performance of the proposed method. The mean False-Positive and the mean False-Negative rates of the proposed method for MIAS and Inbreast datasets were 0.92%, 1.26%, and 2.34%, 1.15%, respectively. The quantitative and qualitative results for the distorted pectoral muscle samples show that the proposed method outperformed the compared methods.