Tea (Camellia sp.) and its plantation are very important on a worldwide scale as it is the second-most consumed beverage after water. Therefore, it becomes necessary to map the widely distributed tea plantations under various geographies and conditions. Remote-sensing techniques are effective tools to map and monitor the impact of tea plantation on land-use/land-cover (LULC). Remote sensing of tea plantations suffers from spectral mixing as these plantation areas are generally surrounded by similar types of green vegetation such as orchards and bushes. This problem is mainly tied to planting style, topography, and spectral characteristics of tea plantations, and the side effects are observed as low classification accuracies after the classification process. In this study, to overcome this problem, a three-step approach was proposed and implemented on a test area with high slope. As a first step, spectral and multi-scale textural features based on Gabor filters were extracted from high resolution multispectral digital aerial images. Similarly, based on the wavelength range of the sensor, a modified normalized difference vegetation index (MNDVI) was applied to distinguish the green vegetation cover from other LULCs. The second step involves the classification of multidimensional textural and spectral feature combinations using a support vector machine (SVM) algorithm. As a final step, two different techniques were applied for evaluating classification accuracy. The first one is a traditional site-specific accuracy assessment based on a confusion matrix calculating statistical metrics for different feature combinations. The overall accuracy and kappa values were calculated as 93.68% and 0.92, 93.82% and 0.92, and 97.40% and 0.97 for LULC maps produced by red, green, and blue (RGB), RGB+MNDVI, and RGB+MNDVI+Gabor features, respectively. The second accuracy assessment technique was the pattern-based accuracy assessment. The technique involves polygon-based fuzzy local matching. Three comparison maps showing local matching indices were obtained and used to compute the global matching index (g) for LULC maps of each feature set combination. The g values were g((RGB)) (0.745), g((RGB+MNDVI)) (0.745), and g((RGB+MNDVI+Gabor)) (0.765) for comparison maps. Finally, based on accuracy assessment metrics, the study area was successfully classified and tea plantation features were extracted with high accuracy.