Sirtuins (SIRTs) are a class of nicotinamide adenine dinucleotide (NAD+)-dependent protein histone deacetylases (HDACs) that are evolutionarily conserved from bacteria to mammals. This group of enzymes catalyses the reversible deacetylation of lysine residues in the histones or non-histone substrates using NAD(+) as a cosubstrate. Numerous studies have demonstrated that the aberrant enzymatic activity of SIRTs has been linked to various diseases like diabetes, cancer, and neurodegenerative disorders. Previously, we performed a pharmacophorebased virtual screening campaign and an aryloxybenzamide derivative (1) displaying SIRT1/2 inhibitory effect was identified as a hit compound. In the current study, the hit-to-lead optimization on the hit compound was explored in order to improve the SIRT binding and inhibition. Fourteen compounds, ten of which were new, have been synthesized and subjected to in vitro biological evaluation for their inhibitory activity against SIRT1-3. By the structural modifications performed, a significant improvement was observed in selective SIRT1 inhibition for ST01, ST02, and ST11 compared to that of the hit compound. The highest SIRT2 inhibitory activity was observed for ST14, which was designed according to compatibility with pharmacophore model developed for SIRT2 inhibitors and thus, providing the interactions required with key residues in SIRT2 active site. Furthermore, ST01, ST02, ST11, and ST14 were subjected to in vitro cytotoxicity assay against MCF-7 human breast cancer cell line to determine the influence of the improvement in SIRT1/2 inhibition along with the structural modifications on the cytotoxic properties of the compounds. The cytotoxicity of the compounds was found to be correlated with their SIRT inhibitory profiles indicating the effects of SIRT1/2 inhibition on cancer cell viability. Overall, this study provides structural insights for further inhibitor improvement.