This article presents a novel fractional order proportional-integral-derivative (FOPID) controller with fractional filter for an automatic voltage regulator (AVR) system. The proposed controller has seven independent parameters to be set. These parameters are optimally tuned by using sine-cosine algorithm (SCA) capable of balancing exploration and exploitation phases to find the optimal solution. The optimally tuned controller named as SCA-FOPIDFF is employed to enhance the performance of the AVR system. In order to show the effectiveness of the proposed controller, a FOPID controller with integer filter called as SCA-FOPIDF, a traditional FOPID controller without filter called as SCA-FOPID, a PID controller with fractional filter called as SCA-PIDFF, and a PID controller with integer filter called as SCA-PIDF are also designed for the AVR system using the SCA with the same objective function. In addition, the superiority of the proposed controller is also validated by the comparison with published studies proposing optimally tuned PID and FOPID controllers for the AVR system. Frequency response characteristics of the proposed controller obtained from bode analysis are presented. Finally, the robustness of the designed controllers are examined separately against both parameter uncertainties in the AVR system and external disturbances injected into the AVR system. Considering the overall results presented, it is clear that the fractional filter in the proposed SCA-FOPIDFF controller has significantly improve the AVR system performance and that the proposed controller can be successfully applied to the AVR system.