Localized surface plasmon resonances (LSPRs) have been widely explored in various research fields because of their excellent ability to condense light into a nanometer scale volume. However, it suffers quite often from the broadening of the LSPR linewidths, resulting in low quality factors. Among the causes of the broadening, fabrication inaccuracies are crucial yet challenging to evaluate. In this paper, we designed a type of metal-insulator-metal structure as an example via the colloidal self-assembly approach. We then demonstrated a facile approach to identify the origin of the discrepancies in between spectra obtained from experiments and simulations. Through a series of simulations in accordance with the experimental results, we could confirm that the predominant influencing factors are the presence of defects, as well as feature size variations, though they impact the spectral response in different ways. For similar plasmonic systems, our results enabled a more cost-effective optimization process in lieu of rather intensive and iterative experimentations, which will pave the way to automated fabrication and optimization, as well as integrated design. Furthermore, our results also indicated that the typical defect ratio that is introduced via the colloidal self-assembly approach has only limited impact on the resulting plasmonic resonances, proving that for similar plasmonic structure designs, colloidal self-assembly methods can provide a reliable and efficient alternative in the field of nanofabrication of plasmonic systems.