We consider the full flavor structure of the $S_1$ leptoquark model and derive conservative constraints on the elements of the left- and right-handed coupling matrices. We focus on the cases where the muon $g-2$ deviation is explained by muon couplings to the top-quark or to the charm-quark or to all up-type quarks. The most significant constraints arise from charged lepton flavor violating decays of the muon and the $\tau$ lepton and from the $\mu-e$ conversion process. Kaon decays and perturbativity provide further constraints. We find strong constraints on almost all coupling matrix elements, implying a very hierarchical matrix structure, where individual entries must differ by at least 4 orders of magnitude. The $\texttt{FlexibleSUSY}$ program was used with appropriate model files incorporating the parameterization of the couplings in the up-type mass diagonal basis. The expressions for the leptonic observables were generated and cross-checked with the help of the $\texttt{NPointFunctions}$ extension of the $\texttt{FlexibleSUSY}$ program.