Due to the smallness of the lepton Yukawa couplings, higher dimensional operators can give a significant contribution to the lepton masses. In this case, the lepton mass matrix and the matrix of lepton-Higgs couplings are misaligned leading to lepton flavor violation (LFV) mediated by the Standard Model Higgs boson. We derive model-independent bounds on the Higgs flavor violating couplings and quantify LFV in decays of leptons and electric dipole moments for a class of lepton-Higgs operators contributing to lepton masses. We find significant Higgs-mediated LFV effects at both 1-loop and 2-loop levels.