Let $G$ be a group and $H \le K \le G$. We say that $H$ is $c$-embedded in $G$ with respect to $K$ if there is a subgroup $B$ of $G$ such that $G = HB$ and $H \cap B \le Z(K)$. Given a finite group $G$, a prime number $p$ and a Sylow $p$-subgroup $P$ of $G$, we investigate the structure of $G$ under the assumption that $N_G(P)$ is $p$-supersolvable or $p$-nilpotent and that certain cyclic subgroups of $P$ with order $p$ or $4$ are $c$-embedded in $G$ with respect to $P$. New characterizations of $p$-supersolvability and $p$-nilpotence of finite groups will be obtained.