Due to the long-range oscillatory character of RKKY exchange interactions, for Dy2Co3Al9 there exist positive and negative couplings between the Dy magnetic moments that lead to magnetic frustration. As a result, the ground state can be easily perturbed, and the system displays a number of spontaneous and field-induced phase transitions. We performed magnetization, magnetic-susceptibility, specific-heat, and electrical-resistivity measurements as well as neutron-diffraction experiments on a single crystal. We find two transitions to distinct incommensurate antiferromagnetic phases at 6.2 and 5.2 K that evolve to a commensurate phase at 3.7 K. In applied magnetic field, new phases emerge. Field-dependent magnetization exhibits a multistep metamagnetic process with sharp transitions accompanied by pronounced magnetoresistance changes. The large number of phases and their complex magnetic structures suggest that the physical properties of Dy2Co3Al9 are ruled by exchange frustration in the presence of a large magnetocrystalline anisotropy.