Polarimetry is a highly sensitive method to quantify changes of the polarization state of light when passing through matter and is therefore widely applied in material science. The progress of synchrotron and X-ray free electron laser (XFEL) sources has led to significant developments of X-ray polarizers, opening perspectives for new applications of polarimetry to study source and beamline parameters as well as sample characteristics. X-ray polarimetry has shown to date a polarization purity of less than, enabling the detection of very small signals from ultrafast phenomena. A prominent application is the detection of vacuum birefringence. Vacuum birefringence is predicted in quantum electrodynamics and is expected to be probed by combining an XFEL with a petawatt-class optical laser. We review how source and optical elements affect X-ray polarimeters in general and which qualities are required for the detection of vacuum birefringence.