This paper presents and evaluates an open source penetration testing framework for finding vulnerabilities in 5G systems under the assumption of malicious end devices. This is achieved by enabling the creation of arbitrary (5G) messages to be transmitted over the air interface of a 5G system. Our framework is modular and scriptable, allowing the easy creation of test cases. It is based on the OpenAirInterface (OAI) open source 5G stack. We evaluated our framework by implementing several tests and running it against a well-known open source 5G system. We were able to identify several vulnerabilities.