Various papers have focused on developing new sensors and technologies for precisely detecting the human presence in the collaborative workspace. However, an important aspect has often been overlooked. It is the context of the appearance/ disappearance of the human concerning the robot activities and workspace circumstances. In other words, which circumstances could prevent the vision system from detecting humans; have they left the cooperation workspace correctly or has a fault event happened E.g. they are covered by another object and not visible to the camera system anymore. This investigation proposes a superordinate safety system for HRI applications. The proposed system consists of several modules. Two of them will be presented in detail in this paper. 1. 'Human-robot states' module: it contains; a. The possible status of the detected and the lost objects based on their position and safety procedures (danger, safe etc.); b. the possible events which could happen for every object based on their activities and their relationship with other objects. 2. 'Events classifiers' module: it analyzes the status of every new and lost object, whether it has entered or left the workspace correctly or an unexpected event has happened. The proposed approach has been tested in a dynamic experimental field with heavy-duty robot.