3D point clouds often require accurate labeling and semantic information. However, in the absence of fully automated methods, such labeling must be performed manually, which can prove extremely time and labour intensive. To address this, we propose a novel hybrid CPU/GPU-based algorithm allowing instantaneous selection and modification of points supporting very large point clouds. Our tool provides a palette of 3D interactions for efficient viewing, selection and labeling of points using head-mounted VR and controllers. We evaluate our method with 25 users on tasks involving large point clouds and find convincing results that support the use case of VR-based point cloud labeling.