Visual Collaborative Simultaneous Localization and Mapping (SLAM) uses multiple mobile devices to capture images of their surroundings and map their environment collectively while simultaneously determining their position. In centralized approaches, a single server merges visual features from these devices to build a shared map. Offloading computationally intensive tasks benefits resource-limited mobile devices. However, due to unpredictable and heterogeneous communication links, maps at the server and on the devices can become asynchronous, potentially leading to real-time localization failures. To address these issues, we present Reliable Agents in Centralized Visual Collaborative SLAM (RAVIC), which employs an optimised server-to-agent communication strategy. This includes a novel keyframe selection method that exploits the server's global map to improve tracking under high agent mobility and network latency. In addition, adjustments to agent-to-server communication and agent tracking algorithm reduce the computational demands on mobile devices, leading to a reduction in average tracking processing time of up to 27.2 ms. Our evaluation using benchmark datasets and different network conditions shows a drastic reduction in track loss probability from 61.4% to only 1.0% utilising merged map data in collaborative scenarios, demonstrating that collaboration increases the reliability of individual agents.