Advanced Air Mobility (AAM) aims to revolutionize transportation by enabling on-demand, low-altitude movement of people and goods using Vertical Take-Off and Landing (VTOL)-capable vehicles (VCAs), but achieving precise and reliable navigation remains a key challenge. The navigation of UAVs typically relies on the integration of GNSS and IMU measurements. When GNSS positioning is degraded or unavailable due to factors such as jamming, spoofing, and multipath effects, these approaches become unreliable. Perception-based methods, such as SLAM based on a combination of RGB camera, event camera, LiDAR, and IMU can lead to resilient navigation with loop closure for drift correction. However, loop closure is often impractical for a point-to point AAM flight. To tackle these challenges, this ongoing research work proposes a multi-modal navigation framework that integrates map-based navigation and semantic SLAM. For the development of this framework, we designed a conceptual setup, which will be presented in this paper. Additionally, we focus on the selection of sensors, their spatio-temporal calibration, and their integration into a holistic onboard system.