Analog in-memory computing (AIMC) with phase-change memory (PCM) offers substantial energy efficiency gains for edge inference by executing matrix–vector multiplications (MVMs) directly in memory, making it well suited to real-time tactile perception in robotics. A fundamental reliability challenge of PCM is conductance drift, a time-dependent stochastic reduction caused by structural relaxation that degrades inference accuracy over time. This paper presents a drift-resilient architecture that couples electrical time-domain reflectometry (ETDR) tactile sensing with PCM-based AIMC for human-robot interaction (HRI). In ETDR, touch-induced deformations create localized impedance discontinuities along an embedded transmission line, producing reflected RF signals that encode force magnitude and contact location in high-dimensional reflectograms. Based on this sensing modality, we evaluate a heterogeneous processing pipeline with three feature-extraction strategies under identical training and drift conditions: parameter-free Robust Segment Statistics (RSS), digitally executed PCA (PCA-D), and PCA mapped to analog crossbars (PCA-A). RSS matches PCA-D within 0.33 percentage points (pp) in digital accuracy, whereas PCA-A exhibits roughly twice the one-year drift degradation of PCA-D, indicating the penalty of mapping learned preprocessing to analog crossbars under the adopted drift model. Drift-aware regularization (DAR) improves RSS one-year force-classification accuracy from 86.87% to 88.03%, recovering 22.4% of the drift-induced loss, while PCA-D achieves a one-year localization error of 29.4 mm. These results suggest that keeping learned preprocessing parameters in the digital domain while applying physics-informed training to the analog classifier is a promising design principle for drift-resilient AIMC tactile inference in resource-constrained HRI systems.