We aim to investigate the nature of doubt regarding voice-based agents by referring to Piaget’s ontological object–subject classification “thing” and “person,” its associated equilibration processes, and influential factors of the situation, the user, and the agent. In two online surveys, we asked 853 and 435 participants, ranging from 17 to 65 years of age, to assess Alexa and the Google Assistant. We discovered that only some people viewed voice-based agents as mere things, whereas the majority classified them into personified things. However, their classification is fragile and depends basically on the imputation of subject-like attributes of agency and mind to the voice-based agents, increased by a dyadic using situation, previous regular interactions, a younger age, and an introverted personality of the user. We discuss these results in a broader context.