In mental healthcare, initial patient assessments function as the essential entry point to treatment. Due to the evident treatment gap in healthcare, solutions for optimizing efficiency and scalability of psychotherapy have to be found. This paper investigates an automated approach to initial patient assessments. Our proposed concept combines rule-based dialog management and topic coverage with the capabilities of large language models for dynamic conversations in a novel chatbot-based system. We hypothesize that automating patient evaluations can shorten the initial therapy phase, thereby increasing access and cost-effectiveness. To investigate the usability, user experience, and perceived feasibility of the system, we conducted a randomized controlled trial with 72 participants using an online form-based control. Additional qualitative feedback was collected through an expert interview. While group differences in quantitative results are non-significant, qualitative feedback provides valuable insights into the potential and shortcomings of the approach and serves as a basis for future work.