Understaffed universities and limited funding for tutors risk students' disengagement due to missing assistance. HCI research increasingly focuses on conversational agents for scalable educational support. This paper presents a course-specific AI chatbot designed to answer students' questions and reduce tutor workload by addressing frequent organizational and content-related questions. Developed as a pilot case study for an introductory Software Technology course, ST-Buddy combines modular LLM-based dialogue with a flexible course-grounded knowledge integration to enable adaptive, context-aware responses grounded in relevant course material and logistical information. A formative evaluation of ST-Buddy (n=32) showed good usability (SUS=77.81), perceived helpfulness particularly when tutors were unavailable, and rated response quality as relevant and understandable. Additionally, students' self-reported everyday academic AI use and lessons learned from the evaluation settings are discussed. These findings outline implications for improving future evaluations and highlight the potential of modular chatbot frameworks to scale personalized support in large courses.