Teaching Astronomy with Large Language Models

Abstract

We present a study of LLM integration in final-year undergraduate astronomy education, examining how students develop AI literacy through structured guidance and documentation requirements. We developed AstroTutor, a domain-specific astronomy tutoring system enhanced with curated arXiv content, and deployed it alongside general-purpose LLMs in the course. Students documented their AI usage through homework reflections and post-course surveys. We analyzed student evolution in AI interaction strategies and conducted experimental comparisons of LLM-assisted versus traditional grading methods. LLM grading showed strong correlation with human evaluation while providing more detailed and consistent feedback. We also piloted LLM-facilitated interview-based examinations as a scalable alternative to traditional assessments, demonstrating potential for individualized evaluation that addresses common testing limitations. Students experienced decreased rather than increased reliance on LLMs over the semester, developing critical evaluation skills and strategic tool selection. They evolved from basic assistance-seeking to verification workflows, with documentation requirements fostering metacognitive awareness. Students developed effective prompting strategies, contextual enrichment techniques, and cross-verification practices. Our findings suggest that structured LLM integration with transparency requirements and domain-specific tools can enhance astronomy education while building essential AI literacy skills. We provide implementation guidelines for educators and make our AstroTutor repository freely available.