A Tutor That Will Not Give the Answer: Prompting AI to Coach Source Evaluation

Press Next step to walk the six steps of the final prompt. Notice what the tutor never does: it never says whether an article is peer-reviewed, never suggests a better article, and never writes the assignment. Then try the sorting game at the bottom. Each move either follows the prompt or breaks one of its rules, and the answer explains which.

The tutor works like a good lab partner. In Step 3 it asks about one clue at a time (the journal’s name, the authors’ universities and degrees, the reference list) and lets you decide what each clue means. Switch between Draft 4 and Draft 5 to see the biggest change: Draft 5 explains why each rule exists, for example that students “might not know what they don’t know.” The authors found that the reasons changed the chatbots’ behavior more than repeating the rules did.

Drafts 1 to 3 listed questions for the model to ask, and the models answered those questions themselves, so no dialogue formed. Draft 4 described a role and a “what you never do” list; Claude followed it, while ChatGPT still offered answers and sources, CoPilot merged steps, and Gemini lectured. Draft 5 separates role, context, and instructions with XML tags and attaches a rationale to each directive. It also adds a writing-support behavior: help with precise wording that never changes the student’s conclusion.

The Draft 4 to Draft 5 change has three parts that the evaluation cannot separate: structure (XML sections), rationale (the “why” behind each directive), and a new writing-support behavior. The authors credit the rationale as most influential, on the basis of transcript review during prompt engineering. The sorting game is this page’s addition; its eight moves are written for the page, and each verdict cites the rule in Appendix A or B that decides it.

The team tested in rounds. First, nine student testers from the UCF Libraries and the DRACO Lab tried Draft 4 on six chatbots. Next, students in two classes for future teachers used Draft 4 on real assignments: an article review in an introductory course and short literature reviews in a reading-education practicum. Finally, after the rewrite to Draft 5, one student researcher ran the same assignment on five chatbots and filled out the survey for each. Grok and Perplexity were dropped after the first round because they behaved too differently from the others.

Each phase answers a different question. Phase 1 developed the survey and found platform-specific failures. Phase 2 tested Draft 4 in authentic, uncontrolled classroom use, where students picked their own platform (mostly ChatGPT and CoPilot) and the two courses used different survey forms (Qualtrics and an abbreviated Google Form). Phase 3 held the assignment and the student’s replies constant to compare platforms on Draft 5. The authors report that whole-course surveys helped least during early drafts and that transcript-level rubric review guided the revisions.

The phases are sequential and confounded by design: prompt draft, respondent pool, setting, and platform mix change together, and Phase 3’s five responses are five platform runs by one technically experienced evaluator. The authors name this in their limitations (single institution, mainly teacher-education students, convenience samples, a single expert evaluator who may inflate scores, and no measure of learning gains). The phase comparison below is therefore descriptive. Phase 1’s count of nine also needs a note: the text describes six student workers, three from each unit, and “three more” DRACO Lab students, while Table 2 lists nine responses.

Pick a measure and compare the three bars. In the classroom round, 70 percent of 40 students said the tutor gave just the right amount of information, and 55 percent rated its overall quality very good or excellent. After the rewrite, the final tester rated all five chatbots favorably on every measure. The course split at the bottom shows a gap worth noticing: 12 of 15 students in the practicum rated the tutor highly, against 10 of 25 in the introductory course.

Each bar shows a fraction. 55.6 percent of nine is five people (5 ÷ 9 = 0.556), and the other four said “a little too much.” Check the course split yourself: 12 of 15 is 80 percent, 10 of 25 is 40 percent, and together they make 22 of 40, which is the 55 percent printed for the whole classroom round. The Phase 3 bars reach 100 percent, but they rest on five chatbot runs by one tester.

Natural language rose the most across phases, from 22.2 percent (2 of 9) to 40 percent (16 of 40) and then 100 percent (5 of 5), matching the transcripts’ complaint that Draft 4 sounded formal and formulaic. Overall quality fell from Phase 1 (78 percent) to Phase 2 (55 percent) on the same Draft 4, and the authors’ lessons name three classroom factors that a tester round lacks: students mostly chose the platforms that handled Draft 4 worst, assignment wording varied in clarity, and students who used AI mainly to generate answers were frustrated by a tutor that withheld them. Information calibration improved from 55.6 to 70 percent, with Phase 2’s critiques split evenly between too little and too much.

Every printed share maps to exactly one whole-number count at its phase’s N, which the bars show. The Wilson intervals (this page’s) make the denominators visible: Phase 1’s 55.6 percent spans 26.7 to 81.1 percent, and Phase 3’s 100 percent of five has a lower bound of 56.6 percent. “Positive” means very good or excellent; the per-course “above average” ratings use the same cut, since 12 + 10 = 22 of 40 reproduces the 55 percent Phase 2 figure. The Fisher test on the course split is the page’s own and, like every phase comparison here, cannot separate the course from its assignment, survey form, and students.

Each square is one chatbot’s score on one rubric line, from 1 to 5; press any square to read what that score means. Claude and DeepSeek scored highest (29 of 30), and Gemini lowest (26), mostly because it kept answering for the student: in the transcript it completed parts of the source check and made the keep-or-discard decision itself. The bars below show how far the rewrite moved the scores on three lines, including “avoiding giving answers,” from 3.08 to 4.2.

A rubric turns a judgment into a number by describing each level in words. Press Gemini’s 3 under No answers and compare it with Claude’s 5: “sometimes gives answers” against “never gives answers.” Add a row to check a total (Gemini: 5 + 4 + 3 + 4 + 5 + 5 = 26). The class-average bars for Draft 4 come from students’ real chats; the Draft 5 bars come from one tester, so the rise mixes a better prompt with a more careful user.

Three criteria separate the platforms: step-by-step guidance (ChatGPT and Gemini 4), avoiding answers (Gemini 3, ChatGPT and CoPilot 4), and transparency (Claude, DeepSeek, and CoPilot 4). The evaluator also found CoPilot rigid, reading “step-by-step” too literally, though its step score was 5. Every platform scored 5 on cognitive congruence and on following directions. The authors recommend Claude and ChatGPT for consistency, and they list DeepSeek separately: it tied Claude, but Florida law bars its use in public institutions, and they suggest local hosting of the open model as a workaround.

Table 1 rests on one transcript per platform, scored by the lead author, so each cell is a single rating with no inter-rater check. The Draft 4 means come from an unstated number of classroom transcripts on mostly two platforms, against Draft 5’s five platforms and one evaluator; the 1.62-point rise in cognitive congruence (3.38 to 5.0) therefore bounds the joint effect of prompt, user, and platform. The text’s Draft 5 means equal Table 1’s five-platform means exactly.

• A prompt can make a chatbot coach a student through judging a source instead of judging it for them, but only after careful revision.
• Explaining the reason behind each rule worked better than repeating the rule.
• Chatbots differ: the same prompt produced a patient tutor on one platform and an answer machine on another.
• Clear assignment directions and students used to treating AI as a thinking partner made the tutor work better.
• The authors conclude that chatbots are not ready for the classroom out of the box.

• Draft 5 rubric totals: Claude 29, DeepSeek 29, CoPilot 28, ChatGPT 28, and Gemini 26 of 30.
• “Just right” amount of information: 5 of 9 (Phase 1), 28 of 40 (Phase 2), and 5 of 5 (Phase 3).
• Overall quality rated very good or excellent: 7 of 9, 22 of 40, and 5 of 5.
• In the classroom round, 12 of 15 practicum students and 10 of 25 introductory students rated quality highly.
• Avoiding giving answers rose from a mean of 3.08 (Draft 4) to 4.2 (Draft 5) on the 1-to-5 rubric.

• Rationale over repetition. Repeating prohibitions reduced ChatGPT’s adherence; XML structure with a stated reason for each directive did.
• Rubric gains. Cognitive congruence 3.38 to 5.0, adaptability 3.18 to 4.8, and avoiding answers 3.08 to 4.2 (Draft 4 classroom transcripts to Draft 5 technical review).
• Survey gains. Natural language rated positively by 22.2, 40, and 100 percent across phases; adaptability by 66.7, 72.5, and 100 percent; information “just right” for 55.6, 70, and 100 percent.
• Platforms matter. Draft 5 totals range from 26 (Gemini 3) to 29 (Claude Sonnet 4.5 and DeepSeek 3.2); CoPilot’s prompt length limit forced splitting the prompt into two entries.
• Assignment design matters. Directions that define acceptable source types worked; vague requests for “scholarly sources” confused both students and the tutor.
• AI literacy mediates. Students who already used AI as a thinking partner engaged; students who used it to generate answers were frustrated.

Students will use chatbots for research whether or not a course plans for it. A tutor prompt that asks instead of answers keeps the thinking with the student, and because the chapter publishes the full prompt, rubric, and survey under an open license, any instructor can copy them, test them on their own assignment, and see whether the chatbot their students use behaves.

The chapter supplies three reusable pieces: a prompt, a six-criterion rubric for judging tutor transcripts, and a survey keyed to the same criteria. Together they let an instructor test a prompt against an assignment before students see it. The authors suggest the same approach for other research tasks (question development, synthesis, gap analysis, outlining, and source integration).

The rubric and survey give LLM tutoring a shared measurement frame that later studies can apply with larger samples, multiple raters, and learning outcomes. The authors’ own future work points that way: longitudinal studies of independent source identification and source quality, student co-design, and tracking platform updates, since every platform result here is a snapshot of one model version.

Notes from building this page

• Table 1’s Average row describes three platforms. The printed averages (5, 4.33, 3.67, 4.67, 4.67, 5; total 27.33) do not match the five rows, whose means are 5, 4.6, 4.2, 4.8, 4.4, 5 (total 28.0). Of the 31 non-empty sets of platforms, only CoPilot, ChatGPT, and Gemini reproduce every printed average; the row may predate the addition of Claude and DeepSeek. The text’s Draft 5 means (5.0, 4.8, and 4.2) use all five.
• Counts behind the shares. Each printed percentage has exactly one whole-number count at its phase’s N (for example, 78 percent of nine is seven). Table 3 lists an Overall Quality row with empty cells; the text supplies 78, 55, and 100 percent.
• “Above average” in the course split. The survey’s scale has no “above average” point; 12 + 10 = 22 of 40 equals the Phase 2 overall-quality share, so the page reads it as very good or excellent.
• Phase 1 respondents. The text describes six student workers plus three more DRACO Lab students, and Table 2 gives N = 9. Stating whether nine people or six people with nine responses took part would settle the denominator.
• Unit of analysis in Phase 3. The five responses are five platforms rated by one evaluator on one assignment, so the 100 percent shares describe platform behavior under expert use rather than student reception; the authors say so in their limitations. A Wilson interval on five of five still runs down to 56.6 percent.
• Draft 4 rubric sample. The Draft 4 means come from a “representative sample” of classroom transcripts whose size and platform mix the text does not give, and Figure 2’s other three Draft 4 means are not stated in the text. Reporting n per criterion and a second rater’s scores would let readers judge the 1.1 to 1.6 point gains.
• The course gap. Fisher’s exact test on 12 of 15 against 10 of 25 gives p = 0.022 (page’s own), but course, assignment, survey form, and student year all differ, so the gap supports the authors’ point about assignment design without isolating it.
• Source record. UCF STARS lists DOI 10.59668/2340.26762 for this chapter; confirming whether it identifies the chapter or the whole book would make the citation exact.

Try this in class

1. Pick one article your class is reading. In pairs, one student plays the tutor and may only ask questions (one at a time, never “yes, it is peer-reviewed”); the other decides, using the clues in the chapter’s Step 3: journal, author affiliations, reference list, and section headings.
2. After five minutes, the observer scores the tutor on two rubric lines from the Scoring the final prompt demo, avoiding giving answers and step-by-step guidance, from 1 to 5.
3. Pool the class’s scores, compute each line’s average, and compare it with the chatbots’ scores in Table 1. Write two sentences on which rule was hardest for a human tutor to follow.