Perturbed Assignments and Subgoal-Labeled AI Literacy Instruction in Undergraduate Discrete Mathematics

Slide 1: Perturbed assignments and subgoal-labeled AI literacy instruction

The talk covers two classroom studies in undergraduate discrete mathematics. The first is the published pilot by Gilson, Tabarsi and Barnes. The second is a Summer 2026 follow-up that places an AI literacy session between two perturbed assignments.

Slide 2: One course setting, two summer studies

Both studies ran in CSC 226, Discrete Mathematics, at NC State, where I was the instructor of record in both summers. Both collected data in the final third of the course, during the unit on computer science applications.

In Summer 2025, 18 students were assigned a perturbed final exam review. Eight used the provided chatbot, and we coded 182 query–response pairs. In Summer 2026, 21 students were enrolled and eight consented. The study used two perturbed lab parts with a session on checking chatbot output between them.

Slide 3: A submitted proof provides no evidence of chatbot aid

A submitted proof shows the answer. It does not show whether the student worked out the structure of an induction proof or asked a chatbot to lay it out and filled in the blanks. Both routes can produce the same page. Golesteanu, Vowinkel and Dougherty (2024) report that ChatGPT passed exams in a theory-of-computing course, while proofs students wrote with ChatGPT as a partner were often nonsensical. Something happens in the exchange between the student and the chatbot, and the chat log is where it can be seen.

Slide 4: Research questions

RQ1 asks how we can make students' chatbot interactions easier to observe from a pedagogical design perspective. RQ2 asks whether students adapt when they receive feedback related to their generative AI use on coursework. The design observes an assignment, teaches, and observes a second assignment.

Slide 5: Design a task where a wrong AI answer can be checked

Salim et al. (2024) perturbed introductory programming assignments to reduce and detect cheating. An instructor has good reasons to prohibit chatbot use, and a researcher who does so loses the data that would show how students use them. These studies use perturbation to observe use: students may use the chatbot, and they have to judge its response.

Slide 6: Three perturbation strategies target different failures

Each strategy changes a question until a chatbot answers it poorly. Spatial injection adds rows, columns and placement conditions to an ordinary counting task. Axiomatic replacement changes the successor rule to S(n) = 2n + 3, so familiar arithmetic rules need checking. Contextual embedding makes the question depend on something from class that the chatbot has not seen. Each question was pasted into the model as written and the answer graded with a rubric; a perturbed question was kept once the model scored below 60% on three attempts.

Slide 7: A salad bar adds spatial constraints to counting

Ten items are placed in a 4–4–2 grid, so the count depends on position. Later parts constrain rows, columns and adjacency. Part 7 adds the rule that if item x is more than six greater than item y, then x's row is lower than y's.

Slide 8: Squemp numbers: prove from the given axioms

The Squemp numbers mirror Peano arithmetic under invented names. The name was chosen because it means nothing, so the model can take no hint about the structure from it. A full-credit proof derives each property from the stated axioms. Models often fell back on the familiar Peano rules, which no longer hold.

Slide 9: The 2025 pilot captured the chatbot exchanges

Students used MerryQuery, a student-facing chatbot built in our lab, running Gemini 2.0 Flash. It logged every conversation in one format, which made the logs straightforward to analyse. The study collected the assessment submissions, the chatbot logs, and exam and homework scores. We chose the DRIVE framework over ICAP because ICAP's broad engagement categories were hard to tell apart in dialogue.

Slide 10: DRIVE: coding students' chatbot requests

DRIVE (Oliveira et al., 2025) codes a student's request in three categories. Writing covers instructions, criteria and evaluation: giving a task or rubric, or asking the chatbot to assess a draft. Argument covers objections, justification and conceptual clarity. Content covers definitions, examples and critical correction of generated content. A DRIVE tag describes the visible request, not what the student did before sending it.

Slide 11: DRIVE coding on the perturbed salad-bar task

"Arrange the ten salad-bar items in the 4–4–2 grid. Here is the assignment" is coded Writing: Instructions. "Explain why your proposed layout satisfies x > y + 6 ⇒ a > c" is Argument: Justify. "Your layout puts i9 above i1; 9 > 1 + 6 means its row must be lower. Correct that" is Content: Critical. The student has checked the output against the constraint and found the error.

Slide 12: AIED 2026: two help-seeking patterns emerged

In cognitive scaffolding, students noticed errors, challenged a claim, and added reasoning or constraints until the chatbot reached a valid solution. The observable move is identifying the failed step and asking the model to reconsider it. In GenAI dependence, students leaned on the generated answer without adequately checking its reasoning, even when it contained errors. The patterns were associated with assignment scores in this sample of eight, and they did not sort students cleanly by grade.

Slide 13: ICA components describe dialogue, not grade groups

An independent component analysis of the coded logs found two axes. The dependence-style axis runs from seeking help with contextual constraints, such as understanding the salad-bar layout, to seeking help with the formal logic of a proof. The scaffolding-style axis runs from requesting scaffolding from the chatbot, such as asking it to write the skeleton of an induction proof, to supplying constraints, reasoning and corrections. The axes do not show grades, and the results are exploratory, with eight students.

Slide 14: The pilot did not establish self-correction

The pilot showed that patterns in chatbot dialogue were associated with perturbed-assignment grades, and that the ICA components did not predict final exam scores. It did not observe self-correction that improved final exam results, though the review's closeness to the exam is one confound. Perturbation made the two patterns easy to tell apart, because each log came down to whether the student caught the induced error. Telling patterns apart is a different thing from teaching a better way to use AI, so the 2026 study teaches checking between two perturbed assignments and compares the logs.

Slide 15: Summer 2026: teach the checking procedures

The study runs from a pre-survey, through the first perturbed lab, a session on checking chatbot solutions and the second perturbed lab, to a post-survey and interviews. The surveys used SNAIL, a scale of non-experts' AI literacy, to account for what students knew about AI before the course. The first lab reused the 2025 perturbed questions. The session fit the course's final unit: with enough probability behind them, students could follow an account of why vague prompts lead to hallucinations.

Slide 16: Summer 2026: eight of 21 students consented

I taught one undergraduate discrete mathematics section under IRB approval. Twenty-one students were enrolled and eight consented. There were two perturbed assignments and one class session of instruction. Students could use Gemini 3.5 Flash under a university license; the questions had been tested against an earlier Gemini model, and the change of model during the assignment is a confound the analysis has to account for. Both lab parts served as the final exam review.

Slide 17: Subgoal labels tell students what to check

Subgoal labels name the job of each step in a worked example (Morrison, Margulieux and Guzdial, 2015). Students tend to retain what a label is for more than its exact wording. Here, students use the labels to appraise a solution a chatbot produced. The labels were drawn from productive usage we saw in the 2025 logs and from the literature.

Slide 18: Hallucination autopsy

The procedure has seven labelled steps, A0 to A6. A student calibrates their reliance on the model, restates the claim, extracts the rule each step uses, and locates the first step where the output diverges from what they expect, testing small cases until the error is found. They then classify the failure and consider why the model was likely to make it. Each label appears with the same wording in the reference box, the worked example and the exercise. Exercise A applies the procedure to a wrong chatbot transcript and marks the first failing line.

Slide 19: Induction example: locate the incorrect hypothesis

The target claim is 2 + 5 + ⋯ + (3n − 1) = n(3n + 1)/2. The flawed step substitutes k(k + 1)/2 for the correct hypothesis k(3k + 1)/2. Checking n = 2 exposes it: the actual sum is 7, and the substituted formula gives 3. The example is a schematic incorrect step, not a verbatim chatbot transcript.

Slide 20: Strong prompting

A strong prompt has four labelled parts: establish context, require step-by-step justification, build in self-questioning, and plan the corrective dialogue. B2 asks each step to name the rule or prior line it uses. B3 has the chatbot list and check its assumptions, including the effects of its own changes, before finishing. Exercise B compares weak and strong prompts and records what changes. A strong prompt reduces hallucination; on its own, it does not make the student learn anything.

Slide 21: Collaborative prompting

Collaborative prompting turns a strong prompt into a way to learn. The student frames the problem and sets the rules, such as never giving the answer. For each step, they ask for one step with its rule named, interrogate it, and verify it against a case they compute by hand, then request the next. The final write-up is the student's own: a chat is rewritten as a rigorous proof. The last step questions the system. A model asked about the "diversity" of a sample may answer in the data-spread sense when the researcher meant who is in the sample; working with a chatbot, as with a human collaborator, starts by aligning what each means.

Slide 22: Summer 2026: interviews completed; log coding pending

At the time of the talk, the two interviews had been analysed and the chat logs were being anonymised and prepared for coding. For RQ2, the interviews give initial accounts of how two students positioned the chatbot relative to their own work. The interviews were office-hours conversations that students chose to attend, and both consented to be recorded.

Slide 23: Students often attempted first, then checked

P2 said, "I'll try and solve it myself, and then kind of use it as a confirmation." P1 said, "I think I've got it … Let me just compare it to the AI so I can be 100% certain." Both reported attempting the problem before asking for confirmation. I read P1's statement as using the model to resolve the student's own uncertainty about an answer.

Slide 24: A confirmation prompt does not reveal the earlier work

The log begins with the prompt, so work done before it does not appear. Written work and interviews help establish that context: when a student began the chat relative to their written work, and whether they open a chatbot at the start of a problem or at a point of difficulty. I will read confirmation requests alongside the submitted work, then compare the coded patterns with the earlier pilot.

Slide 25: Label recall and procedure use need separate measures

P1 said, "Did I apply them explicitly? No, but they were in my head … Is it hallucinating, or … is my prompt strong enough?" P2 described correcting the chatbot as "stuff that I haven't necessarily thought as much about before." The interviews motivate coding what students do separately from which labels they remember.

Slide 26: One student decided to rely on the chatbot less

P2 had been "in my head a lot about whether or not I'm using AI correctly," and the session "felt like confirmation" that they were. P1 said that relying on the chatbot for certainty "hurt me in other areas" and that "it's hard to learn things if you can't make mistakes," and reported a decision to use it less. My interpretation, which is not a study result, is that deciding one's own answer is right is a skill tied to expertise in the subject, and asking a model for that last check skips the practice.

Slide 27: What the follow-up adds to the published pilot

The follow-up reuses the published perturbation method, retested against a newer model; the Perturbation Workbench was built afterwards to run the process. It adds three labelled procedures for checking chatbot output on proofs. It adds interviews that let students explain their prompts beside their submitted work, which is how the timing question in slide 24 arose.

Slide 28: Discussion: what counts as evidence of checking?

When a student asks AI to confirm a proof, what shows checking rather than outsourcing? In the logs I have read so far, the two look similar. And should we measure students remembering the checking labels, actually using them, or both?

Sam Gilson · sagilson@ncsu.edu · Perturbation Workbench · All research