Co-Ideation Across Time
The problem
A sketchnote is a designer’s first prototype of an idea: drawings and words mixed together, capturing function, motivation and context that never make it into the final paper. That’s exactly what makes them valuable to the next generation, and also why nobody reads them. The handwriting is hard to decipher, there’s no reading order, and the private shorthand only makes sense to the person who wrote it.
The questions a search engine can’t answer. “Is this direction worth pursuing?” “What is fascinating about this idea?” “Am I thinking about this in a meaningful way?” Students usually need a mentor for these. Sketchnotes show how mentors worked through the same questions themselves.
Part of the archive: more than 2,000 pages of sketchnotes from the lab’s alumni.
Showing students finished projects tends to cause design fixation. Showing them the process instead invites them to think about methods and strategies. We wanted to make that process readable again, and let students talk back to it.
AI pipeline
We built an Annotation Labeling Interface with three jobs: create an AI Mentor, turn sketchnotes into AI-augmented Knowledge Objects, and test both for accuracy.
Data preparation, cloud storage, backend processing and front end.
1 AI Mentors
Each mentor is built from three sources, run through retrieval-augmented generation, refined with few-shot prompting and compiled into two editable JSON files: personality and academic data.
| Source | What it gives the mentor |
|---|---|
| Publications and theses, with project images | Detailed academic understanding |
| Talk and presentation transcripts | High-level motivation and storytelling |
| Social media posts | Tone of voice |
Why not fine-tune? Keeping the mentor in plain-language JSON keeps it lightweight and easy to edit. Someone without ML experience can create or adjust a mentor persona.
2 AI-augmented Knowledge Objects
A user uploads a sketchnote and draws boxes around regions. For each region they can write a note or ask the mentor, for example “Why didn’t the final project include this?” The coordinates, text and metadata are saved as JSON next to the image in Firebase, so a static page becomes something you can query.
3 Accuracy check
We asked the human mentor to explain their own sketchnotes, then compared those explanations with what the AI Mentor said about the same regions.
BLEU/ROUGE F1 about 0.2 (word overlap); BERTScore F1 0.8 (meaning)
Low word overlap was expected: the mentor was built from polished writing, while the real mentor spoke freely in a second language. The high BERTScore shows the meaning matched.
The “Damascus steel knife” scribble.
It explained things the papers never mention. One sketchnote has a scribble labeled “Damascus steel knife” in the middle of notes on shape-changing materials. The AI Mentor read it as a metaphor: like folded, layered steel, matter could carry embedded intention, memory and transformation. The human mentor confirmed that this was what he meant, and was surprised it got it.
CIAT interface
A node-based whiteboard that feels like brainstorming with a mentor at a shared board. Instead of a chat thread, responses, project references, sketchnote excerpts and generated sketches land on the canvas where you can move and connect them.
Interface features, a–m.
| Feature | What it does |
|---|---|
| Canvas | Pan, zoom and move elements freely |
| Drawing boxes | Nearby strokes merge into one image box that is sent to the AI |
| Connectors | Linked nodes become the “focused topic”; unlinked ones are background context |
| Mentor icons | Drag a mentor onto the canvas and link it to address that persona; use several for different perspectives |
| Generated sketch | Visual replies drawn in the style of the mentor’s work |
| Project reference | The mentor brings up a related past project, as a real mentor would |
| Sketchnote reference | A highlighted excerpt of a labeled sketchnote that matches the user’s idea |
| Circled response | Circle any part of a sketchnote to get it explained |
| History view | Full canvas states over time, for the user and the mentor |
An example session, from a first sketch to a final design.
Study
A within-subject exploratory study with 12 designers (4–10 years of experience) on two topics from the archive: Telepresence and Radical Atoms. We looked at behavior, not idea quality.
Study procedure
| Task | Activity | AI Mentor |
|---|---|---|
| 1 | Interpret a Telepresence sketchnote | Without |
| 2 | Interpret a Radical Atoms sketchnote | With |
| 3 | Create a Radical Atoms idea | With CIAT |
| 4 | Create a Telepresence idea, to see what carries over | Without |
Data: think-aloud transcripts, pre/post definitions, interaction logs, Likert surveys, semi-structured interviews.
Findings
With the mentor, people asked instead of guessing
Without the AI, participants paraphrased what they saw and moved on. With it, they validated and challenged their own reading. Their questions fell into three types:
- Readability. Illegible handwriting and invented terms. “What is ‘I.O. Clay’? Illuminating clay! Oh, cool, clears that up.” (P7)
- Validation. Checking the mentor’s answer against the drawing, and updating or defending their interpretation.
- Structure. Reading order, and how parts of the page relate to each other and to later projects.
| Topic | Most common before | Most common after |
|---|---|---|
| Telepresence (no AI) | Long-distance communication (8), tools like Zoom (4) | Sensory presence (6), emotional connection (3) |
| Radical Atoms (with AI) | Technical shape-changing material (6) | Interaction design framework (6); physics-only framing disappeared |
Ideas moved into new territory
Median ratings out of 7: novel directions 7, felt capable 6, felt motivated 6, ideas evolved 6
Interaction logs showed two styles. Some participants asked many questions (up to 16) and leapt into unexpected ideas; others iterated many times (up to 5) and refined steadily. P12 started with a bento box that changes shape to fit its contents and ended with one that turns throwing out leftovers into a small mourning ritual.
Idea evolution for P6 (organic breathing room), P9 (origami bird that reacts to moisture) and P12 (bento box that mourns leftovers).
Participants sorted the mentor’s feedback into direct suggestions to act on now, long-term advice for their practice, and personal preference, the mentor’s own style rather than a judgment of their idea. That lens is something a generic chatbot doesn’t give you.
It felt safe to ask “obvious” questions
All 12 participants said they felt less social pressure than with a human mentor or even peers, and every participant stayed in control of the design direction.
“I’m 100% in control and did not have to do what it told me to. I still feel ownership over the final idea, even though it was built on top of their work.” — P11
“There is a cognitive shift that happens when I see the sketchnotes; it gives the conversation a backstory.” — P9
“His sketches look quite familiar to how we ideate. It brings the distance closer.” — P6
Design guidelines
Design guidelines
- Interpret structure. Help learners follow the reading order, groupings and flow of a visual artifact, not just explain its symbols.
- Ground the AI in real traces. Anchor feedback in the person’s actual sketches and drafts so users can judge when to trust it.
- Let users set the distance. Offer different mentor personalities, and versions of a mentor from different years, so users choose how much presence they want.
Ethics and limitations
Who gets to be the “voice” of a mentor, and who gets credit when the AI extends an idea? We suggest framing the agent in the third person (“an AI drawing on the mentor’s materials”) rather than “I am your mentor,” involving the original author, and making it visible when the AI is quoting their work versus improvising. The goal is to enrich real mentorship, not replace it.
Participants noted the mentor could be wordy and occasionally random. The task order was fixed so we could see natural behavior first, which means topic familiarity and order effects can’t be fully separated.
