Chieh-Yu Pan

KARINA: Knowledge-Augmented Representation for Ingredient-level Nutrition Analysis from Food Images

Chieh-Yu Pan, Wei-Ta Chu

IEEE Conference on Artificial Intelligence (CAI), 2026

doipdfcodethesisdemobibtex

A large multimodal model holds nutritional knowledge that its own predictions fail to deliver. It recognises that meat is high in protein and that starchy foods are high in carbohydrate, yet asked for the nutrition values of a dish it answers unreliably. KARINA draws on the knowledge rather than the answer. A frozen GPT-4o supplies ingredient-level descriptions with qualitative nutrition profiles, and these are integrated into an RGB-D visual model through bidirectional cross-attention. A mask-based augmentation using Grounded SAM aligns each description with the image region it describes.

Two earlier approaches did not succeed. I fine-tuned open vision-language models with LoRA to predict nutrition directly, and separately supplied the network with food categories assigned by hand. Neither produced a gain. The improvement came from changing what the model was asked to supply rather than how it was trained.

On Nutrition5k, percentage mean absolute error (PMAE) averages 13.3%, against 15.6% for the previous state of the art. I independently drove the survey, method design, implementation, and evaluation under weekly supervision from Prof. Wei-Ta Chu.

Integrating Virtual Reality Technology and Simulation-Based Learning to Enhance Student Learning Performance and Engagement

Chieh-Yu Pan, Xin-Ni Lu, Yu-Ping Cheng, Xin-Yu Li, Yueh-Min Huang

Taiwan Academic Network Conference (TANET), 2022

Undergraduate Research Project Grant and Research Creativity Award, National Science and Technology Council

papervideo

In Chinese. Oral presentation. Wiring a Raspberry Pi wrongly can destroy the board, so beginners are rarely allowed to practise freely. This study rebuilds that practice in virtual reality, where every action is reversible. Built in MagicVR, the environment takes students through wiring the board and then writing a short program to blink three LEDs, with prompts at each step and a role-playing frame to hold attention.

A quasi-experiment with 28 undergraduates, half learning in VR and half from slides. The two groups were indistinguishable on the pre-test, and on the post-test the VR group scored significantly higher (Mann–Whitney U = 47, p = .018). Engagement rose over the session for the VR group (Wilcoxon signed-rank, p = .012).

Enhancing Learning Performance of Engineering Students in Virtual Reality Environment

Chieh-Yu Pan, Yu-Ping Cheng, Yueh-Min Huang

20th International Symposium on Novel and Sustainable Technology (ISNST), 2022

Excellent Poster Presentation Award

paperposter

A single-group evaluation of a virtual reality environment for learning the Raspberry Pi. Students worked through the basics and a practical exercise in the headset, sat a test before and after, and gave a short interview. Mean scores rose from 39.4 to 86.4 (dependent-sample t-test, t = −11.06, p < .001). In the interviews they credited the hands-on practice with making the material stick.

Misty Embodied Agent: a proactive social robot

KTH course project, 2025; independent open-source rebuild, 2026

democodeoriginal course video

I rebuilt our KTH social robotics project as an LLM agent that can initiate an interaction from speech or visual cues. Local wake detection and sustained visual signals decide when to engage; an attention queue gives one interaction at a time to a bounded ReAct loop. The model can use typed tools to speak, listen, inspect the scene, move or respect a request for space, with each action and observation recorded for inspection.

The model chooses a distance category rather than drive speed or duration; a separate controller plans bounded movement from fresh readings and checks for stops and hazards. The repository includes 15 acceptance situations and a local demo with recorded model decisions. This rebuild has been tested with simulated robot effects and synthetic audio and visual inputs, not on a Misty II. Physical behaviour, timing and safety remain unverified.

The original version was a KTH course project with two classmates. I built its perception and planning components; the video above shows that earlier hardware demonstration, while the linked code and demo describe the later rebuild.

Invoice recognition and ERP automation

DIGI+ Talent Program, 2022

Outstanding Award

code

An internship project for a shoe manufacturer, and the first time I built anything with machine learning. YOLOv5 locates the fields on scanned triplicate invoices, OCR reads them, and an RPA workflow files the values in the client’s ERP system. Every field came out correct on 96% of invoices. Built for roughly 55,000 invoices a month, it cut an estimated 1,833 staff hours to 403.

The boxes had been annotated tight against the digits, so YOLO sometimes clipped them and OCR failed. I added a loop that nudges each box outward and inward at random and re-reads until two consecutive reads agree. Five configurations run in parallel, each a different checkpoint with its own offsets, and they vote field by field, so a value is written out only when the readings converge. A crude consistency check, and the first time I had to decide when the output of a model was worth believing.