Chieh-Yu Pan

KARINA: Knowledge-Augmented Representation for Ingredient-level Nutrition Analysis from Food Images

Chieh-Yu Pan, Wei-Ta Chu

IEEE Conference on Artificial Intelligence (CAI), 2026

doipdfcodethesisdemobibtex

A large multimodal model holds nutritional knowledge that its own predictions fail to deliver. It recognises that meat is high in protein and that starchy foods are high in carbohydrate, yet asked for the nutrition values of a dish it answers unreliably. KARINA draws on the knowledge rather than the answer. A frozen GPT-4o supplies ingredient-level descriptions with qualitative nutrition profiles, and these are integrated into an RGB-D visual model through bidirectional cross-attention. A mask-based augmentation using Grounded SAM aligns each description with the image region it describes.

Two earlier approaches did not succeed. I fine-tuned open vision-language models with LoRA to predict nutrition directly, and separately supplied the network with food categories assigned by hand. Neither produced a gain. The improvement came from changing what the model was asked to supply rather than how it was trained.

On Nutrition5k, percentage mean absolute error (PMAE) averages 13.3%, against 15.6% for the previous state of the art. I designed, implemented and evaluated the work independently.

Integrating Virtual Reality Technology and Simulation-Based Learning to Enhance Student Learning Performance and Engagement

Chieh-Yu Pan, Xin-Ni Lu, Yu-Ping Cheng, Xin-Yu Li, Yueh-Min Huang

Taiwan Academic Network Conference (TANET), 2022

Undergraduate Research Project Grant and Research Creativity Award, National Science and Technology Council

papervideo

In Chinese. Oral presentation. Wiring a Raspberry Pi wrongly can destroy the board, so beginners are rarely allowed to practise freely. This study rebuilds that practice in virtual reality, where every action is reversible. Built in MagicVR, the environment takes students through wiring the board and then writing a short program to blink three LEDs, with prompts at each step and a role-playing frame to hold attention.

A quasi-experiment with 28 undergraduates, half learning in VR and half from slides. The two groups were indistinguishable on the pre-test, and on the post-test the VR group scored significantly higher (Mann–Whitney U = 47, p = .018). Engagement rose over the session for the VR group (Wilcoxon signed-rank, p = .012).

Enhancing Learning Performance of Engineering Students in Virtual Reality Environment

Chieh-Yu Pan, Yu-Ping Cheng, Yueh-Min Huang

20th International Symposium on Novel and Sustainable Technology (ISNST), 2022

Excellent Poster Presentation Award

paperposter

A single-group evaluation of a virtual reality environment for learning the Raspberry Pi. Students worked through the basics and a practical exercise in the headset, sat a test before and after, and gave a short interview. Mean scores rose from 39.4 to 86.4 (dependent-sample t-test, t = −11.06, p < .001). In the interviews they credited the hands-on practice with making the material stick.

A proactive multimodal social robot

KTH Royal Institute of Technology, 2025 – present

videocode

A Misty II robot that looks for a reason to help before anyone asks, approaching someone who appears upset rather than waiting to be called. It watches with MediaPipe and listens with Whisper, and frames in which nobody faces the robot never reach GPT-4o, which keeps perception cheap. A course project at KTH with two other students, where I built perception and planning and a teammate built action execution on AutoMisty.

The planner is never asked for a physical quantity. Misty’s drive speed is a percentage of its maximum rather than a unit, so a model multiplying velocity by time overshoots. It returns intent instead, approach or stay or back away, and a closed-loop controller moves in bounded steps, re-measuring the distance after each one. It holds in simulation with the calibration wrong by half, where an open loop misses by around 45 cm.

I have kept working on it since, and am now moving it towards a ReAct loop in place of the fixed perceive, think, act cycle.

Invoice recognition and ERP automation

DIGI+ Talent Program, 2022

Outstanding Award

code

An internship project for a shoe manufacturer, and the first time I built anything with machine learning. YOLOv5 locates the fields on scanned triplicate invoices, OCR reads them, and an RPA workflow files the values in the client’s ERP system. Every field came out correct on 96% of invoices. Built for roughly 55,000 invoices a month, it cut an estimated 1,833 staff hours to 403.

The boxes had been annotated tight against the digits, so YOLO sometimes clipped them and OCR failed. I added a loop that nudges each box outward and inward at random and re-reads until two consecutive reads agree. Five configurations run in parallel, each a different checkpoint with its own offsets, and they vote field by field, so a value is written out only when the readings converge. A crude consistency check, and the first time I had to decide when the output of a model was worth believing.