← all projects

research / 2022-2026 / PhD

Agentic LLM social robot platform

The LLM brain of the humanoid robot Nadine: a ten-agent LangGraph system and a ReAct agent, with long-term and visual memory.

MULTI-AGENT GRAPH · LANGGRAPH intent affect · PAD orchestrator memory retrieval search vision RAG response agent memory update REACT AGENT · LANGCHAIN contextualizer memory knowledge ReAct loop tools answer shared perception and control layers over MQTT
result
4 first-author papers, 1.5B models beating a 22B teacher
role
PhD researcher, University of Geneva
stack
LangGraph, Ollama, QLoRA, ChromaDB, CLIP
  • 10agents in the LangGraph dialogue graph
  • 98.7%accuracy of my fine-tuned 1.5B models, beating their 22B teacher
  • -53.5%latency after moving to the small fine-tuned models
  • 0.87saverage response time with multimodal memory

Problem

A social robot has to talk with strangers, in public, in real time: recognize them, remember them, react to emotion, and answer before the conversation stalls. One prompt to one large model does not cover that, so I treated it as a systems problem.

System

Architecture of the two interaction-layer designs, sharing the same perception and control layers
Three layers over an MQTT bus. Perception (RealSense camera, YOLOv8, InsightFace) and control (speech, lip sync, animation) are shared. The interaction layer, the subject of my PhD, exists in two designs: a LangGraph multi-agent graph (left) and a ReAct agent (right).

Three platforms, three constraints

Same robot, three versions of the interaction layer. Each exists because of one hard constraint.

Built for Design Models Trade-off
Multi-agent, local Privacy and zero API cost: nothing leaves the machine LangGraph graph of ten agents, each step separately testable One LoRA-tuned Qwen2.5-1.5B per agent, served on-device by Ollama Small tuned models stick to their training: new behavior needs new data, not a new prompt
Multi-agent, hybrid cloud A user study on memory: stable understanding, predictable latency Same graph; intent, affect and planning merged into one call bounded at 2.5 s Hosted model for language, vision kept local to protect participants Depends on the network and on a hosted model version
ReAct, cloud Public demos: must not fail in front of an audience Contextualizer, retrieval, then one ReAct loop with a fixed tool set Hosted models No persistent affect, no visual memory

Memory

SUMMER architecture: perception decides which scenes are memorable, interaction retrieves text and visual memories
SUMMER, my selective multimodal memory (AAMAS 2026). People do not remember every moment, so the robot should not either: it keeps a scene only when it is emotionally salient or new, and recalls it when the user asks "do you remember when...".
What was measured SUMMER Reference
Agreement with human ratings of what is memorable (Spearman) 0.506 0.415 between human raters; about 0 for standard image-memorability models
Retrieval over text and image together (Recall@1) up to +13% against text-only or image-only retrieval
Response time with memory 0.87 s 2 s limit for natural conversation

Evaluation

  • Ablations and benchmarks for tool-use recall and long-horizon memory, validated on LongMemEval.
  • A user study on multimodal memory, run on the hybrid-cloud platform with its A/B mode.
  • Two user studies with 123 participants and three years of public demos: see Nadine in the field.

Publications

PhD at MIRALab, University of Geneva (advisors: Prof. N. Magnenat Thalmann, Prof. S. Voloshynovskiy). Full technical documentation: Nadine Social Robot documentation.