quibot/AGENTS.md

QuiBot Project — Agent Guide

Overview

QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of a programmable robot with color-block recognition, gesture control, stepper motors, RGB LED eyes, and multiple input methods (web dashboard, Android voice app). The codebase comprises two independent application layers communicating via HTTP JSON APIs.

[quibot-web Nuxt SPA]  ──HTTP──>  [backend Express]
          │                             │
          ▼                             ├──▶ Raspberry Pi (port 8000) — motor/audio endpoints
    [apk Expo RN app]  ──HTTP──>       ├──▶ LLM (llamacpp)
                                        └──▶ TTS (Piper)

Tech stack: TypeScript/Express | Nuxt 4/Vue 3 | Expo/React Native No database: All state is in-memory, file-based (/tmp/quibot-audio/), or localStorage.

---

Directory Structure

quibot/
├── backend/            # Express server (port 5000), proxy to Pi + LLM/TTS
│   ├── src/
│   │   ├── index.ts           # Express entry: CORS, JSON parser, /health, routes
│   │   ├── config.ts          # Env config: raspi, PIPER_URL, LLAMA_CPP_URL, PORT
│   │   ├── routes/router.ts   # Mounts all controllers
│   │   ├── services/raspi.service.ts  # Axios proxy layer to Pi endpoints
│   │   └── controllers/
│   │       ├── motor.controller.ts      # Motor step/stop/upload
│   │       ├── audio.controller.ts      # Audio file lifecycle (incoming/locked/processed)
│   │       ├── command.controller.ts    # POST /commands proxy to raspi /run
│   │       ├── settings.controller.ts   # GET/PUT /settings runtime config
│   │       └── tts.controller.ts        # TTS synthesis via Piper
│   └── dist/               # Compiled output (generated)
├── quibot-web/             # Nuxt 4 dashboard SPA
│   ├── app/app.vue         # Single-page control panel: block queue, D-pad, eye controls, gesture log
│   ├── server/api/         # Nitro server routes proxying to backend Express
│   │   ├── motor/step/[direction].post.ts
│   │   └── motor/stop.post.ts
│   ├── nuxt.config.ts      # Runtime config: QUIBOT_BASE_URL, QUIBOT_TOKEN
│   └── .output/            # Built Nitro output
├── apk/                    # Expo React Native voice recorder ("VoiceDrop")
│   ├── app/index.tsx       # Recording screen + upload
│   ├── app/settings.tsx    # Backend URL/token configuration (AsyncStorage)
│   └── lib/recorder-settings.ts  # AsyncStorage wrapper
├── .gitea/workflows/       # CI/CD pipelines
│   ├── build.yml           # Web + backend → zip artifacts + Gitea release
│   └── build-apk.yml       # Expo prebuild + signed APK → Gitea release
├── build.sh                # Placeholder
└── README.md               # Project overview (Catalan)

---

Raspberry Pi Layer (remote, port 8000)

The Raspberry Pi runs a lightweight HTTP server exposing hardware control endpoints. The raspi/ source directory is no longer part of this repository — it lives on the Pi itself.

Hardware target: Raspberry Pi Zero 2W controlling a robot with:

• 5 NEMA-style stepper motors (wheels x2, arms x2, syringe) via A4988/TB600 drivers in STEP/DIR mode
• VL53L0X ToF distance sensor (I2C bus 3)
• PAJ7620U2 gesture sensor (I2C bus 3, polled at 50ms)
• TCS34725 color sensor (bit-banged I2C on GPIO22/27)
• ADS1115 ADC for TCRT5000 line-following IR sensors
• WS2811 RGB LED matrix (2x 8x8 = 128 LEDs, GPIO26, pigpio waveforms at -s 1)
• Servo motor (GPIO10 PWM) for block ejection
• Hall-effect endstops on GPIOs 12, 16, 17
• Optional: I2S audio amp (MAX98357A) + mic (SPH0645)

Hardware source files (on Pi)

• pins.py — BCM GPIO pin numbering for every component (STEP, DIR, EN pins, I2C lines, endstops, LED_DATA on GPIO26)
• motion.py — Stepper class with AccelStepper-style acceleration profiling via pigpio.gpio_trigger(). 5 motor instances (wheel_R, wheel_L, arm_R, arm_L, syringe). Continuous stepper daemon thread (_stepper_loop) at ~100Hz. Homing routines read Hall-effect endstops. Line-following with proportional correction on TCRT5000 values via ADS1115.
• gesture.py — Raw I2C via smbus2 (bus 3). Two-register-bank init (~240 total register writes). Polls gesture result registers 0x43/0x44 every 50ms. Returns: GS_NONE, GS_FORWARD, GS_LEFT, GS_RIGHT, GS_UP, GS_DOWN, GS_CLOCKWISE, GS_ANTICLOCKWISE, GS_WAVE
• blocks.py — TCS34725 RGB reads classified via Manhattan distance against calibrated reference table (BK/RD/GN/BU/YE/OG/VT). Smooth servo movement with 3us micro-steps.
• eyes.py — 128 WS2811 LEDs via pigpio waveforms at 1us resolution. Pre-defined shapes: EYES_OPEN, EYES_FW, EYES_DOWN, EYES_GESTURE. Breathing thread oscillates brightness 80-170 at 50ms intervals.
• quibot.py — Main program (equivalent to original Arduino QuiBot.ino). Two threading tasks via FreeRTOS-style pattern: task_read_blocks() (color→action mapping) and task_read_gestures() (WAVE toggles between block/gesture mode). threading.Lock prevents concurrent motor movements. Handles SIGINT/SIGTERM for graceful shutdown.
• main.py — FastAPI server on port 8000 with CORS. File-based state management using /tmp/quibot-audio/ (incoming/locked/processed directories). Token auth via query parameter matching QUIBOT_TOKEN.

Color→Action Mapping (in quibot.py)

Color	Action
RED	Advance forward
GREEN	Turn right
BLUE	Turn left
YELLOW	Take/block pick-up
ORANGE	Leave/eject
VIOLET	Idle
BLACK	Reference / no block

Motor Position Tracking

Stepper class in motion.py tracks absolute position in steps via _pos and _target. Endstops provide physical reference during homing. The stepper loop evaluates acceleration profiles to generate STEP pulses at correct intervals.

---

Backend Layer (backend/)

Role: Express.js HTTP server providing frontend/mobile API, proxying hardware commands to the Raspberry Pi, and managing TTS/LLM integration.

Configuration (.env, loaded by config.ts)

Variable	Default	Purpose
RASPBERRY_PI_HOST	http://raspberrypi.local	Pi API URL
RASPBERRY_PI_PORT	8000	Pi API port
QUIBOT_TOKEN	MY_SECRET_TOKEN	Auth token for all Pi endpoints
PORT	5000	Backend listen port
PIPER_URL	''	Piper TTS service URL
LLAMA_CPP_URL	''	LLM inference service URL
LLAMA_API_KEY	''	LLM API key
LLAMA_PREAMBLE	''	Path or content for LLM preamble

Architecture

index.ts          → Express app, CORS, JSON parser, /health endpoint
routes/router.ts  → Mounts all controllers under /motor, /audio, /commands, /settings, /tts
config.ts         → Mutable getter/setter env vars (runtime update via PUT /settings)
raspi.service.ts  → Axios proxy methods for Pi endpoints + multipart file upload handling

Controllers

• motor.controller.ts — POST /motor/step/forward, /motor/step/backward, /motor/stop. Also POST /motor/upload (multer multipart → proxied as FormData to Pi).
• audio.controller.ts — GET /audio/incoming, POST /audio/lock/:filename, /unlock/:filename, /cancel/:filename, /process/:filename. All proxy to raspi audio file lifecycle endpoints.
• command.controller.ts — POST /commands { task } → proxied to raspi /run?task=...&token=...
• settings.controller.ts — GET /settings returns config; PUT /settings updates raspberryPi.host, raspberryPi.port, token at runtime.
• tts.controller.ts — POST /tts { text, lang } → Synthesizes audio via Piper TTS service. Saves WAV files to /tmp/quibot-audio/tts/.

Build/Run

cd backend
npm install          # Installs express, axios, multer, dotenv, cors, typescript
npx tsc              # Compiles to dist/
node dist/index.js   # Or use tsx/nodemon for dev

---

Web Frontend (quibot-web/)

Stack: Nuxt 4 (Nitro) + Vue 3. Single-page dashboard SPA.

Runtime config (nuxt.config.ts)

Key	Default	Purpose
QUIBOT_BASE_URL	http://quibot:8000	Base URL for backend Express (or Pi)
QUIBOT_TOKEN	MY_SECRET_TOKEN	Auth token

UI Panels (app/app.vue — single-file SPA, 1369 lines)

1. Block Queue Panel — Displays color blocks in a queue (localStorage persistence + demo fallback). Shows action descriptions per color.
2. Motion Controls — D-pad grid: up=forward, down=back, left/right=turns, center=stop. Sends $fetch('/api/motor/step/forward') etc.
3. Eye Controls — Shape selector (open/forward/down/gesture), 8-color picker. Calls POST /api/eye/shape, /api/eye/color, /api/eye/on, /api/eye/off.
4. Gesture Sensor Panel — Toggle between Block Mode / Gesture Mode. Gesture detection history log. Reference table of all 8 gestures.

State & Styling

• Dark/light theme via CSS custom properties, persisted in localStorage.
• Block queue data stored in localStorage with demo fallback.
• Toast notifications for success/error feedback.
• Responsive layout with CSS Grid (mobile-adaptive).

Server Routes (server/api/)

Method	Path	Description
POST	/api/motor/stop	Proxies to backend Express /motor/stop
POST	/api/motor/step/:direction	Proxies to backend Express /motor/step/forward|backwards

Note: The frontend also calls POST /api/eye/shape, /api/eye/color, /api/eye/on, /api/eye/off, /api/gesture/on, /api/gesture/off — server routes for these may need to be created (frontend references them but they don't have explicit server handlers yet).

Build/Run

cd quibot-web
npm install
npx nuxt build   # Produces .output/
# Dev mode:
npx nuxt dev

CI builds the Nuxt output and zips it for distribution.

---

Android APK (apk/)

Stack: Expo SDK ~54 + React Native 0.81.5 + Expo Router ~6.0. App name: "VoiceDrop".

Features

• Records audio via expo-av (Audio.Recording — produces .m4a)
• Auto-uploads on stop as multipart form data to configurable backend URL
• Displays recording timer, status messages, raw server response (first 400 chars)

Screens

• app/index.tsx — Recorder screen: start/stop recording, upload, status
• app/settings.tsx — Backend URL, Bearer token, form field name. Saved to AsyncStorage under recorder.* namespace.

Persistence

Settings stored in @react-native-async-storage/async-storage (keys: recorder.backendUrl, recorder.authToken, recorder.fieldName).

Build/Run

cd apk
npm install
npx expo start              # Dev mode
# Production APK (local):
./build.sh                  # expo prebuild → generates android/ → gradle assembleRelease

CI: build-apk.yml runs expo prebuild, decodes keystore from secrets, builds signed release APK.

---

Complete API Reference

Backend Express (backend/) — port 5000

Method	Path	Body	Description
GET	/health	—	Returns settings object
POST	/commands	{ task }	Proxy to raspi /run
POST	/motor/step/forward	—	Motor forward proxy to Pi
POST	/motor/step/backward	—	Motor backward proxy to Pi (maps to /backwards)
POST	/motor/stop	—	Motor stop proxy to Pi
POST	/motor/upload	multipart file	Audio upload via multer → proxied to Pi
GET	/audio/incoming	—	List incoming audio files from Pi
POST	/audio/lock/:filename	—	Lock audio file on Pi
POST	/audio/unlock/:filename	—	Unlock audio file on Pi
POST	/audio/cancel/:filename	—	Cancel locked audio on Pi
POST	/audio/process/:filename	—	Mark processed on Pi
GET	/settings	—	Returns config
PUT	/settings	{ raspberryPi: { host, port }, token }	Update runtime config
POST	/tts	query: text, lang	Synthesize speech via Piper TTS

Raspberry Pi HTTP Server (remote, port 8000)

Method	Path	Params/Body	Description
POST	/run	query: task, token	Runs whitelisted system commands (restart_nginx, uptime, update)
POST	/motor/step/forward	query: token	Starts motor forward (daemon thread)
POST	/motor/step/backwards	query: token	Starts motor backward (daemon thread)
POST	/motor/stop	query: token	Disables motor driver (GPIO EN HIGH)
POST	/audio/upload	multipart: file, query: format	Saves to /tmp/quibot-audio/incoming/, returns filename + lock_url
GET	/audio/incoming	—	Lists files with size and modified time
POST	/audio/lock/{filename}	—	incoming → locked (claim for processing)
POST	/audio/unlock/{filename}	—	locked → incoming (release)
POST	/audio/cancel/{filename}	—	locked → incoming (cancel)
POST	/audio/process/{filename}	—	locked → processed

Auth: Query parameter token matching QUIBOT_TOKEN env var (default: MY_SECRET_TOKEN).

---

Command Flow Examples

Web → Motion Control

User clicks "Forward" in D-pad
  → $fetch('/api/motor/step/forward', { method: 'POST' })
    → Nuxt Nitro route: server/api/motor/step/[direction].post.ts
      → $fetch(config.quibotBaseUrl + '/motor/step/forward', { query: { token } })
        → Backend Express /motor/step/forward
          → raspi.service.motorStepForward()
            → Pi FastAPI /motor/step/forward?token=...
              → motor_step("forward") in daemon thread on Pi
                → step_motor(200, DIR, 1ms pulses)

Block Processing (internal to Pi)

Child inserts colored block → quibot.py task_read_blocks() polls distance sensor
  → When detected <80mm: read_block_color() via TCS34725
    → Manhattan distance classification against color lookup table
      → RED: eyes_turn_on(EYES_FW, DARK_RED, 2)
         _execute_action(task_move_to, CROSSING)
            → enable_wheels(ON) → follow_line_loop(speed) (proportional on TCRT5000)
     → After action: servo_move_to(EJECT_POSITION)

TTS Synthesis

User triggers speech in web UI
  → POST /tts?text=hello&lang=ca&token=...
    → Backend Express tts.controller.ts
      → piperService.synthesize() → Piper TTS service
        → WAV file saved to /tmp/quibot-audio/tts/{uuid}.wav
          → Returns audioUrl + filename

APK → Audio Upload

User stops recording in VoiceDrop → expo-av .m4a file
  → POST {backendUrl} with FormData {fieldName: "file"} + Bearer auth
    → raspi FastAPI saves to /tmp/quibot-audio/incoming/{uuid}.wav
      → Returns: { status: "received", filename, lock_url }

Gesture Mode Toggle

User toggles mode in web UI
  → _execute_action() locks mutex
    → If gesture mode: eyes_gesture_mode_on() (double cyan flash on 128-LED matrix)
      → Eyes breathing thread at MAX_BR(170) brightness

---

CI/CD (/.gitea/workflows/)

build.yml — Web + Backend

• Triggers: Push to master
• Builds web: npm install && npx nuxt build, zips .output/
• Builds backend: Zips entire backend/ directory
• Creates Gitea release "latest" with both zip artifacts

build-apk.yml — Mobile

• Triggers: Push to master
• expo prebuild → decode keystore from secrets → ./gradlew assembleRelease
• Creates Gitea release with APK artifact

---

Testing

Tests reside on the Pi alongside the hardware source code, not in this repository.

Requirements:

• sudo pigpiod -s 1 daemon running
• Python venv activated with hardware dependencies installed
• Pi connected to robot hardware

Test	What it verifies
test_simple_motor.py	Low-level motor driver via direct GPIO writes
test_motion.py	Wheel steppers, arm/syringe homing, VL53L0X distance, ADS1115 line sensors
test_blocks.py	Servo sweep (open/eject/open), color sensor readings, raw RGB calibration
test_gesture.py	PAJ7620U2 I2C connection + 30-second gesture capture
test_eyes.py	LED shape/color rendering, animation repeat/direction, gesture animations, breathing

---

Key Conventions

1. Token auth is always a query parameter matching QUIBOT_TOKEN (default: MY_SECRET_TOKEN)
2. No database — use filesystem for persistence, localStorage for web client state
3. Backend proxies Pi endpoints — motor/audio commands forwarded via raspi.service.ts
4. Motor commands are fire-and-forget — motor runs in daemon thread on Pi until /motor/stop
5. Audio lifecycle: incoming → locked (claim) → processed OR unlocked (release) / cancelled
6. Eyes breathing runs continuously at MIN_BR(80)-MAX_BR(170) brightness in background
7. quibot.py owns block/gesture autonomy — blocks are processed internally on the Pi without backend/web involvement
8. All paths use forward slashes in URLs; kebab-case for params