Jkdsjksj

2026-06-19 09:34:52 +02:00
parent 023d3c04b9
commit 432df63298
24 changed files with 589 additions and 2644 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -2,16 +2,17 @@

 ## Overview

-QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of a programmable robot with color-block recognition, gesture control, stepper motors, RGB LED eyes, and multiple input methods (web dashboard, Android voice app). The codebase comprises four independent application layers communicating via HTTP JSON APIs.
+QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of a programmable robot with color-block recognition, gesture control, stepper motors, RGB LED eyes, and multiple input methods (web dashboard, Android voice app). The codebase comprises two independent application layers communicating via HTTP JSON APIs.

 ```
-[quibot-web Nuxt SPA]  ──HTTP──>  [backend Express]  ──HTTP──>  [raspi FastAPI (Pi)]
-          │                                         │
-          ▼                                         ▼
-    [apk Expo RN app]  ──HTTP──>  (same backend)       [Python hardware drivers]
+[quibot-web Nuxt SPA]  ──HTTP──>  [backend Express]
+          │                             │
+          ▼                             ├──▶ Raspberry Pi (port 8000) — motor/audio endpoints
+    [apk Expo RN app]  ──HTTP──>       ├──▶ LLM (llamacpp)
+                                        └──▶ TTS (Piper)
 ```

-**Tech stack**: Python (pigpio, FastAPI) | TypeScript/Express | Nuxt 4/Vue 3 | Expo/React Native
+**Tech stack**: TypeScript/Express | Nuxt 4/Vue 3 | Expo/React Native
 **No database**: All state is in-memory, file-based (`/tmp/quibot-audio/`), or localStorage.

 ---
@@ -20,30 +21,22 @@ QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of

 ```
 quibot/
-├── raspi/              # Raspberry Pi brain — Python, controls hardware
-│   ├── main.py         # FastAPI server (port 8000) on the Pi
-│   ├── quibot.py       # Main program: block/gesture threads (like Arduino QuiBot.ino)
-│   ├── motion.py       # Stepper class, homing, line-following, high-level tasks
-│   ├── gesture.py      # PAJ7620U2 gesture sensor (I2C, polled at 50ms)
-│   ├── blocks.py       # TCS34725 color sensor + servo block ejection
-│   ├── eyes.py         # WS2811 LED matrix (128 LEDs, pigpio waveforms, breathing animation)
-│   ├── pins.py         # BCM GPIO pin map for all hardware
-│   └── tests/          # Manual diagnostic scripts (not automated)
-├── backend/            # Local Express server (port 3000), proxy to Pi
+├── backend/            # Express server (port 5000), proxy to Pi + LLM/TTS
 │   ├── src/
-│   │   ├── index.ts           # Express entry: CORS, JSON parser, /health
-│   │   ├── config.ts          # Env config: RASPBERRY_PI_HOST, PORT, QUIBOT_TOKEN
+│   │   ├── index.ts           # Express entry: CORS, JSON parser, /health, routes
+│   │   ├── config.ts          # Env config: raspi, PIPER_URL, LLAMA_CPP_URL, PORT
 │   │   ├── routes/router.ts   # Mounts all controllers
-│   │   ├── services/raspi.service.ts  # Axios proxy layer to Pi FastAPI
+│   │   ├── services/raspi.service.ts  # Axios proxy layer to Pi endpoints
 │   │   └── controllers/
 │   │       ├── motor.controller.ts      # Motor step/stop/upload
 │   │       ├── audio.controller.ts      # Audio file lifecycle (incoming/locked/processed)
 │   │       ├── command.controller.ts    # POST /commands proxy to raspi /run
-│   │       └── settings.controller.ts   # GET/PUT /settings runtime config
+│   │       ├── settings.controller.ts   # GET/PUT /settings runtime config
+│   │       └── tts.controller.ts        # TTS synthesis via Piper
 │   └── dist/               # Compiled output (generated)
 ├── quibot-web/             # Nuxt 4 dashboard SPA
 │   ├── app/app.vue         # Single-page control panel: block queue, D-pad, eye controls, gesture log
-│   ├── server/api/         # Nitro server routes proxying to raspi
+│   ├── server/api/         # Nitro server routes proxying to backend Express
 │   │   ├── motor/step/[direction].post.ts
 │   │   └── motor/stop.post.ts
 │   ├── nuxt.config.ts      # Runtime config: QUIBOT_BASE_URL, QUIBOT_TOKEN
@@ -61,7 +54,9 @@ quibot/

 ---

-## Raspberry Pi Layer (`raspi/`)
+## Raspberry Pi Layer (remote, port 8000)
+
+The Raspberry Pi runs a lightweight HTTP server exposing hardware control endpoints. The `raspi/` source directory is no longer part of this repository — it lives on the Pi itself.

 **Hardware target**: Raspberry Pi Zero 2W controlling a robot with:
 - 5 NEMA-style stepper motors (wheels x2, arms x2, syringe) via A4988/TB600 drivers in STEP/DIR mode
@@ -74,7 +69,7 @@ quibot/
 - Hall-effect endstops on GPIOs 12, 16, 17
 - Optional: I2S audio amp (MAX98357A) + mic (SPH0645)

-### Key files
+### Hardware source files (on Pi)

 - **`pins.py`** — BCM GPIO pin numbering for every component (STEP, DIR, EN pins, I2C lines, endstops, LED_DATA on GPIO26)
 - **`motion.py`** — `Stepper` class with AccelStepper-style acceleration profiling via `pigpio.gpio_trigger()`. 5 motor instances (`wheel_R`, `wheel_L`, `arm_R`, `arm_L`, `syringe`). Continuous stepper daemon thread (`_stepper_loop`) at ~100Hz. Homing routines read Hall-effect endstops. Line-following with proportional correction on TCRT5000 values via ADS1115.
@@ -104,7 +99,7 @@ quibot/

 ## Backend Layer (`backend/`)

-**Role**: Express.js HTTP proxy sitting between frontend/mobile and the Raspberry Pi's FastAPI server. Token passthrough, no business logic.
+**Role**: Express.js HTTP server providing frontend/mobile API, proxying hardware commands to the Raspberry Pi, and managing TTS/LLM integration.

 ### Configuration (`.env`, loaded by `config.ts`)

@@ -113,15 +108,19 @@ quibot/
 | `RASPBERRY_PI_HOST` | `http://raspberrypi.local` | Pi API URL |
 | `RASPBERRY_PI_PORT` | `8000` | Pi API port |
 | `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token for all Pi endpoints |
-| `PORT` | `3000` | Backend listen port |
+| `PORT` | `5000` | Backend listen port |
+| `PIPER_URL` | `''` | Piper TTS service URL |
+| `LLAMA_CPP_URL` | `''` | LLM inference service URL |
+| `LLAMA_API_KEY` | `''` | LLM API key |
+| `LLAMA_PREAMBLE` | `''` | Path or content for LLM preamble |

 ### Architecture

 ```
 index.ts          → Express app, CORS, JSON parser, /health endpoint
-routes/router.ts  → Mounts all controllers under /motor, /audio, /commands, /settings
+routes/router.ts  → Mounts all controllers under /motor, /audio, /commands, /settings, /tts
 config.ts         → Mutable getter/setter env vars (runtime update via PUT /settings)
-raspi.service.ts  → Axios proxy methods for every Pi endpoint + multipart file upload handling
+raspi.service.ts  → Axios proxy methods for Pi endpoints + multipart file upload handling
 ```

 ### Controllers
@@ -130,6 +129,7 @@ raspi.service.ts  → Axios proxy methods for every Pi endpoint + multipart file
 - **`audio.controller.ts`** — `GET /audio/incoming`, `POST /audio/lock/:filename`, `/unlock/:filename`, `/cancel/:filename`, `/process/:filename`. All proxy to raspi audio file lifecycle endpoints.
 - **`command.controller.ts`** — `POST /commands { task }` → proxied to raspi `/run?task=...&token=...`
 - **`settings.controller.ts`** — `GET /settings` returns config; `PUT /settings` updates `raspberryPi.host`, `raspberryPi.port`, `token` at runtime.
+- **`tts.controller.ts`** — `POST /tts { text, lang }` → Synthesizes audio via Piper TTS service. Saves WAV files to `/tmp/quibot-audio/tts/`.

 ### Build/Run

@@ -150,7 +150,7 @@ node dist/index.js   # Or use tsx/nodemon for dev

 | Key | Default | Purpose |
 |-----|---------|---------|
-| `QUIBOT_BASE_URL` | `http://quibot:8000` | Base URL for raspi FastAPI |
+| `QUIBOT_BASE_URL` | `http://quibot:8000` | Base URL for backend Express (or Pi) |
 | `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token |

 ### UI Panels (`app/app.vue` — single-file SPA, 1369 lines)
@@ -170,8 +170,8 @@ node dist/index.js   # Or use tsx/nodemon for dev

 | Method | Path | Description |
 |--------|------|-------------|
-| POST | `/api/motor/stop` | Proxies to raspi `/motor/stop` |
-| POST | `/api/motor/step/:direction` | Proxies to raspi `/motor/step/forward\|backwards` |
+| POST | `/api/motor/stop` | Proxies to backend Express `/motor/stop` |
+| POST | `/api/motor/step/:direction` | Proxies to backend Express `/motor/step/forward\|backwards` |

 **Note**: The frontend also calls `POST /api/eye/shape`, `/api/eye/color`, `/api/eye/on`, `/api/eye/off`, `/api/gesture/on`, `/api/gesture/off` — server routes for these may need to be created (frontend references them but they don't have explicit server handlers yet).

@@ -221,7 +221,26 @@ CI: `build-apk.yml` runs expo prebuild, decodes keystore from secrets, builds si

 ## Complete API Reference

-### Raspberry Pi FastAPI (`raspi/main.py`) — port 8000
+### Backend Express (`backend/`) — port 5000
+
+| Method | Path | Body | Description |
+|--------|------|------|-------------|
+| GET | `/health` | — | Returns settings object |
+| POST | `/commands` | `{ task }` | Proxy to raspi `/run` |
+| POST | `/motor/step/forward` | — | Motor forward proxy to Pi |
+| POST | `/motor/step/backward` | — | Motor backward proxy to Pi (maps to `/backwards`) |
+| POST | `/motor/stop` | — | Motor stop proxy to Pi |
+| POST | `/motor/upload` | multipart file | Audio upload via multer → proxied to Pi |
+| GET | `/audio/incoming` | — | List incoming audio files from Pi |
+| POST | `/audio/lock/:filename` | — | Lock audio file on Pi |
+| POST | `/audio/unlock/:filename` | — | Unlock audio file on Pi |
+| POST | `/audio/cancel/:filename` | — | Cancel locked audio on Pi |
+| POST | `/audio/process/:filename` | — | Mark processed on Pi |
+| GET | `/settings` | — | Returns config |
+| PUT | `/settings` | `{ raspberryPi: { host, port }, token }` | Update runtime config |
+| POST | `/tts` | query: `text`, `lang` | Synthesize speech via Piper TTS |
+
+### Raspberry Pi HTTP Server (remote, port 8000)

 | Method | Path | Params/Body | Description |
 |--------|------|-------------|-------------|
@@ -238,24 +257,6 @@ CI: `build-apk.yml` runs expo prebuild, decodes keystore from secrets, builds si

 **Auth**: Query parameter `token` matching `QUIBOT_TOKEN` env var (default: `MY_SECRET_TOKEN`).

-### Backend Express (`backend/`) — port 3000
-
-| Method | Path | Body | Description |
-|--------|------|------|-------------|
-| GET | `/health` | — | Returns settings object |
-| POST | `/commands` | `{ task }` | Proxy to raspi `/run` |
-| POST | `/motor/step/forward` | — | Motor forward proxy |
-| POST | `/motor/step/backward` | — | Motor backward proxy (maps to raspi `/backwards`) |
-| POST | `/motor/stop` | — | Motor stop proxy |
-| POST | `/motor/upload` | multipart file | Audio upload via multer in-memory buffer |
-| GET | `/audio/incoming` | — | List incoming audio files |
-| POST | `/audio/lock/:filename` | — | Lock audio file |
-| POST | `/audio/unlock/:filename` | — | Unlock audio file |
-| POST | `/audio/cancel/:filename` | — | Cancel locked audio |
-| POST | `/audio/process/:filename` | — | Mark processed |
-| GET | `/settings` | — | Returns config |
-| PUT | `/settings` | `{ raspberryPi: { host, port }, token }` | Update runtime config |
-
 ---

 ## Command Flow Examples
@@ -266,9 +267,11 @@ User clicks "Forward" in D-pad
  → $fetch('/api/motor/step/forward', { method: 'POST' })
    → Nuxt Nitro route: server/api/motor/step/[direction].post.ts
      → $fetch(config.quibotBaseUrl + '/motor/step/forward', { query: { token } })
-        → raspi FastAPI /motor/step/forward
-          → motor_step("forward") in daemon thread
-            → step_motor(200, DIR, 1ms pulses)
+        → Backend Express /motor/step/forward
+          → raspi.service.motorStepForward()
+            → Pi FastAPI /motor/step/forward?token=...
+              → motor_step("forward") in daemon thread on Pi
+                → step_motor(200, DIR, 1ms pulses)
 ```

 ### Block Processing (internal to Pi)
@@ -278,8 +281,18 @@ Child inserts colored block → quibot.py task_read_blocks() polls distance sens
    → Manhattan distance classification against color lookup table
      → RED: eyes_turn_on(EYES_FW, DARK_RED, 2)
         _execute_action(task_move_to, CROSSING)
-           → enable_wheels(ON) → follow_line_loop(speed) (proportional on TCRT5000)
-    → After action: servo_move_to(EJECT_POSITION)
+            → enable_wheels(ON) → follow_line_loop(speed) (proportional on TCRT5000)
+     → After action: servo_move_to(EJECT_POSITION)
+```
+
+### TTS Synthesis
+```
+User triggers speech in web UI
+  → POST /tts?text=hello&lang=ca&token=...
+    → Backend Express tts.controller.ts
+      → piperService.synthesize() → Piper TTS service
+        → WAV file saved to /tmp/quibot-audio/tts/{uuid}.wav
+          → Returns audioUrl + filename
 ```

 ### APK → Audio Upload
@@ -317,8 +330,9 @@ User toggles mode in web UI

 ## Testing

-All tests in `raspi/tests/` are **manual diagnostic scripts** (not automated frameworks). Each test is run independently by uncommenting the desired function call at the bottom of the file. Requirements:
+Tests reside on the Pi alongside the hardware source code, not in this repository.

+Requirements:
 - `sudo pigpiod -s 1` daemon running
 - Python venv activated with hardware dependencies installed
 - Pi connected to robot hardware
@@ -337,8 +351,8 @@ All tests in `raspi/tests/` are **manual diagnostic scripts** (not automated fra

 1. **Token auth** is always a query parameter matching `QUIBOT_TOKEN` (default: `MY_SECRET_TOKEN`)
 2. **No database** — use filesystem for persistence, localStorage for web client state
-3. **Backend is a dumb proxy** — no business logic, just forwards HTTP requests with token passthrough
-4. **Motor commands are fire-and-forget** — motor runs in daemon thread until `/motor/stop`
+3. **Backend proxies Pi endpoints** — motor/audio commands forwarded via raspi.service.ts
+4. **Motor commands are fire-and-forget** — motor runs in daemon thread on Pi until `/motor/stop`
 5. **Audio lifecycle**: incoming → locked (claim) → processed OR unlocked (release) / cancelled
 6. **Eyes breathing** runs continuously at MIN_BR(80)-MAX_BR(170) brightness in background
 7. **`quibot.py` owns block/gesture autonomy** — blocks are processed internally on the Pi without backend/web involvement