TTs whisper

2026-06-18 13:45:32 +02:00
parent 0e7fbbfdca
commit 9a23863320
57 changed files with 10430 additions and 253 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -0,0 +1,345 @@
+# QuiBot Project — Agent Guide
+
+## Overview
+
+QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of a programmable robot with color-block recognition, gesture control, stepper motors, RGB LED eyes, and multiple input methods (web dashboard, Android voice app). The codebase comprises four independent application layers communicating via HTTP JSON APIs.
+
+```
+[quibot-web Nuxt SPA]  ──HTTP──>  [backend Express]  ──HTTP──>  [raspi FastAPI (Pi)]
+          │                                         │
+          ▼                                         ▼
+    [apk Expo RN app]  ──HTTP──>  (same backend)       [Python hardware drivers]
+```
+
+**Tech stack**: Python (pigpio, FastAPI) | TypeScript/Express | Nuxt 4/Vue 3 | Expo/React Native
+**No database**: All state is in-memory, file-based (`/tmp/quibot-audio/`), or localStorage.
+
+---
+
+## Directory Structure
+
+```
+quibot/
+├── raspi/              # Raspberry Pi brain — Python, controls hardware
+│   ├── main.py         # FastAPI server (port 8000) on the Pi
+│   ├── quibot.py       # Main program: block/gesture threads (like Arduino QuiBot.ino)
+│   ├── motion.py       # Stepper class, homing, line-following, high-level tasks
+│   ├── gesture.py      # PAJ7620U2 gesture sensor (I2C, polled at 50ms)
+│   ├── blocks.py       # TCS34725 color sensor + servo block ejection
+│   ├── eyes.py         # WS2811 LED matrix (128 LEDs, pigpio waveforms, breathing animation)
+│   ├── pins.py         # BCM GPIO pin map for all hardware
+│   └── tests/          # Manual diagnostic scripts (not automated)
+├── backend/            # Local Express server (port 3000), proxy to Pi
+│   ├── src/
+│   │   ├── index.ts           # Express entry: CORS, JSON parser, /health
+│   │   ├── config.ts          # Env config: RASPBERRY_PI_HOST, PORT, QUIBOT_TOKEN
+│   │   ├── routes/router.ts   # Mounts all controllers
+│   │   ├── services/raspi.service.ts  # Axios proxy layer to Pi FastAPI
+│   │   └── controllers/
+│   │       ├── motor.controller.ts      # Motor step/stop/upload
+│   │       ├── audio.controller.ts      # Audio file lifecycle (incoming/locked/processed)
+│   │       ├── command.controller.ts    # POST /commands proxy to raspi /run
+│   │       └── settings.controller.ts   # GET/PUT /settings runtime config
+│   └── dist/               # Compiled output (generated)
+├── quibot-web/             # Nuxt 4 dashboard SPA
+│   ├── app/app.vue         # Single-page control panel: block queue, D-pad, eye controls, gesture log
+│   ├── server/api/         # Nitro server routes proxying to raspi
+│   │   ├── motor/step/[direction].post.ts
+│   │   └── motor/stop.post.ts
+│   ├── nuxt.config.ts      # Runtime config: QUIBOT_BASE_URL, QUIBOT_TOKEN
+│   └── .output/            # Built Nitro output
+├── apk/                    # Expo React Native voice recorder ("VoiceDrop")
+│   ├── app/index.tsx       # Recording screen + upload
+│   ├── app/settings.tsx    # Backend URL/token configuration (AsyncStorage)
+│   └── lib/recorder-settings.ts  # AsyncStorage wrapper
+├── .gitea/workflows/       # CI/CD pipelines
+│   ├── build.yml           # Web + backend → zip artifacts + Gitea release
+│   └── build-apk.yml       # Expo prebuild + signed APK → Gitea release
+├── build.sh                # Placeholder
+└── README.md               # Project overview (Catalan)
+```
+
+---
+
+## Raspberry Pi Layer (`raspi/`)
+
+**Hardware target**: Raspberry Pi Zero 2W controlling a robot with:
+- 5 NEMA-style stepper motors (wheels x2, arms x2, syringe) via A4988/TB600 drivers in STEP/DIR mode
+- VL53L0X ToF distance sensor (I2C bus 3)
+- PAJ7620U2 gesture sensor (I2C bus 3, polled at 50ms)
+- TCS34725 color sensor (bit-banged I2C on GPIO22/27)
+- ADS1115 ADC for TCRT5000 line-following IR sensors
+- WS2811 RGB LED matrix (2x 8x8 = 128 LEDs, GPIO26, pigpio waveforms at -s 1)
+- Servo motor (GPIO10 PWM) for block ejection
+- Hall-effect endstops on GPIOs 12, 16, 17
+- Optional: I2S audio amp (MAX98357A) + mic (SPH0645)
+
+### Key files
+
+- **`pins.py`** — BCM GPIO pin numbering for every component (STEP, DIR, EN pins, I2C lines, endstops, LED_DATA on GPIO26)
+- **`motion.py`** — `Stepper` class with AccelStepper-style acceleration profiling via `pigpio.gpio_trigger()`. 5 motor instances (`wheel_R`, `wheel_L`, `arm_R`, `arm_L`, `syringe`). Continuous stepper daemon thread (`_stepper_loop`) at ~100Hz. Homing routines read Hall-effect endstops. Line-following with proportional correction on TCRT5000 values via ADS1115.
+- **`gesture.py`** — Raw I2C via `smbus2` (bus 3). Two-register-bank init (~240 total register writes). Polls gesture result registers 0x43/0x44 every 50ms. Returns: GS_NONE, GS_FORWARD, GS_LEFT, GS_RIGHT, GS_UP, GS_DOWN, GS_CLOCKWISE, GS_ANTICLOCKWISE, GS_WAVE
+- **`blocks.py`** — TCS34725 RGB reads classified via Manhattan distance against calibrated reference table (BK/RD/GN/BU/YE/OG/VT). Smooth servo movement with 3us micro-steps.
+- **`eyes.py`** — 128 WS2811 LEDs via pigpio waveforms at 1us resolution. Pre-defined shapes: EYES_OPEN, EYES_FW, EYES_DOWN, EYES_GESTURE. Breathing thread oscillates brightness 80-170 at 50ms intervals.
+- **`quibot.py`** — Main program (equivalent to original Arduino QuiBot.ino). Two threading tasks via FreeRTOS-style pattern: `task_read_blocks()` (color→action mapping) and `task_read_gestures()` (WAVE toggles between block/gesture mode). `threading.Lock` prevents concurrent motor movements. Handles SIGINT/SIGTERM for graceful shutdown.
+- **`main.py`** — FastAPI server on port 8000 with CORS. File-based state management using `/tmp/quibot-audio/` (incoming/locked/processed directories). Token auth via query parameter matching `QUIBOT_TOKEN`.
+
+### Color→Action Mapping (in quibot.py)
+
+| Color | Action |
+|-------|--------|
+| RED | Advance forward |
+| GREEN | Turn right |
+| BLUE | Turn left |
+| YELLOW | Take/block pick-up |
+| ORANGE | Leave/eject |
+| VIOLET | Idle |
+| BLACK | Reference / no block |
+
+### Motor Position Tracking
+
+`Stepper` class in `motion.py` tracks absolute position in steps via `_pos` and `_target`. Endstops provide physical reference during homing. The stepper loop evaluates acceleration profiles to generate STEP pulses at correct intervals.
+
+---
+
+## Backend Layer (`backend/`)
+
+**Role**: Express.js HTTP proxy sitting between frontend/mobile and the Raspberry Pi's FastAPI server. Token passthrough, no business logic.
+
+### Configuration (`.env`, loaded by `config.ts`)
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `RASPBERRY_PI_HOST` | `http://raspberrypi.local` | Pi API URL |
+| `RASPBERRY_PI_PORT` | `8000` | Pi API port |
+| `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token for all Pi endpoints |
+| `PORT` | `3000` | Backend listen port |
+
+### Architecture
+
+```
+index.ts          → Express app, CORS, JSON parser, /health endpoint
+routes/router.ts  → Mounts all controllers under /motor, /audio, /commands, /settings
+config.ts         → Mutable getter/setter env vars (runtime update via PUT /settings)
+raspi.service.ts  → Axios proxy methods for every Pi endpoint + multipart file upload handling
+```
+
+### Controllers
+
+- **`motor.controller.ts`** — `POST /motor/step/forward`, `/motor/step/backward`, `/motor/stop`. Also `POST /motor/upload` (multer multipart → proxied as FormData to Pi).
+- **`audio.controller.ts`** — `GET /audio/incoming`, `POST /audio/lock/:filename`, `/unlock/:filename`, `/cancel/:filename`, `/process/:filename`. All proxy to raspi audio file lifecycle endpoints.
+- **`command.controller.ts`** — `POST /commands { task }` → proxied to raspi `/run?task=...&token=...`
+- **`settings.controller.ts`** — `GET /settings` returns config; `PUT /settings` updates `raspberryPi.host`, `raspberryPi.port`, `token` at runtime.
+
+### Build/Run
+
+```bash
+cd backend
+npm install          # Installs express, axios, multer, dotenv, cors, typescript
+npx tsc              # Compiles to dist/
+node dist/index.js   # Or use tsx/nodemon for dev
+```
+
+---
+
+## Web Frontend (`quibot-web/`)
+
+**Stack**: Nuxt 4 (Nitro) + Vue 3. Single-page dashboard SPA.
+
+### Runtime config (`nuxt.config.ts`)
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `QUIBOT_BASE_URL` | `http://quibot:8000` | Base URL for raspi FastAPI |
+| `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token |
+
+### UI Panels (`app/app.vue` — single-file SPA, 1369 lines)
+
+1. **Block Queue Panel** — Displays color blocks in a queue (localStorage persistence + demo fallback). Shows action descriptions per color.
+2. **Motion Controls** — D-pad grid: up=forward, down=back, left/right=turns, center=stop. Sends `$fetch('/api/motor/step/forward')` etc.
+3. **Eye Controls** — Shape selector (open/forward/down/gesture), 8-color picker. Calls `POST /api/eye/shape`, `/api/eye/color`, `/api/eye/on`, `/api/eye/off`.
+4. **Gesture Sensor Panel** — Toggle between Block Mode / Gesture Mode. Gesture detection history log. Reference table of all 8 gestures.
+
+### State & Styling
+- Dark/light theme via CSS custom properties, persisted in localStorage.
+- Block queue data stored in localStorage with demo fallback.
+- Toast notifications for success/error feedback.
+- Responsive layout with CSS Grid (mobile-adaptive).
+
+### Server Routes (`server/api/`)
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/api/motor/stop` | Proxies to raspi `/motor/stop` |
+| POST | `/api/motor/step/:direction` | Proxies to raspi `/motor/step/forward\|backwards` |
+
+**Note**: The frontend also calls `POST /api/eye/shape`, `/api/eye/color`, `/api/eye/on`, `/api/eye/off`, `/api/gesture/on`, `/api/gesture/off` — server routes for these may need to be created (frontend references them but they don't have explicit server handlers yet).
+
+### Build/Run
+
+```bash
+cd quibot-web
+npm install
+npx nuxt build   # Produces .output/
+# Dev mode:
+npx nuxt dev
+```
+
+CI builds the Nuxt output and zips it for distribution.
+
+---
+
+## Android APK (`apk/`)
+
+**Stack**: Expo SDK ~54 + React Native 0.81.5 + Expo Router ~6.0. App name: "VoiceDrop".
+
+### Features
+- Records audio via `expo-av` (`Audio.Recording` — produces .m4a)
+- Auto-uploads on stop as multipart form data to configurable backend URL
+- Displays recording timer, status messages, raw server response (first 400 chars)
+
+### Screens
+- **`app/index.tsx`** — Recorder screen: start/stop recording, upload, status
+- **`app/settings.tsx`** — Backend URL, Bearer token, form field name. Saved to AsyncStorage under `recorder.*` namespace.
+
+### Persistence
+Settings stored in `@react-native-async-storage/async-storage` (keys: `recorder.backendUrl`, `recorder.authToken`, `recorder.fieldName`).
+
+### Build/Run
+
+```bash
+cd apk
+npm install
+npx expo start              # Dev mode
+# Production APK (local):
+./build.sh                  # expo prebuild → generates android/ → gradle assembleRelease
+```
+
+CI: `build-apk.yml` runs expo prebuild, decodes keystore from secrets, builds signed release APK.
+
+---
+
+## Complete API Reference
+
+### Raspberry Pi FastAPI (`raspi/main.py`) — port 8000
+
+| Method | Path | Params/Body | Description |
+|--------|------|-------------|-------------|
+| POST | `/run` | query: `task`, `token` | Runs whitelisted system commands (restart_nginx, uptime, update) |
+| POST | `/motor/step/forward` | query: `token` | Starts motor forward (daemon thread) |
+| POST | `/motor/step/backwards` | query: `token` | Starts motor backward (daemon thread) |
+| POST | `/motor/stop` | query: `token` | Disables motor driver (GPIO EN HIGH) |
+| POST | `/audio/upload` | multipart: `file`, query: `format` | Saves to `/tmp/quibot-audio/incoming/`, returns filename + lock_url |
+| GET | `/audio/incoming` | — | Lists files with size and modified time |
+| POST | `/audio/lock/{filename}` | — | incoming → locked (claim for processing) |
+| POST | `/audio/unlock/{filename}` | — | locked → incoming (release) |
+| POST | `/audio/cancel/{filename}` | — | locked → incoming (cancel) |
+| POST | `/audio/process/{filename}` | — | locked → processed |
+
+**Auth**: Query parameter `token` matching `QUIBOT_TOKEN` env var (default: `MY_SECRET_TOKEN`).
+
+### Backend Express (`backend/`) — port 3000
+
+| Method | Path | Body | Description |
+|--------|------|------|-------------|
+| GET | `/health` | — | Returns settings object |
+| POST | `/commands` | `{ task }` | Proxy to raspi `/run` |
+| POST | `/motor/step/forward` | — | Motor forward proxy |
+| POST | `/motor/step/backward` | — | Motor backward proxy (maps to raspi `/backwards`) |
+| POST | `/motor/stop` | — | Motor stop proxy |
+| POST | `/motor/upload` | multipart file | Audio upload via multer in-memory buffer |
+| GET | `/audio/incoming` | — | List incoming audio files |
+| POST | `/audio/lock/:filename` | — | Lock audio file |
+| POST | `/audio/unlock/:filename` | — | Unlock audio file |
+| POST | `/audio/cancel/:filename` | — | Cancel locked audio |
+| POST | `/audio/process/:filename` | — | Mark processed |
+| GET | `/settings` | — | Returns config |
+| PUT | `/settings` | `{ raspberryPi: { host, port }, token }` | Update runtime config |
+
+---
+
+## Command Flow Examples
+
+### Web → Motion Control
+```
+User clicks "Forward" in D-pad
+  → $fetch('/api/motor/step/forward', { method: 'POST' })
+    → Nuxt Nitro route: server/api/motor/step/[direction].post.ts
+      → $fetch(config.quibotBaseUrl + '/motor/step/forward', { query: { token } })
+        → raspi FastAPI /motor/step/forward
+          → motor_step("forward") in daemon thread
+            → step_motor(200, DIR, 1ms pulses)
+```
+
+### Block Processing (internal to Pi)
+```
+Child inserts colored block → quibot.py task_read_blocks() polls distance sensor
+  → When detected <80mm: read_block_color() via TCS34725
+    → Manhattan distance classification against color lookup table
+      → RED: eyes_turn_on(EYES_FW, DARK_RED, 2)
+         _execute_action(task_move_to, CROSSING)
+           → enable_wheels(ON) → follow_line_loop(speed) (proportional on TCRT5000)
+    → After action: servo_move_to(EJECT_POSITION)
+```
+
+### APK → Audio Upload
+```
+User stops recording in VoiceDrop → expo-av .m4a file
+  → POST {backendUrl} with FormData {fieldName: "file"} + Bearer auth
+    → raspi FastAPI saves to /tmp/quibot-audio/incoming/{uuid}.wav
+      → Returns: { status: "received", filename, lock_url }
+```
+
+### Gesture Mode Toggle
+```
+User toggles mode in web UI
+  → _execute_action() locks mutex
+    → If gesture mode: eyes_gesture_mode_on() (double cyan flash on 128-LED matrix)
+      → Eyes breathing thread at MAX_BR(170) brightness
+```
+
+---
+
+## CI/CD (`/.gitea/workflows/`)
+
+### `build.yml` — Web + Backend
+- Triggers: Push to `master`
+- Builds web: `npm install && npx nuxt build`, zips `.output/`
+- Builds backend: Zips entire `backend/` directory
+- Creates Gitea release "latest" with both zip artifacts
+
+### `build-apk.yml` — Mobile
+- Triggers: Push to `master`
+- expo prebuild → decode keystore from secrets → `./gradlew assembleRelease`
+- Creates Gitea release with APK artifact
+
+---
+
+## Testing
+
+All tests in `raspi/tests/` are **manual diagnostic scripts** (not automated frameworks). Each test is run independently by uncommenting the desired function call at the bottom of the file. Requirements:
+
+- `sudo pigpiod -s 1` daemon running
+- Python venv activated with hardware dependencies installed
+- Pi connected to robot hardware
+
+| Test | What it verifies |
+|------|-----------------|
+| `test_simple_motor.py` | Low-level motor driver via direct GPIO writes |
+| `test_motion.py` | Wheel steppers, arm/syringe homing, VL53L0X distance, ADS1115 line sensors |
+| `test_blocks.py` | Servo sweep (open/eject/open), color sensor readings, raw RGB calibration |
+| `test_gesture.py` | PAJ7620U2 I2C connection + 30-second gesture capture |
+| `test_eyes.py` | LED shape/color rendering, animation repeat/direction, gesture animations, breathing |
+
+---
+
+## Key Conventions
+
+1. **Token auth** is always a query parameter matching `QUIBOT_TOKEN` (default: `MY_SECRET_TOKEN`)
+2. **No database** — use filesystem for persistence, localStorage for web client state
+3. **Backend is a dumb proxy** — no business logic, just forwards HTTP requests with token passthrough
+4. **Motor commands are fire-and-forget** — motor runs in daemon thread until `/motor/stop`
+5. **Audio lifecycle**: incoming → locked (claim) → processed OR unlocked (release) / cancelled
+6. **Eyes breathing** runs continuously at MIN_BR(80)-MAX_BR(170) brightness in background
+7. **`quibot.py` owns block/gesture autonomy** — blocks are processed internally on the Pi without backend/web involvement
+8. **All paths use forward slashes** in URLs; kebab-case for params