346 lines
17 KiB
Markdown
346 lines
17 KiB
Markdown
# QuiBot Project — Agent Guide
|
|
|
|
## Overview
|
|
|
|
QuiBot is an educational robotics platform (UPC/UNE collaboration) consisting of a programmable robot with color-block recognition, gesture control, stepper motors, RGB LED eyes, and multiple input methods (web dashboard, Android voice app). The codebase comprises four independent application layers communicating via HTTP JSON APIs.
|
|
|
|
```
|
|
[quibot-web Nuxt SPA] ──HTTP──> [backend Express] ──HTTP──> [raspi FastAPI (Pi)]
|
|
│ │
|
|
▼ ▼
|
|
[apk Expo RN app] ──HTTP──> (same backend) [Python hardware drivers]
|
|
```
|
|
|
|
**Tech stack**: Python (pigpio, FastAPI) | TypeScript/Express | Nuxt 4/Vue 3 | Expo/React Native
|
|
**No database**: All state is in-memory, file-based (`/tmp/quibot-audio/`), or localStorage.
|
|
|
|
---
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
quibot/
|
|
├── raspi/ # Raspberry Pi brain — Python, controls hardware
|
|
│ ├── main.py # FastAPI server (port 8000) on the Pi
|
|
│ ├── quibot.py # Main program: block/gesture threads (like Arduino QuiBot.ino)
|
|
│ ├── motion.py # Stepper class, homing, line-following, high-level tasks
|
|
│ ├── gesture.py # PAJ7620U2 gesture sensor (I2C, polled at 50ms)
|
|
│ ├── blocks.py # TCS34725 color sensor + servo block ejection
|
|
│ ├── eyes.py # WS2811 LED matrix (128 LEDs, pigpio waveforms, breathing animation)
|
|
│ ├── pins.py # BCM GPIO pin map for all hardware
|
|
│ └── tests/ # Manual diagnostic scripts (not automated)
|
|
├── backend/ # Local Express server (port 3000), proxy to Pi
|
|
│ ├── src/
|
|
│ │ ├── index.ts # Express entry: CORS, JSON parser, /health
|
|
│ │ ├── config.ts # Env config: RASPBERRY_PI_HOST, PORT, QUIBOT_TOKEN
|
|
│ │ ├── routes/router.ts # Mounts all controllers
|
|
│ │ ├── services/raspi.service.ts # Axios proxy layer to Pi FastAPI
|
|
│ │ └── controllers/
|
|
│ │ ├── motor.controller.ts # Motor step/stop/upload
|
|
│ │ ├── audio.controller.ts # Audio file lifecycle (incoming/locked/processed)
|
|
│ │ ├── command.controller.ts # POST /commands proxy to raspi /run
|
|
│ │ └── settings.controller.ts # GET/PUT /settings runtime config
|
|
│ └── dist/ # Compiled output (generated)
|
|
├── quibot-web/ # Nuxt 4 dashboard SPA
|
|
│ ├── app/app.vue # Single-page control panel: block queue, D-pad, eye controls, gesture log
|
|
│ ├── server/api/ # Nitro server routes proxying to raspi
|
|
│ │ ├── motor/step/[direction].post.ts
|
|
│ │ └── motor/stop.post.ts
|
|
│ ├── nuxt.config.ts # Runtime config: QUIBOT_BASE_URL, QUIBOT_TOKEN
|
|
│ └── .output/ # Built Nitro output
|
|
├── apk/ # Expo React Native voice recorder ("VoiceDrop")
|
|
│ ├── app/index.tsx # Recording screen + upload
|
|
│ ├── app/settings.tsx # Backend URL/token configuration (AsyncStorage)
|
|
│ └── lib/recorder-settings.ts # AsyncStorage wrapper
|
|
├── .gitea/workflows/ # CI/CD pipelines
|
|
│ ├── build.yml # Web + backend → zip artifacts + Gitea release
|
|
│ └── build-apk.yml # Expo prebuild + signed APK → Gitea release
|
|
├── build.sh # Placeholder
|
|
└── README.md # Project overview (Catalan)
|
|
```
|
|
|
|
---
|
|
|
|
## Raspberry Pi Layer (`raspi/`)
|
|
|
|
**Hardware target**: Raspberry Pi Zero 2W controlling a robot with:
|
|
- 5 NEMA-style stepper motors (wheels x2, arms x2, syringe) via A4988/TB600 drivers in STEP/DIR mode
|
|
- VL53L0X ToF distance sensor (I2C bus 3)
|
|
- PAJ7620U2 gesture sensor (I2C bus 3, polled at 50ms)
|
|
- TCS34725 color sensor (bit-banged I2C on GPIO22/27)
|
|
- ADS1115 ADC for TCRT5000 line-following IR sensors
|
|
- WS2811 RGB LED matrix (2x 8x8 = 128 LEDs, GPIO26, pigpio waveforms at -s 1)
|
|
- Servo motor (GPIO10 PWM) for block ejection
|
|
- Hall-effect endstops on GPIOs 12, 16, 17
|
|
- Optional: I2S audio amp (MAX98357A) + mic (SPH0645)
|
|
|
|
### Key files
|
|
|
|
- **`pins.py`** — BCM GPIO pin numbering for every component (STEP, DIR, EN pins, I2C lines, endstops, LED_DATA on GPIO26)
|
|
- **`motion.py`** — `Stepper` class with AccelStepper-style acceleration profiling via `pigpio.gpio_trigger()`. 5 motor instances (`wheel_R`, `wheel_L`, `arm_R`, `arm_L`, `syringe`). Continuous stepper daemon thread (`_stepper_loop`) at ~100Hz. Homing routines read Hall-effect endstops. Line-following with proportional correction on TCRT5000 values via ADS1115.
|
|
- **`gesture.py`** — Raw I2C via `smbus2` (bus 3). Two-register-bank init (~240 total register writes). Polls gesture result registers 0x43/0x44 every 50ms. Returns: GS_NONE, GS_FORWARD, GS_LEFT, GS_RIGHT, GS_UP, GS_DOWN, GS_CLOCKWISE, GS_ANTICLOCKWISE, GS_WAVE
|
|
- **`blocks.py`** — TCS34725 RGB reads classified via Manhattan distance against calibrated reference table (BK/RD/GN/BU/YE/OG/VT). Smooth servo movement with 3us micro-steps.
|
|
- **`eyes.py`** — 128 WS2811 LEDs via pigpio waveforms at 1us resolution. Pre-defined shapes: EYES_OPEN, EYES_FW, EYES_DOWN, EYES_GESTURE. Breathing thread oscillates brightness 80-170 at 50ms intervals.
|
|
- **`quibot.py`** — Main program (equivalent to original Arduino QuiBot.ino). Two threading tasks via FreeRTOS-style pattern: `task_read_blocks()` (color→action mapping) and `task_read_gestures()` (WAVE toggles between block/gesture mode). `threading.Lock` prevents concurrent motor movements. Handles SIGINT/SIGTERM for graceful shutdown.
|
|
- **`main.py`** — FastAPI server on port 8000 with CORS. File-based state management using `/tmp/quibot-audio/` (incoming/locked/processed directories). Token auth via query parameter matching `QUIBOT_TOKEN`.
|
|
|
|
### Color→Action Mapping (in quibot.py)
|
|
|
|
| Color | Action |
|
|
|-------|--------|
|
|
| RED | Advance forward |
|
|
| GREEN | Turn right |
|
|
| BLUE | Turn left |
|
|
| YELLOW | Take/block pick-up |
|
|
| ORANGE | Leave/eject |
|
|
| VIOLET | Idle |
|
|
| BLACK | Reference / no block |
|
|
|
|
### Motor Position Tracking
|
|
|
|
`Stepper` class in `motion.py` tracks absolute position in steps via `_pos` and `_target`. Endstops provide physical reference during homing. The stepper loop evaluates acceleration profiles to generate STEP pulses at correct intervals.
|
|
|
|
---
|
|
|
|
## Backend Layer (`backend/`)
|
|
|
|
**Role**: Express.js HTTP proxy sitting between frontend/mobile and the Raspberry Pi's FastAPI server. Token passthrough, no business logic.
|
|
|
|
### Configuration (`.env`, loaded by `config.ts`)
|
|
|
|
| Variable | Default | Purpose |
|
|
|----------|---------|---------|
|
|
| `RASPBERRY_PI_HOST` | `http://raspberrypi.local` | Pi API URL |
|
|
| `RASPBERRY_PI_PORT` | `8000` | Pi API port |
|
|
| `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token for all Pi endpoints |
|
|
| `PORT` | `3000` | Backend listen port |
|
|
|
|
### Architecture
|
|
|
|
```
|
|
index.ts → Express app, CORS, JSON parser, /health endpoint
|
|
routes/router.ts → Mounts all controllers under /motor, /audio, /commands, /settings
|
|
config.ts → Mutable getter/setter env vars (runtime update via PUT /settings)
|
|
raspi.service.ts → Axios proxy methods for every Pi endpoint + multipart file upload handling
|
|
```
|
|
|
|
### Controllers
|
|
|
|
- **`motor.controller.ts`** — `POST /motor/step/forward`, `/motor/step/backward`, `/motor/stop`. Also `POST /motor/upload` (multer multipart → proxied as FormData to Pi).
|
|
- **`audio.controller.ts`** — `GET /audio/incoming`, `POST /audio/lock/:filename`, `/unlock/:filename`, `/cancel/:filename`, `/process/:filename`. All proxy to raspi audio file lifecycle endpoints.
|
|
- **`command.controller.ts`** — `POST /commands { task }` → proxied to raspi `/run?task=...&token=...`
|
|
- **`settings.controller.ts`** — `GET /settings` returns config; `PUT /settings` updates `raspberryPi.host`, `raspberryPi.port`, `token` at runtime.
|
|
|
|
### Build/Run
|
|
|
|
```bash
|
|
cd backend
|
|
npm install # Installs express, axios, multer, dotenv, cors, typescript
|
|
npx tsc # Compiles to dist/
|
|
node dist/index.js # Or use tsx/nodemon for dev
|
|
```
|
|
|
|
---
|
|
|
|
## Web Frontend (`quibot-web/`)
|
|
|
|
**Stack**: Nuxt 4 (Nitro) + Vue 3. Single-page dashboard SPA.
|
|
|
|
### Runtime config (`nuxt.config.ts`)
|
|
|
|
| Key | Default | Purpose |
|
|
|-----|---------|---------|
|
|
| `QUIBOT_BASE_URL` | `http://quibot:8000` | Base URL for raspi FastAPI |
|
|
| `QUIBOT_TOKEN` | `MY_SECRET_TOKEN` | Auth token |
|
|
|
|
### UI Panels (`app/app.vue` — single-file SPA, 1369 lines)
|
|
|
|
1. **Block Queue Panel** — Displays color blocks in a queue (localStorage persistence + demo fallback). Shows action descriptions per color.
|
|
2. **Motion Controls** — D-pad grid: up=forward, down=back, left/right=turns, center=stop. Sends `$fetch('/api/motor/step/forward')` etc.
|
|
3. **Eye Controls** — Shape selector (open/forward/down/gesture), 8-color picker. Calls `POST /api/eye/shape`, `/api/eye/color`, `/api/eye/on`, `/api/eye/off`.
|
|
4. **Gesture Sensor Panel** — Toggle between Block Mode / Gesture Mode. Gesture detection history log. Reference table of all 8 gestures.
|
|
|
|
### State & Styling
|
|
- Dark/light theme via CSS custom properties, persisted in localStorage.
|
|
- Block queue data stored in localStorage with demo fallback.
|
|
- Toast notifications for success/error feedback.
|
|
- Responsive layout with CSS Grid (mobile-adaptive).
|
|
|
|
### Server Routes (`server/api/`)
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| POST | `/api/motor/stop` | Proxies to raspi `/motor/stop` |
|
|
| POST | `/api/motor/step/:direction` | Proxies to raspi `/motor/step/forward\|backwards` |
|
|
|
|
**Note**: The frontend also calls `POST /api/eye/shape`, `/api/eye/color`, `/api/eye/on`, `/api/eye/off`, `/api/gesture/on`, `/api/gesture/off` — server routes for these may need to be created (frontend references them but they don't have explicit server handlers yet).
|
|
|
|
### Build/Run
|
|
|
|
```bash
|
|
cd quibot-web
|
|
npm install
|
|
npx nuxt build # Produces .output/
|
|
# Dev mode:
|
|
npx nuxt dev
|
|
```
|
|
|
|
CI builds the Nuxt output and zips it for distribution.
|
|
|
|
---
|
|
|
|
## Android APK (`apk/`)
|
|
|
|
**Stack**: Expo SDK ~54 + React Native 0.81.5 + Expo Router ~6.0. App name: "VoiceDrop".
|
|
|
|
### Features
|
|
- Records audio via `expo-av` (`Audio.Recording` — produces .m4a)
|
|
- Auto-uploads on stop as multipart form data to configurable backend URL
|
|
- Displays recording timer, status messages, raw server response (first 400 chars)
|
|
|
|
### Screens
|
|
- **`app/index.tsx`** — Recorder screen: start/stop recording, upload, status
|
|
- **`app/settings.tsx`** — Backend URL, Bearer token, form field name. Saved to AsyncStorage under `recorder.*` namespace.
|
|
|
|
### Persistence
|
|
Settings stored in `@react-native-async-storage/async-storage` (keys: `recorder.backendUrl`, `recorder.authToken`, `recorder.fieldName`).
|
|
|
|
### Build/Run
|
|
|
|
```bash
|
|
cd apk
|
|
npm install
|
|
npx expo start # Dev mode
|
|
# Production APK (local):
|
|
./build.sh # expo prebuild → generates android/ → gradle assembleRelease
|
|
```
|
|
|
|
CI: `build-apk.yml` runs expo prebuild, decodes keystore from secrets, builds signed release APK.
|
|
|
|
---
|
|
|
|
## Complete API Reference
|
|
|
|
### Raspberry Pi FastAPI (`raspi/main.py`) — port 8000
|
|
|
|
| Method | Path | Params/Body | Description |
|
|
|--------|------|-------------|-------------|
|
|
| POST | `/run` | query: `task`, `token` | Runs whitelisted system commands (restart_nginx, uptime, update) |
|
|
| POST | `/motor/step/forward` | query: `token` | Starts motor forward (daemon thread) |
|
|
| POST | `/motor/step/backwards` | query: `token` | Starts motor backward (daemon thread) |
|
|
| POST | `/motor/stop` | query: `token` | Disables motor driver (GPIO EN HIGH) |
|
|
| POST | `/audio/upload` | multipart: `file`, query: `format` | Saves to `/tmp/quibot-audio/incoming/`, returns filename + lock_url |
|
|
| GET | `/audio/incoming` | — | Lists files with size and modified time |
|
|
| POST | `/audio/lock/{filename}` | — | incoming → locked (claim for processing) |
|
|
| POST | `/audio/unlock/{filename}` | — | locked → incoming (release) |
|
|
| POST | `/audio/cancel/{filename}` | — | locked → incoming (cancel) |
|
|
| POST | `/audio/process/{filename}` | — | locked → processed |
|
|
|
|
**Auth**: Query parameter `token` matching `QUIBOT_TOKEN` env var (default: `MY_SECRET_TOKEN`).
|
|
|
|
### Backend Express (`backend/`) — port 3000
|
|
|
|
| Method | Path | Body | Description |
|
|
|--------|------|------|-------------|
|
|
| GET | `/health` | — | Returns settings object |
|
|
| POST | `/commands` | `{ task }` | Proxy to raspi `/run` |
|
|
| POST | `/motor/step/forward` | — | Motor forward proxy |
|
|
| POST | `/motor/step/backward` | — | Motor backward proxy (maps to raspi `/backwards`) |
|
|
| POST | `/motor/stop` | — | Motor stop proxy |
|
|
| POST | `/motor/upload` | multipart file | Audio upload via multer in-memory buffer |
|
|
| GET | `/audio/incoming` | — | List incoming audio files |
|
|
| POST | `/audio/lock/:filename` | — | Lock audio file |
|
|
| POST | `/audio/unlock/:filename` | — | Unlock audio file |
|
|
| POST | `/audio/cancel/:filename` | — | Cancel locked audio |
|
|
| POST | `/audio/process/:filename` | — | Mark processed |
|
|
| GET | `/settings` | — | Returns config |
|
|
| PUT | `/settings` | `{ raspberryPi: { host, port }, token }` | Update runtime config |
|
|
|
|
---
|
|
|
|
## Command Flow Examples
|
|
|
|
### Web → Motion Control
|
|
```
|
|
User clicks "Forward" in D-pad
|
|
→ $fetch('/api/motor/step/forward', { method: 'POST' })
|
|
→ Nuxt Nitro route: server/api/motor/step/[direction].post.ts
|
|
→ $fetch(config.quibotBaseUrl + '/motor/step/forward', { query: { token } })
|
|
→ raspi FastAPI /motor/step/forward
|
|
→ motor_step("forward") in daemon thread
|
|
→ step_motor(200, DIR, 1ms pulses)
|
|
```
|
|
|
|
### Block Processing (internal to Pi)
|
|
```
|
|
Child inserts colored block → quibot.py task_read_blocks() polls distance sensor
|
|
→ When detected <80mm: read_block_color() via TCS34725
|
|
→ Manhattan distance classification against color lookup table
|
|
→ RED: eyes_turn_on(EYES_FW, DARK_RED, 2)
|
|
_execute_action(task_move_to, CROSSING)
|
|
→ enable_wheels(ON) → follow_line_loop(speed) (proportional on TCRT5000)
|
|
→ After action: servo_move_to(EJECT_POSITION)
|
|
```
|
|
|
|
### APK → Audio Upload
|
|
```
|
|
User stops recording in VoiceDrop → expo-av .m4a file
|
|
→ POST {backendUrl} with FormData {fieldName: "file"} + Bearer auth
|
|
→ raspi FastAPI saves to /tmp/quibot-audio/incoming/{uuid}.wav
|
|
→ Returns: { status: "received", filename, lock_url }
|
|
```
|
|
|
|
### Gesture Mode Toggle
|
|
```
|
|
User toggles mode in web UI
|
|
→ _execute_action() locks mutex
|
|
→ If gesture mode: eyes_gesture_mode_on() (double cyan flash on 128-LED matrix)
|
|
→ Eyes breathing thread at MAX_BR(170) brightness
|
|
```
|
|
|
|
---
|
|
|
|
## CI/CD (`/.gitea/workflows/`)
|
|
|
|
### `build.yml` — Web + Backend
|
|
- Triggers: Push to `master`
|
|
- Builds web: `npm install && npx nuxt build`, zips `.output/`
|
|
- Builds backend: Zips entire `backend/` directory
|
|
- Creates Gitea release "latest" with both zip artifacts
|
|
|
|
### `build-apk.yml` — Mobile
|
|
- Triggers: Push to `master`
|
|
- expo prebuild → decode keystore from secrets → `./gradlew assembleRelease`
|
|
- Creates Gitea release with APK artifact
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
All tests in `raspi/tests/` are **manual diagnostic scripts** (not automated frameworks). Each test is run independently by uncommenting the desired function call at the bottom of the file. Requirements:
|
|
|
|
- `sudo pigpiod -s 1` daemon running
|
|
- Python venv activated with hardware dependencies installed
|
|
- Pi connected to robot hardware
|
|
|
|
| Test | What it verifies |
|
|
|------|-----------------|
|
|
| `test_simple_motor.py` | Low-level motor driver via direct GPIO writes |
|
|
| `test_motion.py` | Wheel steppers, arm/syringe homing, VL53L0X distance, ADS1115 line sensors |
|
|
| `test_blocks.py` | Servo sweep (open/eject/open), color sensor readings, raw RGB calibration |
|
|
| `test_gesture.py` | PAJ7620U2 I2C connection + 30-second gesture capture |
|
|
| `test_eyes.py` | LED shape/color rendering, animation repeat/direction, gesture animations, breathing |
|
|
|
|
---
|
|
|
|
## Key Conventions
|
|
|
|
1. **Token auth** is always a query parameter matching `QUIBOT_TOKEN` (default: `MY_SECRET_TOKEN`)
|
|
2. **No database** — use filesystem for persistence, localStorage for web client state
|
|
3. **Backend is a dumb proxy** — no business logic, just forwards HTTP requests with token passthrough
|
|
4. **Motor commands are fire-and-forget** — motor runs in daemon thread until `/motor/stop`
|
|
5. **Audio lifecycle**: incoming → locked (claim) → processed OR unlocked (release) / cancelled
|
|
6. **Eyes breathing** runs continuously at MIN_BR(80)-MAX_BR(170) brightness in background
|
|
7. **`quibot.py` owns block/gesture autonomy** — blocks are processed internally on the Pi without backend/web involvement
|
|
8. **All paths use forward slashes** in URLs; kebab-case for params
|