A real-time object detection and tracking pipeline that streams normalized coordinates to a robotic arm via WebSocket. YOLOE-11 segmentation models are fine-tuned on custom classes using visual or text prompts, then exported to ONNX for low-latency edge inference on Raspberry Pi or standard USB cameras.
The robot receives continuous JSON payloads — pixel coordinates, normalized (0–1) position, and confidence — which feed directly into an IK solver for end-effector targeting. Demonstrated on coin detection (Mexican pesos, dining tokens, USD) and building/logo recognition.
Demo
Detection Examples
Technical Implementation
Pipeline
- Model: YOLOE-11L-Seg fine-tuned via visual prompts (drawn bounding boxes) or text prompts; exported to ONNX at 640 px for cross-platform deployment
- Inference: OpenCV + Ultralytics ONNX runtime; confidence threshold at 0.2, FPS measured per frame via
results[0].speed['inference'] - Coordinate output: Center point as both pixel coords and normalized (0–1) values, streamed as JSON over WebSocket (port 8765) at ~10 Hz
- IK integration: Normalized coordinates feed a continuous IK control loop for end-effector positioning without hardcoded pixel-to-joint mappings
- Edge deployment: Picamera2 integration enables onboard Raspberry Pi inference; ONNX runtime runs ~2–4× faster than native PyTorch on ARM
- Web dashboard: WebSocket-connected HTML interface for real-time monitoring of detections and robot state