作品Projects / F-01
AI-collaborative Physical AI Lab
從語言到工具，從工具到安全的動作。這是一個實際運作中的實驗台：用 restricted MCP tools、在模型之外執行的 fail-closed 安全 gate 和三相機 vision，把 AI agent 接到 SO-ARM101 leader／follower 機械手臂。這隻 follower 手臂有個暱稱，叫 Crab。 From language to tools, from tools to safe motion. A working lab that connects an AI agent to SO-ARM101 leader/follower arms through restricted MCP tools, fail-closed safety gates enforced outside the model, and a three-camera vision pipeline. The follower arm goes by the nickname Crab.
- Robot
- SO-ARM101 leader + follower
- Nickname
- 「Crab」：follower 手臂的暱稱"Crab", the follower arm's nickname
- Control
- LeRobot 0.3.4 + MCP
- Agent
- Hermes Agent · 3 tools
- Vision
- 三相機 · ArUco／ChArUco · Rerun3 cameras · ArUco / ChArUco · Rerun
- Policy
- 人工把關（human-gated）Human-gated
- As of
- 2026-08-25
數字來自專案 status、測試輸出、serial postflight 檢查與擷取影格；都是實測紀錄，不是目標值。Numbers come from project status, test output, serial postflight checks and captured frames. They are records, not targets.
問題：讓 agent 動手，又不讓它碰馬達Problem: let an agent act without touching the motors
LLM agent 越來越常被賦予工具權限；一旦工具接到實體機器，「模型說做完了」和「手臂安全地做完了」就是兩回事。這個專案要回答的是：怎麼讓語言指令觸發真實動作，同時把每一個動作都限制在可驗證、可停止、有人把關的範圍內。LLM agents are increasingly given tools. Once a tool reaches a physical machine, "the model says it's done" and "the arm finished safely" are different claims. This project asks how a language request can trigger real motion while every motion stays verifiable, stoppable and human-gated.
長期目標是建立可重現的 teleoperate → record → train (ACT) → eval 具身 AI 資料閉環：先在桌上練一個刻意很窄的任務（把一顆紅色泡棉方塊放進固定容器），再延伸到工業 tray 搬運 PoC。The longer goal is a reproducible teleoperate → record → train (ACT) → eval embodied-AI data loop: first a deliberately narrow desktop task (place one red foam cube into a fixed container), then an industrial tray-handling PoC.
原則是每項能力都要一路往上掙得更強的證據：已實作 → 測試通過 → 硬體上觀察到 → 授權動作。沒有第一條 evidence loop 之前，不擴大模型、感測器或 UI 的範圍。The rule: every capability has to earn stronger evidence step by step, from implemented to test passed to observed on hardware to motion authorized. No model, sensor or UI scope grows before the first evidence loop exists.
系統架構：AI 提案，護欄授權，硬體回報Architecture: AI proposes, guardrails authorize, hardware reports
模型只能選一個具名工具；所有安全判斷都在模型之外執行。The model can only pick one named tool. Every safety decision runs outside the model.
statuswavestop呼叫端不能送 raw joints、角度或速度Callers cannot submit raw joints, angles or speed
多相機 visionMulti-camera vision
圖：wave 路徑已在實機跑通；vision 目前只做觀察。Figure: the wave path runs on hardware; vision is observation-only today.
已驗證 vs 待完成Verified vs pending
ACT 訓練與自主 rollout 是計畫中的工作，這一頁刻意不把它們寫成已完成的能力。ACT training and autonomous rollout are planned work. This page deliberately does not present them as completed capabilities.
硬體上已驗證Verified on hardware
- WP01組裝與校正Assembly & calibrationleader／follower 組裝、供電、校正；六軸讀回、伺服身分與唯讀健康檢查通過Leader/follower assembled, powered, calibrated; six-axis readback, servo identity and read-only health checks pass
- WP01有界 teleoperationBounded teleoperation10 分鐘無負載、60 Hz 控制迴圈；遙操作抓取圓形、細長、容器三類物體10-minute no-load run at 60 Hz; teleoperated grasps of round, slender and container-shaped objects
- WP01有界 leader→follower teach-inBounded leader→follower teach-in60 秒、30 Hz、有 relative cap、Rerun telemetry 開啟，結束時關 torque。證明的是有人監看的 teach-in 迴圈可運作，不是自主抓取60 s at 30 Hz with a relative cap and Rerun telemetry, torque shut down at the end. Proves a supervised teach-in loop works, not autonomous grasping
- WP02MCP waveMCP wave單次 live wave、live stop、連續 5/5 wave；每次 postflight 都是 6/6 伺服、torque 0、port 關閉，並用相機複核；Hermes 發現 3 個 toolsSingle live wave, live stop, 5/5 consecutive waves; every postflight showed 6/6 servos, torque 0, port closed, with camera reconfirmation. Hermes discovered all 3 tools
進行中In progress
- WP01多相機校正與 transformMulti-camera calibration & transforms三相機穩定身分、preliminary ChArUco 校正已有；外部相機只核准 observation-only proxy；第一次 hand-eye 求解因殘差過大被退回Stable identity for three cameras and preliminary ChArUco calibration exist; the external camera is approved only as an observation-only proxy; the first hand-eye solve was rejected for high residual
- WP02手機語音端到端Mobile voice end-to-end手機語音到 agent 的完整驗收尚未完成Phone-voice-to-agent acceptance not yet complete
- WP04正式 LeRobotDataset pilotOfficial LeRobotDataset pilot影格、關節狀態、action 時間戳與安全 preflight 對齊前不開始Waits until frames, joint state, action timestamps and safety preflight align
計畫中（未開始）Planned (not started)
- WP04ACT 模仿學習ACT imitation learning先 smoke test 再存 checkpointSmoke test, then checkpoint
- WP04安全包覆的自主 pick-and-placeSafety-wrapped autonomous pick-and-placepolicy 只在安全範圍內執行；評估 10 次，每次失敗都要記原因Policy runs inside the safety envelope; 10-run evaluation with a reason for every failure
- WP03物品遞交安全 demoObject hand-off safety demo
- WP05工業 tray 搬運 PoCIndustrial tray-handling PoC
安全設計Safety design
安全不是事後補上的，而是動作能不能發生的前提條件。Safety is not bolted on afterwards. It is the precondition for any motion.
- S1模型拿不到馬達No motor access for the model
只暴露 status／wave／stop 三個具名工具；不暴露 raw joints、角度、速度或底層 send_action。Only three named tools are exposed. No raw joints, angles, speed or low-level send_action.
- S2固定低速 trajectoryFixed low-speed trajectory
wave 是預先定義、有邊界的動作，約 1.9 秒完成。Wave is a predefined, bounded motion that completes in about 1.9 seconds.
- S3Fail-closed gate 在模型之外Fail-closed gate outside the model
mode、readiness、calibration margin、timeout、run state 任何一項不過就不動；有 run_id 與 busy lock。If any of mode, readiness, calibration margin, timeout or run state fails, nothing moves. Runs carry a run_id and a busy lock.
- S4完成要有 postflight 證據Completion needs postflight evidence
程式在跑不等於動作完成：要有終端結果，加上伺服、torque、port 檢查和相機複核。A running process is not a completed action. Completion needs a terminal result plus servo, torque and port checks and a camera reconfirmation.
- S5Vision 只觀察Vision is observation-only
transform、可見度、誤差 gate 都通過之前，
safe_to_continue=false，vision 不接動作。Until transform, visibility and error gates pass,safe_to_continue=falseand vision is not wired to motion. - S6硬體單一 ownerSingle hardware owner
serial port 只由一個 loop 擁有；dashboard 只收 telemetry，不另開讀取。One loop owns the serial port; the dashboard only receives telemetry and never opens a second reader.
- S7不直接用 stock 流程控制安全No stock flows as safety control
stock 的 connect()／configure() 會寫入馬達設定，所以不拿來當安全控制路徑。Stock connect() / configure() write motor settings, so they are not used as the safety-control path.
- S8影像優先的位置確認Camera-first position checks
手臂位置一律先用相機確認；文字回報與伺服數值只能當補充。Arm position is always confirmed by camera first; text reports and servo values are supplementary.
手機語音不能把手臂切到 live mode、不能覆寫 readiness，也不能傳送 raw motor 指令。實體授權永遠留在現場。Mobile voice cannot switch the robot into live mode, override readiness or transmit raw motor commands. Physical authorization stays local.
影像證據Visual evidence
相機刻意配置成冗餘：每個視角各自回答一個問題，例如工作區脈絡、夾爪可見度、校正板覆蓋或安全觀察。The camera set is intentionally redundant. Each view answers a different question: workspace context, gripper visibility, board coverage or safety observation.
預計 30–90 秒：MCP status → wave → stop → postflight，以及 leader→follower teach-in。Planned 30–90 s: MCP status → wave → stop → postflight, plus leader→follower teach-in.
// 硬體視角Hardware views
// 標註影格Annotated frames
教訓與失敗分析Lessons & failure analysis
每一條都包含失敗機制、解法和未來的觸發條件。「曾經成功」不寫成「永久保證」。Each lesson records the failure mechanism, the fix and when it applies again. "Worked once" is never written as "guaranteed".
程式在跑，不等於動作完成A running process is not a completed action
完成的定義是終端結果加上 postflight 的伺服、torque、port 檢查。Completion means a terminal result plus postflight servo, torque and port checks.
相機身分要撐得過重新列舉Camera identity must survive re-enumeration
數字 index 會在 USB 重插後悄悄指到錯的鏡頭；改用 PnP 身分加上每支相機各自的 capture profile，重插後要用實際影格驗證。Numeric indexes silently point to the wrong lens after a USB reconnect. Switched to PnP identity with per-camera capture profiles, verified with a real frame after every reconnect.
影像品質本身就是校正的一部分Image quality is part of calibration
日光燈造成的 banding 會讓角點偵測不穩；先處理光源，再調相機。fisheye model 不能全域套用，要逐支相機比較。Fluorescent-light banding destabilised corner detection: fix the light source before camera settings. A fisheye model cannot be applied globally; compare per camera.
Agent 延遲和機器人延遲是兩回事Agent latency and robot latency differ
固定動作約 1.9 秒；拖慢語音流程的是一次次的模型往返。The fixed motion took about 1.9 s; repeated model round trips dominated the voice workflow.
透明膠帶會讓 marker 失效Clear tape defeats markers
斜視角下的反光破壞黑白邊界；改用霧面材質後同一影格能同時讀到三個 marker。因為角度也一起改了，這只是強烈支持的推論，不是嚴格的 A/B 實驗。Glare at oblique angles broke black/white edges. With a matte surface, one frame read all three markers. The angle also changed, so this is strongly supported but not a controlled A/B test.
無界錄製會吃光磁碟Unbounded recording fills the disk
raw 影像持續寫入 Rerun，一次測試累積約 14.2 GiB；改成有界執行加 JPEG 壓縮。Rerun 是觀測工具，不是安全控制器。Raw frames streamed into Rerun reached about 14.2 GiB in one test. Switched to bounded runs with JPEG compression. Rerun is observability, not a safety controller.
Dashboard 會和 teleoperation 搶 portThe dashboard contends with teleoperation
兩個程序同時開 follower 的 serial port 會造成 telemetry 停滯；teach-in 期間只讓單一 loop 擁有硬體。Two processes opening the follower port stalled telemetry. During teach-in a single loop owns the hardware.
第一次 hand-eye 求解被退回First hand-eye solve rejected
8 個樣本的離線求解，最大殘差約 160 mm／53°，標記為 REJECTED。停止繼續加樣本，先排查關節角度換算、相機固定方式與校正板穩定性；transform 未授權，不接動作。An 8-sample offline solve left a maximum residual of about 160 mm / 53° and was marked REJECTED. Stopped adding poses to investigate joint-angle conventions, camera mount rigidity and board stability. Transform not authorized; no motion.
AI 協作方式與連結AI collaboration & links
AI 負責加速閱讀原始碼、草擬測試、診斷 log、更新文件，並質疑「完成」的宣稱。曾讓三個獨立的唯讀 AI 審查（vision、安全 gate、LeRobot 資料）分別評估下一步，三份結論一致。實體安全與硬體驗收始終由現場操作者負責。AI accelerated source reading, test drafting, log diagnosis and documentation, and challenged completion claims. Three independent read-only AI reviews (vision, safety gate, LeRobot data) assessed the next step and converged. Physical safety and hardware acceptance stayed with the on-site operator.
工作流：人定義結果與風險 → AI 讀現況 → 修改、測試、加觀測 → 執行軟體或有界硬體動作 → 人確認實體結果 → 寫入 status 與 lessons 作為交接。Loop: human frames outcome and risk → AI inspects current state → patch, test, instrument → run software or bounded hardware → human accepts the physical result → status and lessons become the handoff.