1 arm, 3 cameras, MCP tool backend
The agent acts only through tools that control the arm, enforce a workspace, and does perception using all 3 cameras.
The workcell. Left: the xArm 6 with the tray, brick and a fixed camera. Right: the G2 gripper and wrist RGB-D camera.
The agent's 3 views: 2 fixed side cameras and the wrist camera.
| Robot | UFactory xArm 6, 6-DoF |
|---|---|
| Gripper | UFactory G2 parallel gripper, 84 mm maximum opening |
| Cameras | 3 Intel RealSense RGB-D: 2 fixed (left, right) and 1 on the wrist, 640 x 480 |
| Workspace | TCP box checked on every target and along joint paths |
| Agent | GPT-6 Astra (frozen weights) in the RAX harness |
|---|---|
| Tools | xarm_observe, xarm_move_pose, xarm_move_joints, xarm_gripper, xarm_hold, xarm_pixel_to_position, xarm_status, request_operator |
| Budget | 4 hours and 4,000 model calls per session |
| Operator | Resets scenes on request, answers questions in plain text, holds the e-stop |
Protocol
Solve it manually
The agent drives the arm one tool call at a time until the task succeeds once.
Write and repair a controller
The agent distills what worked into reusable code skills, then runs and patches them on scenes the operator resets.
Run the executable
The agent synthesizes an executable program from its skill library and runs it on fresh scenes. No model is queried during deployment.
Porting RAX from Franka to xArm
RAX was built for a Franka FR3. We ported the RAX backend to xArm 6 with the same 5-method interface; the frontend robot service, audit log, operator channel and agent workflow are unchanged. One challenge we faced during the port: the G2 gripper driver always reports is_grasped: false, so grasps can only be confirmed visually, which the agent figured out during training.
The agent and its controller reach the robot through the same socket and checks. Highlighted parts are new for the xArm.
| Franka FR3 | xArm 6 | |
|---|---|---|
| Arm connection | HTTP bridges to Polymetis | Direct xArm Python SDK |
| Cartesian motion | Repeated targets with a host-side integral correction loop | 1 SDK move, then polling the measured pose until it settles |
| Gripper | Separate bridge with grasp feedback | SDK position commands, standard and G2 |
| Cameras | ZED | Intel RealSense |
Pick and place
The full training session, recordings on robot motion only. Manual episodes at 8× speed, trials at 12× and 8× speed, trial 15 at 4× speed. The clock shows live session time.
How the skills evolved
M1
A manual episode with an upright brick
The agent fits the brick top in wrist depth and descends in stages. Below about 15 cm the brick leaves the wrist camera's depth range; it backs up, refits and finishes in 22 motions.
T1, M2
The operator lays the brick on its side
The first controller finds no upright brick and asks the operator to stand it up, to which the operator responded: “I intentionally laid the brick sideways to test generalization.” The agent measures the 31 x 61 mm footprint by hand and grasps across the narrow side.
T3 to T7
Sharpen perception
Runs stop because the left and wrist cameras disagree on the brick: left depth misses the blue cap. Then the wrist sees only 40% valid depth on the top. Rather than loosen thresholds, the agent adds bounded re-captures, an inspection pose and an RGB fallback where depth is missing.
T8 to T12
Occlusion near contact, and an operator who pushes back
Near contact the fingers hide the brick in the side views and stability checks halt the run. The agent keeps its geometric gates and relaxes only the redundant visual ones.
T13 to T15
Retention from motion, then the first success
With no usable grasp flag, the agent verifies a grasp by checking that the brick rises the expected distance in both side cameras. Trial 15 is the first full autonomous success, in 65 s.
T16 to T19
The tray becomes an obstacle
With the brick beside the tray, a finger lands on the tray wall. The agent writes a clearance model that tests each finger's swept footprint against the tray outline, tries both grasp axes, and refuses when neither clears.
Training paused
Training ends after 19 trials with a 711-line controller and 18 regression tests.
Every training trial
Deployment
The agent synthesized a controller that ran on 18 fresh scenes with no model calls: 6 unassisted successes, mean episode .
None of the 12 failures was a wrong approach. All 9 grasps landed at the planned pose and closed on the brick. Every failure was one of the controller's own checks stopping the run:
- Stopped after a good grasp (3): the fingers hid the brick in 1 view, or the placed brick rested against the tray wall.
- Perception checks before contact (5): a missing silhouette, a split color patch, noisy tray depth.
- Correct refusals (2): no finger sweep cleared the tray wall.
- Robot fault before the start (2).
Approach accuracy
Every deployment grasp, including those in runs stopped later.
The gripper closed at 30 mm every time, on the brick's narrow side; the long side would read about 61 mm and air below 25 mm. Sideways bricks were grasped within 1.3° of the long axis.
All 6 successful deployment runs, 4× speed, on real time. Click a row to play any run.
Where the time goes
Stage durations of the 6 successful runs.
LEGO stacking
The full training session, robot motion only. Manual work at 8× speed, trials 2 to 8 at 12× speed, trial 9 at 4× speed.
From aligned to seated
The agent thought the first manual stack was fully seated but the operator found it only partly seated. With no force sensing and no view of the seam, the agent seated it through presses guided by operator feedback, then compiled them into its learned skills.
seat(): 3 bounded presses, at the center and at 2 offsets of 6 mm, never below 116 mm, then a height gate and a lift test.The manual seating sequence above, 8× speed.
Other things it learned along the way
- Studs, not edges, set the yaw: 4 Hough circles fit to a 16 mm square, instead of unstable depth edges.
- Wrist calibration drifts with yaw: rotate at clear height, then re-localize before targeting.
- Distrust bad depth: the right camera's depth was off by about 0.2 m, so it is used for RGB checks only.
- Use the unoccluded view: checks accept whichever fixed camera can still see the base.
Every training trial
Evaluation after training
The controller was frozen after training. No evaluation run failed because of a bad motion: every grasp and placement was on target.
RAX vs. Astra as the policy
The baseline is the same model, GPT-6 Astra, acting as the policy: it chooses every tool call from the latest observation. RAX uses the model only in training and deploys the synthesized Python program it wrote.
AXIS 1Execution speed on LEGO stacking
RAX controller
Astra as the policy
RAX: full trial 9, from first motion to the verified lift test. Astra: manual episode 1, which ended with the blocks aligned but only partly seated.
AXIS 2Behaviors the controller carries
Each behavior was learned from a training failure and now runs as code in every episode. Clips are from the controller's own runs.
SUMMARY
AXIS 3Success rate influenced by learned skills
On LEGO stacking, each success criterion depends on a learned skill. Alignment comes from reading yaw from the studs, which both policies learned, so both place the block aligned. Seating comes from the 3 bounded presses, which only RAX learned: Astra's lift test passed on a partly seated stack, so it could not tell the difference.
Astra as the policy
Aligned, but the top block is not fully seated.
RAX controller
3 bounded presses, 2× speed. Operator-confirmed fully seated.
RAX: all 5 placements across training and evaluation; fully seated is out of the 3 the operator inspected, since the 2 evaluation placements were not inspected. Astra: 5 attempts as the policy. End-to-end unattended success was 1 of 9 training trials and 0 of 5 evaluations.
RAX
This project is a fork of RAX from Scaling Online Exploration for Robot Agents. The method, the agent harness and the robot service are the original authors' work. We add a branch containing xArm 6 port, Wuji Hand 2 port, and our evals.
BIBTEX
@article{huang2026scaling,
title={Scaling Online Exploration for Robot Agents},
author={Huang, Runhan and Zhang, Xiangcheng and Chen, Haoyu and Li, Yitong and Wang, Yen-Jen and Lyu, Bohan and Ye, Haotian and Hong, Chuye and Wang, Yixuan and Su, Chloe and others},
journal={arXiv preprint arXiv:2610.22496},
year={2026}
}