1. Segmentation rules (RoboCasa365)
- Segments are half-open [start, end), contiguous, cover every frame; the last segment is exactly 'task complete' (idle tail after the final subtask).
- pick = approach + grasp (+ regrasps) + lift, ends when the object is clear of its support. place = arm transport + release + retract. Base motion between them is a navigate segment only if the robot ends up facing a different fixture/location; small repositioning in front of the same fixture folds into the following manipulation segment.
- navigate is used only for base motion (with or without an object in hand). Arm-only reaches are folded into the following pick/place/open/... segment.
- Control semantics: off->on = activate (A1/A2/A3 per activation_controls); on->other level = adjust; on->off = deactivate. Exception: the toaster oven temperature knob is always adjust (J1). If one knob turn both turns on and sets a level, it is one A1 segment with the level as modifier.
- A release without transport (drop the sponge after scrubbing, release the spatula after pressing) is folded into the preceding segment. If the object is carried to a new location first, that is a place segment.
- hold is only for an object kept grasped under running water; turning the faucet on/off are separate activate/deactivate segments. An object set down in the sink under running water is place + wait.
- Pouring: pick up the container -> dump ... -> place the container. The dump segment covers the tilt and untilt.
- agitate (shake/tilt/rotate) covers grasp, motion and release when the object ends on the same support; moving it elsewhere is pick + place. push is translation along a support without lifting.
- One open/close segment per door or drawer (two-door cabinets -> two segments).
- Cognitive phrases in the instruction (locate, select, choose, make sure, leave X where it is) produce no segment.
- Count-based tasks (e.g. 'place two ice cubes in each glass', 'two yogurt containers on each plate'): label each instance as its own pick + place pair; group nouns are FORBIDDEN whenever the task predicate depends on a count or a per-receptacle distribution.
2. Terminal interval
Guideline: the last segment is task complete, the idle tail after the final subtask. In this tool the terminal label is TASK_COMPLETE; it is optional and used only for a distinct trailing interval after the task has succeeded. If the last subtask reaches the end of the episode, end with that subtask.
3. Operations and templates
Operations: navigate, open, close, slide, pick, place, pour, push, hold, activate, deactivate, adjust, stir, scrub, agitate, press, wait. Every segment carries exactly one template; the note column states when the operation starts and ends and what is folded into it.
| id | operation | surface form | roles | examples | note (start/end and folding rules) |
|---|---|---|---|---|---|
| N1 | navigate | navigate to the {destination} | destination: fixture, fixture_part, object | navigate to the fridge navigate to the cup on the dining counter navigate to the tupperware with only the fish in it | Base motion that brings the robot to face a different fixture/location, with or without an object in hand. Returning to an earlier location is still 'navigate to' (no 'back'). Shuffling in front of the same fixture is folded into the following manipulation segment. |
| O1 | open | open the {target} | target: fixture, fixture_part | open the drawer open the left drawer open the fridge open the microwave door open the electric kettle lid open the stand mixer head open the left cabinet door | Hinged/sliding fixture parts only: doors, drawers, hinged lids, the stand mixer head. One segment per door. A button that opens a lid (electric kettle) is O1, not activate. The blender lid is a loose object -> pick/place. |
| C1 | close | close the {target} | target: fixture, fixture_part | close the dishwasher door close the microwave door close the freezer door close the toaster oven door close the right cabinet door | |
| S1 | slide | slide the {target} out | target: fixture_part | slide the top oven rack out slide the toaster oven tray out slide the top dishwasher rack out | Racks and trays only; drawers are open/close. |
| S2 | slide | slide the {target} in | target: fixture_part | slide the oven rack in slide the toaster oven tray in | |
| P1 | pick | pick up the {theme} from the {source} | theme: object, group_noun; source: fixture, fixture_part, object | pick up the straw from the drawer pick up a chicken drumstick from the fridge pick up the larger bowl from the counter pick up the fish from the top rack of the oven pick up the lid from the saucepan pick up the blender lid from the blender pick up the bowls from the counter pick up the bowl | Approach + grasp + lift, including regrasps after a failed grasp; ends when the object is clear of its support. Give the source whenever it is identifiable; the source-less alias exists only for compatibility with official labels. Use 'a' only when several identical instances qualify. Several objects grasped together -> group noun as theme. |
| L1 | place | place the {theme} in the {destination} | theme: object, group_noun; destination: fixture, fixture_part, object | place the bowl in the dishwasher place the straw in the glass cup place the steak in the empty tupperware place the mug in the left sink basin | Optional orientation suffix: 'place the mug in the dish rack upside down'. |
| L2 | place | place the {theme} on the {destination} | theme: object, group_noun; destination: fixture, fixture_part, object | place the pan on the front-right burner place the knife on the cutting board place the fish on the top rack of the oven place the lid on the saucepan place the blender lid on the blender place the mug on the dish rack upside down place the mug on the counter with the handle facing right | Default modifier 'on'; an orientation suffix is appended when the task requires a final orientation. |
| L3 | place | place the {theme} under the {destination} | theme: object; destination: fixture, fixture_part | place the mug under the coffee machine dispenser | |
| L4 | place | place the {theme} {relation} the {destination} | theme: object, group_noun; destination: fixture, fixture_part, object | place the jam next to the plate place the tongs next to the tray place the fork on the left side of the plate place the steak behind the older steak place the mug far away from the sink | |
| L5 | place | stack the {theme} on top of the {destination} | theme: object; destination: object | stack the larger bowl on top of the smaller bowl in the cabinet stack the can on top of the other cans | |
| R1 | pour | dump the {theme} {relation} the {destination} | theme: object, group_noun; destination: fixture, fixture_part, object | dump the vegetables onto the plate dump the leftovers into the sink dump the broccoli into the sink | Tilting the held container so its contents fall out; covers tilt and untilt. The container is picked before and placed after, so it is not repeated in the label. |
| M1 | push | push the {theme} to the {destination} | theme: object; destination: fixture, fixture_part, object | push the cutting board to the sink | Translate an object along its support without lifting it. Rotation-in-place of a grasped pan is G1, not M1. |
| H1 | hold | hold the {theme} under the {target} | theme: object; target: fixture_part | hold the carrot under the sink faucet hold the jug under the sink faucet hold the sponge under the sink faucet | Only while the object stays grasped under running water (rinse / fill / soak). If the object is set down in the sink and water runs over it, label place + wait instead. |
| A1 | activate | turn on the {target} | target: fixture, fixture_part | turn on the front-left burner turn on the front-left burner to medium turn on the sink faucet turn on the microwave turn on the blender turn on the oven turn on the toaster oven | Off -> on for every fixture except the coffee machine (A2), toaster and electric kettle (A3); see activation_controls. If the same knob turn also sets a level, keep it one A1 segment and record the level via the alias. |
| A2 | activate | press the start button on the {target} | target: fixture | press the start button on the coffee machine | Coffee machine only; see activation_controls. |
| A3 | activate | push down the lever of the {target} | target: fixture | push down the lever of the toaster push down the lever of the electric kettle | Toaster and electric kettle only. |
| D1 | deactivate | turn off the {target} | target: fixture, fixture_part | turn off the sink faucet turn off the front-right burner turn off the microwave turn off the stove knob for the kettle | On -> off for every fixture (the microwave stop button is 'turn off the microwave'). |
| J1 | adjust | set the {target} to {setting} | target: fixture_part | set the toaster oven temperature knob to max set the toaster oven temperature knob to lowest set the toaster oven temperature knob to higher set the front-left burner to medium set the front-left burner to low set the faucet handle to hot | Change the level of a control that is already on (burner heat, faucet temperature) — and ALWAYS for the toaster oven temperature knob, which is a level, not an on/off switch (the toaster oven is turned on by its timer -> A1). Use higher/lower when only the direction of change is specified. |
| J2 | adjust | turn the {target} to the {direction} | target: fixture_part | turn the sink spout to the left turn the sink spout to the right sweep the sink spout across the basin | Re-orientation of the sink spout; the direction is task-relevant (TurnSinkSpout, RinseSinkBasin) and is kept. |
| T1 | stir | stir the {theme} in the {target} | theme: object, group_noun; target: object | stir the vegetables in the pot stir the cheese in the pot | The utensil is grasped in a preceding pick segment. |
| B1 | scrub | scrub the {target} | target: fixture, fixture_part, object | scrub the cutting board scrub the bowl scrub the sink wipe the counter wash the broccoli | Repeated contact/agitation on the target. Default variant 'with tool' (held sponge/brush; 'wipe' normalises here). 'wash the {target}' = variant 'by hand': agitating/turning the object in running water with the hand, no tool (e.g. DrainVeggies). |
| G1 | agitate | {verb} the {target} | target: object | shake the pan tilt the pan rotate the pan | Grasp + agitate + release of an object that ends on the same support it started on (lift included). Moving it to another support is pick + place. |
| Q1 | press | press the {theme} against the {target} | theme: object; target: object | press the lemon against the reamer press the spatula against the chicken | Force contact between a held object and another object; 'against' distinguishes it from A2. |
| X1 | wait | wait for the {target} | target: fixture, fixture_part, object | wait for the toaster wait for the digital scale wait for the pot wait for the kettle wait for the cutting board | Robot idle while a process completes, mid-episode; name only the fixture/object being waited on, not the event clause. The idle tail after the last subtask is 'task complete'. |
Escape hatch in the original guideline: other: <free text> — Not a label. Allowed only when no template fits; every use is a template gap that must be reviewed and resolved (new template or relabel) before the task's annotations are frozen. In this audit, choose the nearest template and describe the problem in the segment note.
4. Activation controls
| control | off → on | on → off | level change |
|---|---|---|---|
| sink faucet | A1 turn on the sink faucet | D1 turn off the sink faucet | J1 set the faucet handle to hot|cold |
| burner (stove knob) | A1 turn on the <pos> burner [to <level>] | D1 turn off the <pos> burner | J1 set the <pos> burner to <level> |
| oven | A1 turn on the oven (temperature knob from off) | D1 turn off the oven | J1 set the oven temperature knob to <level> |
| toaster oven | A1 turn on the toaster oven (timer) | D1 turn off the toaster oven | J1 set the toaster oven temperature knob to <level> (always J1, even from off) |
| microwave | A1 turn on the microwave (start button) | D1 turn off the microwave (stop button) | |
| blender | A1 turn on the blender (power button) | D1 turn off the blender | |
| coffee machine | A2 press the start button on the coffee machine | ||
| toaster | A3 push down the lever of the toaster | ||
| electric kettle | A3 push down the lever of the electric kettle |
5. Modifier fields
- relation — spatial relation between theme and destination
- orientation — required final orientation of the theme
- setting — level of a control
- direction — direction of a fixture-part re-orientation
- variant — verb/manner variant of the operation
Slot conventions (noun phrases; informational — the audit tool uses typed templates without free-text slots)
- Noun phrases are lower-case and always definite ('the'); the article carries no information and the parser strips it.
- Slot = [descriptor]* canonical-noun [restrictor]. The canonical noun must come from the slot type's vocabulary (objects use the simulator's rendered names, e.g. 'bread', 'oil/vinegar bottle').
- KEEP descriptors/restrictors that identify the referent among several instances or that the task requires: 'the left drawer', 'the larger bowl', 'the empty tupperware', 'the tupperware with only the fish in it', 'the same tupperware as the cucumber', 'the front-left burner', 'the top rack of the oven'.
- DROP everything else: robot-relative location ('in front of the robot'), restrictors redundant with an already-unique noun ('the front-left burner where the kettle is placed' -> 'the front-left burner'), manner adverbs ('fully', 'lightly', 'briefly', 'securely', 'back'), purpose clauses ('... to make a marinade', '... to wash all locations of the basin'), and event clauses in wait ('... to finish and the lever to pop up').
- Destination- and task-relevant relations are NOT simplified away: in/on/under, next to/behind/left side of, stack on top of, orientation (upside down, handle facing), settings (medium, hot) and directions (left, right) stay, because tasks are defined by them.
- Group nouns are allowed as theme when several objects are grasped/moved/stirred/dumped together.
- Never bundle two actions in one label ('pick up X and place it in Y' is two segments).
- Containers' lids (saucepan lid, blender lid) are objects: 'pick up the lid from the saucepan', 'place the lid on the saucepan'. Hinged fixture doors/lids/heads are fixture parts (open/close); racks and trays slide.
Official label normalization (informational)
Official RC365 labels are template instances after these rewrites; the rewrites remove only adverbs, robot-relative location, redundant restrictors and event clauses — never destination or task-relevant content, and NEVER the operation. Labels whose normalization would change the operation are kept raw and reported as exceptions.
navigate back to the X→navigate to the Xopen the drawer in front of the robot→open the drawerturn on the front-left burner where the kettle is placed→turn on the front-left burnerturn on the left burner where the kettle is placed→turn on the left burnerwait for the toaster to finish and the lever to pop up→wait for the toasterturn on the water in the sink→turn on the sink faucetturn on the sink→turn on the sink faucet
6. How the tool applies this
- Segments must cover every frame of the episode with no gaps or overlaps; the save button checks this.
- Press B to insert a boundary at the playhead, choose the template of each segment from the task's label pool, S saves and advances.
- RoboCasa365 episodes open with NAVIGATE segments prefilled from the robot base motion (kinematics only; the original annotators had the same prefill). Edit them freely.
- The task's label pool lists the templates that occur in that task; the full template list is available in each segment's dropdown.
- Reference examples (📘) show one or two annotated episodes of the same task; they are the convention in action, never the episode you annotate.
7. AgiBot cohort (supermarket pickup, task 327)
Labels: Pick, Place, Idle, and the optional terminal TASK_COMPLETE. The dataset's own labeling scheme alternates Pick and Place: one Pick covers reaching for, grasping and lifting one item; the following Place covers carrying it and putting it down at its destination. Frames before the first Pick and after the last Place are Idle; a clear pause with nothing in progress between a Place and the next Pick is also Idle. The two reference examples show the dataset's provided labels.
8. Allex cohort (conveyor logistics)
Labels: Bring Object, Pass Object, Rotate Box, Rotate PolyBag, and the optional terminal TASK_COMPLETE. Instruction: Bring the package over, orient barcode up, then place it on the conveyor.
Start and end rules for the four Allex labels are pending transcription from the original annotators' guideline; until then the three reference examples define the convention.