Technology

Simulation

Cross-simulator framework unifying MuJoCo and IsaacSim into a single data-to-model pipeline.

The simulation engine is the foundation of the entire Arctivar stack. Built on top of RoboVerse — a unified cross-simulator framework developed by the core team — it integrates multiple physics engines into a single pipeline, connecting simulation environments that are normally isolated from each other.

The Cross-Simulator Problem

Robotics simulation is fragmented. Lightweight simulators like MuJoCo are accessible and fast but lack photorealistic rendering. GPU-powered platforms like IsaacSim offer high fidelity but require specialized hardware and expertise. Most teams pick one and accept its limitations.

Arctivar uses both — and connects them through a common abstraction layer and unified trajectory format.


RoboVerse: Unified Framework

RoboVerse (arXiv:2504.18904) provides a unified interface over multiple simulation backends. The core insight: different simulators excel at different stages. By abstracting their differences behind a common API, each stage of the pipeline routes to the simulator that handles it best.

SimulatorStageRationale
MuJoCo (WASM)Data collection (browser)Lightweight, zero-install, runs on any device
IsaacSimAugmentation and replay (GPU servers)Photorealistic rendering, accurate physics, domain randomization
MuJoCo (native)Rapid prototyping and evaluationFast iteration, broad community tooling

A trajectory collected in MuJoCo WASM can be replayed in IsaacSim without modification. The unified trajectory format and asset specification handle the translation automatically.

Built on MetaSim

RoboVerse's cross-simulator capability is built on MetaSim — a unified scene representation and description language that decouples task definitions from simulator-specific implementations.

MetaSim architecture: one interface for all worlds

MetaSim provides:

  • Standardized scene graphs: objects, articulations, materials, and physics properties described in a simulator-agnostic format
  • Automatic translation: scene descriptions are compiled into native formats for each backend (MJCF for MuJoCo, USD for IsaacSim, URDF for others)
  • Asset normalization: consistent coordinate frames, collision meshes, and visual properties across simulators
  • Programmatic scene generation: composable APIs for procedural task creation, enabling the Task Generation Engine to produce diverse scenarios at scale

This abstraction layer is what allows Arctivar to treat simulation backends as interchangeable — trajectories, assets, and task definitions remain portable across the entire pipeline.


RoboVerse v2: From Platform to Generalization Engine

RoboVerse v2: Accessibility, Scale, Diversity — 7 major upgrades

If RoboVerse v1 was about unifying simulation and learning interfaces — multiple simulators, multiple tasks, RL/IL plug-ins — then RoboVerse v2 is about turning that blueprint into a closed-loop, diversity-driven learning pipeline. The shift is from "a system that runs" to "an engine that manufactures diverse experience."

Why Diversity Is the Bottleneck

Human generalization is remarkably robust to environmental variation. Change the object, the viewpoint, the room, even the task phrasing — and a human still performs reasonably after a glance or two. Robot learning systems, by contrast, perform impressively on a single clean task distribution but collapse as soon as the layout, background, lighting, or object set shifts.

This is not because the algorithms are inherently weak. It is because the data distribution is too narrow, the data generation mechanisms are too brittle, and the bridge between simulation and reality is too thin to support genuine generalization. Real-world generalization lives in a continuous space of changes; most robot datasets are isolated points.

The Answer: Controllable Diversity at Scale

Simulation gives us something reality never can: unconstrained freedom to modify layouts, backgrounds, object combinations, and scene semantics at scale. Decentralized teleoperation lets humans step into these worlds and act with their natural adaptability — for a person, rearranging objects or changing the room barely increases task difficulty, yet those same shifts shatter most trained policies. That gap is human generalization.

When we record it across many simulated worlds and feed it into policy and world-model training, the goal is not to memorize trajectories but to let robots absorb the transferable structure of human strategies. Data becomes the Cartesian product of high-coverage behavior × diverse worlds, rather than a few demonstration traces in a fixed lab setup.

RoboVerse v2 frames this vision along three axes:

Worlds must be easy to create, and humans must be able to enter them effortlessly. Tasks, assets, scenes, and layouts are abstracted so that new variations can be spun up quickly in simulation rather than handcrafted from scratch. Multiple simulators and robots are exposed through a common interface. Human experience becomes an always-available resource, not a scarce artifact tied to one physical lab.

Major Upgrades in v2

  1. Modular Task Plug-and-Play

    A clean task API where any RL / IL / model-training method can train and evaluate consistently. Simulators are reduced to their essential role — producing controllable, reproducible worlds instead of leaking into training logic.

  2. Humanoid and Dexterous System Support

    Where v1 offered almost no humanoid support, v2 integrates industrial-grade embodiments from Unitree (humanoid and quadruped platforms), AgiBot (humanoid robotics), and DexMate (dexterous hand systems). Modern humanoid algorithms such as Beyond-Mimic are fully supported. All robots — humanoids, arms, dexterous hands — share unified observation/action schemas and dataset formats.

  3. Closed-Loop Production Pipeline

    Task Building → Data Collection → DR & Rendering → Training → Real-World Deployment. Tasks can be scripted, RL-generated, transferred from policies, or teleoperated via phone/keyboard/web. Each trajectory is re-rendered in IsaacSim with multi-view, high-fidelity appearance randomization — and the same pipeline deploys directly to real robots.

  4. Multi-Source Data Distribution

    From "a few tasks with limited demos" to a broad distribution defined by teleoperation behaviors, RL policies, long-tail exploration, diverse task families, and varied action semantics. The distribution becomes a Cartesian product of (tasks × worlds × behaviors × visual DR) rather than a fixed set of examples.

  5. Cross-Sim Asset System and Automated Scene Construction

    High-quality reconstructed assets from EmbodiedGen alongside photo-based reconstructions. A unified USD/URDF/MJCF asset pipeline allows consistent cross-sim usage. Automatic layout generation creates kitchens, living rooms, tabletops, offices, and other semantic spaces in one step — scene design → asset selection → layout → data capture becomes a single automated flow.

  6. Realism-Oriented Visual Augmentation

    Beyond simple domain randomization: real-world materials and textures, reconstructed household environments. Observations inherit the complexity, messiness, and visual noise of everyday scenes — bridging the visual gap between simulation and deployment.

  7. Engineering Toolkit

    • Viser / Rerun for lightweight 3D visualization and interactive trajectory replay
    • Pyroki (differentiable IK library) and IK utilities for rapid pose planning and teleoperation assistance
    • Autotest / auto-eval pipelines that validate every task, asset, and simulator backend — eliminating silent failures and improving long-term maintainability

Evaluation: Testing Generalization, Not Leaderboard Scores

Evaluation in v2 shifts from collecting benchmark scores to testing whether a model has actually internalized a structure of generalization. By leveraging structured diversity — multi-sim, multi-layout, multi-object, heavy visual shifts — and checking behavior on real robots, v2 asks a sharper question: not "who scores highest in this one environment?" but which methods continue to behave sensibly as the world moves, and which ones genuinely carry forward the human generalization they were trained on?


Web-Based Teleoperation (MuJoCo WASM)

The data collection frontend compiles MuJoCo to WebAssembly and runs the full physics simulation directly in the browser:

  • Physics: MuJoCo WASM handles all dynamics, collision, and control integration on the user's local CPU.
  • Rendering: Three.js / WebGL provides full-screen real-time 3D visualization with floating semi-transparent UI overlay. The workspace is organized into functional zones including a top global status bar (time, FPS) and a structured controls panel.
  • Control: An embedded inverse kinematics solver translates task-space inputs (end-effector position, rotation, gripper state) into joint-level commands. In V2 the control layer was rebuilt: IK resolves targets rather than driving every frame, and execution follows smooth online interpolation in joint space — eliminating a set of long-standing issues (stutter, latency, interpenetration, unstable grasps, joints locking at their limits) and noticeably steadying bimanual operation.
  • Direct manipulation: Contributors can drag the gripper along the current view plane or double-click a target object to move the arm into a pre-grasp pose automatically — concentrating human input on the informative parts of the trajectory (grasp selection and the fine adjustments around gripper closure) instead of low-information transit across free space.
  • State checkpointing: The simulator state can be saved and restored mid-run (N to save, B to restore), so long-horizon, multi-stage tasks can be recorded in segments and reproduced reliably.
  • In-browser policy inference: For post-training tasks, the task's trained policy — a diffusion-based action denoiser — runs directly in the browser via ONNX Runtime (WASM). Like the physics, policy rollout requires no GPU server on the contributor's side.
  • Seeded rollouts: Each run is initialized from a recorded seed, making the initial state — and, in post-training tasks, the policy's execution — reproducible, so any submission can be replayed and verified exactly as recorded.
  • Performance: The MuJoCo physics loop and Three.js rendering run on separate threads from the React UI. Trajectory data is buffered in memory during the physics loop, and a dedicated broadcaster sends only occasional status updates to the UI — avoiding the overhead of frequent re-renders and ensuring low-latency control during active teleoperation.
  • Trajectory packaging: An embedded checker verifies task completion at runtime. The trajectory manager then automatically compiles buffered state samples and task metadata into a serialized JSON file, ensuring data standardization before upload.

Cross-Embodiment Keyboard Control

To support diverse end-effectors — from simple parallel-jaw grippers to complex dexterous hands — Arctivar employs a unified teleoperation interface based on interpolation mapping.

A library of grasp templates is predefined for various manipulation tasks. Each template specifies a target joint-angle vector corresponding to a specific grasp category (power, precision, tripod, pinch, etc.). The operator adjusts a single scalar parameter using two keyboard keys, interpolating between the fully open rest state and the target template pose:

q(t) = q_rest + α(t) · (q_tmpl − q_rest)

This formulation collapses high-dimensional joint space (often n > 16 for dexterous hands) into a single control axis. New robot embodiments can be integrated by simply defining their rest pose and a task-specific template library — no modifications to the control core required.


Realistic Augmentation (IsaacSim Backend)

Trajectories collected in the browser are uploaded and replayed on Linux GPU servers running IsaacSim. During replay, the augmentation engine applies:

  • Photorealistic rendering: physically-based lighting, material models, and camera simulation
  • Domain randomization: systematic variation across visual and physical dimensions to maximize policy generalization:
  • Lighting: randomized direction, intensity, color temperature, and shadow properties (point, directional, and area lights)
  • Textures and materials: randomized surface properties (albedo, roughness, metallic), procedural textures, and material swaps across objects and backgrounds
  • Backgrounds and distractors: varied tabletop surfaces, surrounding objects, and environmental context
  • Camera parameters: randomized intrinsics (field of view, focal length) and extrinsics (viewpoint position, orientation) to reduce overfitting to a single observation perspective
  • Physics parameter variation: randomized mass, friction, damping, and compliance across objects and joints to improve policy robustness against real-world physical uncertainty
  • Multi-view camera generation: rendering from multiple virtual viewpoints to create diverse RGB, depth, and segmentation streams from a single trajectory

Each original trajectory is multiplied into thousands of augmented samples — visually diverse but behaviorally consistent with the original demonstration.


Domain Randomization System

In robotic Reinforcement Learning (RL) and Imitation Learning (IL), policies often overfit to specific visual or physical parameters within the simulation. The DR system has one job: force the model to learn task-relevant features instead of memorizing environment-specific details.

By injecting diversity across scene composition, material properties, lighting conditions, and camera parameters, the system forces the model to learn generalized, underlying physical and geometric features — enabling robust transfer from pure simulation to the physical world.

Core Design: Separation of Concerns

In traditional simulators, object creation and property modification are often entangled, leading to exponential code complexity. The central design principle of the RoboVerse DR system is the Separation of Concerns — strictly dividing simulation variations into two distinct operations:

Handled by the SceneRandomizer, which directly manipulates the USD (Universal Scene Description) Stage. It focuses purely on the creation, deletion, and switching of dynamic objects — for example, swapping between different table models or adding/removing visual distractors.

This decoupling keeps interfaces clean and allows different randomizers to be freely composed like building blocks:

# Lifecycle Management: handles "life and death" (spawning different tables, backgrounds)
scene_rand = SceneRandomizer(scene_cfg, seed=42)

# Property Editing: handles "feature modification" (changing materials, lighting)
mat_rand = MaterialRandomizer(mat_cfg, seed=43)
light_rand = LightRandomizer(light_cfg, seed=44)

Object Abstraction

Due to physics engine limitations, the system splits objects into two categories:

TypeManaged ByPhysicsExamples
Static ObjectsSimulation Handler (at init)Full collision and dynamicsRobots, core task objects
Dynamic ObjectsSceneRandomizer (at runtime)Visual only — no physicsBackgrounds, tabletop distractors

To bridge this divide, RoboVerse introduces the ObjectRegistry singleton. All randomizers query target objects through a unified interface, eliminating the burden of manually managing complex USD Prim paths.

Three-Layer Scene Hierarchy

Scene construction follows a declarative, semantically layered structure:

LayerRoleExamples
Layer 0 — EnvironmentMacro backgroundFloors, walls, complete indoor room layouts
Layer 1 — WorkspaceManipulation surfaceTables of varying styles, materials, and dimensions
Layer 2 — ObjectsVisual diversityRandomly scattered desktop clutter and distractors
scene_cfg = SceneRandomCfg(
    # Layer 0: Macro background (e.g., Kujiale interior room layouts)
    environment_layer=EnvironmentLayerCfg(
        elements=[USDAssetPoolCfg(name="kujiale_scene", usd_paths=scene_paths, selection_strategy="random")]
    ),
    # Layer 1: Manipulation surface (e.g., EmbodiedGen table assets)
    workspace_layer=WorkspaceLayerCfg(
        elements=[USDAssetPoolCfg(name="table", usd_paths=table_paths, selection_strategy="random")]
    ),
    # Layer 2: Visual distractors (e.g., randomized desktop clutter)
    # objects_layer=ObjectsLayerCfg(...)
)

Multi-Level Progressive Randomization

For reproducibility and controlled experimentation, the system employs a progressive randomization strategy. Researchers can systematically isolate and quantify the impact of each factor on policy generalization:

LevelWhat ChangesPurpose
Level 0Nothing (baseline)Control condition
Level 1Scene geometry + materialsVisual diversity baseline
Level 2+ LightingIllumination robustness
Level 3+ Camera perturbationViewpoint invariance

During each environment reset(), randomizers execute in strict order — the SceneRandomizer must first define the scene geometry before its properties can be modified:

def reset(self, env_ids=None):
    self.scene_rand()   # Level 1: reconstruct scene geometry
    self.mat_rand()     # Level 1+: assign random materials
    self.light_rand()   # Level 2+: randomize lighting
    self.cam_rand()     # Level 3+: perturb camera pose and FOV
    return super().reset(env_ids)

System Advantages


Cross-Simulator Verification

Replaying the same trajectory across different simulation environments achieves a stronger form of domain randomization than any single simulator can provide. Discrepancies between simulators expose assumptions the model has memorized rather than learned, forcing generalization to the underlying logic of the task.

Next: Data Pipeline →
Simulation