Robot Pick-and-Place Simulation with SimReady Assets12 min read2,369 words

Robot Pick-and-Place Simulation with SimReady Assets

Build a robot pick-and-place simulation in Isaac Sim with SimReady assets, agent-assisted setup, validation checks, and practical limits to test before scaling.

robot pick-and-place simulationSimReady assetsIsaac Simrobotics simulationAI agents for robotics
Robot Pick-and-Place Simulation with SimReady Assets

Build a Robot Pick-and-Place Simulation Without Skipping Validation

A robot can look ready in a 3D scene and still fall apart once physics is enabled. Joints might not respond the way you expect, a gripper can miss its object, or the object can end up slipping or falling through the work surface. Visual correctness is a starting point - not proof that your simulation is actually ready to run.

This walkthrough shows how to build a beverage-can pick-and-place simulation in NVIDIA Isaac Sim using SimReady assets: a UR17G arm, a Robotiq 2F-140 gripper, and a can. It also covers how an AI agent can help assemble assets and generate task controls - and why you still need to sanity-check the behavior you see in the simulator. The goal is a repeatable test, not a single successful run that convinces you the robot is production-ready.

What does SimReady mean for robotics simulation?

A SimReady asset is an OpenUSD asset prepared with simulation properties for a specific use case. For robotics, those properties can include joints, joint drives, collision geometry, and mass. The practical point is simple: you're not starting from a "pretty model" alone - you get settings intended to behave correctly in a simulation workflow.

SimReady Foundation organizes this into three levels:

  • Profiles define the use case (for example, a robot body or robot gripper).
  • Features describe what the asset needs to support (like controllable joints).
  • Requirements list checks that should pass for those features (like joint-drive settings that are appropriate for the profile).

This structure makes "readiness" easier to inspect. Instead of only asking "does it look right?", you can review the profile and the validation evidence to see which checks were performed. In this example, the arm and gripper need usable joints, and the can needs collision geometry and mass so it can interact with the robot and the work surface.

That matters because validation is scoped. The UR17G arm's report uses the Robot-Body 4.0.0 profile, and the Robotiq gripper's report uses Robot-Gripper 3.0.0. Both reports record passing PhysX driven-joint checks. That's good evidence about joint-drive configuration - but it doesn't prove that a grasp, controller, or full pick-and-place task will actually work the way you want.

I like to think of readiness as a chain of tests:

  1. Asset-level checks: Are the required simulation properties present and valid for the profile?
  2. Assembly checks: Do the arm and gripper connect and behave like one controllable mechanism?
  3. Task checks: Can the assembled system pick up, move, and release the object under your setup?

Passing one step doesn't automatically mean the next will pass. Profiles give you a baseline you can reuse; the assembled simulation tells you whether that baseline is enough for your particular task.

What do you need before building the scene?

This example uses three SimReady Central assets: a UR17G robot arm, a Robotiq 2F-140 gripper, and a beverage can. Download each asset package and keep its folder structure intact. USD can reference other files inside the package, so copying only the "main" USD file can lead to missing materials or broken references when you load the scene.

You can download the packages through the asset library interface or with the Hugging Face command-line option. Use whichever fits your workflow. In either case, note the local USD file path for each asset - you'll need those paths when composing the scene or when asking an agent to work with the assets.

The software setup in the example includes:

  • Isaac Sim 6.2, the version used for this walkthrough. Check the system requirements for the version you install.
  • An agent that can use Isaac Sim skills, if you want agent-assisted assembly and task generation. The agent needs access to the relevant skills and a way to run Python against the simulator.
  • The Hugging Face CLI, only if you prefer command-line downloads.
  • Access to the USD Search endpoint, if you want to search for available content through that service.

Before you write task logic, open the assets in Isaac Sim and confirm they load without missing-file errors. Then inspect the robot and gripper joints: check their limits and drive settings, and confirm the can has collision geometry and mass. You can do this yourself in Isaac Sim, or ask an agent to inspect the scene and report what it finds.

Also keep a small setup record: asset versions, simulator version, local paths, and any scene changes you make. This isn't "paperwork for paperwork's sake." If the behavior changes later, the record helps you figure out what actually changed - code, assets, or simulator settings. It also makes it easier for someone else to reproduce your initial conditions instead of guessing.

How do you assemble the robot and create the task?

The asset packages give you models and simulation settings. They don't automatically define your whole task. You still need to assemble the arm and gripper, place the robot and the can in a scene, set up a work surface, choose starting positions, and create controls for moving the arm and operating the gripper.

If your agent needs to interact with the simulator, start Isaac Sim with its remote Python server enabled:

./isaac-sim.sh --enable isaacsim.code_editor.python_server

For agent-assisted work, start the agent from the Isaac Sim installation directory so it can access the skills packaged with that installation. Then provide the local USD paths and ask it to assemble the arm and gripper using the Isaac Sim Robot Assembler workflow. Ask for validation of the resulting assembly, not just a scene that "looks connected."

A good prompt spells out what inputs you're giving and what evidence you expect back. For example:

Assemble the UR17G robot USD at the supplied local path with the Robotiq 2F-140 gripper USD at its supplied path. Use the Isaac Sim Robot Assembly workflow. Validate the assembly and report any joint or physics issues.

The agent can use Isaac Sim's remote interaction and USD articulation skills to make changes and inspect results. Even then, you should verify the basics yourself: that both assets load, that the gripper is attached where you expect, and that the assembled articulation exposes the joints you plan to control. A validation report is a checkpoint - not a replacement for actually running small movement tests in the assembled scene.

Once assembly is in place, add the can and a flat work surface. Pick a fixed starting position for the can and a target location for release. In this example, the robot is anchored and the workspace is clear, which makes debugging less confusing. If the grasp fails, you can focus on the controller, contact behavior, or asset configuration rather than also wondering whether the object started somewhere weird.

Before you run the full sequence, test each control separately. Move the arm a small distance, then open and close the gripper. Confirm the motion direction is right and that the simulation responds through the joint drives. This incremental approach saves time - you're not trying to diagnose a long scripted sequence where the first failure shows up late.

How do you validate a pick-and-place run?

The task sequence is straightforward to describe: approach the can, grasp it, lift it, move it to the target, and release it. The real work is checking what happens at each step, not assuming the final position alone tells the whole story.

The example uses a contact-based grasp. The gripper closes on the can, and contact plus friction keep it in place as the arm follows fixed waypoints. The controller coordinates arm movement with opening and closing the gripper. Agent-generated controls can help you get a first script, but the simulator behavior is what matters - not whether the code ran without throwing an exception.

Use a repeatable procedure:

  1. Reset the robot and can to the same starting state.
  2. Run the approach and confirm the gripper is aligned with the can.
  3. Close the gripper and check whether contact is established.
  4. Lift the can and watch for slipping, bouncing, or unexpected rotation.
  5. Transfer it to the target while monitoring whether the grasp holds.
  6. Release it and measure or inspect the placement error.
  7. Record where the failure happens before changing anything else.

Before you run the full task, also test the basic physical interactions. The can should rest on the work surface rather than fall through it. The arm should respond to small commanded motions, and the gripper should open and close. These checks catch common setup issues early, which makes the full-task result easier to interpret.

If the result isn't what you expected, reset before you retry. Changing multiple variables at once - starting position, gripper effort, controller timing, and waypoints - makes it hard to tell what actually fixed the issue. Change one factor, rerun from the same initial state, and record what happened. It's a basic feedback loop, but it beats repeatedly asking an agent for a "different script" without preserving what changed.

In this demonstration, the horizontal grasp uses a 2 N·m drive-effort cap, selected through earlier simulation tests. That value isn't a calibrated fingertip force, and you shouldn't treat it like a manufacturer recommendation. The setup also uses fixed waypoints rather than replanning around obstacles. So these results describe this scene and controller - not a general guarantee across payloads or object positions.

Where do AI agents help - and where do they add risk?

An AI agent can reduce the overhead of moving between scene inspection, asset assembly, and initial task code. In this workflow, an agent with Isaac Sim skills can connect to the simulator through Python, inspect or assemble USD assets, and generate controls for the pick-and-place sequence. That's most useful when the work is repetitive or when you need a faster starting point for an unfamiliar simulation setup.

The practical benefit is faster iteration - not "hands-off" engineering. An agent may produce a usable first draft, but you still need to inspect the scene, review the generated code, run the simulation, and decide whether the results meet your task requirements. Generated code and outcomes can vary depending on the agent and model combination, so treat the output as a change that needs review and keep notes on your setup.

A sensible division of work looks like this:

  • Let the agent handle: repetitive scene tasks, initial assembly steps, code scaffolding, and summarizing validation output.
  • Keep human review for: joint configuration, controller assumptions, grasp quality, collision behavior, and whether the test actually matches your intended use case.
  • Require observable evidence: successful joint motion, stable contact, object retention during transfer, and acceptable placement error.

This is also where simulation ethics and safety show up in practice. A successful simulation isn't proof that a real robot is safe to deploy. This example assumes a known object position, an anchored robot, a clear and flat workspace, and limited collision checking. It doesn't use vision, flexible can deformation, or liquid motion, and it doesn't test uncertain object locations. Those omissions are fine for a bounded example - but they matter if you start making real-world decisions from the result.

If you're thinking about a production workflow, make the gap explicit. Spell out what the simulation covers, what it doesn't, and what separate testing you'll do before you touch hardware. Simulation can help you catch errors earlier and make experiments more repeatable, but it can't, by itself, establish robustness to different payloads, real-world friction, perception errors, or mechanical variation.

Key takeaway: Use the agent to speed up the build loop. Use validation and repeatable tests to decide whether the outcome is trustworthy.

Conclusion

SimReady assets help you start with simulation-oriented properties and profile validation evidence. In Isaac Sim, the UR17G, Robotiq 2F-140, and beverage can can be assembled into a useful pick-and-place test - but you still have to build the scene, controls, and task checks.

Keep the workflow incremental: confirm the assets load, validate the assembly, test joints and contact, then run the full sequence from a repeatable starting state. An AI agent can speed up setup and code generation, but your validation process determines what the result actually demonstrates. Start with a bounded simulation, document its assumptions, and expand the tests before using the outcome to inform hardware deployment.

Frequently Asked Questions

SimReady describes an OpenUSD asset prepared with simulation properties for a defined use case. Depending on the asset and profile, those properties can include joints, drives, collision geometry, and mass. Profile validation checks specified requirements, but does not prove that an entire robot task will succeed.
An agent with Isaac Sim skills can help assemble assets, inspect a scene, and generate task controls. You still need to review its changes, test joint motion and contact, and evaluate the complete task. Agent-generated code is a starting point, not evidence that the simulation is correct or safe for hardware.
No. The example uses a fixed starting position, simplified object physics, and scripted waypoints. It does not test perception, uncertain object locations, all collision conditions, or real hardware behavior. Treat the result as evidence for the specific simulated setup, then plan additional testing for the intended deployment.
First reset the scene and reproduce the failure from the same starting state. Check gripper alignment, contact, joint drives, collision geometry, and the motion sequence. Change one factor at a time and observe where the can slips. Avoid treating the example’s effort cap as a calibrated grip-force recommendation.

Related articles