Show HN: Kinematic Hand Skeleton Optimization in Jax

Summary

Pi0 and Lerobot with Rerun A repo to explore training robots with Pi0 and Lerobot and human pose motion retargeting Installation Currently only linux is supported Using Pixi Make sure you have the Pixi package manager installed git clone https://github.com/rerun-io/pi0-lerobot.git cd pi0-lerobot Usage Camera Calibration using VGGT Get camera intrinsics/extrinsics, depth maps, and pointcloud for a set of multiview images from hocap To run camera calibration on a sequence run pixi run calibrate-cameras-vggt Human Pose and Kinematics For HOCap dataset, sample is automatically downloaded from dataset on command run. Checkout example rrd file here For the full assembly101 dataset go to this link to get/download dataset To run dataset visualization pixi run visualize-hocap-dataset # recommended pixi run visualize-assembly101-dataset Current Pipeline To run 2D pose estimation, tracking, and triangulation pixi run pose-estimation-assembly101 Kinematic Hand Skeleton The goal is to create a pipeline that converts time-synced multi-camera footage into axis-angle joint angles (θ) and metric 3D positions for every finger joint. This provides "motion fuel" for robotic retargeting and learning fine motor skills. Why use a kinematic skeleton with joint angles? Rig-agnostic Pose: Captures relative bone rotations, allowing transfer to any avatar or robot hand. Captures relative bone rotations, allowing transfer to any avatar or robot hand. Easy Retargeting: A single Forward Kinematics (FK) pass transfers motion. A single Forward Kinematics (FK) pass transfers motion. Compact Storage: Requires only ~60 floats per frame. Requires only ~60 floats per frame. Built-in Constraints: Easily enforce bone lengths, joint limits, etc. Easily enforce bone lengths, joint limits, etc. Gradient-Friendly: Differentiable FK enables optimization from 2D errors. Differentiable FK enables optimization from 2D errors. Sensor Fusion-Ready: Integrates well with IMUs, mocap markers, etc. Capture-to-Angles Pi...