Physical AI Lab

Move your finger.
A robot arm follows.

From raw webcam pixels to a robot arm you control with your hand.

Jim Seelan · 3 hours · 6 modules · 1 closed loop

This is basically how every robot works.
Look → think → move. Repeat.

Every module covers one stage. The dashed red path is the feedback — each action changes the world, which changes the next camera frame.

Using this deck

Two modes. One file.

Presentation mode

Arrow keys or Space bar to navigate
S — speaker notes
F — fullscreen
B — blank screen (useful mid-session)
? — all shortcuts

Handbook / self-study

Scroll mode — add ?view=scroll to the URL
physical-ai-labs.vercel.app/slides?view=scroll
All slides become one continuous scrollable page — read at your own pace
PDF export — open in Chrome, add ?print-pdf, then File → Print → Save as PDF
Works offline after the first load

What's in here

Part	Content	When to read
1 — Concept foundations	11 concepts with plain-language explanations, analogies, references, self-check	Before or after the session
2 — The lab	Every module: scripts, exercises, diagrams, speaker notes	During and after
3 — After today	9 capstone projects across laptop and hardware tiers	After the session
4 — Quick reference	Key numbers, all demo scripts, common issues, fallback behaviours	Keep open during the session

Part 1 of 4

Concept Foundations

Eleven concepts you need to understand the lab. Each has a plain-language explanation, an analogy, and links to go deeper. During a live session, skip to Part 2. On your own, read these first.

Self-check After reading each section, close it and try to explain the concept in one sentence. If you can't, re-read. Understanding — not memorisation — is the goal.

Concept 1 of 11

What is Physical AI?

One sentence Physical AI is AI that acts on the physical world in a continuous feedback loop — perceive → decide → act → perceive again — rather than processing a static input and producing a static output.

Static AI

You give it an image, it returns a label. The world does not change because of what it said.

Examples: image classifier, chatbot, recommendation engine.

Physical AI

The agent acts → the world changes → those changes become the next input. A continuous conversation with the physical world.

Examples: robot arm, self-driving car, drone.

Concept 1 — Physical AI

The analogy

Reading a book is static — the book doesn't change based on what you understood. Having a conversation is dynamic — what you say changes what the other person says next. Physical AI systems are always in a conversation with the physical world.

Why it matters for Module 4 Your hand moves → MediaPipe detects it → joint angles computed → torque sent → robot arm moves → camera sees new frame. Every stage feeds the next. Remove any stage and the loop breaks. The loop is not a detail — it is the capability.

Go deeper

NVIDIA: What is Physical AI? — 5-minute overview from the company defining the field
OpenAI: Solving Rubik's Cube with a Robot Hand — watch the video, read the overview. This is what Physical AI looks like.

Concept 2 of 11

From pixels to numbers

One sentence A camera frame is a 3D array of integers (height × width × 3 colour channels). All perception work is about compressing that array into a small, meaningful list of numbers the robot can act on.

What a frame actually is

cap.read()  # returns numpy array, shape (480, 640, 3)
# 480 rows × 640 columns × 3 (B, G, R values)
# = 921,600 numbers per frame

A robot cannot act on 921,600 numbers directly. Too much, most of it irrelevant. The perception layer compresses this down to 3 or 4 joint angle values.

Concept 2 — Pixels to numbers

The compression pipeline

Camera frame (480×640×3 ints)
  → MediaPipe model
  → 21 landmark (x, y, z) coordinates
  → dot-product angle formula
  → 3–4 joint angles in degrees
  → numpy array shape (4,) float32  ← the state vector

Analogy Describing your hand position with every pixel = thousands of numbers. Saying "my index finger is at (0.3, 0.4) and slightly bent" = a state vector. Tiny, precise, actionable.

Go deeper

NumPy ndarray docs — understand shape, dtype, slicing
OpenCV Python tutorials — Core Operations and Image basics

Concept 3 of 11

MediaPipe & landmarks

One sentence MediaPipe runs a pre-trained neural network on your camera frame and outputs the (x, y, z) positions of 21 specific points on your hand — called landmarks. You don't train anything.

What "pre-trained" means

Training from scratch: millions of images, weeks of GPU, deep CV expertise. We use Google's model (hand_landmarker.task — 7.5 MB) and run inference on it. The model was trained elsewhere. We just use it.

Normalised coordinates

Landmark at x=0.5, y=0.3 means: halfway across the frame, 30% down from top. Always in [0.0, 1.0] regardless of camera resolution. Code is resolution-independent.

Concept 3 — MediaPipe

LIVE_STREAM mode — why async

The pattern Feed frames via detect_async() → model calls your on_result callback when done. Main loop keeps reading camera while model processes the previous frame in a background thread. queue.Queue(maxsize=1) safely passes results between threads.

Analogy Like sending photos to a photo lab. You drop off a photo (detect_async), keep taking more, the lab calls you (callback) when each one is developed. You don't wait around.

Go deeper

MediaPipe Hand Landmarker guide — Overview and Configurations sections
MediaPipe Solutions overview — the Tasks API architecture, three running modes

Concept 4 of 11

State vectors

One sentence A state vector is a fixed-size numpy float32 array that captures everything the agent needs to know right now in order to decide what to do next.

Why fixed size?

Neural networks require fixed-size inputs. The state vector must always have the same number of elements. CartPole: always (4,). Reacher: always (11,). Non-negotiable.

What goes in it?

Exactly what's needed — no more. CartPole: cart position, cart velocity, pole angle, pole angular velocity. Not the pixel render. Not the time elapsed. Just those 4 numbers.

Concept 4 — State vectors

State vectors in this lab

Context	Shape	Contents
M1 exercise result	`(4,) float32`	Index, middle, ring, thumb MCP angles
CartPole observation	`(4,) float32`	cart pos, cart vel, pole angle, pole vel
Reacher observation	`(11,) float32`	Joint cos/sin, target pos, velocities, fingertip pos + dist
M4 action	`(2,) float32`	Joint 0 torque, joint 1 torque

Analogy A chess player's state isn't the tablecloth colour or who's watching. Just the piece positions. Minimal sufficient description to decide the next move.

Observation design — the most important decision you make Choosing what to include in a state vector is called observation design. Leave out something important: the agent can never learn. Include irrelevant noise: training slows and generalisation suffers. The examples above each reflect careful observation design — not arbitrary choices.

Why float32, not float64? Neural networks run measurably faster with 32-bit floats. The precision difference (7 vs 15 significant digits) does not matter for robot control — joint angles don't need nanometre precision. Using float32 everywhere is standard practice in RL and robotics.

Go deeper

Gymnasium: Basic Usage — obs, action, reward and the step loop
NumPy absolute beginners guide — arrays, shapes, dtypes

Concept 5 of 11

Gymnasium & MuJoCo

One sentence Gymnasium is a Python API for physics simulations an RL agent can interact with. MuJoCo is the physics engine that makes those simulations realistic. We simulate because training on real hardware is slow, expensive, and dangerous.

Why simulate at all?

RL needs millions of trials
Real hardware: slow, expensive, can break
Simulation: 1,000 trials/sec, crashes are free, run overnight
The catch: the sim-to-real gap (Concept 11)

What Gymnasium provides

A standardised interface — the same three calls — for hundreds of different environments. Whether you're training on CartPole or a humanoid robot, your RL code stays the same. Only the environment changes.

obs, info = env.reset()
obs, reward, done, trunc, info = env.step(action)
env.close()

Concept 5 — Gymnasium/MuJoCo

The episode loop

render_mode="rgb_array" — how the lab hub streams simulation frames When you request render_mode="rgb_array", each env.render() call returns the simulation frame as a numpy array (height × width × 3 RGB) instead of opening a display window. The FastAPI server in api/server.py uses this: render → encode to JPEG → stream over HTTP to your browser. No display hardware, GPU, or desktop required.

Go deeper

Gymnasium: Basic Usage — explains reset, step, the obs/reward/done/info tuple
Reacher-v5 environment docs — Observation Space and Action Space sections
MuJoCo homepage — gallery and "Why MuJoCo?"

Concept 6 of 11

Reinforcement Learning

One sentence RL is a way of training an agent by letting it try things, measuring how well it does with a reward signal, and adjusting its behaviour to get more reward over time — without ever telling it the right answer.

The core loop

Agent observes current state ("pole tilted 5° right, moving at 0.3 rad/s")
Agent picks an action ("push cart right")
Environment transitions to new state (cart moves, pole responds)
Agent receives reward (+1 because pole still upright)
Repeat millions of times

Concept 6 — RL

RL vs supervised learning

Supervised learning You have labelled examples: this image IS a cat. The correct answer is provided for every training example.

Reinforcement learning Nobody tells the agent "in this situation, push right." It figures it out from reward signals alone. Harder — and more powerful.

Analogy Teaching a dog to sit. You don't explain what "sit" means. You give a treat when it does something resembling sitting. The dog figures out what behaviour produces treats. RL is the formal mathematical version of this.

Go deeper

HuggingFace Deep RL Course — Unit 1 — free, interactive, written for beginners. Start here.
OpenAI Spinning Up: Key Concepts — Part 1: Key Concepts and Terminology

Concept 7 of 11

Markov Decision Processes

One sentence An MDP is a mathematical framework for describing any sequential decision-making problem — it formalises "who decides, what they can decide, what happens next, and what reward they get."

Symbol	Name	CartPole example
S	State space	All possible (cart_pos, cart_vel, pole_angle, pole_vel) combinations
A	Action space	{0, 1} — push left or push right
R(s,a)	Reward function	+1 if pole upright; episode ends if it falls
T(s′\|s,a)	Transition function	Physics of the system — deterministic for CartPole
γ	Discount factor	0.99 — near rewards worth slightly more than far rewards

Concept 7 — MDPs

The Markov property — why it matters

The property The next state depends only on the current state and action — not on the history of how you got there. P(s′ | s, a, all_previous_states) = P(s′ | s, a).

For CartPole: knowing current (position, velocity, angle, angular_velocity) is sufficient to predict the next state. No history needed. This is what makes the problem mathematically tractable.

What the agent maximises G = R₁ + γ·R₂ + γ²·R₃ + … (expected cumulative discounted reward)

Go deeper

HuggingFace RL Course: MDPs — clearest beginner explanation
CartPole-v1 docs — exact S, A, R for the environment you run

Concept 8 of 11

PPO — Proximal Policy Optimisation

One sentence PPO trains a neural network (the policy) by collecting experience, estimating which actions were better, and making small, stable updates to the network weights — clipping the update to prevent collapse.

The problem it solves

Naive policy gradient methods: update too aggressively → policy "collapses" → unrecoverable. PPO's fix: a clipped objective function that constrains how much the policy can change in one update. "Proximal" = stay close to where you were.

Steps per update

Run policy for n_steps timesteps, collect (s, a, r, s′)
Compute advantage via GAE (Generalised Advantage Estimation) — was this action better or worse than average?
Update policy toward high-advantage actions — but clip the update
Repeat

Why PPO and not SAC? SAC (Soft Actor-Critic) is often better for purely continuous action spaces at scale. PPO works well on both discrete (CartPole) and continuous (Reacher) tasks, is stable on a laptop, and is the de facto standard starting point for robotics RL. SAC is the next algorithm to learn after this.

Key SB3 hyperparameters

Parameter	What it controls	Default
`n_steps`	Timesteps collected per rollout before an update	2048
`batch_size`	Mini-batch size for each gradient update	64
`clip_range`	Max policy change per update (the "proximal" constraint)	0.2
`gamma`	Discount factor γ — how much future rewards are worth	0.99

Concept 8 — PPO

The reward curve

The flat phase is random exploration. The sharp climb is the moment the agent discovers balance. Stop talking when it climbs — let everyone watch.

Go deeper

HuggingFace RL Course: Unit 8 — PPO — most accessible PPO explanation, uses CartPole
Stable-Baselines3 PPO docs — Parameters: n_steps, batch_size, clip_range, gamma
Original PPO paper (Schulman et al. 2017) — just read the abstract and introduction

Concept 9 of 11

Closing the loop

One sentence The loop is "closed" when the agent's action changes the world, and that changed world feeds back into the agent's perception — creating a continuous cycle of sense → act → sense → act.

Open loop Acts without feedback. Washing machine runs a fixed cycle regardless of whether clothes are clean. Most AI you've used is open-loop: input → output, but output doesn't change input.

Closed loop Continuously adjusts based on what it observes. Thermostat: measures temperature → turns on heat → measures again. Output (heat) changes input (temperature).

Concept 9 — Closed loop

The exact loop you build in Module 4

Your hand moves slightly
  → camera captures new frame
  → MediaPipe detects landmark 8 at (x=0.6, y=0.4)
  → scale_landmark_to_action → action = [0.2, -0.2]
  → env.step([0.2, -0.2]) → Reacher arm moves
  → env.render() → new frame sent to browser
  → you see the arm respond
  → your hand adjusts
  → loop continues at ~20 Hz

50–100ms latency between hand movement and arm response. That latency is the loop time. When it works, you feel it.

Go deeper

Wikipedia: Open-loop and closed-loop control
RT-2 project page — this is what Module 4 looks like with a real robot arm and a language model

Concept 10 of 11

Foundation Models & VLMs

One sentence A Foundation Model is a large neural network trained on massive diverse data that develops broad general capabilities and can be prompted to reason about specific tasks without task-specific retraining.

Three characteristics

Very large scale — billions of parameters, internet-scale data
Emergent capabilities — abilities not explicitly trained for
Adaptable via prompting or fine-tuning without retraining from scratch

VLM — Vision-Language Model

Gemini is a VLM: it understands both images and text. Give it a JPEG and a question — it reasons about the image in context. Not labelling — reasoning.

In this lab: camera frame + prompt → JSON action dict

Concept 10 — Foundation Models

VLA — Vision-Language-Action models

The frontier VLA models output robot action commands (joint angles, torques) directly from vision and language inputs. RT-2, OpenVLA, π0 are VLA models. Instead of a human parsing JSON and deciding what to do — the model outputs motor commands directly.

System	Organisation	What it does
RT-2	Google DeepMind	VLA on real robot hardware
OpenVLA	Stanford / Berkeley	Open-source VLA model
π0	Physical Intelligence	Dexterous manipulation VLA
Our M5	This lab	Camera → Gemini → JSON action

Go deeper

Gemini API: Vision capabilities
Google Gemini Cookbook — find the vision examples
RT-2 blog post — most accessible VLA introduction

Concept 11 of 11

The sim-to-real gap

One sentence Policies trained purely in simulation often fail on real hardware because the physics, sensors, and appearance of the real world differ from the simulation.

Three sources

Physics mismatch — simulated friction, contacts, and inertia are approximations. Real motors have backlash, thermal drift.
Sensor noise — real cameras have blur, distortion, lighting variation. Simulated cameras are noiseless.
Appearance gap — MuJoCo renders look nothing like reality. A policy trained on sim visuals won't recognise real scenes.

Concept 11 — Sim-to-real

How to bridge the gap

Domain randomisation Randomise everything during training: lighting, colours, friction, masses. The policy learns to be robust to variation. The real world becomes "just another sample in the distribution."

Sim-to-real transfer Train in simulation (fast, cheap), then fine-tune on real robot data (slow, expensive, but much less needed).

High-fidelity simulation Use better simulators that model real-world physics more accurately. NVIDIA Isaac Sim / Isaac Lab — the industry standard for sim-to-real robotics research, with domain randomisation at scale. Genesis — a newer, fast, differentiable physics engine. Runs significantly faster than MuJoCo for some tasks and supports gradient-based policy optimisation directly through the simulator. Both are the logical next step after this lab.

Go deeper

OpenAI: Learning Dexterity — the definitive sim-to-real example. Read "Our approach."
NVIDIA Isaac Lab docs — where to go after this lab
Genesis physics engine — fast, differentiable physics; the emerging alternative to Isaac Sim

Part 1 — Self-check

Can you answer these without looking?

What is a state vector, and what shape is CartPole's?
What does MediaPipe's LIVE_STREAM mode do differently from synchronous inference?
What happens if the pole in CartPole tilts beyond 12°?
Why does the PPO reward curve stay flat before it climbs?
What does 2x − 1 do, and why is it the right formula for Module 4?
What is the Markov property, and why does it matter for RL?
What is the sim-to-real gap? Name one technique to bridge it.
What's the difference between a VLM and a VLA model?
What is observation design, and why does it matter more than model architecture?
Why does PPO use float32 not float64? What is GAE?
What does render_mode="rgb_array" do, and why does the lab hub need it?

Part 2 of 4

The Lab

Six modules. Every one builds one stage of the pipeline. By Module 4, your hand controls a robot arm in real time. That's the point.

Module	Duration	What you build
M0 — Kickoff	15 min	Draw the pipeline. Print your first state vector.
M1 — Perception	30 min	Webcam → 21 landmarks → 4 joint angles → state vector
M2 — Simulation	20 min	Step environments, explore Reacher's observation space
M3 — RL	30 min	Watch PPO learn CartPole live. Reward curve climbs.
M4 — Loop closes	40 min	Your hand controls the robot. The loop is closed.
M5 — Foundation Models	25 min	Gemini sees the camera and outputs robot actions
M6 — Wrap-up	15 min	Map everything to real-world research tools

Module 0 · Kickoff

The pipeline. Everything builds toward this. 15 min

We're not building from scratch. We're wiring together pre-trained pieces. The glue code is what you're writing today.

modules/00_kickoff/overview.ipynb   # kernel: Physical AI Lab — run all cells

Core concept — State Vector A fixed-length numpy float32 array encoding everything the agent needs to decide its next action. Module 1 produces one. CartPole has one (4 elements). Reacher has one (11 elements). Module 4 maps one into another. It is the common format of the pipeline.

M0 — Exercise

Print your first state vector

import numpy as np
print(np.array([145.2, 163.8, 171.4, 158.0], dtype=np.float32))
# [145.2 163.8 171.4 158.0]
# That printed line is a state vector.

What these numbers mean Four joint angles, in degrees. Index finger MCP, middle MCP, ring MCP, thumb MCP. This is all the "intelligence" the robot needs to start — four floats.

Allow ~3 minutes. Then draw the pipeline on the whiteboard and name the module for each stage.

Module 1 · Perception

921,600 numbers. We need four. 30 min

Your webcam sees 921,600 numbers per frame. MediaPipe compresses that to 21 landmarks. The angle formula compresses that to 4 joint angles. That's your state vector.

python modules/01_perception/01_webcam_basics.py
# Prints frame shape: (480, 640, 3)

python modules/01_perception/02_hand_tracking.py
# 21 green dots appear. Hold your hand up.

python modules/01_perception/03_joint_angles.py
# Prints [145.2, 163.8, 171.4] once per second
# "That is a state vector."

The MediaPipe Tasks API We use mediapipe.tasks.vision.HandLandmarker. The older mp.solutions.hands still appears in most tutorials but is deprecated. Key: coordinates already normalised to [0,1].

Fallback If camera unavailable, fallback_hand_demo.mp4 loops silently. Code path is identical. Don't spend time on this.

M1 — Hand landmarks

21 landmarks. Normalised to [0, 1].

Coordinates already normalised to [0,1] — no pixel division needed.

M1 — Angle formula

The dot-product angle

Exercise — add ring finger In 03_joint_angles.py, add landmarks 13→14 to produce a 4-element state vector. Allow ~10 min. Ask 1–2 volunteers to share output.

Module 2 · Simulation

We never write the physics. 20 min

CartPole: 4 observations, 2 discrete actions. Reacher: 11 observations, 2 continuous torques. Those torques are what your hand controls in 20 minutes.

python modules/02_simulation/01_gym_intro.py
# Observation space: Box(-4.8, 4.8, (4,), float32)
# Action space: Discrete(2)

python modules/02_simulation/02_mujoco_reacher.py
# Observation space: Box(-inf, inf, (11,), float64)
# Action space: Box(-1.0, 1.0, (2,), float32)

Exercise Print observation_space.low and observation_space.high for Reacher. Understand the bounds before you try to send actions. ~8 min.

Before the session — 03_pretrained_agent.py Run python modules/02_simulation/03_pretrained_agent.py before the lab starts. It loads a pre-trained policy and runs HalfCheetah (a MuJoCo humanoid) for 5 seconds. When participants ask "what does a million training steps look like?" — that video is the answer. Don't run this live; it takes 30 seconds to load and derails the pace.

M2 — Reacher state vector

11 observations. You'll control 2 of them.

In Module 4, we bypass the RL agent — we compute the action directly from the hand landmark position.

Module 3 · Reinforcement Learning

It starts random. Watch it figure it out. 30 min

python modules/03_rl/02_train_cartpole.py
# ~2–3 min. Keep matplotlib window visible.
# Walk the MDP cycle diagram while it trains.
# STOP TALKING when the curve climbs — let everyone watch.

What PPO does in plain English Collect experience → estimate which actions led to better outcomes → update policy weights → clip the update so it doesn't change too drastically. Most stable RL algorithm for continuous control. Used in real robot locomotion controllers today.

Exercise Change total_timesteps and re-run. ~10 min. Watch the curve shape change.

M3 — The MDP cycle

State → Policy → Environment → Reward → repeat

Module 4 · Perception → Action

One line of code. The loop closes. 40 min

python modules/04_perception_to_action/01_hand_to_reacher.py
# Two windows: camera + Reacher sim.
# Move your index finger slowly.
# x position → joint 0 torque. y position → joint 1 torque.

The mapping is one line action = 2 * norm_x − 1 That's the entire "intelligence" of the Module 4 controller. Landmark x in [0, 1] → torque in [−1, +1].

Exercise — third torque channel (15 min) Add landmark 4 (thumb tip) as a third torque channel. ActionSliceWrapper is provided. It wraps Reacher's 2-element action space so you can pass a 3-element array and it slices the first two elements for the actual env.step() call — your extra channel controls a visual indicator without breaking the environment interface.

M4 — The mapping

Landmark x → robot torque

Why this is imitation learning data collection A human drives via hand position while (state, action) pairs could be recorded. Those recordings become the training dataset for Behaviour Cloning, ACT, or Diffusion Policy. The teleoperation loop you built today is a data collection rig.

Module 5 · Foundation Models

We swapped the hand for a language model. 25 min

python modules/05_foundation_models/01_gemini_vision.py
# Captures one frame, sends "Describe this scene." ~2 sec.

python modules/05_foundation_models/02_gemini_robot_brain.py
# "Calling Gemini..." → "Done." → {"action":"FORWARD","reason":"..."}

Exercise — prompt engineering (10 min) Write your own MY_PROMPT. Different prompts → different responses. How you phrase the question dramatically changes reliability of the output.

No Gemini API key? All scripts fall back to cached_responses.json automatically. Exercise still works.

The fallback cache is a real production pattern Network calls to cloud models introduce latency, rate limits, and failure modes. Production robot systems always have a local fallback — typically a smaller on-device model. Our cached_responses.json is the simplified version of this: if the API is unavailable, the robot keeps operating from pre-computed responses rather than stopping. Offline-first is a hard requirement in physical systems.

M5 — VLM pipeline

Camera → Gemini → JSON action

The markdown fence stripping before json.loads is necessary — models often wrap JSON in ``` blocks even when asked not to.

Module 6 · Wrap-up

This is what researchers actually use. 15 min

Module	Pipeline stage	Real-world equivalent
M1 — Perception	Webcam → State Vector	Industrial vision systems; MediaPipe in production robot demos
M2 — Simulation	State → Action → World	MuJoCo used in Boston Dynamics, DeepMind, academic labs
M3 — RL	Training the policy	PPO in real locomotion controllers and game-playing agents
M4 — Closed loop	Perception to Action	Teleoperation data collection for BC / ACT / Diffusion Policy
M5 — Foundation Models	Reasoning layer	RT-2 (Google), OpenVLA (Stanford/Berkeley), π0 (Physical Intelligence)

M6 — The sim-to-real gap

Every policy you trained today lives in simulation.

Three sources of failure on real hardware (1) Physics mismatch — friction, inertia, contacts are approximations. (2) Sensor noise — real cameras have blur, distortion, lighting variation. (3) Appearance gap — MuJoCo renders look nothing like real scenes.

Domain randomisation Randomise physics parameters during training. Real world becomes "just another sample."

Where to go next NVIDIA Isaac Lab — high-fidelity simulation. Hugging Face LeRobot — real robot imitation learning.

Part 3 of 4

After Today

Nine independent projects. Every one has a closed feedback loop — that's the Physical AI criterion. None are theory-only. Pick one from your hardware tier and start with the perception layer.

The Physical AI test Every project below fails or becomes meaningless if the feedback loop is removed. A classifier that labels a photo is not Physical AI. An agent whose output changes what it perceives next is.

Projects 1–6 · Laptop only

No extra hardware needed

#	Project	Level	Extends
1	Adaptive Posture Coach — Webcam detects shoulder/neck landmarks. Agent adapts alert timing from your correction history — not a fixed threshold.	Beginner	M1 · M3
2	Sim-to-Sim Transfer — Train PPO on CartPole, deploy into modified env (heavier pole, wind noise). Measure degradation. Apply domain randomisation.	Intermediate	M2 · M3
3	Behaviour Cloning Agent — Use hand control (M4) to play a Pygame environment. Record (obs, action) pairs. Train an MLP clone. Observe covariate shift.	Intermediate	M1 · M4 · M5
4	Sign Language Recogniser + Corrective Loop — Real-time letter recognition from landmarks. Model confidence feeds back to prompt the user to re-sign ambiguous letters.	Beginner–Int	M1 · M5
5	PID Tuner via RL — Simulated pendulum or water level. Use RL to learn PID gain parameters as the action space. Compare learned vs hand-tuned.	Intermediate	M2 · M3
6	Multi-Step VLA Planner — Give Gemini a verbal instruction. It decomposes into sub-goals. Each executed via the M4 pipeline. Chain 3+ steps.	Intermediate	M4 · M5

Projects 7–9 · Pi + servo

With a Raspberry Pi and two SG90 servos

~£20 total hardware. Projects 7–8 are the best introductions to real-world Physical AI.

#	Project	Level
7	Gaze-Tracked Servo Turret — Pi Camera + MediaPipe detects face or hand. Two SG90 servos pan/tilt to track. Apply the M4 mapping (2x−1) directly to servo angle. Servo moves → camera points somewhere new → perception updates → next command. Physically broken without the loop.	Intermediate
8	Imitation Learning Arm — 2-servo arm on Pi. Record (landmark state, servo angle) demonstrations. Train small MLP to replicate. Deploy. Document where it drifts. Covariate shift is physically observable — the arm drifts into states it never saw. The failure mode is the educational content.	Int–Advanced
9	RL Balancer — Train PPO on MuJoCo pendulum. Deploy onto physical pendulum with Pi + servo + encoder. Measure the sim-to-real gap quantitatively. 4–6 week project. This is what labs like Tedrake's at MIT study at scale.	Advanced

After today — next steps

The tools, the platforms, the papers

Simulation & training

NVIDIA Isaac Lab — high-fidelity simulation, domain randomisation at scale
Genesis — fast, differentiable physics simulator
Stable-Baselines3 — what we used today, goes much further

Real robot data + imitation

Hugging Face LeRobot — real robot datasets, ACT and Diffusion Policy
ACT (Action Chunking with Transformers) — imitation learning on real hardware

VLA models to study

RT-2 (Google DeepMind) — the VLA paper that defined the field
OpenVLA (Stanford/Berkeley) — open-source VLA
π0 (Physical Intelligence) — dexterous manipulation

Courses

HuggingFace Deep RL Course — free, units 1–8 directly relevant
Berkeley CS285: Deep RL — the graduate course
OpenAI Spinning Up — hands-on RL from scratch

Part 4 of 4

Quick Reference

Key numbers, demo scripts, exercises at a glance, common issues. Keep this open during the session.

Quick Reference

Key numbers

# Key landmarks
lm 0  = wrist
lm 4  = thumb tip    (M4 exercise)
lm 8  = index tip    (M4 control)
lm 13 = ring MCP     (M1 exercise)
lm 14 = ring PIP     (M1 exercise)

# State shapes
M1 exercise  : (4,)  float32
CartPole obs : (4,)  float32
Reacher obs  : (11,) float64
M4 action    : (2,)  float32

# Training
50k steps ≈ 2–3 min on laptop
CartPole clicks at 20k–40k steps — don't stop early

# CartPole terminal conditions
|pole angle| > 12°  → episode ends
|cart pos|  > 2.4m  → episode ends
max reward  = 500   (500 steps)

# Reacher action space
torque ∈ [−1, +1] for each joint

# M4 mapping
action = 2 * landmark_x − 1

# MDP components
S  = state space
A  = action space
R  = reward function
T  = transition function
γ  = discount factor (0.99)

Quick Reference

Demo scripts

# Start the hub
.venv\Scripts\activate
python api/server.py
# → http://localhost:8000

# Verify install
python verify_install.py
# All 14 should be PASS or WARN

# M1 — Perception
python modules/01_perception/01_webcam_basics.py
python modules/01_perception/02_hand_tracking.py
python modules/01_perception/03_joint_angles.py

# M2 — Simulation
python modules/02_simulation/01_gym_intro.py
python modules/02_simulation/02_mujoco_reacher.py

# M3 — RL
python modules/03_rl/02_train_cartpole.py

# M4 — Loop closes
python modules/04_perception_to_action/01_hand_to_reacher.py

# M5 — Foundation Models
python modules/05_foundation_models/01_gemini_vision.py
python modules/05_foundation_models/02_gemini_robot_brain.py

# M6 — Wrap-up
modules/06_wrapup/concepts_map.ipynb

Quick Reference

Exercises at a glance

Module	Exercise	Time
M0	Print a state vector: `np.array([145.2, 163.8, 171.4, 158.0], dtype=np.float32)`	3 min
M1	Add ring finger angle (landmarks 13→14) to make a 4-element vector	10 min
M2	Print `observation_space.low` and `observation_space.high` for Reacher	8 min
M3	Change `total_timesteps` and re-run. Watch the curve shape change.	10 min
M4	Add landmark 4 (thumb tip) as a 3rd torque channel via `ActionSliceWrapper`	15 min
M5	Write your own `MY_PROMPT`. Observe how phrasing changes output reliability.	10 min

Quick Reference

Common issues

Symptom	Fix
MediaPipe window black	`assets/hand_landmarker.task` missing — re-run setup.bat
MuJoCo window doesn't open	Normal — render fallback active. Check terminal for rgb_array message.
Reward curve never climbs	Let it run — CartPole clicks at 20k–40k steps. Don't stop early.
Port 8000 in use	`netstat -ano \| findstr :8000` → `taskkill /PID [n] /F`
Hub page blank	Chrome or Edge — not Firefox.
Gemini 429 rate limit	Cache fallback activates automatically. Continue.
ARM64 / mujoco fails	WSL2: `wsl --install` (restart) → run setup.sh inside Ubuntu
[WARN] Camera unavailable	Fallback video looping — code path identical. Not a problem.

Quick Reference

Fallback behaviours

What fails	What happens automatically
Webcam unavailable	Loops `fallback_hand_demo.mp4` silently — code path identical
OpenGL / display	`make_env` tries "human" → "rgb_array" → None
Gemini key missing	Random entry from `cached_responses.json`
Gemini API call fails	Any exception → cache fallback
No internet during session	All assets pre-bundled — everything works

Philosophy Every hardware/network dependency has a transparent fallback. When you see [WARN], say: "That's the fallback activating — your code path is identical."

The loop is closed.

You've touched every piece of the modern Physical AI stack — perception, simulation, reinforcement learning, the closed loop, foundation model reasoning. Clone the repo. Take it home. Keep building.

GitHub Repository → Jim Seelan on LinkedIn →

Move your finger.A robot arm follows.

This is basically how every robot works. Look → think → move. Repeat.

Two modes. One file.

Presentation mode

Handbook / self-study

Concept Foundations

What is Physical AI?

Static AI

Physical AI

The analogy

From pixels to numbers

What a frame actually is

The compression pipeline

MediaPipe & landmarks

What "pre-trained" means

Normalised coordinates

LIVE_STREAM mode — why async

State vectors

Why fixed size?

What goes in it?

State vectors in this lab

Gymnasium & MuJoCo

Why simulate at all?

What Gymnasium provides

The episode loop

Reinforcement Learning

The core loop

RL vs supervised learning

Markov Decision Processes

The Markov property — why it matters

PPO — Proximal Policy Optimisation

The problem it solves

Steps per update

The reward curve

Closing the loop

The exact loop you build in Module 4

Foundation Models & VLMs

Three characteristics

VLM — Vision-Language Model

VLA — Vision-Language-Action models

The sim-to-real gap

Three sources

How to bridge the gap

Can you answer these without looking?

The Lab

The pipeline. Everything builds toward this. 15 min

Print your first state vector

921,600 numbers. We need four. 30 min

21 landmarks. Normalised to [0, 1].

The dot-product angle

We never write the physics. 20 min

11 observations. You'll control 2 of them.

It starts random. Watch it figure it out. 30 min

State → Policy → Environment → Reward → repeat

One line of code. The loop closes. 40 min

Landmark x → robot torque

We swapped the hand for a language model. 25 min

Camera → Gemini → JSON action

This is what researchers actually use. 15 min

Every policy you trained today lives in simulation.

After Today

No extra hardware needed

With a Raspberry Pi and two SG90 servos

The tools, the platforms, the papers

Simulation & training

Real robot data + imitation

VLA models to study

Courses

Quick Reference

Key numbers

Demo scripts

Exercises at a glance

Common issues

Fallback behaviours

The loop is closed.

Move your finger.
A robot arm follows.

This is basically how every robot works.
Look → think → move. Repeat.