A Raspberry Pi TPU pairs a CPU-bound single-board computer with a small accelerator — usually a Coral Edge TPU over USB or PCIe. The datasheet promises TOPS; the pipeline delivers whatever the slowest stage allows. That gap is the whole story, and most people building edge inference on a Pi walk straight into it. The naive reading goes like this: the Coral Edge TPU is rated at roughly 4 TOPS (per Google’s published Coral specifications), so a quantised MobileNet or EfficientDet should run at accelerator speed and the Pi becomes an inference box. The rating is real. The conclusion is wrong. On a Pi, the CPU still owns frame capture, colour conversion, resize, normalisation, and the copy across the USB or PCIe boundary to the TPU. The accelerator only owns the matrix-multiply-heavy body of a quantised model. If the CPU stage is slow, the TPU sits idle between frames waiting for data — and idle silicon is not fast silicon. What does a Raspberry Pi TPU actually do, and where is the handoff? Strip away the marketing and a Pi TPU setup is a two-processor pipeline with a hard boundary in the middle. The Pi’s CPU — an Arm core, on a Pi 4 or Pi 5 — handles everything up to and after the model. The Edge TPU handles the model’s quantised operations. Between them is a transport: USB 3.0 on the Coral USB Accelerator, or PCIe on the Coral M.2 / dual Edge TPU boards. The handoff is not free, and it happens at least twice per inference: CPU → TPU: the preprocessed input tensor is copied from host memory across the transport into the accelerator. TPU → CPU: the output tensor (detections, logits, embeddings) is copied back for post-processing. The model only runs between those copies. A common way this surfaces in practice: the TPU finishes a 3-millisecond inference, then waits several milliseconds for the CPU to preprocess the next frame and push it across USB. Rated throughput assumes the accelerator is fed continuously. A real pipeline rarely is. This is the same hardware-acceleration discord that shows up in GPU video pipelines, scaled down to the edge. When a datacentre GPU stalls because CPU-side decode or preprocessing can’t keep pace, you see it as low GPU utilisation; the same idle-accelerator pattern is described in our breakdown of what GPU/CPU stage economics mean for inference pipelines. On a Pi you see it as the TPU sitting idle between frames while the CPU catches up. Different silicon, identical failure class. What runs on the CPU versus the TPU? The split is not a matter of preference — the Edge TPU only executes a specific subset of TensorFlow Lite operations, quantised to 8-bit integers, and compiled by Google’s Edge TPU compiler. Anything the compiler can’t map stays on the CPU. That includes most of what surrounds the model. Here is the honest division of labour for a typical detection pipeline: Stage Runs on Why Frame capture (camera / RTSP decode) CPU I/O and codec work, not a TPU operation Colour convert + resize + normalise CPU Elementwise/reshape ops, often un-accelerated Quantise input tensor CPU Prep for the 8-bit TPU model Host → device copy Transport (USB/PCIe) The boundary cost the datasheet ignores Quantised model body TPU The stage the 4 TOPS rating describes Any unsupported op mid-model CPU (fallback) Forces a round-trip, kills duty cycle Device → host copy Transport Return path NMS / decode / tracking / drawing CPU Control-flow-heavy, stays on host The trap sits in the “unsupported op mid-model” row. If a single operation in the middle of your graph isn’t Edge-TPU-compatible, the compiler splits the model: everything up to that op runs on the TPU, the op runs on the CPU, and everything after it runs on the CPU too — because once you’re back on the host, moving the intermediate tensor to the TPU again rarely pays. A model that looks fully accelerated on paper can run mostly on the Arm core in practice. Checking the compiler’s partition log is the first thing we do when a Pi TPU pipeline underperforms. How does the USB or PCIe copy become the bottleneck? The copy is a latency phenomenon, not just a bandwidth one. A 300×300×3 uint8 input tensor is only about 270 KB — trivial for USB 3.0’s bandwidth. The problem is per-transfer overhead: setup, synchronisation, and the fact that inference can’t start until the copy completes and can’t return until the result copies back. This is the edge-scale version of the host-device latency framing we apply to GPUs, where the copy across the PCIe boundary is treated as a first-class cost rather than an afterthought. Two conditions turn the copy from a rounding error into the dominant cost: Small, frequent inferences. If each inference is only a few milliseconds, a copy of comparable duration roughly halves throughput. The accelerator is fast; the trip to reach it is not. Redundant copies. Preprocessing on the CPU, copying to the TPU, getting a result, then copying data back to the CPU for a second model, then back again — each boundary crossing adds fixed overhead. Pipelines that chain two small models across the transport often lose more time to copies than to compute. The USB Coral makes this worse than the PCIe variants because USB adds protocol overhead and is more sensitive to host controller sharing. On a Pi 4, the USB 3.0 bus is shared, and contention with a camera or network adapter shows up directly as jittery inference latency. The PCIe M.2 Coral on a Pi 5 has a cleaner path, which is why the same model can show a tighter frame-time distribution there — a difference you only see if you profile, not if you read TOPS. Why is a low TPU duty cycle a routing signal, not a buying signal? Here’s the reframe that saves money: when you measure your Edge TPU and find it busy only 30% of the time, the instinct is to conclude you need a bigger accelerator or a second one. Usually you don’t. A low duty cycle means the TPU is waiting, and adding more waiting silicon changes nothing. The bottleneck is upstream — in CPU preprocessing or the copy — and a second accelerator will idle just as much. Read TPU utilisation honestly. Idle gaps between frames are telling you where the boundary is, not how much accelerator you’re short of. The measurable outcome you actually want is frames-per-second-per-watt holding near the accelerator’s rated throughput instead of collapsing to CPU-preprocessing speed. In our experience profiling edge nodes, eliminating redundant host-device copies and trimming CPU preprocessing recovers the TPU’s duty cycle so the node hits its inference target on the board it already has — no second Coral required. (This is an observed pattern across edge engagements, not a benchmarked figure for your specific model and camera.) The economics only survive if you route deliberately. A worked example makes the logic concrete: Assume a 30 fps camera, a quantised detector with a 4 ms TPU inference, and 12 ms of CPU preprocessing per frame. The CPU stage caps the pipeline at roughly 80 fps of preprocessing throughput on paper, but with a single-threaded capture-preprocess-infer loop the stages run serially: 12 + copy + 4 + copy ≈ 20 ms, giving about 50 fps. The TPU is busy 4 ms of every 20 ms — a 20% duty cycle. Buying a second TPU does nothing; the CPU still produces one frame every 12+ ms. Fix the pipeline, not the hardware: overlap preprocessing with inference (a producer-consumer queue so the CPU preps frame N+1 while the TPU runs frame N), drop resolution to what the model needs, and remove any redundant colour conversions. Now the CPU’s 12 ms is the wall, the TPU fills the gaps, and you approach 80 fps on the same board. Which stages belong on the TPU, and which should stay on the CPU? The routing rule is economic, not architectural: a stage belongs on the TPU only if its accelerator speed-up survives the round-trip copy cost. A large convolutional backbone with dense 8-bit matrix multiplies easily clears that bar — the compute dwarfs the copy. A tiny post-processing head, or a model small enough that inference and copy take similar time, often does not. Concretely, keep on the CPU: capture and decode, resize and normalise, non-maximum suppression, tracking, and any control-flow-heavy logic. Route to the TPU: the quantised, dense body of the model where the operation set is fully supported and the compute-to-copy ratio is high. When object detection runs on an edge node, the detection backbone is the TPU’s job while the box decoding and tracking stay on the host — the same division we describe in how object detection works and what it costs on the decode path and in how object tracking software fits a video-analysis pipeline. Whether a Pi TPU is even the right accelerator against a small GPU-based edge box is a separate decision — but the profiling discipline is identical to what we bring to any GPU workload: measure the stage boundary before you size the silicon. How do you profile a Pi TPU pipeline to find the stall? You cannot fix a boundary you haven’t measured. The tooling is modest but sufficient. Instrument three timestamps per frame — preprocessing start, inference start (post host→device copy), and result available (post device→host copy) — and log the deltas. The gap between “inference done for frame N” and “inference start for frame N+1” is your idle time; that number is the routing signal. A pragmatic checklist for diagnosing a stalled Pi TPU: Read the Edge TPU compiler log. Confirm the whole model mapped to the TPU. Any CPU-fallback op mid-graph is the first suspect. Time preprocessing in isolation. Run capture + preprocess with no inference. If it can’t sustain your target fps, the CPU is the wall, full stop. Measure per-inference wall time including copies, not just the model’s compute time. The difference is the transport cost. Compute duty cycle = (inference compute time) ÷ (frame period). Low duty cycle plus a slow preprocessing measurement confirms an upstream bottleneck. Test overlap. Move preprocessing to a separate thread with a queue. If throughput jumps, you were stalling on serial execution, not on accelerator capacity. Check bus contention. On USB Coral, watch for a camera or NIC sharing the controller; the jitter shows up in the frame-time histogram, not the mean. FAQ How does raspberry pi tpu work in practice? A Raspberry Pi TPU setup pairs the Pi’s Arm CPU with a small accelerator — typically a Coral Edge TPU over USB or PCIe — to run quantised inference at the edge. The Pi’s CPU handles capture, preprocessing, and the copy across the transport; the TPU runs only the quantised body of the model. In practice, the accelerator’s rated throughput holds only if the CPU keeps it fed, so real-world speed is set by the slowest stage, not by the TOPS rating. What runs on the Pi’s CPU versus the TPU, and where does the handoff happen? The CPU owns frame capture, colour conversion, resize, normalisation, input quantisation, and all post-processing like NMS and tracking. The Edge TPU owns only the quantised, TPU-compatible operations of the model. The handoff happens across the USB or PCIe boundary — at least once to send the input tensor and once to return the output — and any unsupported operation mid-model forces the graph back onto the CPU. How does the USB or PCIe copy between the Pi and the TPU become a bottleneck? The copy is a latency cost, not just a bandwidth one: inference cannot start until the input arrives and cannot return until the output copies back. When inferences are small and frequent, a copy of comparable duration can roughly halve throughput, and chaining models across the boundary multiplies the fixed per-transfer overhead. USB Coral is more exposed than the PCIe variants because of protocol overhead and host-controller contention. Why can a low TPU duty cycle be a routing signal rather than a reason to add accelerators? A low duty cycle means the TPU is waiting on an upstream stage — CPU preprocessing or the copy — so adding a second accelerator only adds more idle silicon. The idle gaps between frames identify where the boundary is, not how much accelerator you lack. Fixing the pipeline (overlapping preprocessing with inference, trimming redundant copies) usually recovers the duty cycle on the board you already have. Which inference stages benefit from the TPU and which should stay on the CPU? A stage belongs on the TPU only if its speed-up survives the round-trip copy cost — dense, quantised model backbones clear that bar; tiny heads and copy-comparable models often do not. Keep capture, decode, resize, normalisation, NMS, and tracking on the CPU. Route the compute-heavy, fully-supported model body to the TPU. How do you profile a Pi TPU pipeline to find where preprocessing stalls the accelerator? Instrument three timestamps per frame — preprocessing start, inference start after the host-to-device copy, and result available after the return copy — and log the deltas. Read the Edge TPU compiler log to confirm the full model mapped, time preprocessing in isolation, and compute duty cycle as inference compute time over the frame period. A low duty cycle plus slow standalone preprocessing confirms an upstream bottleneck. How does resolving the CPU-accelerator discord change frames-per-second and power efficiency at the edge? Once preprocessing overlaps with inference and redundant copies are removed, the TPU’s duty cycle rises and frames-per-second climbs toward the accelerator’s rated capacity rather than collapsing to CPU-preprocessing speed. Because you avoid a larger board or a second accelerator, frames-per-second-per-watt improves — the edge node hits its inference target on the hardware it already has. The Edge TPU rating tells you how fast the model can run. Whether it does is decided at the CPU boundary — in the preprocessing you didn’t profile and the copies you didn’t count. Before you conclude a Pi is underpowered, measure the stage boundary; the same stage-boundary profiling that surfaces acceleration discord in a GPU video pipeline (the A1 GPU Performance Audit, scoped to edge and accelerator workloads) usually reveals that the TPU was fast all along, just starved.
