AI glossary · Basic · Efficiency & infra
TPU
A TPU (Tensor Processing Unit) is an AI-specific chip that Google designed in-house to speed up deep learning computation, especially matrix multiplication.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
Most deep learning computation consists of multiplying large matrices together. A TPU puts a large circuit for matrix multiplication at the center of the chip so it can do more matrix operations with the same power. Large models are trained on 'Pods', units that link many TPU chips through a dedicated interconnect. Programs are converted to fit the TPU and run by a compiler called XLA.
Where you'll see it
TPUs are not a product you buy separately; they are mostly rented on Google Cloud. You can find them in Google Cloud price lists and instance lists, and in the runtime selection menu of Google Colab. On the code side, JAX is the most natural fit, and TensorFlow and PyTorch (via XLA integration) can also be used. Research papers sometimes include a line such as 'trained on TPUs' when describing their experimental setup.
Why it matters
It is one of the main options besides NVIDIA GPUs for training and serving large-scale AI. Google is known to train and serve its own models on TPUs. It is also a leading example of big tech companies building their own AI chips. It is considered as an alternative when GPU supply is tight.
Things to know
Code written in CUDA and CUDA-only kernels do not run on TPUs as is. Open-source inference engines and new techniques often come out for GPUs first, so support on TPUs has to be checked separately. For performance comparisons, it is more realistic to look at how much of the same workload gets done at the same cost than to compare single chips. It is also worth considering that usage is effectively tied to Google Cloud.
GPU vs. TPU
GPU(NVIDIA)General-purpose parallel chip
- Broad CUDA ecosystem
- Available on many clouds or for purchase
- New techniques tend to be supported first
TPU (Google)AI-specific chip
- Specialized for matrix multiplication
- Mostly used on Google Cloud
- Works well with JAX and XLA
공통 Both are used for large-scale model training and inference
Example
When a startup chooses infrastructure for model training and has already been writing its code in JAX, it adds TPUs on Google Cloud to its shortlist. By contrast, a service that relies on PyTorch and a CUDA-only inference engine may weigh the burden of changing its code and choose to stay on a GPU cloud.