{{Short description|Hardware acceleration unit for artificial intelligence tasks}} {{Use American English|date=January 2019}} {{Use mdy dates|date=October 2021}} [[File:Raspberry Pi 5 Hailo AI Accelerator Module.jpg|thumb|A Hailo AI Accelerator Module attached to a Raspberry Pi 5 via an M.2 adapter hat (2024)]]
A '''neural processing unit''' ('''NPU'''), also known as an '''AI accelerator''' or '''deep learning processor''', is a class of specialized hardware accelerator<ref>{{cite web |url=https://www.v3.co.uk/v3-uk/news/3014293/intel-unveils-movidius-compute-stick-usb-ai-accelerator |title=Intel unveils Movidius Compute Stick USB AI Accelerator |date=July 21, 2017 |access-date=August 11, 2017 |archive-url=https://web.archive.org/web/20170811193632/https://www.v3.co.uk/v3-uk/news/3014293/intel-unveils-movidius-compute-stick-usb-ai-accelerator |archive-date=August 11, 2017 }}</ref> or computer system<ref>{{cite web |url=https://insidehpc.com/2017/06/inspurs-unveils-gx4-ai-accelerator/ |title=Inspurs unveils GX4 AI Accelerator |date=June 21, 2017}}</ref><ref>{{citation |title=Neural Magic raises $15 million to boost AI inferencing speed on off-the-shelf processors |last=Wiggers |first=Kyle |date=November 6, 2019 |url=https://venturebeat.com/2019/11/06/neural-magic-raises-15-million-to-boost-ai-training-speed-on-off-the-shelf-processors/ |publication-date=November 6, 2019 |orig-date=2019 |archive-url=https://web.archive.org/web/20200306120524/https://venturebeat.com/2019/11/06/neural-magic-raises-15-million-to-boost-ai-training-speed-on-off-the-shelf-processors/ |archive-date=March 6, 2020 |access-date=March 14, 2020}}</ref> designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and computer vision.
==Use== Their purpose is either to efficiently execute already trained AI models (inference) or to train AI models. NPUs can be more efficient in terms of speed or power consumption.
NPU applications include algorithms for robotics, Internet of things, and data-intensive or sensor-driven tasks.<ref>{{cite web |url=https://www.eetimes.com/google-designing-ai-processors/ |title=Google Designing AI Processors|date=May 18, 2016 }} Google using its own AI accelerators.</ref> They are often manycore or spatial designs and focus on low-precision arithmetic, novel dataflow architectures, or in-memory computing capability. {{As of|2024}}, a widely used datacenter-grade AI integrated circuit chip, the Nvidia H100 GPU, contains tens of billions of MOSFETs.<ref>{{cite web|url=https://www.datacenterdynamics.com/en/news/nvidia-reveals-new-hopper-h100-gpu-with-80-billion-transistors/|title=Nvidia reveals new Hopper H100 GPU, with 80 billion transistors|last=Moss|first=Sebastian|date=2022-03-23|website=Data Center Dynamics|access-date=2024-01-30}}</ref>
=== Consumer devices === AI accelerators are used in Apple silicon, Qualcomm, Samsung, Huawei,<ref>{{Cite web|url=https://consumer.huawei.com/en/press/news/2017/ifa2017-kirin970|title=HUAWEI Reveals the Future of Mobile AI at IFA|access-date=January 28, 2024|archive-date=November 10, 2021|archive-url=https://web.archive.org/web/20211110131401/https://consumer.huawei.com/en/press/news/2017/ifa2017-kirin970/|url-status=dead}}</ref> and Google Tensor smartphone processors.<ref>{{Cite web| title=Snapdragon 8 Gen 3 mobile platform | url=https://docs.qualcomm.com/bundle/publicresource/87-71408-1_REV_B_Snapdragon_8_gen_3_Mobile_Platform_Product_Brief.pdf | archive-url=https://web.archive.org/web/20231025162610/https://docs.qualcomm.com/bundle/publicresource/87-71408-1_REV_B_Snapdragon_8_gen_3_Mobile_Platform_Product_Brief.pdf | archive-date=2023-10-25}}</ref>
Vision processing units are accelerators specialized for machine vision algorithms such as CNN (convolutional neural networks) and SIFT (scale-invariant feature transform). They are used in devices that need to keep track of objects visually such as AR headsets and drones.<ref name="RiseOfVPUs">{{cite web|last1=Weckler|first1=Adrian|title=Dublin tech firm Movidius to power Google's new virtual reality headset|url=http://www.independent.ie/business/technology/news/dublin-tech-firm-movidius-to-power-googles-new-virtual-reality-headset-34449883.html|website=Independent.ie|date=14 February 2016 |access-date=15 March 2016}}</ref><ref>{{cite web|url=https://www.movidius.com/news/dji-brings-two-new-flagship-drones-to-lineup-featuring-myriad-2-vpus|title=DJI Brings Two New Flagship Drones to Lineup Featuring Myriad 2 VPUs - Machine Vision Technology - Movidius|website=www.movidius.com}}</ref><ref>{{cite web|url=http://www.pcworld.com/article/2917512/microsoft-designed-a-special-processor-to-handle-hololens-data.html|title=Microsoft dives deeper into HoloLens details: 'Holographic processor' role revealed|date=May 1, 2015|author=Fred O'Connor|work=PCWorld}}</ref>
It is more recently (circa 2017) added to processors from Apple<ref>{{Cite web|title=iPhone X Press Release|url=https://www.apple.com/newsroom/2017/09/the-future-is-here-iphone-x/|website=Apple|date=September 2017}}</ref> and (circa 2022) to processors from Intel<ref>{{Cite web|url=https://www.intel.com/content/www/us/en/newsroom/news/intels-lunar-lake-processors-arriving-q3-2024.html|title=Intel's Lunar Lake Processors Arriving Q3 2024|website=Intel|date=May 20, 2024 }}</ref> and AMD.<ref>{{cite web|title=AMD XDNA Architecture|url=https://www.amd.com/en/technologies/xdna.html}}</ref> All models of Intel Meteor Lake processors have a built-in ''versatile processor unit'' (''VPU'') for accelerating inference for computer vision and deep learning.<ref>{{Cite web|url=https://www.pcmag.com/news/intel-to-bring-a-vpu-processor-unit-to-14th-gen-meteor-lake-chips|title=Intel to Bring a 'VPU' Processor Unit to 14th Gen Meteor Lake Chips|website=PCMAG|date=August 2022 }}</ref>
{{Anchor|TOPS}}On consumer devices, the NPU is intended to be small, power-efficient, but reasonably fast when used to run small models. To do this they are designed to support low-bitwidth operations using data types such as INT4, INT8, FP8, and FP16. A common metric is trillions of operations per second (TOPS). Although TOPS does not explicitly specify the kind of operations, it is typically INT8 additions and multiplications.<ref>{{cite web |title=A guide to AI TOPS and NPU performance metrics |url=https://www.qualcomm.com/news/onq/2024/04/a-guide-to-ai-tops-and-npu-performance-metrics}}</ref>
=== Datacenters === [[File:TPU v4.png|thumb|The Google Tensor Processing Unit (TPU) v4 package (ASIC in center plus 4 HBM stacks) and printed circuit board (PCB) with 4 liquid-cooled packages; the board's front panel has 4 top-side PCIe connectors (2023).]]Accelerators are used in cloud computing servers: e.g., tensor processing units (TPU) for Google Cloud Platform,<ref>{{Cite journal|date=2017-06-24|title=In-Datacenter Performance Analysis of a Tensor Processing Unit|journal=ACM SIGARCH Computer Architecture News|volume=45|issue=2|pages=1–12|language=EN|doi=10.1145/3140659.3080246|doi-access=free |last1=Jouppi |first1=Norman P. |last2=Young |first2=Cliff |last3=Patil |first3=Nishant |last4=Patterson |first4=David |last5=Agrawal |first5=Gaurav |last6=Bajwa |first6=Raminder |last7=Bates |first7=Sarah |last8=Bhatia |first8=Suresh |last9=Boden |first9=Nan |last10=Borchers |first10=Al |last11=Boyle |first11=Rick |last12=Cantin |first12=Pierre-luc |last13=Chao |first13=Clifford |last14=Clark |first14=Chris |last15=Coriell |first15=Jeremy |last16=Daley |first16=Mike |last17=Dau |first17=Matt |last18=Dean |first18=Jeffrey |last19=Gelb |first19=Ben |last20=Ghaemmaghami |first20=Tara Vazir |last21=Gottipati |first21=Rajendra |last22=Gulland |first22=William |last23=Hagmann |first23=Robert |last24=Ho |first24=C. Richard |last25=Hogberg |first25=Doug |last26=Hu |first26=John |last27=Hundt |first27=Robert |last28=Hurt |first28=Dan |last29=Ibarz |first29=Julian |last30=Jaffey |first30=Aaron |display-authors=1 |arxiv=1704.04760 }}</ref> and Trainium and Inferentia chips for Amazon Web Services.<ref>{{cite web | title = How silicon innovation became the 'secret sauce' behind AWS's success| website = Amazon Science| date = July 27, 2022| url = https://www.amazon.science/how-silicon-innovation-became-the-secret-sauce-behind-awss-success| access-date = July 19, 2024}}</ref> Many vendor-specific terms exist for devices in this category, and it is an emerging technology without a dominant design.
Since the late 2010s, graphics processing units designed by companies such as Nvidia and AMD often include AI-specific hardware in the form of dedicated functional units for low-precision matrix-multiplication operations. These GPUs are commonly used as AI accelerators, both for training and inference.<ref>{{cite web| last1 = Patel| first1 = Dylan| last2 = Nishball| first2 = Daniel| last3 = Xie| first3 = Myron| title = Nvidia's New China AI Chips Circumvent US Restrictions| url=https://www.semianalysis.com/p/nvidias-new-china-ai-chips-circumvent| website = SemiAnalysis| date=2023-11-09| access-date=2024-02-07}}</ref>
=== Scientific computation === Although NPUs are tailored for low-precision (e.g., FP16, INT8) matrix multiplication operations, they can be used to emulate higher-precision matrix multiplications in scientific computing. As modern GPUs place much focus on making the NPU part fast, using emulated FP64 (Ozaki scheme) on NPUs can potentially outperform native FP64. This has been demonstrated using FP16-emulated FP64 on NVIDIA TITAN RTX and using INT8-emulated FP64 on NVIDIA consumer GPUs and the A100 GPU. Consumer GPUs especially benefited as they have limited FP64 hardware capacity, showing a 6× speedup.<ref>{{cite journal |last1=Ootomo |first1=Hiroyuki |last2=Ozaki |first2=Katsuhisa |last3=Yokota |first3=Rio |title=DGEMM on integer matrix multiplication unit |journal=The International Journal of High Performance Computing Applications |date=July 2024 |volume=38 |issue=4 |pages=297–313 |doi=10.1177/10943420241239588 |arxiv=2306.11975}}</ref> Since CUDA Toolkit 13.0 Update 2, cuBLAS automatically uses INT8-emulated FP64 matrix multiplication of the equivalent precision if it is faster than native. This is in addition to the FP16-emulated FP32 feature introduced in version 12.9.<ref>{{cite web |title=Unlocking Tensor Core Performance with Floating Point Emulation in cuBLAS |url=https://developer.nvidia.com/blog/unlocking-tensor-core-performance-with-floating-point-emulation-in-cublas/ |website=NVIDIA Technical Blog |date=24 October 2025}}</ref>
== Programming == An operating system or a higher-level library may provide application programming interfaces such as TensorFlow with LiteRT Next (Android), CoreML (iOS, macOS) or DirectML (Windows). Formats such as ONNX are used to represent trained neural networks.
Consumer CPU-integrated NPUs are accessible through vendor-specific APIs. AMD (Ryzen AI), Intel (OpenVINO), Apple silicon (CoreML),{{efn|MLX builds atop the CPU and GPU parts, not the Apple Neural Engine (ANE) part of Apple Silicon chips. The relatively good performance is due to the use of a large, fast unified memory design.}} and Qualcomm (SNPE) each have their own APIs, which can be built upon by a higher-level library.
GPUs generally use existing GPGPU pipelines such as CUDA and OpenCL adapted for lower precisions and specialized matrix-multiplication operations. Vulkan is also being used. Custom-built systems such as the Google TPU use private interfaces.
There are a large number of separate underlying acceleration APIs and compilers/runtimes in use in the AI field, causing a great increase in software development effort due to the many combinations involved. As of 2025, the open standard organization Khronos Group is pursuing standardization of AI-related interfaces to reduce the amount of work needed. Khronos is working on three separate fronts: expansion of data types and intrinsic operations in OpenCL and Vulkan, inclusion of compute graphs in SPIR-V, and a NNEF/SkriptND file format for describing a neural network.<ref>{{cite conference|last1=Tavenrath |first1=Markus |title=Current Status of AI-related Standardization in the Khronos Group |url=https://www.khronos.org/assets/uploads/developers/presentations/Current_Status_of_AI-related_Standardization_in_the_Khronos_Group_GISC25.pdf|conference=Global ICT Standards Conference 2025|date=2025}}</ref>
==Notes== {{notelist}}
== See also == * Wetware computer
== References == {{Reflist}}
== External links == *[https://www.nextplatform.com/2016/04/05/nvidia-puts-accelerator-metal-pascal/ Nvidia Puts The Accelerator To The Metal With Pascal], The Next Platform *[http://eyeriss.mit.edu/ Eyeriss Project], Massachusetts Institute of Technology
{{Hardware acceleration}} Category:Application-specific integrated circuits Category:Neural processing units Category:Coprocessors Category:Computer optimization Category:Gate arrays Category:Deep learning