# Automatic image annotation

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Automatic_image_annotation
> Markdown URL: https://mediated.wiki/source/Automatic_image_annotation.md
> Source: https://en.wikipedia.org/wiki/Automatic_image_annotation
> Source revision: 1343431511
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

Process which assigns captioning to a digital image

Output of DenseCap "dense captioning" software, analysing a photograph of a man riding an elephant

**Automatic image annotation** (also known as **automatic image tagging** or **linguistic indexing**) is the process by which a computer system automatically assigns [metadata](/source/Metadata) in the form of [captioning](/source/Photo_caption) or [keywords](/source/Index_term) to a [digital image](/source/Digital_image). This application of [computer vision](/source/Computer_vision) techniques is used in [image retrieval](/source/Image_retrieval) systems to organize and locate images of interest from a [database](/source/Database).

This method can be regarded as a type of [multi-class](/source/Multiclass_classification) [image classification](/source/Image_recognition) with a very large number of classes - as large as the vocabulary size. Typically, [image analysis](/source/Image_analysis) in the form of extracted [feature vectors](/source/Feature_vector) and the training annotation words are used by [machine learning](/source/Machine_learning) techniques to attempt to automatically apply annotations to new images.[1] The first methods learned the correlations between [image features](/source/Feature_(computer_vision)) and training annotations. Subsequently, techniques were developed using [machine translation](/source/Machine_translation) to attempt to translate the textual vocabulary into the 'visual vocabulary,' represented by clustered regions known as *blobs.* Subsequent work has included classification approaches, relevance models, and other related methods.

The advantages of automatic image annotation versus [content-based image retrieval](/source/Content-based_image_retrieval) (CBIR) are that queries can be more naturally specified by the user.[2] At present, Content-Based Image Retrieval (CBIR) generally requires users to search by image concepts such as color and [texture](/source/Texture_(visual_arts)) or by finding example queries. However, certain image features in example images may override the concept that the user is truly focusing on. Traditional methods of image retrieval, such as those used by libraries, have relied on manually annotated images, which is expensive and time-consuming, especially given the large and constantly growing image databases in existence.

## See also

- [Content-based image retrieval](/source/Content-based_image_retrieval)

- [Object categorization from image search](/source/Object_categorization_from_image_search)

- [Object detection](/source/Object_detection)

- [Outline of object recognition](/source/Outline_of_object_recognition)

## References

1. **[^](#cite_ref-1)** Barrat, Sabine; Tabbone, Salvatore (2010-05-01). ["Modeling, classifying and annotating weakly annotated images using Bayesian network"](https://www.sciencedirect.com/science/article/abs/pii/S1047320310000301). *Journal of Visual Communication and Image Representation*. **21** (4): 355–363. [doi](/source/Doi_(identifier)):[10.1016/j.jvcir.2010.02.010](https://doi.org/10.1016%2Fj.jvcir.2010.02.010). [ISSN](/source/ISSN_(identifier)) [1047-3203](https://search.worldcat.org/issn/1047-3203).

1. **[^](#cite_ref-2)** ["Archived copy"](https://web.archive.org/web/20140808134447/http://i.yz.yamagata-u.ac.jp/paper/inoue04irix.pdf) (PDF). *i.yz.yamagata-u.ac.jp*. Archived from [the original](http://i.yz.yamagata-u.ac.jp/paper/inoue04irix.pdf) (PDF) on 8 August 2014. Retrieved 13 January 2022.{{[cite web](https://en.wikipedia.org/wiki/Template:Cite_web)}}: CS1 maint: archived copy as title ([link](https://en.wikipedia.org/wiki/Category:CS1_maint:_archived_copy_as_title))

- Datta, Ritendra; Dhiraj Joshi; [Jia Li](/source/Jia_Li); James Z. Wang (2008). ["Image Retrieval: Ideas, Influences, and Trends of the New Age"](http://infolab.stanford.edu/~wangz/project/imsearch/review/JOUR/). *ACM Computing Surveys*. **40** (2): 1–60. [doi](/source/Doi_(identifier)):[10.1145/1348246.1348248](https://doi.org/10.1145%2F1348246.1348248). [S2CID](/source/S2CID_(identifier)) [7060187](https://api.semanticscholar.org/CorpusID:7060187).

- Nicolas Hervé; Nozha Boujemaa (2007). ["Image annotation : which approach for realistic databases ?"](https://web.archive.org/web/20110520140240/http://www-rocq.inria.fr/~nherve/nherve_civr2007.pdf) (PDF). *ACM International Conference on Image and Video Retrieval*. Archived from [the original](http://www-rocq.inria.fr/~nherve/nherve_civr2007.pdf) (PDF) on 2011-05-20.

- M Inoue (2004). ["On the need for annotation-based image retrieval"](https://web.archive.org/web/20140808134447/http://i.yz.yamagata-u.ac.jp/paper/inoue04irix.pdf) (PDF). *Workshop on Information Retrieval in Context*. pp. 44–46. Archived from [the original](http://i.yz.yamagata-u.ac.jp/paper/inoue04irix.pdf) (PDF) on 2014-08-08.

## Further reading

This section may require cleanup to meet Wikipedia's quality standards. The specific problem is: long and multiline format. Please help improve this section if you can. (November 2022) (Learn how and when to remove this message)

- Word co-occurrence model

- Y Mori; H Takahashi & R Oka (1999). "Image-to-word transformation based on dividing and vector quantizing images with words.". *Proceedings of the International Workshop on Multimedia Intelligent Storage and Retrieval Management*. [CiteSeerX](/source/CiteSeerX_(identifier)) [10.1.1.31.1704](https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.1704).

- Annotation as machine translation

- P Duygulu; K Barnard; N de Fretias & D Forsyth (2002). ["Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary"](https://web.archive.org/web/20050305174408/http://vision.cs.arizona.edu/kobus/research/publications/ECCV-02-1/). *Proceedings of the European Conference on Computer Vision*. pp. 97–112. Archived from [the original](http://vision.cs.arizona.edu/kobus/research/publications/ECCV-02-1/) on 2005-03-05.

- Statistical models

- J Li & J Z Wang (2006). ["Real-time Computerized Annotation of Pictures"](http://www-db.stanford.edu/~wangz/project/imsearch/ALIP/ACMMM06/). *Proc. ACM Multimedia*. pp. 911–920.

- J Z Wang & J Li (2002). ["Learning-Based Linguistic Indexing of Pictures with 2-D MHMMs"](http://www-db.stanford.edu/~wangz/project/imsearch/ALIP/ACM02/). *Proc. ACM Multimedia*. pp. 436–445.

- Automatic linguistic indexing of pictures

- J Li & J Z Wang (2008). ["Real-time Computerized Annotation of Pictures"](http://infolab.stanford.edu/~wangz/project/imsearch/ALIP/PAMI08/). *IEEE Transactions on Pattern Analysis and Machine Intelligence*.

- J Li & J Z Wang (2003). ["Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach"](http://www-db.stanford.edu/~wangz/project/imsearch/ALIP/PAMI03/). *IEEE Transactions on Pattern Analysis and Machine Intelligence*. pp. 1075–1088.

- Hierarchical Aspect Cluster Model

- K Barnard; D A Forsyth (2001). ["Learning the Semantics of Words and Pictures"](https://web.archive.org/web/20070928161148/http://kobus.ca/research/publications/ICCV-01/). *Proceedings of International Conference on Computer Vision*. pp. 408–415. Archived from [the original](http://kobus.ca/research/publications/ICCV-01/) on 2007-09-28.

- Latent Dirichlet Allocation model

- D Blei; A Ng & M Jordan (2003). ["Latent Dirichlet allocation"](https://web.archive.org/web/20050316213517/http://www.ics.uci.edu/~liang/seminars/win05/papers/blei03-latent-dirichlet.pdf) (PDF). *Journal of Machine Learning Research*. pp. 3:993–1022. Archived from [the original](http://www.ics.uci.edu/~liang/seminars/win05/papers/blei03-latent-dirichlet.pdf) (PDF) on March 16, 2005.

- [Supervised](/source/Supervised_learning) [multiclass labeling](/source/Multiclass_labeling)

- G Carneiro; A B Chan; P Moreno & N Vasconcelos (2006). ["Supervised Learning of Semantic Classes for Image Annotation and Retrieval"](http://www.svcl.ucsd.edu/publications/journal/2007/pami/pami07-semantics.pdf) (PDF). *IEEE Transactions on Pattern Analysis and Machine Intelligence*. pp. 394–410.

- Texture similarity

- R W Picard & T P Minka (1995). ["Vision Texture for Annotation"](http://citeseer.ist.psu.edu/picard95vision.html). *Multimedia Systems*.

- Support Vector Machines

- C Cusano; G Ciocca & R Scettini (2004). Santini, Simone & Schettini, Raimondo (eds.). "Image Annotation Using SVM". *Internet Imaging V*. **5304**: 330–338. [Bibcode](/source/Bibcode_(identifier)):[2003SPIE.5304..330C](https://ui.adsabs.harvard.edu/abs/2003SPIE.5304..330C). [doi](/source/Doi_(identifier)):[10.1117/12.526746](https://doi.org/10.1117%2F12.526746). [S2CID](/source/S2CID_(identifier)) [16246057](https://api.semanticscholar.org/CorpusID:16246057).

- Ensemble of Decision Trees and Random Subwindows

- R Maree; P Geurts; J Piater & L Wehenkel (2005). ["Random Subwindows for Robust Image Classification"](http://www.montefiore.ulg.ac.be/~maree/#publications). *Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition*. pp. 1:34–30.

- Maximum Entropy

- J Jeon; R Manmatha (2004). ["Using Maximum Entropy for Automatic Image Annotation"](http://ciir.cs.umass.edu/pubfiles/mm-355.pdf) (PDF). *Int'l Conf on Image and Video Retrieval (CIVR 2004)*. pp. 24–32.

- Relevance models

- J Jeon; V Lavrenko & R Manmatha (2003). ["Automatic image annotation and retrieval using cross-media relevance models"](http://ciir.cs.umass.edu/pubfiles/mm-41.pdf) (PDF). *Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval*. pp. 119–126.

- Relevance models using continuous probability density functions

- V Lavrenko; R Manmatha & J Jeon (2003). ["A model for learning the semantics of pictures"](http://ciir.cs.umass.edu/pubfiles/mm-46.pdf) (PDF). *Proceedings of the 16th Conference on Advances in Neural Information Processing Systems NIPS*.

- Coherent Language Model

- R Jin; J Y Chai; L Si (2004). ["Effective Automatic Image Annotation via A Coherent Language Model and Active Learning"](http://www.cse.msu.edu/~rongjin/publications/acmmm04.jin.pdf) (PDF). *Proceedings of MM'04*.

- Inference networks

- D Metzler & R Manmatha (2004). ["An inference network approach to image retrieval"](http://ciir.cs.umass.edu/pubfiles/mm-346.pdf) (PDF). *Proceedings of the International Conference on Image and Video Retrieval*. pp. 42–50.

- Multiple Bernoulli distribution

- S Feng; R Manmatha & V Lavrenko (2004). ["Multiple Bernoulli relevance models for image and video annotation"](http://ciir.cs.umass.edu/pubfiles/mm-333.pdf) (PDF). *IEEE Conference on Computer Vision and Pattern Recognition*. pp. 1002–1009.

- Multiple design alternatives

- J Y Pan; H-J Yang; P Duygulu; C Faloutsos (2004). ["Automatic Image Captioning"](https://web.archive.org/web/20041209191242/http://www.informedia.cs.cmu.edu/documents/ICME04AutoICap.pdf) (PDF). *Proceedings of the 2004 IEEE International Conference on Multimedia and Expo (ICME'04)*. Archived from [the original](http://www.informedia.cs.cmu.edu/documents/ICME04AutoICap.pdf) (PDF) on 2004-12-09.

- Image captioning

- Quan Hoang Lam; Quang Duy Le; Kiet Van Nguyen; Ngan Luu-Thuy Nguyen (2020). ["UIT-ViIC: A Dataset for the First Evaluation on Vietnamese Image Captioning"](https://link.springer.com/chapter/10.1007/978-3-030-63007-2_57). *Proceedings of the 2020 International Conference on Computational Collective Intelligence (ICCCI 2020)*. [arXiv](/source/ArXiv_(identifier)):[2002.00175](https://arxiv.org/abs/2002.00175). [doi](/source/Doi_(identifier)):[10.1007/978-3-030-63007-2_57](https://doi.org/10.1007%2F978-3-030-63007-2_57).

- Natural scene annotation

- J Fan; Y Gao; H Luo; G Xu (2004). ["Automatic Image Annotation by Using Concept-Sensitive Salient Objects for Image Content Representation"](http://portal.acm.org/ft_gateway.cfm?id=1009055&type=pdf&coll=GUIDE&dl=GUIDE&CFID=1581830&CFTOKEN=99651762). *Proceedings of the 27th annual international conference on Research and development in information retrieval*. pp. 361–368.

- Relevant low-level global filters

- A Oliva & A Torralba (2001). ["Modeling the shape of the scene: a holistic representation of the spatial envelope"](http://cvcl.mit.edu/Papers/IJCV01-Oliva-Torralba.pdf) (PDF). *International Journal of Computer Vision*. pp. 42:145–175.

- Global image features and nonparametric density estimation

- A Yavlinsky, E Schofield & S Rüger (2005). ["Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation"](https://web.archive.org/web/20051220164354/http://km.doc.ic.ac.uk/www-pub/civr05-annotation.pdf) (PDF). *Int'l Conf on Image and Video Retrieval (CIVR, Singapore, Jul 2005)*. Archived from [the original](http://km.doc.ic.ac.uk/www-pub/civr05-annotation.pdf) (PDF) on 2005-12-20.

- Video semantics

- N Vasconcelos & A Lippman (2001). ["Statistical Models of Video Structure for Content Analysis and Characterization"](http://www.svcl.ucsd.edu/publications/journal/2000/ip/ip00.pdf) (PDF). *IEEE Transactions on Image Processing*. pp. 1–17.

- Ilaria Bartolini; Marco Patella & Corrado Romani (2010). ["Shiatsu: Semantic-based Hierarchical Automatic Tagging of Videos by Segmentation Using Cuts"](http://dl.acm.org/citation.cfm?doid=1862344.1862364). *3rd ACM International Multimedia Workshop on Automated Information Extraction in Media Production (AIEMPro10)*.

- Image Annotation Refinement

- Yohan Jin; [Latifur Khan](/source/Latifur_Khan); Lei Wang & Mamoun Awad (2005). ["Image annotations by combining multiple evidence & wordNet"](http://portal.acm.org/citation.cfm?id=1101305&dl=GUIDE,). *13th Annual ACM International Conference on Multimedia (MM 05)*. pp. 706–715.

- Changhu Wang; Feng Jing; Lei Zhang & Hong-Jiang Zhang (2006). ["Image annotation refinement using random walk with restarts"](http://portal.acm.org/citation.cfm?id=1180639.1180774#,). *14th Annual ACM International Conference on Multimedia (MM 06)*.

- Changhu Wang; Feng Jing; Lei Zhang & Hong-Jiang Zhang (2007). "content-based image annotation refinement". *IEEE Conference on Computer Vision and Pattern Recognition (CVPR 07)*. [doi](/source/Doi_(identifier)):[10.1109/CVPR.2007.383221](https://doi.org/10.1109%2FCVPR.2007.383221).

- Ilaria Bartolini & Paolo Ciaccia (2007). "Imagination: Exploiting Link Analysis for Accurate Image Annotation". *Springer Adaptive Multimedia Retrieval*. [doi](/source/Doi_(identifier)):[10.1007/978-3-540-79860-6_3](https://doi.org/10.1007%2F978-3-540-79860-6_3).

- Ilaria Bartolini & Paolo Ciaccia (2010). ["Multi-dimensional Keyword-based Image Annotation and Search"](http://dl.acm.org/citation.cfm?doid=1868366.1868371). *2nd ACM International Workshop on Keyword Search on Structured Data (KEYS 2010)*.

- Automatic Image Annotation by Ensemble of Visual Descriptors

- Emre Akbas & Fatos Y. Vural (2007). "Automatic Image Annotation by Ensemble of Visual Descriptors". *Intl. Conf. on Computer Vision (CVPR) 2007, Workshop on Semantic Learning Applications in Multimedia*. [doi](/source/Doi_(identifier)):[10.1109/CVPR.2007.383484](https://doi.org/10.1109%2FCVPR.2007.383484). [hdl](/source/Hdl_(identifier)):[11511/16027](https://hdl.handle.net/11511%2F16027).

- A New Baseline for Image Annotation

- Ameesh Makadia and Vladimir Pavlovic and Sanjiv Kumar (2008). ["A New Baseline for Image Annotation"](http://www.cs.rutgers.edu/~vladimir/pub/makadia08eccv.pdf) (PDF). *European Conference on Computer Vision (ECCV)*.

Simultaneous Image Classification and Annotation

- Chong Wang and David Blei and Li Fei-Fei (2009). ["Simultaneous Image Classification and Annotation"](http://cs.stanford.edu/groups/vision/documents/WangBleiFei-Fei_CVPR2009.pdf) (PDF). *Conf. on Computer Vision and Pattern Recognition (CVPR)*.

- TagProp: Discriminative Metric Learning in Nearest Neighbor Models for Image Auto-Annotation

- Matthieu Guillaumin and Thomas Mensink and Jakob Verbeek and Cordelia Schmid (2009). ["TagProp: Discriminative Metric Learning in Nearest Neighbor Models for Image Auto-Annotation"](https://lear.inrialpes.fr/pubs/2009/GMVS09/GMVS09.pdf) (PDF). *Intl. Conf. on Computer Vision (ICCV)*.

- Image Annotation Using Metric Learning in Semantic Neighbourhoods

- Yashaswi Verma & C. V. Jawahar (2012). ["Image Annotation Using Metric Learning in Semantic Neighbourhoods"](https://web.archive.org/web/20130514202446/http://researchweb.iiit.ac.in/%7Eyashaswi.verma/eccv12/vj_eccv12.pdf) (PDF). *European Conference on Computer Vision (ECCV)*. Archived from [the original](http://researchweb.iiit.ac.in/~yashaswi.verma/eccv12/vj_eccv12.pdf) (PDF) on 2013-05-14. Retrieved 2014-02-26.

- Automatic Image Annotation Using Deep Learning Representations

- Venkatesh N. Murthy & Subhransu Maji and R. Manmatha (2015). ["Automatic Image Annotation Using Deep Learning Representations"](https://people.cs.umass.edu/~smaji/papers/embeddings-icmr15s.pdf) (PDF). *International Conference on Multimedia (ICMR)*.

- Holistic Image Annotation using Salient Regions and Background Image Information

- Sarin, Supheakmungkol; Fahrmair, Michael; Wagner, Matthias & Kameyama, Wataru (2012). [*Leveraging Features from Background and Salient Regions for Automatic Image Annotation*](https://www.jstage.jst.go.jp/article/ipsjjip/20/1/20_1_250/_pdf/-char/en). Journal of Information Processing. Vol. 20. pp. 250–266.

- Medical Image Annotation using bayesian networks and active learning

- N. B. Marvasti & E. Yörük and B. Acar (2018). ["Computer-Aided Medical Image Annotation: Preliminary Results With Liver Lesions in CT"](https://www.researchgate.net/publication/320935564). *IEEE Journal of Biomedical and Health Informatics*.

v t e Computer vision Categories Datasets Digital geometry Commercial systems Feature detection Geometry Image sensor technology Learning Morphology Motion analysis Noise reduction techniques Recognition and categorization Research infrastructure Researchers Segmentation Software Technologies Computer stereo vision Motion capture Object recognition 3D object recognition Applications 3D reconstruction 3D reconstruction from multiple images 2D to 3D conversion Gaussian splatting Neural radiance field Shape from focus Simultaneous localization and mapping Structure from motion View synthesis Visual hull 4D reconstruction Free viewpoint television Volumetric capture 3D pose estimation Activity recognition Audio-visual speech recognition Automatic image annotation Automatic number-plate recognition Automated species identification Augmented reality Bioimage informatics Bin picking Blob detection Computer-aided diagnosis Content-based image retrieval Reverse image search Eye tracking Face recognition Foreground detection Gesture recognition Image denoising Image restoration Landmark detection Medical image computing Object detection Moving object detection Pedestrian detection Small object detection Optical character recognition Pose tracking Remote sensing Robotic mapping Autonomous vehicles Video browsing Video content analysis Video motion analysis Video surveillance Video synopsis Video tracking Main category

---
Adapted from the Wikipedia article [Automatic image annotation](https://en.wikipedia.org/wiki/Automatic_image_annotation) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Automatic_image_annotation?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.