Perceptrons (book)

{{Short description|Book by Marvin Minsky and Seymour Papert}} {{Infobox book <!-- |italic title = (see above) --> | name = Perceptrons: An Introduction to Computational Geometry | image = File:Perceptrons_(book).jpg | caption = | author = Marvin Minsky, Seymour Papert | title_orig = | translator = | illustrator = | cover_artist = | country = | language = | series = | subject = | genre = | publisher = | pub_date = 1969 | english_pub_date = | media_type = | pages = | isbn = 0 262 13043 2 | oclc = | dewey = | congress = | preceded_by = | followed_by = }} '''''Perceptrons: An Introduction to Computational Geometry''''' is a book written by Marvin Minsky and Seymour Papert and published in 1969. An edition with handwritten corrections and additions was released in the early 1970s. An expanded edition was further published in 1988 ({{ISBN|9780262631112}}) after the revival of neural networks, containing a chapter dedicated to countering the criticisms made of it in the 1980s.

The main subject of the book is the perceptron, a type of artificial neural network developed in the late 1950s and early 1960s. The book was dedicated to psychologist Frank Rosenblatt, who in 1957 had published the first model of a "Perceptron".<ref>{{cite report |last1=Rosenblatt |first1=Frank |title=The Perceptron: A Perceiving and Recognizing Automaton (Project PARA) |date=January 1957 |id=Report No. 85–460–1 |url=https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf |access-date=29 December 2019 |publisher=Cornell Aeronautical Laboratory, Inc. |archive-date=7 April 2023 |archive-url=https://web.archive.org/web/20230407120421/https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf |url-status=dead }} Memorialized at Joe Pater, [https://blogs.umass.edu/brain-wars/ Brain Wars: How does the mind work? And why is that so important?], UmassAmherst.</ref> Rosenblatt and Minsky knew each other since adolescence, having studied with a one-year difference at the Bronx High School of Science.<ref>{{Harvnb|Crevier|1993}}</ref> They became at one point central figures of a debate inside the AI research community, and are known to have promoted loud discussions in conferences, yet remained friendly.{{sfn|Olazaran|1996}}

This book is the center of a long-standing controversy in the study of artificial intelligence. It is claimed that pessimistic predictions made by the authors were responsible for a change in the direction of research in AI, concentrating efforts on so-called "symbolic" systems, a line of research that petered out and contributed to the so-called AI winter of the 1980s, when AI's promise was not realized.<ref>{{cite book |last1=Mitchell|first1=Melanie|title= Artificial Intelligence: A Guide for Thinking Humans |date=October 2019 |url=https://melaniemitchell.me/aibook/ |publisher= Farrar, Straus and Giroux |isbn=978-0-374-25783-5}}</ref>

The crux of ''Perceptrons'' is a number of mathematical proofs which acknowledge some of the perceptrons' strengths while also showing major limitations.{{sfn|Olazaran|1996}} The most important one is related to the computation of some predicates, such as the XOR function, and also the important connectedness predicate. The problem of connectedness is illustrated at the awkwardly colored cover of the book, intended to show how humans themselves have difficulties in computing this predicate.<ref>Minsky-Papert 1972:74 shows the figures in black and white. The cover of the 1972 paperback edition has them printed purple on a red background, and this makes the connectivity even more difficult to discern without the use of a finger or other means to follow the patterns mechanically. This problem is discussed in detail on pp.136ff and indeed involves tracing the boundary.</ref> One reviewer, Earl Hunt, noted that the XOR function is difficult for humans to acquire as well during concept learning experiments.<ref>{{Cite journal |last=Hunt |first=Earl |date=1971 |title=Review of Perceptrons |url=https://www.jstor.org/stable/1420478 |journal=The American Journal of Psychology |volume=84 |issue=3 |pages=445–447 |doi=10.2307/1420478 |jstor=1420478 |issn=0002-9556|url-access=subscription }}</ref>

== Publication history == When Papert arrived at MIT in 1963, Minsky and Papert decided to write a theoretical account of the limitations of perceptrons. It took until 1969 for them to finish solving the mathematical problems that unexpectedly turned up as they wrote. The first edition was printed in 1969. Handwritten alterations were made by the authors for the second printing in 1972. The handwritten notes include some references to the reviews of the first edition.<ref>{{Cite journal |last=Block |first=H.D. |date=December 1970 |title=A review of "perceptrons: An introduction to computational geometry≓ |url=https://linkinghub.elsevier.com/retrieve/pii/S0019995870904092 |journal=Information and Control |language=en |volume=17 |issue=5 |pages=501–522 |doi=10.1016/S0019-9958(70)90409-2|url-access=subscription }}</ref><ref>{{Cite journal |last=Newell |first=Allen |date=1969-08-22 |title=A Step toward the Understanding of Information Processes: Perceptrons . An Introduction to Computational Geometry. Marvin Minsky and Seymour Papert. M.I.T. Press, Cambridge, Mass., 1969. vi + 258 pp., illus. Cloth, $12; paper, $4.95. |url=https://www.science.org/doi/10.1126/science.165.3895.780 |journal=Science |language=en |volume=165 |issue=3895 |pages=780–782 |doi=10.1126/science.165.3895.780 |issn=0036-8075|url-access=subscription }}</ref><ref>{{Cite journal |last=Mycielski |first=Jan |date=January 1972 |title=Review: Marvin Minsky and Seymour Papert, Perceptrons, An Introduction to Computational Geometry |url=https://projecteuclid.org/journals/bulletin-of-the-american-mathematical-society/volume-78/issue-1/Review--Marvin-Minsky-and-Seymour-Papert-Perceptrons-An-Introduction/bams/1183533389.full |journal=Bulletin of the American Mathematical Society |volume=78 |issue=1 |pages=12–15 |doi=10.1090/S0002-9904-1972-12831-3 |issn=0002-9904|doi-access=free |url-access=subscription }}</ref>

An "expanded edition" was published in 1988, which adds a prologue and an epilogue to discuss the revival of neural networks in the 1980s, but no new scientific results.<ref>Stephen Grossberg. "Review of Perceptrons", ''AI Magazine'', 10(2) (1989).</ref> In 2017, the expanded edition was reprinted, with a foreword by Léon Bottou that discusses the book from the perspective of someone working in deep learning.

== Background == The perceptron is a neural net developed by psychologist Frank Rosenblatt in 1958 and is one of the most famous machines of its period.<ref>{{cite journal |last=Rosenblatt |first=Frank |title=The perceptron: A probabilistic model for information storage and organization in the brain |journal=Psychological Review |volume=65 |issue=6 |year=1958 |pages=386–408 |doi=10.1037/h0042519 |pmid=13602029 |citeseerx=10.1.1.588.3775|s2cid=12781225 }}</ref><ref name=":0">{{harvnb|Olazaran|1996|p=618}}</ref> In 1960, Rosenblatt and colleagues were able to show that the perceptron could in finitely many training cycles learn any task that its parameters could embody. The perceptron convergence theorem was proved for single-layer neural nets.<ref name=":0" />

During this period, neural net research was a major approach to the brain-machine issue that had been taken by a significant number of persons.<ref name=":0" /> Reports by the New York Times and statements by Rosenblatt claimed that neural nets would soon be able to see images, beat humans at chess,<ref>Deep Blue was not based on neural networks</ref> and reproduce.{{sfn|Olazaran|1996}} At the same time, other new approaches including symbolic AI emerged.<ref>{{Cite book|title=Artificial Intelligence: The Very Idea|last=Haugeland|first=John|publisher=MIT Press|year=1985|isbn=978-0-262-08153-5|location=Cambridge, Mass}}</ref> Different groups found themselves competing for funding and people, and their demand for computing power far outpaced the available supply.<ref>{{cite arXiv |last=Hwang |first=Tim |title=Computational Power and the Social Impact of Artificial Intelligence |eprint=1803.08971v1|class=cs.AI |year=2018 }}</ref>

== Contents == ''Perceptrons: An Introduction to Computational Geometry'' is a book of thirteen chapters grouped into three sections. Chapters 1–10 present the authors' perceptron theory through proofs, Chapter 11 involves learning, Chapter 12 treats linear separation problems, and Chapter 13 discusses some of the authors' thoughts on simple and multilayer perceptrons and pattern recognition.<ref name=":2" /><ref name=":1" />

=== Definition of perceptron === Minsky and Papert took as their subject the abstract versions of a class of learning devices which they called perceptrons, "in recognition of the pioneer work of Frank Rosenblatt".<ref name=":1">{{Cite book|title=Perceptrons: An Introduction to Computational Geometry|last1=Minsky|first1=Marvin|last2=Papert|first2=Seymour|publisher=MIT Press|year=1988}}</ref> These perceptrons were modified forms of the perceptrons introduced by Rosenblatt in 1958. They consisted of a retina, a single layer of input functions and a single output.<ref name=":2">{{Cite journal|last=Block|first=H. D.|year=1970|title=A Review of 'Perceptrons: An Introduction to Computational Geometry'|journal=Information and Control|volume=17|issue=1|pages=501–522|doi=10.1016/S0019-9958(70)90409-2|doi-access=free}}</ref><ref name=":0" />

Besides this, the authors restricted the "order", or maximum number of incoming connections, of their perceptrons. Sociologist Mikel Olazaran explains that Minsky and Papert "maintained that the interest of neural computing came from the fact that it was a parallel combination of ''local'' information", which, in order to be effective, had to be a simple computation. To the authors, this implied that "each association unit could receive connections only from a small part of the input area".<ref name=":0" /> Minsky and Papert called this concept "conjunctive localness".<ref name=":1" />

=== Parity and connectedness === Two main examples analyzed by the authors were parity and connectedness. Parity involves determining whether the number of activated inputs in the input retina is odd or even, and connectedness refers to the figure-ground problem. Minsky and Papert proved that the single-layer perceptron could not compute parity under the condition of conjunctive localness (Theorem 3.1.1), and showed that the order required for a perceptron to compute connectivity grew with the input size (Theorem 5.5).<ref name=":3">{{harvnb|Olazaran|1996|p=630}}</ref><ref name=":1" />

=== The XOR affair === Some critics of the book {{citation needed|date=April 2023}} state that the authors imply that, since a single artificial neuron is incapable of implementing some functions such as the XOR logical function, larger networks also have similar limitations, and therefore should be dropped. Research on three-layered perceptrons showed how to implement such functions. Rosenblatt in his book proved that the "elementary perceptron" with an ''a priori'' unlimited number of hidden layer A-elements (neurons) and one output neuron can solve any classification problem. (Existence theorem.<ref>Theorem 1 in Rosenblatt, F. (1961) Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan. Washington DC.</ref>) Minsky and Papert used perceptrons with restricted numbers of inputs of the hidden layer A-elements and a locality condition: each element of the hidden layer receives the input signals from a small circle. These restricted perceptrons cannot define whether the image is a connected figure or is the number of pixels in the image even (the parity predicate).

There are many mistakes in this story{{citation needed|date=April 2023}}. Although a single neuron can in fact compute only a small number of logical predicates, it was widely known{{citation needed|date=April 2023}} that networks of such elements can compute any possible Boolean function. This was known by Warren McCulloch and Walter Pitts, who even proposed how to create a Turing machine with their formal neurons (Section III of <ref>{{Cite journal |last1=McCulloch |first1=Warren S. |last2=Pitts |first2=Walter |date=1943-12-01 |title=A logical calculus of the ideas immanent in nervous activity |url=https://doi.org/10.1007/BF02478259 |journal=The Bulletin of Mathematical Biophysics |language=en |volume=5 |issue=4 |pages=115–133 |doi=10.1007/BF02478259 |issn=1522-9602|url-access=subscription }}</ref>), is mentioned in Rosenblatt's book, mentioned in a typical paper in 1961 (Figure 15 <ref>{{Cite journal |last=Hawkins |first=J. |date=January 1961 |title=Self-Organizing Systems-A Review and Commentary |journal=Proceedings of the IRE |volume=49 |issue=1 |pages=31–48 |doi=10.1109/JRPROC.1961.287776 |bibcode=1961PIRE...49...31H |s2cid=51640615 |issn=0096-8390}}</ref>), and is even mentioned in the book Perceptrons.<ref>Cf. Minsky-Papert (1972:232): "... a universal computer could be built entirely out of linear threshold modules. This does not in any sense reduce the theory of computation and programming to the theory of perceptrons."</ref> Minsky also extensively uses formal neurons to create simple theoretical computers in Chapter 3 of his book ''Computation: Finite and Infinite Machines''.

In the 1960s, a special case of the perceptron network is studied as "linear threshold logic", for applications in digital logic circuits.<ref>Hu, Sze-Tsen. ''Threshold logic''. Vol. 32. Univ of California Press, 1965.</ref> The classical theory is summarized in <ref>{{Cite book |last=Muroga |first=Saburo |title=Threshold logic and its applications |date=1971 |publisher=Wiley-Interscience |isbn=978-0-471-62530-8 |location=New York}}</ref> according to Donald Knuth.<ref>{{Cite book |last=Knuth |first=Donald Ervin |title=The art of computer programming, Volume 4A |date=2011 |publisher=Addison-Wesley |isbn=978-0-201-03804-0 |location=Upper Saddle River |pages=75–79}}</ref> In this special case, perceptron learning was called "Single-Threshold-Element Synthesis by Iteration", and constructing a perceptron network was "Network Synthesis".<ref>Dertouzos, Michael L. "Threshold logic: a synthesis approach." (1965).</ref> Other names included ''linearly separable logic, linear-input logic, threshold logic, majority logic,'' and ''voting logic''. Hardware for realizing linear threshold logic included magnetic core, resistor-transistor, parametron, resistor-tunnel diode, and multiple coil relay.<ref>{{Cite journal |last=Minnick |first=Robert C. |date=March 1961 |title=Linear-Input Logic |journal=IEEE Transactions on Electronic Computers |volume=EC-10 |issue=1 |pages=6–16 |doi=10.1109/TEC.1961.5219146 |bibcode=1961IRTEC..10....6M |issn=0367-7508}}</ref> There were also theoretical studies on the upper and lower bounds on the minimum number of perceptron units necessary to realize any Boolean function.<ref>See references within Cover, Thomas M. "[https://isl.stanford.edu/~cover/papers/paper12.pdf Capacity problems for linear machines]." ''Pattern recognition'' (1968): 283-289.</ref><ref>{{Cite journal |last1=Šíma |first1=Jiří |last2=Orponen |first2=Pekka |date=2003-12-01 |title=General-Purpose Computation with Neural Networks: A Survey of Complexity Theoretic Results |url=https://direct.mit.edu/neco/article/15/12/2727-2778/6791 |journal=Neural Computation |language=en |volume=15 |issue=12 |pages=2727–2778 |doi=10.1162/089976603322518731 |pmid=14629867 |s2cid=264603251 |issn=0899-7667|url-access=subscription }}</ref>

What the book does prove is that in three-layered feed-forward perceptrons (with a so-called "hidden" or "intermediary" layer), it is not possible to compute some predicates unless at least one of the neurons in the first layer of neurons (the "intermediary" layer) is connected with a non-null weight to each and every input (Theorem 3.1.1, reproduced below). This was contrary to a hope held by some researchers {{citation needed|date=April 2023}} in relying mostly on networks with a few layers of "local" neurons, each one connected only to a small number of inputs. A feed-forward machine with "local" neurons is much easier to build and use than a larger, fully connected neural network, so researchers at the time concentrated on these instead of on more complicated models{{citation needed|date=April 2023}}.

Some other critics, notably Jordan Pollack, note that what was a small proof concerning a global issue (parity) not being detectable by local detectors was interpreted by the community as a rather successful attempt to bury the whole idea.<ref name=":4">{{cite journal |last=Pollack |first=J. B. |year=1989 |title=No Harm Intended: A Review of the Perceptrons expanded edition |journal=Journal of Mathematical Psychology |volume=33 |issue=3 |pages=358–365 |doi=10.1016/0022-2496(89)90015-1}}</ref>

=== Critique of perceptrons and their extensions === In the prologue and the epilogue, added to the 1988 edition, the authors react to the 1980s revival of neural networks by discussing multilayer neural nets and Gamba perceptrons.<ref name=":5">From the name of the Italian neural network researcher Augusto Gamba (1923–1996), designer of the PAPA perceptron. PAPA is acronym for "Programmatore e Analizzatore Probabilistico Automatico" ("Automatic Probabilistic Programmer and Analyzer").</ref><ref>{{Cite journal |last1=Borsellino |first1=A. |last2=Gamba |first2=A. |date=1961-09-01 |title=An outline of a mathematical theory of PAPA |url=https://doi.org/10.1007/BF02822644 |journal=Il Nuovo Cimento |language=en |volume=20 |issue=2 |pages=221–231 |doi=10.1007/BF02822644 |bibcode=1961NCim...20S.221B |issn=1827-6121|url-access=subscription }}</ref><ref>{{Cite journal |last1=Gamba |first1=A. |last2=Gamberini |first2=L. |last3=Palmieri |first3=G. |last4=Sanna |first4=R. |date=1961-09-01 |title=Further experiments with PAPA |url=https://doi.org/10.1007/BF02822639 |journal=Il Nuovo Cimento |language=en |volume=20 |issue=2 |pages=112–115 |doi=10.1007/BF02822639 |bibcode=1961NCim...20S.112G |issn=1827-6121|url-access=subscription }}</ref><ref>{{Cite journal |last=Gamba |first=A. |date=1962-10-01 |title=A multilevel PAPA |url=https://doi.org/10.1007/BF02782996 |journal=Il Nuovo Cimento |language=en |volume=26 |issue=1 |pages=176–177 |doi=10.1007/BF02782996 |bibcode=1962NCim...26S.176G |issn=1827-6121|url-access=subscription }}</ref> By "Gamba perceptrons", they meant two-layered perceptron machines where the first layer is also made of perceptron units ("Gamba-masks"). In contrast, most of the book discusses two-layered perceptrons where the first layer is made of boolean units. They conjecture that Gamba machines would require "an enormous number" of Gamba-masks and that multilayer neural nets are a "sterile" extension. Additionally, they note that many of the "impossible" problems for perceptrons had already been solved using other methods.<ref name=":1" />

The Gamba perceptron machine was similar to the perceptron machine of Rosenblatt. Its input was an image. The image is passed through binary masks (randomly generated) in parallel. Behind each mask is a photoreceiver that fires if the input, after masking, is bright enough. The second layer is made of standard perceptron units.

They claimed that perceptron research waned in the 1970s not because of their book, but because of inherent problems: no perceptron learning machines could perform credit assignment any better than Rosenblatt's perceptron learning rule, and perceptrons cannot represent the knowledge required for solving certain problems.<ref name=":4" />

In the final chapter, they claimed that for the 1980s neural networks, "little of significance [has] changed since 1969". They predicted that any single, homogeneous machine must fail to scale up. Neural networks trained by gradient descent would fail to scale up, due to local minima, extremely large weights, and slow convergence. General learning algorithms for neural networks must all be impractical, because a general, domain-independent theory of "how neural networks work" does not exist. Only a society of mind can work. Specifically, they thought there are many different kinds of little problems in the world, each is on the scale of a "toy problem". Large problems are always decomposable into little problems. Each requires a different algorithm to solve, some being perceptrons, others being logical programs, and so on. Any homogeneous machine must fail to solve all but a small number of the little problems. Human intelligence consists of nothing but a collection of many little different algorithms organized like a society.<ref name=":4" />

== Mathematical content ==

=== Preliminary definitions === Let <math display="inline">R</math> be a finite set. A '''predicate''' on <math display="inline">R</math> is a boolean function that takes in a subset of <math display="inline">R</math> and outputs either <math display="inline">0</math> or <math display="inline">1</math>. In particular, a perceptron unit is a predicate.

A predicate <math display="inline">\psi</math> has '''support''' <math display="inline">S \subset R</math>, iff any <math display="inline">X \subset S</math>, we have <math display="inline">\psi(X) = \psi(X \cap S)</math>. In words, it means that if we know how <math display="inline">\psi</math> works on subsets of <math display="inline">S</math>, then we know how it works on subsets of all of <math display="inline">R</math>.

A predicate can have many different supports. The '''support size''' of a predicate <math display="inline">\psi</math> is the minimal number of elements necessary in its support. For example, the constant-0 and constant-1 functions both are supported on the empty set, thus they both have support size 0.

A '''perceptron''' (the kind studied by Minsky and Papert) over <math display="inline">R</math> is a function of form<math display="block">\theta\left(\sum_i a_i \psi_i\right)</math>where <math display="inline">\psi_i</math> are predicates, and <math display="inline">a_i</math> are real numbers.

If <math display="inline">\Phi</math> is a set of predicates, then <math display="inline">L(\Phi)</math> is the set of all perceptrons using just predicates in <math display="inline">\Phi</math>.

The '''order''' of a perceptron <math display="inline">\theta\left(\sum_i a_i \psi_i\right)</math> is the maximal support size of its component predicates <math display="inline">\{\psi_i\}_i</math>.

The '''order''' of a boolean function on <math display="inline">R</math> is the minimal order possible for a perceptron implementing the boolean function.

A boolean function is '''conjunctively local''' iff its order does not increase to infinity as <math>|R|</math> increases to infinity.

The '''mask''' of <math display="inline">A \subset R</math> is the predicate <math display="inline">1_A</math> defined by<math display="block">1_A(X) = \begin{cases} 1 & \text{ if }A \subset X,\\ 0 & \text{ else.} \end{cases}</math>

=== Main theorems === {{Math theorem | name = Theorem 1.5.1, Positive Normal Form | note = |math_statement=

If a perceptron is of order <math display="inline">k</math>, then it is of order <math display="inline">k</math> using only masks. }}

{{Math proof|title=Proof|proof=

Let the perceptron be <math display="inline">\theta\left(\sum_i a_i \psi_i\right)</math>, where each <math display="inline">\psi_i</math> is of support size at most <math display="inline">k</math>. We convert it into a linear sum of masks, each having size at most <math display="inline">k</math>.

Let <math display="inline">\psi_i</math> be supported on set <math display="inline">A</math>. Write it in disjunctive normal form, with one clause for each subset of <math display="inline">A</math> on which <math display="inline">\psi_i</math> returns <math display="inline">1</math>, and for each subset, write one positive literal for each element in the subset, and one negative literal otherwise.

For example, suppose <math display="inline">\psi_i</math> is supported on <math display="inline">\{1,2\}</math>, and is <math display="inline">1</math> on all odd-sized subsets, then we can write it as<math display="block">(x_1 \land \neg x_2) \lor (\neg x_1 \land x_2)</math>

Now, convert this formula to a Boolean algebra formula, then expand, yielding a linear sum of masks. For example, the above formula is converted to<math display="block">x_1(1-x_2) + (1-x_1)x_2 = x_1 + x_2 - 2x_1x_2</math>

Repeat this for each predicate used in the perceptron, and sum them up, we obtain an equivalent perceptron using just masks. }}

Let <math display="inline">S_R</math> be the permutation group on the elements of <math display="inline">R</math>, and <math display="inline">G</math> be a subgroup of <math display="inline">S_R</math>.

We say that a predicate <math display="inline">\psi</math> is <math display="inline">G</math> -invariant iff <math display="inline">\psi \circ g = \psi</math> for any <math display="inline">g \in G</math>. That is, any <math display="inline">X\subset R</math>, we have <math display="inline">\psi(X) = \psi(g(X))</math>.

For example, the parity function is <math display="inline">S_R</math> -invariant, since any permutation of the set preserves the size, and thus parity, of any of its subsets. {{Math theorem | name = Theorem 2.3, group invariance theorem | note = |math_statement=

If <math display="inline">\Phi</math> is closed under action by <math display="inline">G</math>, and <math display="inline">\psi\in L(\Phi)</math> is <math display="inline">G</math> -invariant, there exists a perceptron<math display="block">\theta\left(\sum_i a_i \psi_i\right) = \psi</math>such that if <math display="inline">\psi_i = \psi_j \circ g</math> for some <math display="inline">g\in G</math>, then <math display="inline">a_i = a_j</math>. }}

{{Math proof|title=Proof|proof= The proof idea is to take the average over all elements of <math display="inline">G</math>.

Enumerate the predicates in <math display="inline">\Phi</math> as <math display="inline">\psi_1, \psi_2, ...</math>, and write <math display="inline">g(j)</math> for the index of the predicate such that <math display="inline">\psi_{g(j)} = \psi_j \circ g</math>, for any <math display="inline">g\in G</math>. That is, we have defined a group action on the set <math display="inline">\Phi</math>.

Define <math display="inline">a_j := \sum_{g\in G}b_{g^{-1}(j)}</math>. We claim this is the desired perceptron.

Since <math display="inline">\psi \in L(\Phi)</math>, there exists some real numbers <math display="inline">b_j</math> such that<math display="block">\theta\left(\sum_j b_j \psi_j\right) = \psi</math>

By definition of <math display="inline">G</math> -invariance, if <math display="inline">\psi(A) = 1</math>, then <math display="inline">\psi(g(A)) = 1</math> for all <math display="inline">g\in G</math>. That is,<math display="block">\sum_j b_j (\psi_j\circ g)(A) > 0; \quad g \in G</math>and so, taking the average over all elements in <math display="inline">G</math>, we have<math display="block">0 < \sum_{g\in G}\sum_j b_j (\psi_j\circ g)(A) = \sum_{g\in G}\sum_j b_{g^{-1}(j)} \psi_j (A) =\sum_j \left(\sum_{g\in G}b_{g^{-1}(j)}\right) \psi_j (A)= \sum_j a_j \psi_j(A)</math>

Similarly for the case where <math display="inline">\psi(A) = 0</math>. }}

{{Math theorem | name = Theorem 3.1.1 | note = |math_statement=

The parity function has order <math display="inline">|R|</math>. }} {{Math proof|title=Proof|proof=

Let <math display="inline">\psi_{parity}</math> be the parity function, and <math display="inline">\Phi</math> be the set of all masks of size <math display="inline">\leq |R| -1</math>. Clearly both <math display="inline">\psi_{parity}</math> and <math display="inline">\Phi</math> are invariant under all permutations.

Suppose <math display="inline">\psi_{parity}</math> has order <math display="inline">\leq |R|-1</math>, then by the positive normal form theorem, <math display="inline">\psi_{parity} \in L(\Phi)</math>.

By the group invariance theorem, there exists a perceptron<math display="block">\theta\left(\sum_i a_i \psi_i\right) = \psi_{parity}</math>such that <math display="inline">a_i</math> depends only on the <math display="inline">S_R</math> equivalence class of the mask <math display="inline">\psi_i</math>, and thus, only depends on the size of the mask <math display="inline">\psi_i</math>. That is, there exists real numbers <math display="inline">b_0, b_1, ..., b_{|R|-1}</math> such that if <math display="inline">\psi_i</math> is the mask on <math display="inline">A</math>, then <math display="inline">a_i = b_{|A|}</math>.

Now we can explicitly calculate the perceptron on any subset <math display="inline">X \subset R</math>.

Since <math display="inline">X</math> contains <math display="inline">\binom{|X|}{k}</math> subsets of size <math display="inline">k</math>, we plug in the perceptron’s formula and calculate:<math display="block">\psi_{parity}(X) = \theta\left(\sum_{k=0}^{|R|-1} b_k \binom{|X|}{k} \right)</math>

Now, define the polynomial function<math display="block">p(x) := \sum_{k=0}^{|R|-1} b_k \binom{x}{k}</math>where <math display="inline">\binom{x}{k} = \frac{x(x-1) \cdots(x-k+1)}{k!}</math>. It has at most degree <math display="inline">|R|-1</math>. then since <math display="inline">\theta(p(|X|)) = \psi_{parity}(X)</math>, for each <math display="inline">|X| = 0, 1, 2, ..., |R|</math>, we have<math display="block">p(0) - \epsilon > 0, \quad p(1) - \epsilon < 0, \quad p(2) - \epsilon > 0, \quad \cdots</math>for a small positive <math display="inline">\epsilon</math>.

Thus, the degree <math display="inline">\leq |R|-1</math> polynomial <math display="inline">p-\epsilon</math> has at least <math display="inline">|R|</math> different roots, one on each <math display="inline">(0, 1), (1, 2), ..., (|R|-1, |R|)</math>, contradiction. }}

{{Math theorem | name = Theorem 5.9 | note = | math_statement = The only topologically invariant predicates of finite order are functions of the Euler number <math display="inline">E</math>.

That is, if <math display="inline">\psi</math> is a boolean function that depends on topology can be implemented by a perceptron of order <math display="inline">k</math>, such that <math display="inline">k</math> is fixed, and does not grow as <math display="inline">R</math> grows into a larger and larger rectangle, then <math display="inline">\psi</math> is of form <math display="inline">f \circ E</math>, for some function <math display="inline">f: \N \to 2</math>. }}Proof: omitted.{{Math theorem | name = Section 5.5, due to David A. Huffman | note = |math_statement=

Let <math display="inline">R_n</math> be the rectangle of shape <math display="inline">5n \times (2n+12)</math>, then as <math display="inline">n\to\infty</math>, the connectedness function on <math display="inline">R_n</math> has order growing at least as fast as <math display="inline">\Omega(|R_n|^{1/2})</math>. }}

Proof sketch: By reducing the parity function to the connectness function, using circuit gadgets. It is in a similar style as the one showing that Sokoban is NP-hard.<ref>{{Cite journal |last1=Dor |first1=Dorit |last2=Zwick |first2=Uri |date=1999-10-01 |title=SOKOBAN and other motion planning problems |url=https://www.sciencedirect.com/science/article/pii/S0925772199000176 |journal=Computational Geometry |volume=13 |issue=4 |pages=215–228 |doi=10.1016/S0925-7721(99)00017-6 |issn=0925-7721|url-access=subscription }}</ref>

== Reception and legacy == ''Perceptrons'' received a number of positive reviews in the years after publication. In 1969, Stanford professor Michael A. Arbib stated, "[t]his book has been widely hailed as an exciting new chapter in the theory of pattern recognition."<ref>{{Cite journal|last=Arbib|first=Michael|date=November 1969|title=Review of 'Perceptrons: An Introduction to Computational Geometry'|journal=IEEE Transactions on Information Theory|volume=15|issue=6|pages=738–739|doi=10.1109/TIT.1969.1054388}}</ref> Earlier that year, CMU professor Allen Newell composed a review of the book for ''Science'', opening the piece by declaring "[t]his is a great book."<ref>{{cite journal |last=Newell |first=Allen |title=A Step toward the Understanding of Information Processes |journal=Science |volume=165 |issue=3895 |year=1969 |pages=780–782 |jstor=1727364|doi=10.1126/science.165.3895.780 }}</ref>

On the other hand, H.D. Block expressed concern at the authors' narrow definition of perceptrons. He argued that they "study a severely limited class of machines from a viewpoint quite alien to Rosenblatt's", and thus the title of the book was "seriously misleading".<ref name=":2" /> Contemporary neural net researchers shared some of these objections: Bernard Widrow complained that the authors had defined perceptrons too narrowly, but also said that Minsky and Papert's proofs were "pretty much irrelevant", coming a full decade after Rosenblatt's perceptron.<ref name=":3" />

''Perceptrons'' is often thought to have caused a decline in neural net research in the 1970s and early 1980s.{{sfn|Olazaran|1996}}<ref>{{cite arXiv |title=The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches |last=Alom |first=Md Zahangir |display-authors=etal |eprint=1803.01164v1 |quote=1969: Minsky & Papert show the limitations of perceptron's, killing research in neural networks for a decade|class=cs.CV |year=2018 }}</ref> During this period, neural net researchers continued smaller projects outside the mainstream, while symbolic AI research saw explosive growth.<ref>{{cite journal |last=Bechtel |first=William |title=The Case for Connectionism |journal=Philosophical Studies |volume=71 |issue=2 |year=1993 |pages=119–154 |jstor=4320426 |doi=10.1007/BF00989853|s2cid=170812977 }}</ref>{{sfn|Olazaran|1996}}

With the revival of connectionism in the late 80s, PDP researcher David Rumelhart and his colleagues returned to ''Perceptrons''. In a 1986 report, they claimed to have overcome the problems presented by Minsky and Papert, and that "their pessimism about learning in multilayer machines was misplaced".{{sfn|Olazaran|1996}}

== Analysis of the controversy == It is most instructive to learn what Minsky and Papert themselves said in the 1970s as to what were the broader implications of their book. On his website Harvey Cohen,<ref>{{cite web| url=http://harveycohen.net/image/perceptron.html |title=The Perceptron Controversy}}</ref> a researcher at the MIT AI Labs 1974+,<ref>{{cite web|url=http://harveycohen.net/papers/TheArtOfSnaringDragons-MIT-AI-Memo338.pdf|title=Author of MIT AI Memo 338}}</ref> quotes Minsky and Papert in the 1971 Report of Project MAC, directed at funding agencies, on "Gamba networks":<ref name=":5" /> "Virtually nothing is known about the computational capabilities of this latter kind of machine. We believe that it can do little more than can a low order perceptron." In the preceding page Minsky and Papert make clear that "Gamba networks" are networks with hidden layers.

Minsky has compared the book to the fictional book ''Necronomicon'' in H. P. Lovecraft's tales, a book known to many, but read only by a few.<ref>{{cite web|url=http://www.ucs.louisiana.edu/~isb9112/dept/phil341/histconn.html |title=History: The Past |publisher=Ucs.louisiana.edu |access-date=2013-07-10}}</ref> The authors talk in the expanded edition about the criticism of the book that started in the 1980s, with a new wave of research symbolized by the PDP book.

How ''Perceptrons'' was explored first by one group of scientists to drive research in AI in one direction, and then later by a new group in another direction, has been the subject of a sociological study of scientific development.{{sfn|Olazaran|1996}}

== Notes == {{reflist}}

== References ==

* {{McCorduck 2004}}, pp.&nbsp;104−107 * {{Crevier 1993}}, pp.&nbsp;102−105 * {{Russell Norvig 2003}} p.&nbsp;22 * Marvin Minsky and Seymour Papert, 1972 (2nd edition with corrections, first edition 1969) ''Perceptrons: An Introduction to Computational Geometry'', The MIT Press, Cambridge MA, {{ISBN|0-262-63022-2}}. * {{cite journal |last=Olazaran |first=Mikel |year=1996 |title=A Sociological Study of the Official History of the Perceptrons Controversy |journal=Social Studies of Science |volume=26 |issue=3 |pages=611–659 |doi=10.1177/030631296026003005 |jstor=285702 |s2cid=16786738}} * {{Citation |last=Olazaran |first=Mikel |title=Advances in Computers Volume 37 |chapter=A Sociological History of the Neural Network Controversy |date=1993-01-01 |chapter-url=https://www.sciencedirect.com/science/article/pii/S0065245808604088 |volume=37 |pages=335–425 |editor-last=Yovits |editor-first=Marshall C. |access-date=2023-10-31 |publisher=Elsevier|doi=10.1016/S0065-2458(08)60408-8 |isbn=9780120121373 |chapter-url-access=subscription }}

Category:Computer science books