The contentious/storied history of the 𝐛𝐚𝐜𝐤𝐩𝐫𝐨𝐩𝐚𝐠𝐚𝐭𝐢𝐨𝐧 algorithm that ignited the 𝐝𝐞𝐞𝐩 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 revolution: Geoff Hinton and co. are often credited for it, but even he acknowledges the algorithm’s precedents. Why then do we regard the 𝐑𝐮𝐦𝐞𝐥𝐡𝐚𝐫𝐭, 𝐇𝐢𝐧𝐭𝐨𝐧 𝐚𝐧𝐝 𝐖𝐢𝐥𝐥𝐢𝐚𝐦𝐬 𝟏𝟗𝟖𝟔 𝐩𝐚𝐩𝐞𝐫 as the one that ignited modern ML?⁣ ⁣ In 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, I tell the story of this paper and point out that this short 3-pager was the first to clearly connect the training of hidden-layer neural networks to backpropagation 𝘢𝘯𝘥 point out 𝘸𝘩𝘢𝘵 the network learns as a result: it learns the features/representations necessary to make inferences.⁣ ⁣ But the history of backpropagation goes back to 𝐅𝐫𝐚𝐧𝐤 𝐑𝐨𝐬𝐞𝐧𝐛𝐥𝐚𝐭𝐭, who in his 1961 tome, 𝘛𝘩𝘦 𝘗𝘳𝘪𝘯𝘤𝘪𝘱𝘭𝘦𝘴 𝘰𝘧 𝘕𝘦𝘶𝘳𝘰𝘥𝘺𝘯𝘢𝘮𝘪𝘤𝘴, realized that training multilayer neural networks would require one to develop a ‘𝘣𝘢𝘤𝘬𝘱𝘳𝘰𝘱𝘢𝘨𝘢𝘵𝘪𝘯𝘨 𝘦𝘳𝘳𝘰𝘳 𝘤𝘰𝘳𝘳𝘦𝘤𝘵𝘪𝘰𝘯 𝘱𝘳𝘰𝘤𝘦𝘥𝘶𝘳𝘦’. He just didn’t know how to do it.⁣ ⁣ The procedure was elucidated in stages, and for other uses besides neural networks, including, but not limited to: control and aeronautics engineers 𝐇𝐞𝐧𝐫𝐲 𝐊𝐞𝐥𝐥𝐲 𝐚𝐧𝐝 𝐀𝐫𝐭𝐡𝐮𝐫 𝐁𝐫𝐲𝐬𝐨𝐧 (1960-61); 𝐒𝐭𝐮𝐚𝐫𝐭 𝐃𝐫𝐞𝐲𝐟𝐮𝐬 (1962); 𝐒𝐡𝐮𝐧’ 𝐢𝐜𝐡𝐢 𝐀𝐦𝐚𝐫𝐢 (1967); 𝐒𝐞𝐩𝐩𝐨 𝐋𝐢𝐧𝐧𝐚𝐢𝐧𝐦𝐚𝐚 (1970); and 𝐏𝐚𝐮𝐥 𝐖𝐞𝐫𝐛𝐨𝐬 (1974)…and then finally, 𝐑𝐮𝐦𝐞𝐥𝐡𝐚𝐫𝐭, 𝐇𝐢𝐧𝐭𝐨𝐧 and 𝐖𝐢𝐥𝐥𝐢𝐚𝐦𝐬 (1986).⁣ ⁣ Rumelhart et al. made some key technical advances: using the 𝐬𝐢𝐠𝐦𝐨𝐢𝐝 activation function, understanding the need for 𝘣𝘳𝘦𝘢𝘬𝘪𝘯𝘨 𝘴𝘺𝘮𝘮𝘦𝘵𝘳𝘺 during the initialization of the weights, and showing how the network learned interesting representations. Rumelhart designed the algorithm, Hinton refined it, and implemented and tested it, Williams focused on the math.⁣ ⁣ Meanwhile, 𝐘𝐚𝐧𝐧 𝐋𝐞𝐂𝐮𝐧, then still a student in France was developing his own version of the algorithm.⁣ ⁣ In 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍, we learn not just of this story of backprop, but the math to train a very simple neural network with one hidden layer, and one neuron per layer. We do it step-by-step. It doesn’t get any easier than that, but what we learn holds regardless of the complexity of the network. The power of backprop is astonishing—and it all comes to the 𝐜𝐡𝐚𝐢𝐧 𝐫𝐮𝐥𝐞 𝐨𝐟 𝐜𝐚𝐥𝐜𝐮𝐥𝐮𝐬!⁣ Hinton said this about Rumelhart, who died in 2011 of a progressive, neurodegenerative disease: “𝘐𝘧 𝘩𝘦 𝘩𝘢𝘥 𝘭𝘪𝘷𝘦𝘥, 𝘩𝘦 𝘸𝘰𝘶𝘭𝘥 𝘩𝘢𝘷𝘦 𝘣𝘦𝘦𝘯 𝘵𝘩𝘦 𝘱𝘦𝘳𝘴𝘰𝘯 𝘸𝘩𝘰 𝘨𝘰𝘵 𝘮𝘰𝘴𝘵 𝘰𝘧 𝘵𝘩𝘦 𝘤𝘳𝘦𝘥𝘪𝘵 𝘧𝘰𝘳 𝘣𝘢𝘤𝘬𝘱𝘳𝘰𝘱.” Here's more info about 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍:⁣ ⁣⁣ 𝐔𝐒: penguinrandomhouse.com/books… 𝐔𝐊: penguin.co.uk/books/446849/w… #ArtificialIntelligence #MachineLearning #deeplearning #neuralnetworks #AI #ML

"𝐘𝐨𝐮 𝐰𝐢𝐥𝐥 𝐜𝐨𝐦𝐞 𝐚𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐭𝐡𝐢𝐬 𝐛𝐨𝐨𝐤 𝐤𝐧𝐨𝐰𝐢𝐧𝐠 𝐦𝐨𝐫𝐞 𝐚𝐛𝐨𝐮𝐭 𝐡𝐨𝐰 [𝘮𝘢𝘤𝘩𝘪𝘯𝘦 𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘢𝘭𝘨𝘰𝘳𝘪𝘵𝘩𝘮𝘴] 𝐰𝐨𝐫𝐤." - New Scientist⁣ ⁣ I think of @newscientist as my "𝐡𝐨𝐦𝐞" -- where I began my writing career and learned my craft from some amazing colleagues (many of whom are dear friends). So, it's lovely to see 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍 reviewed in last week's issue of the mag.⁣ ⁣ Of course, excerpting from reviews is a bit of game, to make oneself look good :-) So, with that in mind, here's another blurb from the review about 𝐖𝐇𝐘 𝐌𝐀𝐂𝐇𝐈𝐍𝐄𝐒 𝐋𝐄𝐀𝐑𝐍: "𝘈 𝘴𝘰𝘭𝘪𝘥 𝘧𝘰𝘶𝘯𝘥𝘢𝘵𝘪𝘰𝘯 𝘵𝘰 𝘣𝘦𝘢𝘳 𝘺𝘰𝘶 𝘪𝘯𝘵𝘰 𝘵𝘩𝘦 𝘸𝘦𝘪𝘳𝘥, 𝘪𝘯𝘤𝘳𝘦𝘢𝘴𝘪𝘯𝘨𝘭𝘺 𝘴𝘺𝘯𝘵𝘩𝘦𝘵𝘪𝘤-𝘧𝘦𝘦𝘭𝘪𝘯𝘨 𝘧𝘶𝘵𝘶𝘳𝘦." For a more nuanced take, the review is here: newscientist.com/article/mg2… #ArtificialIntelligence #deeplearning #MachineLearning #AI #neuralnetworks #math #STEM @DuttonBooks @PenguinUKBooks @AllenLaneBooks