Brief History of AI

McCulloch and Pitts, in "A Logical Calculus of the Ideas Immanent in Nervous Activity", proposed logical models to replicate the activity of biological neurons. The mathematical neuron was a fascinating concept even without learning mechanisms. Yet it will form the basis for artificial neural networks and Deep Learning.

Rosenblatt, in his technical report "The Perceptron: A Perceiving and Recognizing Automaton", developed a practical version of the McCulloch-Pitts neuron, "Perceptron", capable of performing binary classification automatically. A breakthrough that was to lead to a revolution in artificial neural networks in the years to come.

Alexey Grigoryevich Ivakhnenko and Valentin Grigoryevich Lapa develop a hierarchical representation of a neural network that incorporates a polynomial activation function and is thus trained using the Group Method of Data Handling (GMDH). It is now considered the first multilayer perceptron and Ivakhnenko is widely regarded as the father of Deep Learning.

Alexey Grigoryevich Ivakhnenko describes in "Polynomial Theory of Complex Systems", a Deep Learning network with 8 layers trained with his much-cited method Group Method of Data Handling, which is still popular in the new millennium, where much of machine learning originated.

Kunihiko Fukushima proposed the Neocognitron, a hierarchical, multilayer artificial neural network, for a Mechanism of Pattern Recognition. It was used for handwritten Japanese character recognition and other pattern recognition tasks, and inspired convolutional neural networks.

John Hopfield introduced an artificial neural network, Hopfield Network, to collect and retrieve memories like the human brain. It has only one layer of neurons related to the size of input and output, which must be the same. It serves as a content-addressable memory system and would be crucial for further Recurrent Neural Network (RNN) models in the modern Deep Learning era.

Terrence Sejnowski and Charles Rosenberg developed NETtalk, an artificial neural network, to create simplified models that illustrate the complexity of learning human-level cognitive tasks. NETtalk is a program implemented as a connectionist model that can learn to perform a comparable task. For example, it can learn to pronounce written English text by displaying text as input and comparing phonetic transcriptions.

Geoffrey Hinton, Rumelhart, and Williams describe a new learning procedure, back-propagation, for networks of neuron-like units. The procedure iteratively adjusts the weights of the network's connections in order to minimize a measure of the difference between the net's actual output vector and the desired output vector. Back-propagation differs from earlier, simpler methods such as the perceptron-convergence procedure in its ability to generate useful new features.

Paul Smolensky has developed a variant of the Boltzmann machine in which there are no connections between the input and hidden layers. It is known as the Restricted Boltzmann Machine (RBM). It will become popular in the coming years, especially for building recommender systems.

Yann LeCun uses backpropagation to train a convolutional neural network that has been successfully applied to the recognition of handwritten ZIP codes from the U.S. Postal Service. A single network learns the entire recognition process, from the normalized image of the character to the final classification. This is a groundbreaking moment, as it lays the foundation for modern computer vision with Deep Learning.

Sepp Hochreiter and Juergen Schmidhuber publish a groundbreaking paper on "Long Short-Term Memory" (LSTM). This is an advanced architecture for recurrent neural networks that enables information preservation and solves the vanishing gradient problem for RNNs. LSTM will revolutionize Deep Learning in the coming decades.

Geoffrey Hinton, Ruslan Salakhutdinov, Osindero, and Teh publish the paper "A fast learning algorithm for deep belief nets" in which they layer multiple RBMs and call them Deep Belief Networks. DBNs can be viewed as a composition of simple unsupervised networks such as Restricted Boltzmann Machines (RBMs) or autoencoders. The hidden layer of each subnet serves as the visible layer for the next layer and the connections between layers, but not within layers. This composition leads to a fast, layer-by-layer unsupervised training procedure, and the training process is much more efficient for large datasets.

Stanford University researchers Rajat Raina, Anand Madhavan and Andrew Ng wrote a seminal paper in 2009 discussing the promise of this technology for machine learning applications. It was not long before GPUs became the de facto standard for processing deep machine learning algorithms. These systems were considered more effective at massively parallel processing of repeatable, identical computations for machine learning compared to their CPU counterparts, which process multiple, more complex computations simultaneously.

The ImageNet project, inspired by AI researcher Fei-Fei Li, names a large visual database developed for visual object recognition research. More than 14 million images have been hand-annotated by the project, of which at least one million of the images labeled in more than 20,000 categories. The ImageNet project holds an annual software competition, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), in which software programs compete to correctly classify and recognize objects and scenes. Availability of this free database is critical for extending and improving AI algorithms.

In their paper "Deep Sparse Rectifier Neural Networks," Yoshua Bengio, Antoine Bordes, and Xavier Glorot demonstrate that the ReLU activation function can avoid the vanishing gradient problem. This means that, in addition to GPUs, the deep learning community now has another tool to avoid issues with long and inefficient deep neural network training times.

AlexNet is the name of a convolutional neural network (CNN) architecture developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. The key finding of the original work was that the depth of the model was critical to its high performance, which was computationally intensive but enabled by the use of graphics processing units (GPUs) during training. AlexNet is widely considered to be one of the most influential works in the field of computer vision and inspired many other papers to use CNNs and GPUs to accelerate Deep Learning.

The Generative Adversarial Neural Network, also known as GAN, was developed by Ian Goodfellow. Given a training set, this technique learns to generate new data with the same statistics as the training set. GANs open up entirely new applications for Deep Learning in fashion, art, and science because of their ability to synthesize real-world data.

DeepMind made headlines in 2016 after its program AlphaGo beat human Go pro Lee Sedol, the world champion. A more general program, AlphaZero, beat the most powerful programs playing Go, chess and shogi (Japanese chess) after a few days of playing against itself using reinforcement learning, taking the promise of Deep Learning to a whole new level.

The Transformer, a recurrent neural network, is designed to process sequential input data, such as natural language. However, unlike RNNs, Transformers do not necessarily process data sequentially. Rather, the attention mechanism provides context for each position in the input sequence by weighting the importance of each piece of input data differently. 

For example, if the input data is a natural language sentence, the transformer does not need to process the beginning of the sentence before the end. Rather, it identifies the context that confers meaning to each word in the sentence. This feature allows for more parallelization than RNNs and thus reduces training times. 

It is used primarily in natural language processing (NLP) and computer vision (CV).

Open AI has unveiled the GPT-3 natural language processing algorithm, which has the remarkable ability to produce human-like text in response to a prompt. GPT-3 is now considered the largest and most advanced language model in the world, with 175 billion parameters trained on Microsoft Azure's AI supercomputer.

GPT-3 has 175 billion parameters—10 times more than its predecessor, GPT-2. But GPT-3 is dwarfed by the class of 2021. Jurassic-1, a commercially available large language model launched by US startup AI21 Labs in September, edged out GPT-3 with 178 billion parameters. Gopher, a new model released by DeepMind in December, has 280 billion parameters. Megatron-Turing NLG has 530 billion. Google’s Switch-Transformer and GLaM models have one and 1.2 trillion parameters, respectively.

The trend is not just in the US. This year the Chinese tech giant Huawei built a 200-billion-parameter language model called PanGu. Inspur, another Chinese firm, built Yuan 1.0, a 245-billion-parameter model. Baidu and Peng Cheng Laboratory, a research institute in Shenzhen, announced PCL-BAIDU Wenxin, a model with 280 billion parameters that Baidu is already using in a variety of applications, including internet search, news feeds, and smart speakers. And the Beijing Academy of AI announced Wu Dao 2.0, which has 1.75 trillion parameters.

Meanwhile, South Korean internet search firm Naver announced a model called HyperCLOVA, with 204 billion parameters.

For all the effort put into building new language models this year, AI is still stuck in GPT-3’s shadow. “We thought we needed a new idea, but we got there just by scale”. Yet, “In 10 or 20 years, large-scale models will be the norm”. (Will Douglas Heaven)