> For the complete documentation index, see [llms.txt](https://smousavi05.gitbook.io/mlseismology/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://smousavi05.gitbook.io/mlseismology/machine-learning-terms-and-methods.md). # Machine Learning Terms and Methods ### Active Learning A [**training**](https://developers.google.com/machine-learning/glossary/#training) approach in which the algorithm *chooses* some of the data it learns from. Active learning is particularly valuable when [**labeled examples**](https://developers.google.com/machine-learning/glossary/#labeled_example) are scarce or expensive to obtain. Instead of blindly seeking a diverse range of labeled examples, an active learning algorithm selectively seeks the particular range of examples it needs for learning. ### **AlexNet** AlexNet is the name of a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN) architecture [AlexNet: The Architecture that Challenged CNNs](https://towardsdatascience.com/alexnet-the-architecture-that-challenged-cnns-e406d5297951) ### ### Attention Attention Mechanisms are inspired by human visual attention, the ability to focus on specific parts of an image. Attention mechanisms can be incorporated in both Language Processing and Image Recognition architectures to help the network learn what to “focus” on when making predictions. * [Attention and Memory in Deep Learning and NLP](http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/) Any of a wide range of [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network) architecture mechanisms that aggregate information from a set of inputs in a data-dependent manner. A typical attention mechanism might consist of a weighted sum over a set of inputs, where the [**weight**](https://developers.google.com/machine-learning/glossary/#weight) for each input is computed by another part of the neural network. Refer also to [**self-attention**](https://developers.google.com/machine-learning/glossary/#self-attention) and [**multi-head self-attention**](https://developers.google.com/machine-learning/glossary/#multi-head-self-attention), which are the building blocks of [**Transformers**](https://developers.google.com/machine-learning/glossary/#Transformer). ### **Artificial Neural Networks (ANN)** A model that, taking inspiration from the brain, is composed of layers (at least one of which is [**hidden**](https://developers.google.com/machine-learning/glossary/#hidden_layer)) consisting of simple connected units or [**neurons**](https://developers.google.com/machine-learning/glossary/#neuron) followed by nonlinearities. ### **Autoencoder** An Autoencoder is a Neural Network model whose goal is to predict the input itself, typically through a “bottleneck” somewhere in the network. By introducing a bottleneck, we force the network to learn a lower-dimensional representation of the input, effectively compressing the input into a good representation. Autoencoders are related to PCA and other dimensionality reduction techniques, but can learn more complex mappings due to their nonlinear nature. A wide range of autoencoder architectures exist, including [Denoising Autoencoders](http://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf), [Variational Autoencoders](http://arxiv.org/abs/1312.6114), or [Sequence Autoencoders](http://arxiv.org/abs/1511.01432). ### Backpropagation Backpropagation is an algorithm to efficiently calculate the gradients in a Neural Network, or more generally, a feedforward computational graph. It boils down to applying the chain rule of differentiation starting from the network output and propagating the gradients backward. The first uses of backpropagation go back to Vapnik in the 1960’s, but [Learning representations by back-propagating errors](http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html) is often cited as the source. * [Calculus on Computational Graphs: Backpropagation](http://colah.github.io/posts/2015-08-Backprop/) The primary algorithm for performing [**gradient descent**](https://developers.google.com/machine-learning/glossary/#gradient_descent) on [**neural networks**](https://developers.google.com/machine-learning/glossary/#neural_network). First, the output values of each node are calculated (and cached) in a forward pass. Then, the [**partial derivative**](https://developers.google.com/machine-learning/glossary/#partial_derivative) of the error with respect to each parameter is calculated in a backward pass through the graph. ### **CapsNet** A Capsule Neural Network (CapsNet) is a machine learning system that is a type of [artificial neural network](https://en.wikipedia.org/wiki/Artificial_neural_network) (ANN) that can be used to better model hierarchical relationships. The idea is to add structures called “capsules” to a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN), and to reuse output from several of those capsules to form more stable (with respect to various perturbations) representations for higher capsules. The output is a vector consisting of the [probability of an observation](https://en.wikipedia.org/wiki/Realization_\(probability\)), and a [pose for that observation](https://en.wikipedia.org/wiki/Pose_\(computer_vision\)). This vector is similar to what is done for example when doing [classification with localization](https://en.wikipedia.org/w/index.php?title=Classification_with_localization\&action=edit\&redlink=1) in CNNs. Among other benefits, capsnets address the "Picasso problem" in image recognition: images that have all the right parts but that are not in the correct spatial relationship (e.g., in a "face", the positions of the mouth and one eye are switched). For image recognition, capsnets exploit the fact that while viewpoint changes have nonlinear effects at the pixel level, they have linear effects at the part/object level. [Hinton and Google Brain - Capsule Networks](https://www.youtube.com/watch?v=x5Vxk9twXlE) ### Clustering Grouping related [**examples**](https://developers.google.com/machine-learning/glossary/#example), particularly during [**unsupervised learning**](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning). Once all the examples are grouped, a human can optionally supply meaning to each cluster. Many clustering algorithms exist. For example, the [**k-means**](https://developers.google.com/machine-learning/glossary/#k-means) algorithm clusters examples based on their proximity to a [**centroid**](https://developers.google.com/machine-learning/glossary/#centroid). ### Convolutional Layer A layer of a [**deep neural network**](https://developers.google.com/machine-learning/glossary/#deep_model) in which a [**convolutional filter**](https://developers.google.com/machine-learning/glossary/#convolutional_filter) passes along an input matrix. ### Convolutional Neural Network (CNN) A [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network) in which at least one layer is a [**convolutional layer**](https://developers.google.com/machine-learning/glossary/#convolutional_layer). A typical convolutional neural network consists of some combination of the following layers: * [**convolutional layers**](https://developers.google.com/machine-learning/glossary/#convolutional_layer) * [**pooling layers**](https://developers.google.com/machine-learning/glossary/#pooling) * [**dense layers**](https://developers.google.com/machine-learning/glossary/#dense_layer) Convolutional neural networks have had great success in certain kinds of problems, such as image recognition. A CNN uses [convolutions](https://en.wikipedia.org/wiki/Convolution) to connected extract features from local regions of an input. Most CNNs contain a combination of convolutional, [pooling](http://www.wildml.com/deep-learning-glossary/#pooling) and [affine](http://www.wildml.com/deep-learning-glossary/#affine-layer) layers. CNNs have gained popularity particularly through their excellent performance on visual recognition tasks, where they have been setting the state of the art for several years. * [Stanford CS231n class – Convolutional Neural Networks for Visual Recognition](http://cs231n.github.io/) * [Understanding Convolutional Neural Networks for NLP](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/) ### Data Augmentation Artificially boosting the range and number of [**training**](https://developers.google.com/machine-learning/glossary/#training) examples by transforming existing examples to create additional examples. For example, suppose images are one of your features, but your dataset doesn't contain enough image examples for the model to learn useful associations. Ideally, you'd add enough [**labeled**](https://developers.google.com/machine-learning/glossary/#label) images to your dataset to enable your model to train properly. If that's not possible, data augmentation can rotate, stretch, and reflect each image to produce many variants of the original picture, possibly yielding enough labeled data to enable excellent training. ### Decision Tree A model represented as a sequence of branching statements. For example, the following over-simplified decision tree branches a few times to predict the price of a house (in thousands of USD). According to this decision tree, a house larger than 160 square meters, having more than three bedrooms, and built less than 10 years ago would have a predicted price of 510 thousand USD. ### Decoder In general, any ML system that converts from a processed, dense, or internal representation to a more raw, sparse, or external representation. Decoders are often a component of a larger model, where they are frequently paired with an [**encoder**](https://developers.google.com/machine-learning/glossary/#encoder). In [**sequence-to-sequence tasks**](https://developers.google.com/machine-learning/glossary/#sequence-to-sequence-task), a decoder starts with the internal state generated by the encoder to predict the next sequence. Refer to [**Transformer**](https://developers.google.com/machine-learning/glossary/#Transformer) for the definition of a decoder within the Transformer architecture. ### **DenseNet** [Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993) ### **DeepLab** A deep CNN developed for Semantic Image Segmentation. [DeepLab](http://liangchiehchen.com/projects/DeepLab.html) [Semantic Image Segmentation with DeepLab in TensorFlow](http://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html) ### DnCNN It is an efficient deep learning model to estimate a residual image from the input image with the Gaussian noise. [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://arxiv.org/abs/1608.03981) ### ### **Dictionary Learning** It is a branch of signal processing and machine learning that aims at finding a frame (called dictionary) in which some training data admits a sparse representation. The sparser the representation, the better the dictionary. ### ### **Dropout** Dropout is a regularization technique for Neural Networks that prevents overfitting. It prevents neurons from co-adapting by randomly setting a fraction of them to 0 at each training iteration. Dropout can be interpreted in various ways, such as randomly sampling from an exponential number of different networks. Dropout layers first gained popularity through their use in [CNNs](http://www.wildml.com/deep-learning-glossary/#cnn), but have since been applied to other layers, including input embeddings or recurrent networks. * [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) * [Recurrent Neural Network Regularization](http://arxiv.org/abs/1409.2329) A form of [**regularization**](https://developers.google.com/machine-learning/glossary/#regularization) useful in training [**neural networks**](https://developers.google.com/machine-learning/glossary/#neural_network). Dropout regularization works by removing a random selection of a fixed number of the units in a network layer for a single gradient step. The more units dropped out, the stronger the regularization. This is analogous to training the network to emulate an exponentially large ensemble of smaller networks. For full details, see [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf). ### **Dynamic Bayesian Networks** A Dynamic Bayesian Network (DBN) is a [Bayesian network](https://en.wikipedia.org/wiki/Bayesian_network) (BN) which relates variables to each other over adjacent time steps. ### ### Encoder In general, any ML system that converts from a raw, sparse, or external representation into a more processed, denser, or more internal representation. Encoders are often a component of a larger model, where they are frequently paired with a [**decoder**](https://developers.google.com/machine-learning/glossary/#decoder). Some [**Transformers**](https://developers.google.com/machine-learning/glossary/#Transformer) pair encoders with decoders, though other Transformers use only the encoder or only the decoder. Some systems use the encoder's output as the input to a classification or regression network. In [**sequence-to-sequence tasks**](https://developers.google.com/machine-learning/glossary/#sequence-to-sequence-task), an encoder takes an input sequence and returns an internal state (a vector). Then, the [**decoder**](https://developers.google.com/machine-learning/glossary/#decoder) uses that internal state to predict the next sequence. Refer to [**Transformer**](https://developers.google.com/machine-learning/glossary/#Transformer) for the definition of an encoder in the Transformer architecture. ### Ensemble A merger of the predictions of multiple [**models**](https://developers.google.com/machine-learning/glossary/#model). You can create an ensemble via one or more of the following: * different initializations * different [**hyperparameters**](https://developers.google.com/machine-learning/glossary/#hyperparameter) * different overall structure Deep and wide models are a kind of ensemble. ### Feature An input variable used in making [**predictions**](https://developers.google.com/machine-learning/glossary/#prediction). ### Feature Extraction Overloaded term having either of the following definitions: * Retrieving intermediate feature representations calculated by an [**unsupervised**](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning) or pretrained model (for example, [**hidden layer**](https://developers.google.com/machine-learning/glossary/#hidden_layer) values in a [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network)) for use in another model as input. * Synonym for [**feature engineering**](https://developers.google.com/machine-learning/glossary/#feature_engineering). ### ### Federated Learning A distributed machine learning approach that [**trains**](https://developers.google.com/machine-learning/glossary/#training) machine learning [**models**](https://developers.google.com/machine-learning/glossary/#model) using decentralized [**examples**](https://developers.google.com/machine-learning/glossary/#example) residing on devices such as smartphones. In federated learning, a subset of devices downloads the current model from a central coordinating server. The devices use the examples stored on the devices to make improvements to the model. The devices then upload the model improvements (but not the training examples) to the coordinating server, where they are aggregated with other updates to yield an improved global model. After the aggregation, the model updates computed by devices are no longer needed, and can be discarded. Since the training examples are never uploaded, federated learning follows the privacy principles of focused data collection and data minimization. For more information about federated learning, see [this tutorial](https://federated.withgoogle.com/). ### Fine Tuning Perform a secondary optimization to adjust the parameters of an already trained [**model**](https://developers.google.com/machine-learning/glossary/#model) to fit a new problem. Fine tuning often refers to refitting the weights of a trained [**unsupervised**](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning) model to a [**supervised**](https://developers.google.com/machine-learning/glossary/#supervised_machine_learning) model. ### GAN Abbreviation for [**generative adversarial network**](https://developers.google.com/machine-learning/glossary/#generative_adversarial_network). ### **Gaussian Mixture Model (GMM)** A Gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians. ### **Gaussian Process** Gaussian process is a [stochastic process](https://en.wikipedia.org/wiki/Stochastic_process) (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a [multivariate normal distribution](https://en.wikipedia.org/wiki/Multivariate_normal_distribution), i.e. every finite [linear combination](https://en.wikipedia.org/wiki/Linear_combination) of them is normally distributed. The distribution of a Gaussian process is the [joint distribution](https://en.wikipedia.org/wiki/Joint_distribution) of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space. ### ### **G**enerative Adversarial Network (GAN) A system to create new data in which a [**generator**](https://developers.google.com/machine-learning/glossary/#generator) creates data and a [**discriminator**](https://developers.google.com/machine-learning/glossary/#discriminator) determines whether that created data is valid or invalid. ### **GoogleNet** The name of the Convolutional Neural Network architecture that won the ILSVRC 2014 challenge. The network uses [Inception modules](http://www.wildml.com/deep-learning-glossary/#inception-module) to reduce the parameters and improve the utilization of the computing resources inside the network. * [Going Deeper with Convolutions](http://arxiv.org/abs/1409.4842) ### **GRU** The Gated Recurrent Unit is a simplified version of an LSTM unit with fewer parameters. Just like an LSTM cell, it uses a gating mechanism to allow RNNs to efficiently learn long-range dependency by preventing the [vanishing gradient problem](http://www.wildml.com/deep-learning-glossary/#vanishing-gradient-problem). The GRU consists of a reset and update gate that determine which part of the old memory to keep vs. update with new values at the current time step. * [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](http://arxiv.org/abs/1406.1078v3) * [Recurrent Neural Network Tutorial, Part 4 – Implementing a GRU/LSTM RNN with Python and Theano](http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/) ### **Gradient Boosting Machine** It is a [machine learning](https://en.wikipedia.org/wiki/Machine_learning) technique for [regression](https://en.wikipedia.org/wiki/Regression_\(machine_learning\)), [classification](https://en.wikipedia.org/wiki/Classification_\(machine_learning\)) and other tasks, which produces a prediction model in the form of an [ensemble](https://en.wikipedia.org/wiki/Ensemble_learning) of weak prediction models, typically [decision trees](https://en.wikipedia.org/wiki/Decision_tree_learning). When a decision tree is the weak learner, the resulting algorithm is called gradient boosted trees, which usually outperforms [random forest](https://en.wikipedia.org/wiki/Random_forest). It builds the model in a stage-wise fashion like other [boosting](https://en.wikipedia.org/wiki/Boosting_\(machine_learning\)) methods do, and it generalizes them by allowing optimization of an arbitrary [differentiable](https://en.wikipedia.org/wiki/Differentiable_function) [loss function](https://en.wikipedia.org/wiki/Loss_function).
### Ground Truth The correct answer. Reality. Since reality is often subjective, expert [**raters**](https://developers.google.com/machine-learning/glossary/#rater) typically are the proxy for ground truth. ### Hierarchical Clustering A category of [**clustering**](https://developers.google.com/machine-learning/glossary/#clustering) algorithms that create a tree of clusters. Hierarchical clustering is well-suited to hierarchical data, such as botanical taxonomies. There are two types of hierarchical clustering algorithms: * **Agglomerative clustering** first assigns every example to its own cluster, and iteratively merges the closest clusters to create a hierarchical tree. * **Divisive clustering** first groups all examples into one cluster and then iteratively divides the cluster into a hierarchical tree. Contrast with [**centroid-based clustering**](https://developers.google.com/machine-learning/glossary/#centroid_based_clustering). ### **Hidden Markov Model (HMM)** Hidden Markov Model (HMM) is a [statistical](https://en.wikipedia.org/wiki/Statistical_model) [Markov model](https://en.wikipedia.org/wiki/Markov_model) in which the system being [modeled](https://en.wikipedia.org/wiki/Mathematical_model) is assumed to be a [Markov process](https://en.wikipedia.org/wiki/Markov_process). ### Hyperparameter The "knobs" that you tweak during successive runs of training a model. For example, [**learning rate**](https://developers.google.com/machine-learning/glossary/#learning_rate) is a hyperparameter. Contrast with [**parameter**](https://developers.google.com/machine-learning/glossary/#parameter). ### **Inception** Inception Modules are used in Convolutional Neural Networks to allow for more efficient computation and deeper Networks trough a dimensionality reduction with stacked 1×1 convolutions. * [Going Deeper with Convolutions](http://arxiv.org/abs/1409.4842) ### ### Inference In machine learning, often refers to the process of making predictions by applying the trained model to [**unlabeled examples**](https://developers.google.com/machine-learning/glossary/#unlabeled_example). In statistics, inference refers to the process of fitting the parameters of a distribution conditioned on some observed data. (See the [Wikipedia article on statistical inference](https://wikipedia.org/wiki/Statistical_inference).) ### Interpretability The ability to explain or to present an ML model's reasoning in understandable terms to a human. ### K-Means A popular [**clustering**](https://developers.google.com/machine-learning/glossary/#clustering) algorithm that groups examples in unsupervised learning. The k-means algorithm basically does the following: * Iteratively determines the best k center points (known as [**centroids**](https://developers.google.com/machine-learning/glossary/#centroid)). * Assigns each example to the closest centroid. Those examples nearest the same centroid belong to the same group. The k-means algorithm picks centroid locations to minimize the cumulative *square* of the distances from each example to its closest centroid. ### **K-Nearest Neighbors (KNN)** It is used for [classification](https://en.wikipedia.org/wiki/Statistical_classification) and [regression](https://en.wikipedia.org/wiki/Regression_analysis). In both cases, the input consists of the k closest training examples in [data set](https://en.wikipedia.org/wiki/Data_set).. k-NN is a type of [classification](https://en.wikipedia.org/wiki/Classification) where the function is only approximated locally and all computation is deferred until function evaluation. Since this algorithm relies on distance for classification, if the features represent different physical units or come in vastly different scales then [normalizing](https://en.wikipedia.org/wiki/Normalization_\(statistics\)) the training data can improve its accuracy dramatically. Both for classification and regression, a useful technique can be to assign weights to the contributions of the neighbors, so that the nearer neighbors contribute more to the average than the more distant ones. For example, a common weighting scheme consists in giving each neighbor a weight of 1/d, where d is the distance to the neighbor. The neighbors are taken from a set of objects for which the class (for k-NN classification) or the object property value (for k-NN regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required. ### Label In supervised learning, the "answer" or "result" portion of an [**example**](https://developers.google.com/machine-learning/glossary/#example). Each example in a labeled dataset consists of one or more features and a label. For instance, in a housing dataset, the features might include the number of bedrooms, the number of bathrooms, and the age of the house, while the label might be the house's price. In a spam detection dataset, the features might include the subject line, the sender, and the email message itself, while the label would probably be either "spam" or "not spam." ### Layer A set of [**neurons**](https://developers.google.com/machine-learning/glossary/#neuron) in a [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network) that process a set of input features, or the output of those neurons. Also, an abstraction in TensorFlow. Layers are Python functions that take [**Tensors**](https://developers.google.com/machine-learning/glossary/#tensor) and configuration options as input and produce other tensors as output. ### **LeNet** LeNet is a **convolutional neural network structure** proposed by Yann LeCun et al. ... Convolutional neural networks are a kind of feed-forward neural network whose artificial neurons can respond to a part of the surrounding cells in the coverage range and perform well in large-scale image processing. ### ### Linear Discriminant Analysis **(LDA)** Linear discriminant analysis (LDA), normal discriminant analysis (NDA), or discriminant function analysis is a generalization of Fisher's linear discriminant, a method used in [statistics](https://en.wikipedia.org/wiki/Statistics) and other fields, to find a [linear combination](https://en.wikipedia.org/wiki/Linear_combination) of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a [linear classifier](https://en.wikipedia.org/wiki/Linear_classifier), or, more commonly, for [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction) before later [classification](https://en.wikipedia.org/wiki/Statistical_classification) ### ### **Logistic Regression (LR)** A [**classification model**](https://developers.google.com/machine-learning/glossary/#classification_model) that uses a [**sigmoid function**](https://developers.google.com/machine-learning/glossary/#sigmoid_function) to convert a [**linear model's**](https://developers.google.com/machine-learning/glossary/#linear_model) raw prediction (y′) into a value between 0 and 1. You can interpret the value between 0 and 1 in either of the following two ways: * As a probability that the example belongs to the [**positive class**](https://developers.google.com/machine-learning/glossary/#positive_class) in a binary classification problem. * As a value to be compared against a [**classification threshold**](https://developers.google.com/machine-learning/glossary/#classification_threshold). If the value is equal to or above the classification threshold, the system classifies the example as the positive class. Conversely, if the value is below the given threshold, the system classifies the example as the [**negative class**](https://developers.google.com/machine-learning/glossary/#negative_class). For example, suppose the classification threshold is 0.82: * Imagine an example that produces a raw prediction (y′) of 2.6. The sigmoid of 2.6 is 0.93. Since 0.93 is greater than 0.82, the system classifies this example as the positive class. * Imagine a different example that produces a raw prediction of 1.3. The sigmoid of 1.3 is 0.79. Since 0.79 is less than 0.82, the system classifies that example as the negative class. Although logistic regression is often used in [**binary classification**](https://developers.google.com/machine-learning/glossary/#binary_classification) problems, logistic regression can also be used in [**multi-class classification**](https://developers.google.com/machine-learning/glossary/#multi-class) problems (where it becomes called **multi-class logistic regression** or **multinomial regression**). ### Long Short-Term Memory (LSTM) A type of cell in a [**recurrent neural network**](https://developers.google.com/machine-learning/glossary/#recurrent_neural_network) used to process sequences of data in applications such as handwriting recognition, machine translation, and image captioning. LSTMs address the [**vanishing gradient problem**](https://developers.google.com/machine-learning/glossary/#vanishing_gradient_problem) that occurs when training RNNs due to long data sequences by maintaining history in an internal memory state based on new input and context from previous cells in the RNN. Long Short-Term Memory networks were invented to prevent the [vanishing gradient problem](http://www.wildml.com/deep-learning-glossary/#vanishing-gradient-problem) in Recurrent Neural Networks by using a memory gating mechanism. Using LSTM units to calculate the hidden state in an RNN we help to the network to efficiently propagate gradients and learn long-range dependencies. * [Long Short-Term Memory](http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf) * [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) * [Recurrent Neural Network Tutorial, Part 4 – Implementing a GRU/LSTM RNN with Python and Theano](http://www.wildml.com/2015/10/recurrent-neural-network-tutorial-part-4-implementing-a-grulstm-rnn-with-python-and-theano/) ### Loss A measure of how far a model's [**predictions**](https://developers.google.com/machine-learning/glossary/#prediction) are from its [**label**](https://developers.google.com/machine-learning/glossary/#label). Or, to phrase it more pessimistically, a measure of how bad the model is. To determine this value, a model must define a loss function. For example, linear regression models typically use [**mean squared error**](https://developers.google.com/machine-learning/glossary/#MSE) for a loss function, while logistic regression models use [**Log Loss**](https://developers.google.com/machine-learning/glossary/#Log_Loss). ### LSTM Abbreviation for [**Long Short-Term Memory**](https://developers.google.com/machine-learning/glossary/#Long_Short-Term_Memory). ### Meta-Learning A subset of machine learning that discovers or improves a learning algorithm. A meta-learning system can also aim to train a model to quickly learn a new task from a small amount of data or from experience gained in previous tasks. Meta-learning algorithms generally try to achieve the following: * Improve/learn hand-engineered features (such as an initializer or an optimizer). * Be more data-efficient and compute-efficient. * Improve generalization. Meta-learning is related to [**few-shot learning**](https://developers.google.com/machine-learning/glossary/#few-shot_learning). ### Model Training The process of determining the best [**model**](https://developers.google.com/machine-learning/glossary/#model). ### **Multilayer Perceptron (MLP)** A Multilayer Perceptron is a Feedforward Neural Network with multiple fully-connected layers that use nonlinear [activation functions](http://www.wildml.com/deep-learning-glossary/#activation-function) to deal with data which is not linearly separable. An MLP is the most basic form of a multilayer Neural Network, or a deep Neural Networks if it has more than 2 layers. ### Naive Bayes Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable. ### Objective Function The mathematical formula or metric that a model aims to optimize. For example, the objective function for [**linear regression**](https://developers.google.com/machine-learning/glossary/#linear_regression) is usually [**squared loss**](https://developers.google.com/machine-learning/glossary/#squared_loss). Therefore, when training a linear regression model, the goal is to minimize squared loss. In some cases, the goal is to maximize the objective function. For example, if the objective function is accuracy, the goal is to maximize accuracy. See also [**loss**](https://developers.google.com/machine-learning/glossary/#loss). ### Performance Overloaded term with the following meanings: * The traditional meaning within software engineering. Namely: How fast (or efficiently) does this piece of software run? * The meaning within machine learning. Here, performance answers the following question: How correct is this [**model**](https://developers.google.com/machine-learning/glossary/#model)? That is, how good are the model's predictions? ### ### **PINN** Physics informed neural networks, neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear [partial differential equations](https://en.wikipedia.org/wiki/Partial_differential_equation). [Data-driven solutions and discovery of Nonlinear Partial Differential Equations](https://maziarraissi.github.io/PINNs/) [
](https://github.com/maziarraissi/PINNs) ### **Principal Component Analysis (PCA)** The principal components of a collection of points in a [real coordinate space](https://en.wikipedia.org/wiki/Real_coordinate_space) are a sequence of {\displaystyle p}![p](https://wikimedia.org/api/rest_v1/media/math/render/svg/81eac1e205430d1f40810df36a0edffdc367af36) [unit vectors](https://en.wikipedia.org/wiki/Unit_vector), where the {\displaystyle i}![i](https://wikimedia.org/api/rest_v1/media/math/render/svg/add78d8608ad86e54951b8c8bd6c8d8416533d20)-th vector is the direction of a line that best fits the data while being [orthogonal](https://en.wikipedia.org/wiki/Orthogonal) to the first {\displaystyle i-1}![i-1](https://wikimedia.org/api/rest_v1/media/math/render/svg/9d2ca5c639f26340e0e80f5883cc93a00254513c) vectors. Here, a best-fitting line is defined as one that minimizes the average squared [distance from the points to the line](https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line). These directions constitute an [orthonormal basis](https://en.wikipedia.org/wiki/Orthonormal_basis) in which different individual dimensions of the data are [linearly uncorrelated](https://en.wikipedia.org/wiki/Linear_correlation). Principal component analysis (PCA) is the process of computing the principal components and using them to perform a [change of basis](https://en.wikipedia.org/wiki/Change_of_basis) on the data, sometimes using only the first few principal components and ignoring the rest. PCA is used in [exploratory data analysis](https://en.wikipedia.org/wiki/Exploratory_data_analysis) and for making [predictive models](https://en.wikipedia.org/wiki/Predictive_modeling). It is commonly used for [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction) by projecting each data point onto only the first few principal components to obtain lower-dimensional data while preserving as much of the data's variation as possible. The first principal component can equivalently be defined as a direction that maximizes the variance of the projected data. The {\displaystyle i}![i](https://wikimedia.org/api/rest_v1/media/math/render/svg/add78d8608ad86e54951b8c8bd6c8d8416533d20)-th principal component can be taken as a direction orthogonal to the first {\displaystyle i-1}![i-1](https://wikimedia.org/api/rest_v1/media/math/render/svg/9d2ca5c639f26340e0e80f5883cc93a00254513c) principal components that maximizes the variance of the projected data. ### Q-learning In reinforcement learning, an algorithm that allows an [**agent**](https://developers.google.com/machine-learning/glossary/#agent) to learn the optimal [**Q-function**](https://developers.google.com/machine-learning/glossary/#q-function) of a [**Markov decision process**](https://developers.google.com/machine-learning/glossary/#markov_decision_process) by applying the [**Bellman equation**](https://developers.google.com/machine-learning/glossary/#bellman_equation). The Markov decision process models an [**environment**](https://developers.google.com/machine-learning/glossary/#environment). ### **R-CNN** Region Based Convolutional Neural Networks (R-CNN) are a family of machine learning models for computer vision and specifically object detection. ### ### Random Forest (RF) An ensemble approach to finding the [**decision tree**](https://developers.google.com/machine-learning/glossary/#decision_tree) that best fits the training data by creating many decision trees and then determining the "average" one. The "random" part of the term refers to building each of the decision trees from a random selection of features; the "forest" refers to the set of decision trees. ### Recurrent Neural Network (RNN) A [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network) that is intentionally run multiple times, where parts of each run feed into the next run. Specifically, hidden layers from the previous run provide part of the input to the same hidden layer in the next run. Recurrent neural networks are particularly useful for evaluating sequences, so that the hidden layers can learn from previous runs of the neural network on earlier parts of the sequence. A RNN models sequential interactions through a hidden state, or memory. It can take up to N inputs and produce up to N outputs. For example, an input sequence may be a sentence with the outputs being the part-of-speech tag for each word (N-to-N). An input could be a sentence, and the output a sentiment classification of the sentence (N-to-1). An input could be a single image, and the output could be a sequence of words corresponding to the description of an image (1-to-N). At each time step, an RNN calculates a new hidden state (“memory”) based on the current input and the previous hidden state. The “recurrent” stems from the facts that at each step the same parameters are used and the network performs the same calculations based on different inputs. * [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) * [Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) ### Reinforcement Learning (RL) A family of algorithms that learn an optimal [**policy**](https://developers.google.com/machine-learning/glossary/#policy), whose goal is to maximize [**return**](https://developers.google.com/machine-learning/glossary/#return) when interacting with an [**environment**](https://developers.google.com/machine-learning/glossary/#environment). For example, the ultimate reward of most games is victory. Reinforcement learning systems can become expert at playing complex games by evaluating sequences of previous game moves that ultimately led to wins and sequences that ultimately led to losses. ### Representation The process of mapping data to useful [**features**](https://developers.google.com/machine-learning/glossary/#feature). ### **ResNet** Deep Residual Networks won the ILSVRC 2015 challenge. These networks work by introducing shortcut connection across stacks of layers, allowing the optimizer to learn “easier” residual mappings instead of the more complicated original mappings. These shortcut connections are similar to [Highway Layers](http://www.wildml.com/deep-learning-glossary/#highway-layer), but they are data-independent and don’t introduce additional parameters or training complexity. ResNets achieved a 3.57% error rate on the ImageNet test set. * [Deep Residual Learning for Image Recognition](http://arxiv.org/abs/1512.03385) ### RNN Abbreviation for [**recurrent neural networks**](https://developers.google.com/machine-learning/glossary/#recurrent_neural_network). ### **SegNet** Similar to U-Net but the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling unlike the U-Net in which the entire features from lower-resolution are passed to the higher-resolution layers. [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://arxiv.org/abs/1511.00561) ### **SENET** [Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507) ### ### **Self Organized Mapping (SOM)** An SOM is a type of [artificial neural network](https://en.wikipedia.org/wiki/Artificial_neural_network) but is trained using [competitive learning](https://en.wikipedia.org/wiki/Competitive_learning) rather than the error-correction learning (e.g., [backpropagation](https://en.wikipedia.org/wiki/Backpropagation) with [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)) used by other artificial neural networks. The SOM is an [unsupervised](https://en.wikipedia.org/wiki/Unsupervised_learning) [machine learning](https://en.wikipedia.org/wiki/Machine_learning) technique used to produce a [low-dimensional](https://en.wikipedia.org/wiki/Dimensionality_reduction) (typically two-dimensional) representation of a higher dimensional data set while preserving the [topological structure](https://en.wikipedia.org/wiki/Topology) of the data. For example, a data set with p variables measured in n observations could be represented as clusters of observations with similar values for the variables. These clusters then could be visualized as a two-dimensional "map" such that observations in proximal clusters have more similar values than observations in distal clusters. This can make high-dimensional data easier to visualize and analyze. ### Self-Supervised Learning A family of techniques for converting an [**unsupervised machine learning**](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning) problem into a [**supervised machine learning**](https://developers.google.com/machine-learning/glossary/#supervised_machine_learning) problem by creating surrogate [**labels**](https://developers.google.com/machine-learning/glossary/#label) from [**unlabeled examples**](https://developers.google.com/machine-learning/glossary/#unlabeled_example). Some [**Transformer**](https://developers.google.com/machine-learning/glossary/#Transformer)-based models such as [**BERT**](https://developers.google.com/machine-learning/glossary/#BERT) use self-supervised learning. Self-supervised training is a [**semi-supervised learning**](https://developers.google.com/machine-learning/glossary/#semi-supervised_learning) approach. ### Semi-Supervised Learning Training a model on data where some of the training examples have labels but others don't. One technique for semi-supervised learning is to infer labels for the unlabeled examples, and then to train on the inferred labels to create a new model. Semi-supervised learning can be useful if labels are expensive to obtain but unlabeled examples are plentiful. [**Self-training**](https://developers.google.com/machine-learning/glossary/#self-training) is one technique for semi-supervised learning. ### Sequence Model A model whose inputs have a sequential dependence. For example, predicting the next video watched from a sequence of previously watched videos. ### **SqueezeNet** SqueezeNet is the name of a deep neural network for computer vision that was released in 2016. [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://arxiv.org/abs/1602.07360) ### ### Sparse Representation A [**representation**](https://developers.google.com/machine-learning/glossary/#representation) of a tensor that only stores nonzero elements. For example, the English language consists of about a million words. Consider two ways to represent a count of the words used in one English sentence: * A **dense representation** of this sentence must set an integer for all one million cells, placing a 0 in most of them, and a low integer into a few of them. * A sparse representation of this sentence stores only those cells symbolizing a word actually in the sentence. So, if the sentence contained only 20 unique words, then the sparse representation for the sentence would store an integer in only 20 cells. ### **Siamese** A Siamese neural network (sometimes called a twin neural network) is an [artificial neural network](https://en.wikipedia.org/wiki/Artificial_neural_network) that uses the same weights while working in tandem on two different input vectors to compute comparable output vectors. ### ### Sparsity The number of elements set to zero (or null) in a vector or matrix divided by the total number of entries in that vector or matrix. For example, consider a 10x10 matrix in which 98 cells contain zero. The calculation of sparsity is as follows:sparsity=98100=0.98 **Feature sparsity** refers to the sparsity of a feature vector; **model sparsity** refers to the sparsity of the model weights. ### **Stochastic Configuration Networks** Stochastic [configuration networks](https://www.sciencedirect.com/topics/computer-science/network-configuration) (SCNs) that employ a supervisory mechanism to automatically and fast construct universal approximators can achieve promising performance for resolving regression problems. ### **Support Vector Machine (SVM)** **A**re [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) models with associated learning [algorithms](https://en.wikipedia.org/wiki/Algorithm) that analyze data for [classification](https://en.wikipedia.org/wiki/Statistical_classification) and [regression analysis](https://en.wikipedia.org/wiki/Regression_analysis). SVMs are one of the most robust prediction methods, being based on statistical learning frameworks. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-[probabilistic](https://en.wikipedia.org/wiki/Probabilistic_classification) [binary](https://en.wikipedia.org/wiki/Binary_classifier) [linear classifier](https://en.wikipedia.org/wiki/Linear_classifier) (although methods such as [Platt scaling](https://en.wikipedia.org/wiki/Platt_scaling) exist to use SVM in a probabilistic classification setting). SVM maps training examples to points in space so as to maximise the width of the gap between the two categories. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall. In addition to performing [linear classification](https://en.wikipedia.org/wiki/Linear_classifier), SVMs can efficiently perform a non-linear classification using what is called the [kernel trick](https://en.wikipedia.org/wiki/Kernel_method#Mathematics:_the_kernel_trick), implicitly mapping their inputs into high-dimensional feature spaces.
**TCN**\ Temporal Convolutional Networks, are convolutional neural networks with dilations used for time series data modeling particularly. [**Temporal Convolutional Networks, The Next Revolution for Time-Series?**](https://towardsdatascience.com/temporal-convolutional-networks-the-next-revolution-for-time-series-8990af826567) ### ### Supervised Learning Training a [**model**](https://developers.google.com/machine-learning/glossary/#model) from input data and its corresponding [**labels**](https://developers.google.com/machine-learning/glossary/#label). Supervised machine learning is analogous to a student learning a subject by studying a set of questions and their corresponding answers. After mastering the mapping between questions and answers, the student can then provide answers to new (never-before-seen) questions on the same topic. Compare with [**unsupervised machine learning**](https://developers.google.com/machine-learning/glossary/#unsupervised_machine_learning). ### Transfer Learning Transferring information from one machine learning task to another. For example, in multi-task learning, a single model solves multiple tasks, such as a [**deep model**](https://developers.google.com/machine-learning/glossary/#deep_model) that has different output nodes for different tasks. Transfer learning might involve transferring knowledge from the solution of a simpler task to a more complex one, or involve transferring knowledge from a task where there is more data to one where there is less data. Most machine learning systems solve a *single* task. Transfer learning is a baby step towards artificial intelligence in which a single program can solve *multiple* tasks. ### Transformer A [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network) architecture developed at Google that relies on [**self-attention**](https://developers.google.com/machine-learning/glossary/#self-attention) mechanisms to transform a sequence of input [**embeddings**](https://developers.google.com/machine-learning/glossary/#embeddings) into a sequence of output embeddings without relying on [**convolutions**](https://developers.google.com/machine-learning/glossary/#convolution) or [**recurrent neural networks**](https://developers.google.com/machine-learning/glossary/#recurrent_neural_network). A Transformer can be viewed as a stack of self-attention layers. A Transformer can include any of the following: * an [**encoder**](https://developers.google.com/machine-learning/glossary/#encoder) * a [**decoder**](https://developers.google.com/machine-learning/glossary/#decoder) * both an encoder and decoder An **encoder** transforms a sequence of embeddings into a new sequence of the same length. An encoder includes N identical layers, each of which contains two sub-layers. These two sub-layers are applied at each position of the input embedding sequence, transforming each element of the sequence into a new embedding. The first encoder sub-layer aggregates information from across the input sequence. The second encoder sub-layer transforms the aggregated information into an output embedding. A **decoder** transforms a sequence of input embeddings into a sequence of output embeddings, possibly with a different length. A decoder also includes N identical layers with three sub-layers, two of which are similar to the encoder sub-layers. The third decoder sub-layer takes the output of the encoder and applies the [**self-attention**](https://developers.google.com/machine-learning/glossary/#self-attention) mechanism to gather information from it. The blog post [Transformer: A Novel Neural Network Architecture for Language Understanding](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html) provides a good introduction to Transformers. ### Translational Invariance In an image classification problem, an algorithm's ability to successfully classify images even when the position of objects within the image changes. For example, the algorithm can still identify a dog, whether it is in the center of the frame or at the left end of the frame. See also [**size invariance**](https://developers.google.com/machine-learning/glossary/#size_invariance) and [**rotational invariance**](https://developers.google.com/machine-learning/glossary/#rotational_invariance). ### **U-Net** **U-Net** is a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) that was developed for biomedical [image segmentation](https://en.wikipedia.org/wiki/Image_segmentation)**.** The network is based on the fully convolutional network and its architecture was modified and extended to work with fewer training images and to yield more precise segmentations. [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597) ### ### Underfitting Producing a model with poor predictive ability because the model hasn't captured the complexity of the training data. Many problems can cause underfitting, including: * Training on the wrong set of features. * Training for too few epochs or at too low a learning rate. * Training with too high a regularization rate. * Providing too few hidden layers in a deep neural network. ### Unsupervised Learning Training a [**model**](https://developers.google.com/machine-learning/glossary/#model) to find patterns in a dataset, typically an unlabeled dataset. The most common use of unsupervised machine learning is to cluster data into groups of similar examples. For example, an unsupervised machine learning algorithm can cluster songs together based on various properties of the music. The resulting clusters can become an input to other machine learning algorithms (for example, to a music recommendation service). Clustering can be helpful in domains where true labels are hard to obtain. For example, in domains such as anti-abuse and fraud, clusters can help humans better understand the data. Another example of unsupervised machine learning is [principal component analysis (PCA)](https://wikipedia.org/wiki/Principal_component_analysis). For example, applying PCA on a dataset containing the contents of millions of shopping carts might reveal that shopping carts containing lemons frequently also contain antacids. Compare with [**supervised machine learning**](https://developers.google.com/machine-learning/glossary/#supervised_machine_learning). ### Vanishing Gradient The tendency for the gradients of early [**hidden layers**](https://developers.google.com/machine-learning/glossary/#hidden_layer) of some [**deep neural networks**](https://developers.google.com/machine-learning/glossary/#deep_neural_network) to become surprisingly flat (low). Increasingly lower gradients result in increasingly smaller changes to the weights on nodes in a deep neural network, leading to little or no learning. Models suffering from the vanishing gradient problem become difficult or impossible to train. [**Long Short-Term Memory**](https://developers.google.com/machine-learning/glossary/#Long_Short-Term_Memory) cells address this issue. Compare to [**exploding gradient problem**](https://developers.google.com/machine-learning/glossary/#exploding_gradient_problem). ### **VGG** VGG refers to convolutional neural network model that secured the first and second place in the 2014 ImageNet localization and classification tracks, respectively. The VGG model consist of 16–19 weight layers and uses small convolutional filters of size 3×3 and 1×1. * [Very Deep Convolutional Networks for Large-Scale Image Recognition](http://arxiv.org/abs/1409.1556) ### **WaveNet** A deep generative model of raw audio waveforms. We show that WaveNets are able to generate speech which mimics any human voice and which sounds more natural than the best existing Text-to-Speech systems, reducing the gap with human performance by over 50%. [WaveNet: A generative model for raw audio](https://deepmind.com/blog/article/wavenet-generative-model-raw-audio) ### ### Wasserstein loss One of the loss functions commonly used in [**generative adversarial networks**](https://developers.google.com/machine-learning/glossary/#generative_adversarial_network), based on the [**earth mover's distance**](https://developers.google.com/machine-learning/glossary/#earth-movers-distance) between the distribution of generated data and real data. ### Weight A coefficient for a [**feature**](https://developers.google.com/machine-learning/glossary/#feature) in a linear model, or an edge in a deep network. The goal of training a linear model is to determine the ideal weight for each feature. If a weight is 0, then its corresponding feature does not contribute to the model. ### Width The number of [**neurons**](https://developers.google.com/machine-learning/glossary/#neuron) in a particular [**layer**](https://developers.google.com/machine-learning/glossary/#layer) of a [**neural network**](https://developers.google.com/machine-learning/glossary/#neural_network). ### **YOLO** \ The “You Only Look Once,” or YOLO, family of models are a series of end-to-end deep learning models designed for fast object detection, developed by Joseph Redmon, et al. and first described in the 2015 paper titled “[You Only Look Once: Unified, Real-Time Object Detection](https://pjreddie.com/darknet/yolo/).”