Shortcuts

nets

NETS is divided in five main sub-packages:

  • autograd refers to the autograd system used,

  • nn refers to the core of the package, ie deep network models,

  • optim refers to optimizers used during the backpropagation to update weights and biases,

  • data refers to array container, like training and testing examples with their labels,

  • datasets refers to popular machine learning libraries for image analysis and NLP.

nets

nets.tensor

Defines tensors for deep learning application. A tensor is multi-dimensional array, similar to numpy arrays.

class nets.tensor.Tensor(data, requires_grad=False, hooks=None)[source]

A Tensor is a multi dimensional array that track and record previous gradients, creating a dynamic computational graph.

append(t, axis=0)[source]

Append a value(- or tensor) to a Tensor.

Note

The operation takes place in-place and does not support autograd.

Parameters

t (scale, Tensor) – object to add

Returns

None

astype(new_type)[source]

Set a new type to the Tensor’s data.

Parameters

new_type (type) – new type to convert the data

Returns

None

backward(grad=None)[source]

Compute a single backward pass on all Tensor linked to this one. The Tensor depending to this top-level Tensor``are stored in the ``_hooks attribute. The backward pass compute a gradient back-propagation on all Tensor registered in _hooks. The backward pass gradient in grad attribute (and add upstream gradient if the Tensor is used multiple times).

Parameters

grad (Tensor) – upstream gradient. Default is None, and will be set to Tensor(1.0), a 0-dimensional Tensor.

Returns

None

Note

To be able to back-propagate, the top-level Tensor must have requires_grad set to True to propagate the gradient.

detach()[source]

Unlink the Tensor to the computational graph. By calling this method, the attribute _hooks and grad are set to their default values, None.

Returns

None

flatten()[source]

Flatten a Tensor with a new shape. The transformation is not made in-place.

Returns

Tensor

item()[source]

Get the item (float, int etc.) of a 0-dimensional Tensor. It will detach the tensor from the computational graph by setting _hooks = [] and grad = None to free memory and send this graph to the garbage collector.

Returns

Any

numpy()[source]

Convert the Tensor data to a numpy.ndarray object.

Returns

numpy.ndarray

register_hook(hook)[source]

Register a hook to a Tensor

Parameters

hook (Hook) – hook to register

Returns

None

reshape(*shapes)[source]

Reshape a Tensor with a new shape. The transformation is not made in-place.

Note

The new shape must have the same size of the actual shape. If its not the case, the reshape method will raise an error.

Parameters

int (*shapes) – permutation

Returns

Tensor

sum(axis=None)[source]

Sum the data along a given axis. If no axis are specified, all values within the Tensor will be summed.

Parameters

axis (int) – the index of the axis to sum on.

Returns

Tensor

tolist()[source]

Convert the Tensor data to a list (of list eventually).

Returns

list

transpose(*indices)[source]

Transpose the Tensor. The operation is not in-place.

Parameters

indices (tuple) – permutation

Returns

Tensor

zero_grad()[source]

Set to a zero Tensor the gradient. This is call when initializing a Tensor that requires gradient tracking, or re-initialize parameters’s gradient after a training loop as they accumulate on each other.

Returns

None

nets.tensor.to_array(arrayable)[source]

Convert an object to a numpy.ndarray if possible.

Parameters

arrayable – object to convert

Returns

numpy.ndarray

nets.tensor.to_tensor(tensorable)[source]

Convert an object to a Tensor if possible.

Parameters

tensorable – object to convert

Returns

Tensor

nets.numeric

The functional modules defines basic functions to generate tensors and transform data.

nets.functional

nets.autograd

nets.autograd.functions

This modules defines more complex operations, and defines the most popular functions such as:

  • exp,

  • log,

  • tanh,

  • sigmoid;

  • relu…

nets.autograd.functions.exp(t)[source]

Exponentiation f a tensor.

\[T_{out} = \text{exp}(T)\]
Parameters

t (Tensor) – input tensor to transform

Returns

Tensor

nets.autograd.functions.leaky_relu(x: Array, alpha: Optional[float] = 0.01) → Array[source]

Variation from the original relu function, which can prevent ‘dying’ relu thanks to a slope alpha.

\[\text{leaky_relu(x)} = \max{(\alpha \times x, x)}\]
Parameters

alpha (float, optional) – slope towards \(-\infty\).

Shape:
  • input: x (numpy.array): input to compute the leaky_relu function on.

  • output: y (numpy.array): leaky relu output, with the same shape than x.

../_images/functional_leaky_relu.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> alpha = 0.1
>>> out_array = leaky_relu(in_array, alpha)

See LeakyReLU for the activation implementation.

nets.autograd.functions.leaky_relu_prime(x: Array, alpha: Optional[float] = 0.01) → Array[source]

First order derivative of leaky_relu function, defined as:

\[\begin{split}\text{leaky_relu'(x)} = \begin{cases} 1, &\quad x \ge 0 \\ \alpha, &\quad x < 0. \end{cases}\end{split}\]
Parameters

alpha (float, optional) – slope towards \(-\infty\).

Shape:
  • input: x (numpy.array): input to compute the leaky_relu_prime function on.

  • output: y (numpy.array): derivative of the input, with the same shape than x.

../_images/functional_leaky_relu_prime.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> alpha = 0.1
>>> out_array = leaky_relu_prime(in_array, alpha)

See LeakyReLU for the activation implementation.

nets.autograd.functions.log(t)[source]

Compute the logarithm of a tensor object.

\[T_{out} = \text{log}(T)\]
Parameters

t (Tensor like) – input tensor

Returns

Tensor

nets.autograd.functions.maximum(t1, t2)[source]

Get the maximum between two tensors.

Parameters
  • t1 (Tensor) – input tensor to merge

  • t2 (Tensor) – input tensor to merge

Returns

Tensor

nets.autograd.functions.minimum(t1, t2)[source]

Get the minimum between two tensors.

Parameters
  • t1 (Tensor) – input tensor to merge

  • t2 (Tensor) – input tensor to merge

Returns

Tensor

nets.autograd.functions.pow(t, power)[source]

Power a tensor-like object.

\[T_{out} = T^2\]
Parameters
  • t (Tensor like) – reference tensor

  • power (int) – power to elevate a tensor

Returns

Tensor

nets.autograd.functions.relu(x: Array) → Array[source]

relu is a standard activation function, defined as:

\[\text{relu(x)} = \max{(0, x)}\]
Shape:
  • input: x (numpy.array): input to compute the relu function on.

  • output: y (numpy.array): relu output, with the same shape than x.

../_images/functional_relu.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = relu(in_array)

See ReLU for the activation implementation.

nets.autograd.functions.relu_prime(x: Array) → Array[source]

First order derivative of the relu function.

\[\begin{split}\text{relu'(x)} = \begin{cases} 1, &\quad x \ge 0 \\ 0, &\quad x < 0. \end{cases}\end{split}\]
Shape:
  • input: x (numpy.array): input to compute the leaky_relu function on.

  • output: y (numpy.array): gradient of the input, with the same shape than x.

../_images/functional_relu_prime.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = relu_prime(in_array)

See ReLU for the activation implementation.

nets.autograd.functions.sigmoid(x: Array) → Array[source]

Vanilla sigmoid function, defined as:

\[\text{sigmoid}(x) = \frac{1}{1 + e^{-x}}\]
Shape:
  • input: x (numpy.array): input to compute the sigmoid function on.

  • output: y (numpy.array): sigmoid output, with the same shape than x.

../_images/functional_sigmoid.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = sigmoid(in_array)

See Sigmoid for the activation implementation.

nets.autograd.functions.sigmoid_prime(x: Array) → Array[source]

First order derivative of sigmoid function, defined as:

\[\text{sigmoid'}(x) = (1 - \text{sigmoid}(x))\]
Shape:
  • input: x (numpy.array): input to compute the sigmoid derivative function on.

  • output: y (numpy.array): gradient of the input, with the same shape than x.

../_images/functional_sigmoid_prime.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = sigmoid_prime(in_array)

See Sigmoid for the activation implementation.

nets.autograd.functions.softmax(x: Array, axis: Optional[int] = 0) → Tensor[source]

softmax function. The implementation chosen is not straight forward to prevent numerical instability.

The \(i^{th}\) element of a softmax function evaluated on a vector \(x \in \mathbb{R}^n\) is given by:

\[\text{softmax}(x_{i}) = \frac{e^{x_i}}{\sum_{j=1}^{n} e^{x_j}}\]
Parameters

axis (int) – axis to consider for the softmax. Default is 0.

Shape:
  • input: x (numpy.array): input to compute the softmax function on.

  • output: y (numpy.array): hyper-plane of the input.

See Softmax for the activation implementation.

nets.autograd.functions.sqrt(t)[source]

Square root of a tensor-like object.

\[T_{out} = T^2\]
Parameters
  • t (Tensor like) – reference tensor

  • power (int) – power to elevate a tensor

Returns

Tensor

nets.autograd.functions.tanh(t: Tensor) → Tensor[source]

tanh standard function, definead as:

\[\text{tanh}(T) = \frac{e^{T} - e^{-T}}{e^{T} + e^{-T}}\]
Shape:
  • input: x (numpy.array): input to compute the tanh function on.

  • output: y (numpy.array): tanh output, with the same shape than \(T\).

../_images/functional_tanh.png

Examples:

>>> import nets
>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = nets.tanh(in_array)

See Tanh for the activation implementation.

nets.autograd.functions.tanh_prime(x: Array) → Array[source]

First order derivative of tanh function, defined as:

\[\text{tanh'}(x) = 1 - \text{tanh}^2(x)\]
Shape:
  • input: x (numpy.array): input to compute the tanh derivative function on.

  • output: y (numpy.array): gradient of the input, with the same shape than x.

../_images/functional_tanh_prime.png

Examples:

>>> in_array = np.array([-5, 2, 6, -2, 4])
>>> out_array = tanh(in_array)

See Tanh for the activation implementation.

nets.autograd.functions.where(cond, t1, t2)[source]

Transformation regarding a condition.

\[\begin{split}T_{out} = \begin{cases} T_1, &\quad if \quad condition \\ T_2, &\quad else. \end{cases}\end{split}\]
Parameters
  • cond (bool) – condition to merge two tensors

  • t1 (Tensor) – input tensor to merge

  • t2 (Tensor) – input tensor to merge

Returns

Tensor

nets.autograd.hook

A Hook keeps track of gradients and operations.

class nets.autograd.hook.Hook(tensor, grad_fn)[source]

A hook is a collection of tensors and gradients.

nets.autograd.numeric

This modules defines basic transformations on a Tensor like transpose or reshape.

nets.autograd.numeric.append(t, value)[source]

Append multiples Tensor from an iterable.

Note

The Tensor in iterable should and must have the same shape.

Parameters

t (Tensor) – list containing Tensor to concatenate.

Returns

the concatenation of all Tensor.

Return type

Tensor

nets.autograd.numeric.argmax(t, axis=None)[source]

Get the indices of maximum elements from a Tensor.

Parameters
  • t (Tensor) – tensor get maximum indices from

  • axis (int, optional) – index of the axis. Default is None.

Returns

Tensor

nets.autograd.numeric.concatenate(iterable)[source]

Concatenate multiples Tensor from an iterable.

Note

The Tensor in iterable should and must have the same shape.

Parameters

iterable (tuple, list) – list containing Tensor to concatenate.

Returns

the concatenation of all Tensor.

Return type

Tensor

nets.autograd.numeric.flatten(t)[source]

Reshape in 1-dimensional a Tensor.

Parameters

t (Tensor) – tensor get reshape.

Returns

Tensor

nets.autograd.numeric.max(t, axis=None)[source]

Get the maximum from a Tensor.

Parameters
  • t (Tensor) – tensor to transform

  • axis (int, optional) – index of the axis to search. Default is None.

Returns

Tensor

nets.autograd.numeric.pad(t, padding, constant_values=0)[source]

Reshape a Tensor to a bigger size and add a padding on the side, with a 0 constant value.

Parameters
  • t (Tensor) – tensor to transform

  • padding (tuple) – padding dimensions

  • constant_values (scalar, optional) – scalar affected in the padding

Returns

Tensor

nets.autograd.numeric.reshape(t, shape)[source]

Reshape a Tensor.

Parameters
  • t (Tensor) – tensor to transform

  • shape (tuple) – new shape of t

Returns

Tensor

nets.autograd.numeric.transpose(t, indices=None)[source]

Permutation a tensor object.

Parameters
  • t (Tensor) –

  • indices (tuple, optional) – index to transpose.

Returns

Tensor

nets.autograd.ops

Defines basic operations between two tensors, like addition, subtraction, dot product etc.

nets.autograd.ops.add(t1, t2)[source]

Add two tensor-like together.

\[T_{out} = T_1 + T_2\]
Parameters
  • t1 (Tensor like) – tensor to add

  • t2 (Tensor like) – second tensor to add with

Returns

the sum of two Tensor-like object

Return type

Tensor

nets.autograd.ops.div(t1, t2)[source]

Divide two tensor-like object.

\[T_{out} = T_1 \times \frac{1}{T_2}\]
Parameters
  • t1 (Tensor like) – tensor to multiply

  • t2 (Tensor like) – tensor to invert

Returns

Tensor

nets.autograd.ops.dot(t1, t2)[source]

Dot product of two matrices.

\[T_{out} = (t_{i, j}^{[out]})_{i, j} \quad where \quad t_{i, j}^{[out]} = \sum_{k=1}^{n} t_{i, k}^{[1]} \times t_{k, j}^{[2]}\]
Parameters
  • t1 (Tensor like) –

  • t2 (Tensor like) –

Returns

Tensor

nets.autograd.ops.eq(t, other)[source]

Return a boolean tensor for equal condition.

\[condition = T == other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.ge(t, other)[source]

Return a boolean tensor for greater or equal condition.

\[condition = T \ge other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.gt(t, other)[source]

Return a boolean tensor for greater than condition.

\[condition = T > other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.inverse(t)[source]

Inverse a tensor-like object.

\[T_{out} = \frac{1}{T}\]
Parameters

t (Tensor like) – tensor to inverse.

Returns

Tensor

nets.autograd.ops.le(t, other)[source]

Return a boolean tensor for lower or equal condition.

\[condition = T \le other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.lt(t, other)[source]

Return a boolean tensor for lower than condition.

\[condition = T < other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.multiply(t1, t2)[source]

Elementwise multiplication of two tensors-like object.

\[T_{out} = T_1 \times T_2\]
Parameters
  • t1 (Tensor like) – tensor to multiply

  • t2 (Tensor like) – second tensor to multiply with

Returns

Tensor

nets.autograd.ops.ne(t, other)[source]

Return a boolean tensor for not equal condition.

\[condition = T not other\]
Parameters
  • t (Tensor) – tensor to compare

  • other (Tensor like) – object to compare the tensor

Returns

Tensor

nets.autograd.ops.neg(t)[source]

Oppose the values of a tensor.

\[T_{out} = - T\]
Parameters

t (Tensor) – tensor to oppose.

Returns

Tensor

nets.autograd.ops.slice(t, indices)[source]

Slice a tensor from given indices.

Parameters
  • t (Tensor) – tensor to slice

  • (tuple, int, (idxs) – ): indices to extract data

Returns

Tensor

nets.autograd.ops.sub(t1, t2)[source]

Subtract two tensor-like object

\[T_{out} = T_1 - T_2\]
Parameters
  • t1 (Tensor like) – tensor to subtract

  • t2 (Tensor like) – second tensor to subtract with

Returns

Tensor

nets.autograd.ops.sum(t, axis=None, keepdims=False)[source]

Compute the sum of all elements in a tensor, and update the gradients and hooks.

\[\text{sum} = \sum_{idx} t_{idx}\]
Parameters
  • t (Tensor) – tensor to get the sum from

  • axis (int) – axis to sum

  • keepdims (bool) – keep the same dimension in the resulting Tensor as the input if set to True. Default is False.

Returns

summed tensor

Return type

Tensor

nets.autograd.parameter

A parameter is a trainable tensor.

class nets.autograd.parameter.Parameter(data=None, shape=None)[source]

Instantiate a parameter, made of trainable data. A trainable data is a value that will be updated during the back-propagation, usually it refers to weights and biases of a layer.

classmethod normal(shape, mu=0, sigma=1)[source]

Generate a Parameter following a normal distribution center at mu with a standard deviation of sigma.

Parameters
  • shape (tuple) – shape of the Parameter

  • mu (scalar) – mean of the normal distribution. Default is 0.

  • sigma (scalar) – standard deviation of the normal distribution. Default is 1.

Returns

Parameter

classmethod orthogonal(shape)[source]

Initializes weight parameters orthogonally. From the [exercise 02456 from DTU course](https://github.com/DeepLearningDTU/02456-deep-learning-with-PyTorch).

Note

Refer to [this paper](https://arxiv.org/abs/1312.6120) for an explanation of this initialization.

Parameters

shape (tuple) – shape of dimensionality greater than 2 (weight matrix)

Returns

Parameter

classmethod scaled_weight(input_dim, output_dim)[source]

Scaled initialization from \(He et al.\)

Parameters
  • input_dim (int) – dimension of the input layer

  • output_dim (int) – dimension of the output layer

Returns

Parameter

classmethod uniform(shape, low=-1, high=1)[source]

Generate a Parameter with data following a uniform distribution between lower and upper.

Parameters
  • shape (tuple) – shape of the Parameter

  • low (scalar) – lower bound of the uniform distribution. Default is -1.

  • high (scalar) – upper bound of the uniform distribution. Default is 1.

Returns

Parameter

classmethod zeros(shape)[source]

Generate a zero-Parameter

Parameters

shape (tuple) – shape of the Parameter

Returns

Parameter

nets.nn

The main core of NETS is located in nn package. The neural networks implemented are the most popular ones:

nets.nn.activation

This modules defines all activation function, used in neural networks to add non-linearity from one layer to another.

Usage:

import numpy as np
import nets.nn.activation as A

# Define a ReLU activation function
# This activation function is popular as an hidden activation function
activation_function = A.ReLU()
# Defines 3 1-D array of length 5, stacked together.
batch_input = np.array([[1, -2, 5, 10, -7],
                        [2, 4, 3, -2, 5],
                        [-3, 2, 8, 4, 0]])

# Check the result for one single pass
batch_output = activation_function(batch_input)
print(batch_output)

# Check the backward pass
backward = ReLU.backward(batch_ouput)
print(backward)

Out:

array([[1, 0, 5, 10, 0],
       [2, 4, 3, 0, 5],
       [0, 2, 8, 4, 0]])
array([[1, 0, 1, 1, 0],
       [1, 1, 1, 0, 1],
       [0, 1, 1, 1, 0]])
class nets.nn.activation.Activation(func, func_prime)[source]

An activation modules is a transformation that modify its inputs element wise, usually it uses non-linearity functions.

backward(grad)[source]

One backward step.

forward(inputs)[source]

One forward step. Gradients and outputs should be saved in the _cache when training, to be able to perform the backward pass.

class nets.nn.activation.LeakyReLU[source]

LeakyReLU activation function.

\(\text{LeakyReLU.forward}(x) = \text{leaky_relu(x)} = \max{(\alpha \times x, x)}\)

\(\text{LeakyReLU.backward}(x) = \text{leaky_relu'(x)} = \begin{cases} 1, &\quad x \ge 0 \\ \alpha, &\quad x < 0. \end{cases}\)

../_images/activation_LeakyReLU.png

Examples:

>>> activation = LeakyReLU()
>>> batch_input = np.array([[-5, 2, 6, -2, 4],
...                         [2, 5, -6, 7, -3]])
>>> output = activation(batch_input)
>>> gradient = activation.backward(output)

See leaky_relu() and leaky_relu_derivative() for the functional implementation.

class nets.nn.activation.ReLU[source]

ReLU activation function.

\(\text{ReLU.forward}(x) = \text{relu(x)} = \max{(0, x)}\)

\(\text{ReLU.backward}(x) = \text{relu'(x)} = \begin{cases} 1, &\quad x \ge 0 \\ 0, &\quad x < 0. \end{cases}\)

../_images/activation_ReLU.png

Examples:

>>> activation = LeakyReLU()
>>> batch_input = np.array([[-5, 2, 6, -2, 4],
...                         [2, 5, -6, 7, -3]])
>>> output = activation(batch_input)
>>> gradient = activation.backward(output)

See relu() and relu_derivative() for the functional implementation.

class nets.nn.activation.Sigmoid[source]

Sigmoid activation function.

\(\text{Sigmoid.forward}(x) = \text{sigmoid}(x) = \frac{1}{1 + e^{-x}}\)

\(\text{Sigmoid.backward}(x) = \text{sigmoid'}(x) = (1 - \text{sigmoid}(x))\)

../_images/activation_Sigmoid.png

Examples:

>>> activation = Sigmoid()
>>> batch_input = np.array([[-5, 2, 6, -2, 4],
...                         [2, 5, -6, 7, -3]])
>>> output = activation(batch_input)
>>> gradient = activation.backward(output)

See sigmoid() and sigmoid_derivative() for the functional implementation.

class nets.nn.activation.Softmax(axis=0)[source]

Softmax activation function, which should be used only at the last layer to normalize outputs values between \([0, 1]\).

backward(grad)[source]

One backward step.

forward(inputs)[source]

One forward step. Gradients and outputs should be saved in the _cache when training, to be able to perform the backward pass.

class nets.nn.activation.Tanh[source]

Tanh activation function.

\(\text{Tanh.forward}(x) = \text{tanh}(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}\)

\(\text{Tanh.backward}(x) = \text{tanh'}(x) = 1 - \text{tanh}^2(x)\)

../_images/activation_Tanh.png

Examples:

>>> activation = Tanh()
>>> batch_input = np.array([[-5, 2, 6, -2, 4],
...                         [2, 5, -6, 7, -3]])
>>> output = activation(batch_input)
>>> gradient = activation.backward(output)

See tanh() and tanh_derivative() for the functional implementation.

nets.nn.functional

Defines elementary functions used in Neural Network layers.

nets.nn.functional.dropout(t, prob=0.5)[source]

Zeros elements from a Tensor with a probability prob.

\[\begin{split}\text{dropout}(T) = T \times Z \quad \text{where} Z = (z_{i})_{i} \quad and z_i = \begin{cases} 1, &\quad p \ge prob \\ 0, &\quad else. \end{cases}\end{split}\]
Parameters
  • t (Tensor) – tensor to zeros

  • prob (float [0, 1]) – probability to zero an element

Returns

input tensor with some zeros

Return type

Tensor

nets.nn.loss

Loss functions evaluate the precision and correctness of a model’s predictions. The most popular ones are MSE, CrossEntropyLoss and Adam. Each loss functions presents advantages, and results may vary when choosing one from another.

class nets.nn.loss.CrossEntropyLoss[source]

Cross Entropy Loss. First, a softmax transformation is used to map the predictions between \([0, 1]\), then the cost is computed:

\[\text{CrossEntropyLoss} = - \frac{1}{N} \sum_{i=1}^{c}labels_{i}\log(pred_{i})\]
backward(predictions, labels)[source]

One backward step.

forward(predictions, labels)[source]

Compute the cross entropy cost function.

Parameters
  • predictions (numpy.array) – tensor of un-normalized floats with shape \((N, c)\).

  • labels (numpy.array) – tensor of integer values with shape \((N)\).

Returns

the cost regarding the loss function.

Return type

cost (float)

class nets.nn.loss.Loss[source]

A loss function evaluate the correctness of a set of predictions regarding gold-labels. The predictions should be un-corrected, ie no transformations like Softmax should have been used before. The loss function will do the transformation if necessary. The attribute history keeps track of the cost when the loss function is called.

forward(predictions, labels)[source]

Compute the cross entropy cost function.

Parameters
  • predictions (numpy.array) – tensor of un-normalized floats with shape \((N, c)\).

  • labels (numpy.array) – tensor of integer values with shape \((N)\).

Returns

the cost regarding the loss function.

Return type

cost (float)

class nets.nn.loss.MSELoss[source]

Mean Square Error Loss, defined as:

\[\text{MSE} = \frac{1}{N}\sum_{i=1}^{c}(predictions - labels)^2\]
backward(predictions, labels)[source]

One backward step.

forward(predictions, labels)[source]

Compute the cross entropy cost function.

Parameters
  • predictions (numpy.array) – tensor of un-normalized floats with shape \((N, c)\).

  • labels (numpy.array) – tensor of integer values with shape \((N)\).

Returns

the cost regarding the loss function.

Return type

cost (float)

nets.nn.utils

nets.nn.utils.one_hot(Y, num_classes)[source]

Perform one-hot encoding on input Y.

\[\begin{split}\text{Y'}_{i, j} = \begin{cases} 1, &\quad if \quad Y_i = 0 \\ 0, &\quad else \end{cases}\end{split}\]
Parameters
  • Y (Tensor) – 1D tensor of classes indices of length \(N\)

  • num_classes (int) – number of classes \(c\)

Returns

one hot encoded tensor of shape \((N, c)\)

Return type

Tensor

nets.nn.modules

nets.nn.conv

This modules defines a Convolution Neural Network (CNN) naively. This CNN is for test and comparisons purposes. If you wan to use a more appropriate CNN for your models, use the CNN instead.

class nets.nn.modules.conv.Conv2d(in_channels, out_channels, filter_size, stride=1, pad=0)[source]

Convolutional Neural Networks (CNN) are a class of Neural Networks that use convolution filters. Their particularity is their ability to synthesis information and learn spatial features. They are mainly used in Image Analysis, but are also known as sliding windows in Natural Language Processing.

Conv2d networks applies a 2-d convolution on a 4-d tensor.

backward(dout)[source]

One backward step.

forward(inputs)[source]

One forward step. Gradients and outputs should be saved in the _cache when training, to be able to perform the backward pass.

inner_repr()[source]

Display the inner parameter of a CNN

nets.nn.dnn

class nets.nn.modules.dnn.DNN(layer_dimensions, activation_hidden=ReLU( (parameters): ))[source]

Dense Neural Network.

layer_dimensions

list of int containing all layer dimensions. The dimension at index i is the value of layer_dimensions at this index.

Type

list(int)

hidden_dimensions

list of int containing only hidden dimensions.

Type

list(int)

training

Boolean to indicate if we are training or not. This function can namely be used for inference only, in which case we do not need to store the features values.

Type

bool

activation_hidden

activation function used in hidden layers.

Type

Activation

activation_output

activation function used in the last layer.

Type

Activation

_parameters

dict containing values of weights and biases for each layers. - weight_{i}: contains the weight matrix of layer i - bias_{i}: contains the bias array of layer i

Type

dict

_cache

dict with - the linear combinations Z^[l] = W^[l]a^[l-1] + b^[l] for l in [1, L]. - the activations A^[l] = activation(Z^[l]) for l in [1, L]. We cache them in order to use them when computing gradients in the back propagation.

Type

dict

backward(outputs, labels)[source]

Update parameters using backpropagation algorithm.

Parameters
  • outputs (numpy.array) – matrix of floats with shape (num_classes, batch_size).

  • labels (numpy.array) – numpy array of integers with shape (num_classes, batch_size). Collection of one-hot encoded true input labels.

Returns

Dictionary with matrices that is to be used in the parameter update. Contains
  • the gradient of the weights, grad_W^[l] for l in [1, L].

  • the gradient of the biases grad_b^[l] for l in [1, L].

Return type

grad_params

forward(inputs)[source]

One forward step.

Parameters

inputs (numpy.array) – float numpy array with shape (n^[0], batch_size). Input image batch.

Returns

float numpy array with shape (n^[L], batch_size). The output predictions of the

network, where n^[L] is the number of prediction classes. For each input i in the batch, output[c, i] gives the probability that input i belongs to class c.

Return type

outputs (numpy.array)

init_parameters()[source]

Initialize the parameters dictionary. For Dense Neural Network, the parameters are either weights or biases. They are saved in the dictionary with the following keys:

  • w_{i}: weight matrix at layer i,

  • b_{i}: bias vector at layer i.

nets.nn.dropout

Define a Dropout layer.

class nets.nn.modules.dropout.Dropout(prob=0.5)[source]

Applies a dropout to an input Tensor with probability prob. The effect of this layer is to zeros elements of the incoming tensors, and cancel some neighbours effect / interactions from one layer to another.

backward(outputs)[source]

One backward step.

forward(inputs)[source]

One forward step. Gradients and outputs should be saved in the _cache when training, to be able to perform the backward pass.

nets.nn.linear

Linear layers are widely used either for Dense Neural Module or Convolutional Neural Module, to cite the most popular. The architecture is made of two sets of neurons aka perceptrons, all connected with weights and biases.

class nets.nn.modules.linear.Linear(input_dim, output_dim)[source]

A linear layer is made of a weight matrix \(W\) of shape \((\text{input_dim}, \text{output_dim})\) and a bias vector \(b\) of shape \((\text{output_dim})\). The linear transformation for an incoming vector \(x\) of shape \((N, \text{input_dim})\) results in a vector \(y\) of shape \((N, \text{output_dim})\):

\[y = x W + b\]

Examples:

>>> model = Linear(5, 10)
>>> batch_input = np.array([[-5, 2, 6, -2, 4],
...                         [2, 5, -6, 7, -3]])
>>> batch_pred = model(batch_input)
backward(dout)[source]

Backward pass for a single Linear layer.

Shape:
  • input (Tensor): upstream gradient.

  • output (Tensor): downstream gradient after a linear transformation.

forward(x)[source]

Forward pass. Compute the linear transformation and save the results in the _cache, which will be used during the backward pass.

Shape:
  • input (numpy.array): batch inputs of shape.

  • output (numpy.array): results of the linear transformation,

    of shape \((N, \text{input_dim})\), with \(N = \text{batch_size}\).

init_params(mode=None)[source]

Initialize the parameters dictionary. For Dense Neural Module, the parameters are either weights or biases. They are saved in the dictionary _params with the following keys: weight matrix w, bias vector b. The initialization can be changed with mode parameter, between the default uniform \(\mathcal{U}(0, 1)\) initialization or use \(\text{He et al.} \quad \mathcal{N} (0, \frac{1}{input_dim})\).

inner_repr()[source]

Display the inner parameter of a Linear layer

nets.nn.module

Modules are the main architecture for all transformations from one tensor to another. In other words, a neural network is a succession of modules (layer, convolution, activation…). When building a custom neural network, your model must inherits from Module abstract class and override the forward method. Moreover, you can specify the back-propagation rule in backward method. Usually, the backward method computes the naive back-propagation using only local gradients saved in the modules’s _cache. If you don’t specify it, NETS will uses autograd functionality to compute all gradients.

class nets.nn.modules.module.Module[source]

Abstract Module architecture. All models used to transform tensors should extends from this class to benefits forward and backward propagation rules.

add(*modules)[source]

Add modules to the current one.

Parameters

modules (Module) – modules to add

backward(*outputs)[source]

One backward step.

cache()[source]

Iterator through all cache dict

eval()[source]

Set the training attribute to evaluation mode.

abstract forward(*inputs)[source]

One forward step. Gradients and outputs should be saved in the _cache when training, to be able to perform the backward pass.

get_name()[source]

Quick access to get the name of a modules.

Returns

modules’s name

Return type

string

gradients()[source]

Iterator through all gradients

inner_repr()[source]

Return the representation of a single modules. This method should be unique for each modules.

Returns

the representation of one modules.

Return type

string

load_state(state_dict)[source]

Load parameters from a state_dict dictionary.

modules()[source]

Iterator through all gradients

parameters()[source]

Iterator through all parameters

save(filename='model.pickle')[source]

Save a model as a PICKLE file.

save_dict(filename='state_dict.json')[source]

Save the state as a JSON file.

state_dict()[source]

Save all parameters in a dictionary.

train()[source]

Set the training attribute to training mode.

zero_grad()[source]

Zero grad all parameters within a modules

nets.nn.pool

This modules defines a Pooling layer. Usually such layer is used after a convolutional layer.

class nets.nn.modules.pool.MaxPool2d(pool_size, stride=1, pad=0)[source]

A Pooling layer extract features from a multi dimensional Tensor and map them into another one. This extraction is used to decrease the dimension of the input, and often used after a convolutional layer.

backward(dout)[source]

Manual backward pass for a MaxPool2d layer.

forward(x)[source]

Forward pass.

inner_repr()[source]

Display the inner parameter of a CNN

nets.nn.rnn

Defines a basic Recurrent Neural Network.

class nets.nn.modules.rnn.RNN(input_dim, hidden_dim)[source]

Recurrent neural network (RNN) is a type of neural network that has been successful in modelling sequential data, e.g. language, speech, protein sequences, etc.

A RNN performs its computations in a cyclic manner, where the same computation is applied to every sample of a given sequence. The idea is that the network should be able to use the previous computations as some form of memory and apply this to future computations. From the [exercise 02456 from DTU course](https://github.com/DeepLearningDTU/02456-deep-learning-with-PyTorch).

backward(dout)[source]

Computes the backward pass of a vanilla RNN. Save gradients parameters in the _grads parameter.

Parameters

dout (Tensor) – upstream gradient.

Returns

downstream gradient

Return type

Tensor

forward(inputs)[source]

Computes the forward pass of a vanilla RNN.

\[ \begin{align}\begin{aligned}h_{0} = 0\\h_t = ext{tanh}(x W_{ih} + h_{t-1} W_{hh} + b_{h})\\y = h_t W_{ho} + b_{o}\end{aligned}\end{align} \]
Parameters

inputs – sequence of inputs to be processed

set_hidden_0(hidden_cell)[source]

Set the first hidden cell.

nets.nn.rnnbase

Defines the general structure of a Recurrent Network, which can takes the form of a:

  • Recurrent Neural Network,

  • Long Short Term Memory,

  • Gated Recurrent Unit.

class nets.nn.modules.rnnbase.RNNBase(*args, **kwargs)[source]

Base architecture for Recurrent Networks. Every general recurrent layers should extends this class to inherits private attributes and helper methods.

nets.nn.sequential

Sequential models are an ordered succession of modules.

class nets.nn.modules.sequential.Sequential(*modules)[source]

A Sequential model is build from a succession of Modules. All of them must have a forward method. In addition, a backward pass is created by default, running the back-propagation in all modules previously added. If for one of the modules the backward pass is not implemented, it will raise an error.

backward(grad)[source]

Vanilla backward pass. This pass computes local gradients from parameters saved in its _cache.

Shape:
  • inputs (Tensor): upstream gradient. The first downstream gradient is usually the loss.

  • outputs (Tensor): last downstream gradient.

forward(inputs)[source]

Compute the forward pass for all modules within the sequential modules.

Shape:
  • inputs (Tensor): incoming data.

  • outputs (Tensor): result of all forward pass.

nets.optim

nets.optim.optimizer

class nets.optim.optimizer.Optimizer(parameters)[source]

Optimizer. Modify a model’s parameters, and update its weights / biases.

backward()[source]

Update rules without autograd

abstract step()[source]

Update the parameters. Should be used only with autograd system

zero_grad()[source]

Zero grad all parameters contained in parameters attribute.

Returns

None

nets.optim.adagrad

class nets.optim.adagrad.Adagrad(parameters, lr=0.01, epsilon=1e-08)[source]

Adagrad optimizer. More sophisticated version of SGD, with learning rate decay.

step()[source]

Performs Adagrad update rules.

nets.optim.adam

class nets.optim.adam.Adam(parameters, lr=0.01, beta1=0.9, beta2=0.999, epsilon=1e-08)[source]

Uses the Adam update rule, which incorporates moving averages of both the gradient and its square and a bias correction term.

step()[source]

Performs Adam update rules.

nets.optim.rmsprop

Uses the RMSProp update rule, which uses a moving average of squared gradient values to set adaptive per-parameter learning rates.

class nets.optim.rmsprop.RMSprop(parameters, lr=0.01, decay=0.99, epsilon=1e-08)[source]

Uses the RMSProp update rule, which uses a moving average of squared gradient values to set adaptive per-parameter learning rates.

step()[source]

Performs RMSProp update.

nets.optim.sgd

Stochastic Gradient Descent is a popular optimizer for machine learning purposes.

class nets.optim.sgd.SGD(parameters, lr=0.01, momentum=0)[source]

Stochastic Gradient Descent optimizer follows the parameters optimization:

Note

A momentum can be added to this process.

step()[source]

Performs stochastic gradient descent with momentum.

nets.solver

nets.solver.solver

class nets.solver.solver.Solver(model=None, criterion=None, optimizer=None)[source]

Train and evaluate models.

Parameters
  • model (Module) – model to optimize or test.

  • criterion (Loss) – loss function.

  • optimizer (Optimizer) – optimizer for weights and biases.

Attributes::

model (Module): model to optimize or test. checkpoint (dict): checkpoint of the best model tested. criterion (Loss): loss function. optimizer (Optimizer): optimizer for weights and biases. performance (dict):

  • train (dict):
    • loss (list(float))

    • accuracy (list(float))

    • precision (list(float))

    • recall (list(float))

    • macro_f1 (list(float))

    • confusion_matrix (list(list(list(int))))

  • eval (dict):
    • loss (list(float))

    • accuracy (list(float))

    • precision (list(float))

    • recall (list(float))

    • macro_f1 (list(float))

    • confusion_matrix (list(list(list(int))))

abstract evaluate(iterator, *args, **kwargs)[source]

Evaluate one time the model on iterator data.

Parameters

iterator (Iterator) – iterator containing batch samples of data.

Returns

the performance and metrics of the training session.

Return type

dict

get_accuracy(y_tilde, y)[source]

Compute accuracy from predicted classes and gold labels.

Parameters
  • y_tilde (Tensor) – 1D tensor containing the predicted classes for each predictions in the batch. This tensor should be computed through get_predicted_classes(y_hat) method.

  • y (Tensor) – gold labels. Note that y_tilde an y must have the same shape.

Returns

the mean of correct answers.

Return type

float

reset()[source]

Reset the performance dictionary.

Returns

None

run(epochs, train_iterator, eval_iterator, *args, **kwargs)[source]

Train and evaluate a model X times. During the training, both training and evaluation results are saved under the performance attribute.

Parameters
  • epochs (int) – number of times the model will be trained.

  • train_iterator (Iterator) – iterator containing batch samples of data.

  • eval_iterator (Iterator) – iterator containing batch samples of data.

  • verbose (bool, optional) – if True display a progress bar and metrics at each epoch. The default is True.

Returns

None

Examples:

>>> performer = MySolver(model, criterion, optimizer)
>>> # Train & eval EPOCHS times
>>> EPOCHS = 10
>>> performer.run(EPOCHS, train_iterator, eval_iterator, verbose=True)
    Epoch:        1/10
    Training:     100% | [==================================================] | Time: 2m 26s
    Validation:   100% | [==================================================] | Time: 0m 4s
    Stats Training:    | Loss: 0.349 | Acc: 84.33% | Prec.: 84.26% | Rec.: 84.33% | F1: 84.26%
    Stats Evaluation:  | Loss: 0.627 | Acc: 72.04% | Prec.: 72.22% | Rec.: 72.17% | F1: 72.22%
>>> # ...
save(filename=None, dirpath='.', checkpoint=True)[source]

Save the best torch model.

Parameters
  • filename (str, optional) – name of the model. The default is “model.pt”.

  • dirpath (str, optional) – path to the desired foldre location. The default is “.”.

Returns

None.

abstract train(iterator, *args, **kwargs)[source]

Train one time the model on iterator data.

Parameters

iterator (Iterator) – iterator containing batch samples of data.

Returns

the performance and metrics of the training session.

Return type

dict

nets.solver.solver.get_metrics(gold_labels, predictions, labels=None)[source]

Compute metrics for given predictions. This method returns basic metrics as a dictionary.

Parameters
  • gold_labels (Tensor) – true labels.

  • predictions (Tensor) – predicted labels.

  • labels (list, optional) – list of classes indices.

Returns

dict

nets.data

nets.data.batch

A Batch is a set of examples, usually normalized and scaled for faster predictions.

class nets.data.batch.Batch(example, batch_size)[source]

A Batch``depends on a ``Dataset and is made of batch_size Example.

nets.data.dataset

class nets.data.dataset.Dataset(examples, fields)[source]

Abstract Dataset class. All dataset for machine learning purposes can inherits from this architecture, for convenience.

classmethod download(root)[source]

Download and unzip a web archive (.zip, .gz, or .tgz).

Parameters

root (str) – Folder to download data to.

Returns

Path to extracted dataset.

Return type

string

nets.data.example

class nets.data.example.Example[source]

Store a single training / testing example, and store it as an attribute. Highly inspired from PyTorch [example](https://github.com/pytorch/text/blob/master/torchtext/data/example.py)

classmethod fromlist(values, fields)[source]

Add an example from a list of data with respect to the fields.

Parameters
  • values – raw data

  • fields (tuple(string, Field)) – fields to preprocess the data on

Returns

None

class nets.data.example.Field(transform=None, dtype=None)[source]

A Field defines the data to process from a raw dataset. It will convert the data into a tensor. The data can be a string, integers, float etc. The data is meant to be preprocessed and the attribute transform handles the way the user want to process the raw data.

process(value)[source]

Applies the transformation and changes the data’s type if necessary.

Parameters

value (Tensor) –

Returns:

nets.data.iterator

This modules defines how the data should be called from a datasets. This takes into account shuffle mode.

class nets.data.iterator.Iterator(dataset, batch_size=32, shuffle=False)[source]

An Iterator call the data in batches. These batches can be shuffled and normalized. Usually, you want to feed a model with these batches.

nets.datasets

nets.datasets.cifar

Load and preprocess the CIFAR-10 dataset.

class nets.datasets.cifar.CIFAR10(filepath, transform=None)[source]

CIFAR-10 dataset, available at https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz

The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images, with the following classes:

Label

Description

0

airplane

1

automobile

2

bird

3

cat

4

deer

5

dog

6

frog

7

horse

8

ship

9

truck

Note

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

classmethod splits(root='.data', train='data_batch_', test='test_batch', **kwargs)[source]

Loads training, validation, and test partitions of the cifar10 dataset (https://www.cs.toronto.edu/~kriz/cifar.html). If the data is not already contained in root folder, it will download it.

Parameters
  • root (string) – relative or absolute path of the dataset.

  • train (string) – training data path

  • test (string) – testing data path

Returns

training and testing datasets

Return type

tuple(Dataset)

nets.datasets.mnist

Defines and pre-process the MNIST dataset. The data will be converted into Tensor objects.

class nets.datasets.mnist.MNIST(path_data, path_label, transform=None)[source]

Loads training, validation, and test partitions of the mnist dataset (http://yann.lecun.com/exdb/mnist/). If the data is not already contained in data_dir, it will try to download it.

This dataset contains 60000 training examples, and 10000 test examples of handwritten digits in {0, …, 9} and corresponding labels. Each handwritten image has an “original” dimension of 28x28x1, and is stored row-wise as a string of 784x1 bytes. Pixel values are in range 0 to 255 (inclusive).

Parameters
  • data_dir – String. Relative or absolute path of the dataset.

  • devel_size – Integer. Size of the development (validation) dataset partition.

Returns

float64 numpy array with shape [784, 60000-devel_size] with values in [0, 1]. Y_train: uint8 numpy array with shape [60000-devel_size]. Labels. X_devel: float64 numpy array with shape [784, devel_size] with values in [0, 1]. Y_devel: uint8 numpy array with shape [devel_size]. Labels. X_test: float64 numpy array with shape [784, 10000] with values in [0, 1]. Y_test: uint8 numpy array with shape [10000]. Labels.

Return type

X_train

classmethod splits(root='.data', train_data='train-images-idx3-ubyte.gz', train_label='train-labels-idx1-ubyte.gz', test_data='t10k-images-idx3-ubyte.gz', test_label='t10k-labels-idx1-ubyte.gz', **kwargs)[source]

Loads training and test partitions of the [mnist dataset](https://www.cs.toronto.edu/~kriz/cifar.html). If the data is not already contained in the root folder, it will download it.

Parameters

root (str) – relative or absolute path of the dataset.

Returns

training and testing datasets

Return type

tuple(Dataset)

nets.datasets.svhn

nets.utils

nets.utils.display

This modules defines basic function to render a simulation, like progress bar and statistics table.

nets.utils.display.describe_stats(state_dict)[source]

Describe and render a dictionary. Usually, this function is called on a Solver state dictionary, and merged with a progress bar.

Parameters

state_dict (dict) – the dictionary to showcase

Returns

the dictionary to render.

Return type

string

nets.utils.display.get_time(start_time, end_time)[source]

Get ellapsed time in minutes and seconds.

Parameters
  • start_time (float) – strarting time

  • end_time (float) – ending time

Returns

elapsed time in minutes elapsed_secs (float): elapsed time in seconds.

Return type

elapsed_mins (float)

nets.utils.display.progress_bar(current_index, max_index, prefix=None, suffix=None, start_time=None)[source]

Display a progress bar and duration.

Parameters
  • current_index (int) – current state index (or epoch number).

  • max_index (int) – maximal numbers of state.

  • prefix (str, optional) – prefix of the progress bar. The default is None.

  • suffix (str, optional) – suffix of the progress bar. The default is None.

  • start_time (float, optional) – starting time of the progress bar. If not None, it will display the time spent from the beginning to the current state. The default is None.

Returns

None. Display the progress bar in the console.

nets.utils.errors

Defines custom errors.

exception nets.utils.errors.BackwardCallError[source]

This error is used when the call to backward method is not legit.

nets.utils.functions

Utility functions.

nets.utils.functions.append2dict(dict1, *dicts)[source]

Append key values to another dict with the same keys.

Args: dict1 (dict): dictionary where values will be added. dict2 (dict) dictionaries to extract values and append to another one.

This dictionary should have the same keys as dict1.

Returns

None

Examples:

>>> dict1 = {"key1": [], "key2": []}
>>> dict2 = {"key1": 0, "key1": 1}
>>> append2dict(dict1, dict2)
>>> dict1
    {"key1": [0], "key2": [1]}

>>> dict3 = {"key1": 2, "key1": 3}
>>> dict4 = {"key1": 4, "key1": 5}
>>> append2dict(dict1, dict3, dict4)
>>> dict1
    {"key1": [0, 2, 4], "key2": [1, 3, 5]}

Docs

Access comprehensive developer documentation for Nets

View Docs

Tutorials

Get beginners tutorials and create state-of-the-art models

View Tutorials

Resources

Check the GitHub page and contribute to the project

View GitHub