When Equations Learn

The Perceptron is based on a matrix of trainable parameters, \( W\), and also a vector of other trainable parameters, \( b\).

and

These parameters are called trainable because they change during the perceptron’s training, mysteriously accumulating the network’s learning until they reach an optimal value. At that point, the network is ready to perform the task it was created for.

The elements of the matrix \( W\) are the perceptron’s weights, while the vector \( b\) is the bias. The elements of \( b\) are the biases for each neuron.

To put it very simply and directly, the number of rows in \( W\) is the number of neurons in the Perceptron, and the number of columns is the number of weights for each neuron. All neurons (in the same layer) have the same number of weights.

I referred to the perceptron as the building block of most neural networks. Neural networks are built with basic structures called layers, and the Perceptron is this layer in a vast majority of network architectures. Furthermore, the Perceptron itself can have layers, as we will see in Chapter 3.

Note that if the Perceptron consisted of a single neuron, then the matrix \( W\) representing this layer would, in fact, be a row-vector! That is, \( W\) would be a \( 1\times m\) matrix:

Just to give an example, as incredible as it may seem, even Deep Learning projects with more complex and deep architectures that perform binary classification are likely using a Perceptron with a single row of weights as their final layer. This would be the case for a network whose goal is to read bone x-rays and determine whether they indicate fractures. Such a well-developed network could be trained to identify fissures or fractures that are difficult for the human eye to detect.

A perceptron, like any other neural network, is created to perform a task. It must learn to perform its task. It must produce a result, \( z\), that corresponds to each element, \( x\), both in a set of data or information, \( D\). We say that \( D\) is a set of vectors or tensors and that the perceptron is trained on this set. We will talk more about \( D\) when we get to Section Training.

If \( x\in D\) is one of the training vectors, then the perceptron’s response, \( y\), to this vector is:

The vector \( x\) is one of the perceptron’s inputs, and \( y\) is the corresponding output.

Any of the expressions in (6) are sometimes simply called the perceptron’s linearity.

Below, I will briefly present two alternative ways the Perceptron can be represented. The reader may encounter these in other books, and knowing they exist can broaden one’s ability to manipulate the mathematical tools that describe and model it. I include them here in passing, but we will not use them in this book.

It is extremely common to pass each of the values of \( y\) to the same activation function. This activation function is also known as a "non-linearity," because it "breaks," in a way, the linear behavior produced in 6. It can take one of several commonly used forms. For now, let’s just designate it with a symbol: \( a\). Thus, in its most general form, the single-layer perceptron is:

This last expression might seem a bit confusing at the moment, perhaps because of the expressions inside the brackets, but don’t be alarmed. It displays the mathematical symbols that mirror the conceptual structure of a single-layer Perceptron. It also shows how the vector \( x\) is "absorbed" and processed by the network. The vectors \( W_i\) are the rows of \( W\) and the scalars \( b_i\) are elements of \( b\). See how the vector \( x\) is processed by each of the rows of \( W\). The equation shows how the signal \( x\) "flows" through the perceptron until it is transformed into its response, \( P\).

Try to firmly grasp the fact that \( a\) is a vector and that its elements are functions whose independent variables are, respectively, the elements of the vector \( y\). In the Subsection below, I have placed a table with some well-known and commonly used activation functions.

We have already said that a perceptron must learn to perform a certain task and that it is trained on a set of data or information, \( D\). This set can be considered a set of ordered pairs, \( (x,z)\in D\), such that \( z\in Z\subset D\) and \( x\in X\subset D\) are, respectively, the set of desired responses and the set of vectors representing any domain of things we are interested in relating to the desired responses.

We have already seen that \( x\) is a vector. The second component of the ordered pair, \( (x,z)\), is the response we want the perceptron to learn to give to the input vector \( x\). The ordinate \( z\) can be a scalar number, a vector, a matrix, or a tensor. This will depend on how we mathematically encode the task. In this book, our desired responses, \( z\), will only be scalars or vectors.

Initially, the perceptron gives some arbitrary response, \( P\), to the vector \( x\). Throughout the training, this response gets closer to the correct or desired response, \( z\). We check this approximation with a Cost Function, which we will denote with the symbol \( E\) (we will see more about it in Section The Cost Function), which tells us how far or how close \( P(x)\) is from the desired response \( z\). Throughout the training, the value of the Cost Function decreases because \( P(x)\) becomes increasingly close to \( z\). Our goal during training is to make the error go to zero, meaning that the equality \( P(x)=z\) becomes true, or almost true, as a sufficiently small error is usually enough.

From a certain point of view, this entire book is about how to reintegrate into the Perceptron, during training, the information contained in the error, \( E\), to make the Perceptron more accurate in its task. That is, to create the relationship we desire, \( P(x_i)=z_i\).

There are several functions that can be used as a cost or loss function. The choice depends on the project and sometimes on the preference of the person training the network. In Section Some Cost Functions, there is a table with some useful and commonly used cost functions. The important thing to know is that the Cost Function, whatever it may be, must comply with the mathematical definition of a norm. This is a topic we won’t delve into in this book, but the interested reader can find the definition in Appendix Norm on a Vector Space, along with a small proof that if a norm is zero, then its argument must also be zero.

We will see in Section Updating the Trainable Parameters that the process we will use to approximate \( P(x)\) and \( z\) is based on the gradient of the Error Function. This process, called Gradient Descent or Stochastic Gradient Descent, gradually indicates the direction of the lowest value of \( E\) and, thus, also indicates the path to a smaller separation between \( P\) and \( z\).

We can say that the Perceptron is a mapping that learns to give an appropriate response \( z\) to each \( x\) in a learning process that is carried out over one or several training sessions. Usually, several! In a training session, the Perceptron receives all the elements \( x\) from \( D\), one after the other, and for each \( x\), the corresponding \( P(x)\) is calculated. After that, the error function is calculated on \( P(x)\) and \( z\), so we can express it as: \( E(P,z)\). As many training sessions are performed as necessary to make \( E(P,z)\) sufficiently close to zero. This is the reason for requiring the Error or Cost Function to be a norm, because then, when \( E(P,z)\longrightarrow 0\), it will also be true that \( P-z\longrightarrow 0\), meaning the network’s response is becoming equal to the desired response.

Artificial learning is an optimization process.

What is optimized in artificial learning? A function that is usually called a Cost Function, Loss Function, or even Error Function! I personally call it, in this context of artificial learning, a Pedagogical Function, since it measures how far the Perceptron’s response is from the desired response, and by this means, we know if the network is learning or not. It is from the derivation of the Loss Function that learning happens.

Anyone who has ever derived a function to then find its maximum or minimum is in a perfect position to understand how artificial learning happens.

Learning occurs in one or more training sessions, where the Perceptron’s trainable parameters are repeatedly updated (See Section Training). These parameters are updated at each step of the training, that is, after each training batch is presented to the network. We will see more about training in batches in the chapter Training in Batches.

The description of how artificial learning happens is the most important and interesting part of neural networks in this author’s opinion. Without this, there is no machine learning.

When we go to school, our learning is measured by assessments. The learning of neural networks is also measured by performance evaluations.

The Perceptron’s school is the training session.

Just as a final school grade is obtained from a formula, neural networks also use formulas that "grade" their performance.

In the case of neural networks, such formulas are known as cost functions, loss functions, or error functions. In this book, I refer to them much more often as error functions. They, in fact, measure the error made by the network when trying to predict a response to a corresponding input.

The error function can have several forms, but I will not address any specific form now, as we are interested in how its general form fits into the learning formulas. For now, we will just symbolize any error function with the letter \( E\). In Subsection Some Cost Functions, right below, you can find a table with some cost functions.

There is much to say about \( E\), but for now, let’s stick to the operational aspects that make it possible for the perceptron to learn.

The function \( E\) takes the Perceptron’s output as its argument. So,

But, you see, the perceptron’s output depends on its trainable parameters, meaning \( E\) also depends on the weights and bias. So, it is more common to write:

This notation is very useful because the perceptron’s learning depends on the derivative of \( E\) with respect to its weights and bias.

Here, we need to consider the compositional structure of the error function,

and keep in mind that \( a\) and \( y\) are vectors—the data from 6 and 11—and that \( E\) is a real-valued function.

Let’s calculate the derivative of the error function with respect to the Perceptron’s weights. This derivative is also known as the gradient of the Error.

We know that \(\frac{\partial E}{\partial a}\) is the gradient of \( E\) with respect to the activation vector \( a\), because \( E\) is a real-valued function with a vector domain. Thus, \(\frac{\partial E}{\partial a}=\nabla_a E\).

The derivative \(\frac{\partial a}{\partial y}\) generates a matrix. This comes from the fact that both \( a\) and \( y\) are vector functions. See appendix Derivative of Vector Functions for more details on derivatives of vector functions.

The derivatives \(\frac{da_i}{d y_i}\) cannot yet be calculated or fully reduced. We don’t yet have any definite form for a. This will happen when we are dealing with specific examples or architectures.

If we analyze equation 11, we will see that each \( a_i\) depends only on \( y_i\). Therefore, we must have \(\frac{d a_i}{d y_j}=0\) if \( i \ne j\). Consequently, \(\frac{\partial a}{\partial y}\) will be a diagonal matrix. The elements, \(\frac{da_i}{d y_i}\), of this diagonal matrix will depend on the specific form of \( a\).

The differential \(\frac{\partial y}{\partial W}\) has the form of a column-vector with \( n\) elements. However, these elements are, in turn, \( n\times m\) matrices.

Now, notice that each element \( i\) of the column-vector on the right side of 16 is the derivative of a real-valued function whose arguments coincide only with the \( i\)-th row of \( W\). These real-valued functions are defined in equation 6, from which we know that \( y_i(W_i)=\sum_{j=1}^{m} w_{ij}x_j +b_i\) (See Appendix The Derivatives of \( y_i\) to review the procedure for deriving this equation). Therefore, the elements of the column-vector are matrices with null entries, with the sole exception of their \( i\)-th row.

Column-vectors or row-vectors of matrices will appear many times in this presentation. This is due to the fact that we are deriving the error, \( E\), with respect to the entire weight matrix at once.

Thus, the formula 15, which calculates the partial derivative of the perceptron’s error \( E\) with respect to its weights \( W\), is:

We now need to calculate the derivative of E with respect to the bias vector, \( b\).

From 6, we see that the biases are embedded at the deepest level of the perceptron, along with the weights.

Before, we considered the error as a function of only the weights. Now, let’s consider it as a function of only the biases.

Based on the calculations we’ve already done in the previous section, we can very simply write:

If necessary, see Appendix The Derivatives of \( y_i\) for more considerations on the calculation of \(\frac{\partial y}{\partial b}\).

Finally, we’ve arrived where we wanted: \( \frac{\partial E}{\partial W}\) and \( \frac{\partial E}{\partial b}\) will be used to update the perceptron’s weights and bias. The process where this is done is called backpropagation and is based on the Stochastic Gradient Descent technique. This technique, in turn, is based on the fact that \( \frac{\partial E}{\partial W}\) is a gradient vector and, therefore, always points in the direction of the greatest rate of change of \( E\). Consequently, its negative, \( -\frac{\partial E}{\partial W}\) (see formulas 21 and 23), will point in the direction of the smallest variation. I won’t go into further detail on this point here, but the interested reader can find a few other interesting and pertinent observations about our use of it in Appendix Some Observations on the Gradient. I will just mention that the aforementioned negative sign is very important. If you have ever coded this formula, say in Python or TensorFlow or any other language or framework, to train a Perceptron, but mistakenly used a positive sign instead of a negative one, you may have noticed that the error made by the Perceptron actually only increases instead of decreasing!

At this moment, we have everything necessary to present the formula that allows learning to happen. This formula has a simplicity and beauty that is only matched by its power to make the Perceptron’s learning possible.

During a training session, the Perceptron’s weights are updated many times. Each update happens at a moment, \( t\), of the training. As the training evolves, the weights are altered in search of better performance, that is, in search of a lower cost, \( E\). At a given moment, \( t\), of the training, the perceptron has the weight matrix \( W_t\), which is updated by adding \( \Delta W_t\). The result becomes the new current weight matrix of the Perceptron, \( W_{t+1}\).

where

The symbol \( \eta\) is called the learning rate. It is a parameter whose importance lies in dictating the pace of the training, as it allows for adjusting the "speed" of the training. However, it is difficult to know what the optimal speed is for each step of a neural network’s training, although there are general guidelines and useful calculation methods, which we will not discuss at this time. It is a somewhat delicate parameter to handle, as are other defining parameters of neural networks. In practice, small values like \( \eta=0.01\) or \( \eta=0.001\) are always used as a first alternative. Other approaches alter the value of \( \eta\) throughout the training so that its value decreases as the training progresses.

The bias update is done with formulas very similar to those for updating the weights:

where

An input signal, \( x\), will "flow" through the network’s layers, entering the first layer, passing through each one until it exits through the activation functions of the final layer.

We have already seen that a single-layer perceptron is defined by its weights and bias, so we can view it as the object:

Let’s represent the stacking of layers, that is, the juxtaposition of several single-layer perceptrons, simply like this:

where

The superscripts in 26 indicate the layer number to which the weights and bias belong.

For now, let’s consider a two-layer perceptron, \( P=P_1\rightarrow P_2\). Perceptron 1 has \( n\) neurons, while perceptron 2 will have \( p\) neurons. We will consider an input vector, \( x\), with \( m\) elements.

Perceptron \( P_1\) will receive the signal \( x\), but \( P_2\) will receive the output of \( P_1\), that is, the activation functions, \( a^1\), of \( P_1\).

The output of \( P_2\) is delivered to the error function. In other words, the activation functions, \( a^2\), of \( P_2\) are the arguments of the Error Function.

Now that we have gained a better understanding of a Perceptron’s structure, let’s quickly write the equation for one, \( P\), with any number of layers, \( L\). We again use the symbol \( \circ\) for function composition.

The ellipsis, naturally, indicates that any number of layers can be in its place, and each pair \( a^l\circ y^l\) indicates the elements of layer \( l\), namely, the activation vector whose argument is its linearity vector, \( a^l( y^l)\).

With the exception of the linearity of layer 1, every other linearity, \( L\ge l\ge 2\), has the following form:

where \( n_l\) and \( p_l\) are, respectively, the number of rows and columns of \( W^l\). Since the number of columns of matrix \( W^l\) and the number of rows of vector \( a^{l-1}\) coincide, the number of elements in \( a^{l-1}\) is also \( p_l\).

The linearity of layer 1 has a very similar form to the other linearities, with the exception of its incoming signal, \( x\).

The error function for the case of \( L\) layers is the same as in the other cases. It takes the Perceptron’s output, \( P\), as its argument.

Next, we will display the formula for the derivative of \( E\) with respect to the weights of a layer \( l\). Its form is perfectly understandable when considering expression 48, because from this, we know we have to use the chain rule as the derivation method to obtain:

while the derivative with respect to the weights of layer 1, \( W^1\), is:

Note once again how far the derivation process of 49 descends through the layers of 45.

It turns out that, as beautiful and elegant as 49 and 50 may be, in many cases, they could not be calculated in their entirety at each training step of a multi-layer Perceptron!

The more layers a perceptron has, the longer 49 and 50 become. Let’s remember that each derivative in these formulas is a matrix or vector, or even a vector of matrices, whose dimensions can take on very large values. This makes using these formulas in their current form impractical.

Consider two successive calculations, that of \( \frac{\partial E}{\partial W^{l+1}}\) and \( \frac{\partial E}{\partial W^{l}}\). If we were to use formula 49 for these two calculations, we would have calculated all the first \( L-(l+1)+1=L-l\) rates of change from 49 twice!

Fortunately, there is a practical solution to this problem.

The solution to the problem presented in the previous section is to calculate the derivative of the weights of a layer, \( l\), by leveraging all the calculations already made for layers \( L\) down to \( l+1\). At each step down through the layers, the last performed calculation is stored in memory.

Let’s do a quick analysis of the dimension of the matrices involved in 65 and then explicitly perform the products indicated in it. This will give us a picture of the final result of the calculations we have been performing up to this point. Furthermore, this result will be used in 20 for updating the weights, and for this, it is necessary that the dimensions of \( W^l\) and \( \frac{\partial E}{\partial W^l}\) are equal.

Considering the structural elements and the layers indicated in 65, let’s assume that layer \( l+1\) has \( n\) neurons, layer \( l\) has \( p\) neurons, and that the number of elements in the incoming vector, \( s\), is \( m\). The vector \( s\) is the signal that enters layer \( l\). This signal can be either the activation vector of layer \( l-1\), or it can be the vector \( x\) on which the network is being trained. If layer \( l\) is the first layer of the network, then \( s=x\), otherwise, \( s=a^{l-1}\). For this reason, I will use the symbol \( s\) for the remainder of this Section to indicate that we could be dealing with any of these cases.

In this case, \( \frac{\partial E}{\partial a^l}\) is a row vector of \( p\) elements, \( \frac{\partial a^{l}}{\partial y^{l}}\) is a square matrix \( p\times p\), while \( \frac{\partial y^l}{\partial W^l}\) is a column-vector with \( p\) positions whose elements are matrices with the same dimension as \( W^l\), i.e., \( p\times m\).

Now, \( \frac{\partial E}{\partial y^{l+1}}\) is a row vector of \( n\) elements, \( \frac{\partial y^{l+1}}{\partial a^{l}}\) is an \( n\times p\) matrix, and last but not least, \( \frac{\partial E}{\partial W^l}\) is a matrix with the dimensions of \( W^l\).

If necessary, consult Section Derivative of Vector Functions in the Appendix for a brief explanation of the dimension of objects resulting from the derivation of vector functions.

So, in the case where \( l=L\), we have a product of three matrices with the following dimensions: \( 1\times p\), \( p\times p\), and \( p\times 1\). This is the minimum we would expect for the product to be possible, which is, the number of columns of the matrix on the left being equal to the number of rows of the matrix on the right.

In the case where \( L>l\ge 1\), we have four matrices with the following dimensions, from left to right: \( 1\times n\), \( n\times p\), \( p\times p\), and finally, \( p\times 1\). Again, we have the minimum we would need.

In both cases, the final dimension \( p\times 1\) is that of a column-vector with \( p\) rows and \( 1\) column, whose \( p\) positions are matrices that have the dimension of \( W^l\), as we have seen several times now. Thus, in both cases, the \( 1\times 1\) dimension of the result of the matrix products is not a scalar, but rather a single matrix that, as we’ve seen, has the dimension of the weight matrix of layer \( l\), i.e., \( p\times m\).

So, when \( l\) is the last layer of the network:

It is easy to see that the last line in 67, and indeed its entire development since 66, is essentially the one that was started in 35, with the exception of the symbols for the incoming signal and the layer number.

The outer product is not as common as the normal product of vectors, but its concept is just as simple. I present its definition in Appendix Outer Product.

Now, let’s move on to the explicit form of \( \frac{\partial E}{\partial W^l}\) for any layer \( l\), except the last one.

In equation 38 and the following, we had already seen that the derivative of a linearity with respect to its input signal is the weight matrix of that linearity’s layer. So, in 68, we have already used the fact that \( \frac{\partial y^{l+1}}{\partial a^l}=W^{l+1}\). Continuing,

Up to this point, we have performed the indicated product of matrices or vectors, from left to right. The reader should follow these developments very closely, as they are responsible for producing the mathematical forms that make learning possible in principle. The rightmost column-vector is, in fact, the vertical vector of matrices we have been talking about. These matrices will appear explicitly shortly below.

Next, we will re-encounter the row vector we found in the second expression of 69, but this time, it will be enclosed within a transpose operation. This transposed expression is responsible for the final form we will arrive at below.

Now, let’s look at the corresponding formulas for the biases. We have already seen that the form of \( \frac{\partial E}{\partial b^l}\) is simpler than that of \( \frac{\partial E}{\partial W^l}\). This happens because \( \frac{\partial y^l_i}{\partial b^l}\) are unitary matrices and can be "disregarded" in the calculation of \( \frac{\partial E}{\partial b^l}\). Thus, below, I present the formulas related to the biases without detailing their full development since it is analogous and simpler than what we have just done.

Thus, the formula for the biases, if \( l\) is the last layer, is:

If \( l\) is any other layer except the last, then:

We have already seen in 20, 21, 22, and 23 how the update of weights and biases is done. Let’s reproduce those formulas here, loaded with the information we have just obtained about \( \frac{\partial E}{\partial W^l}\) and \( \frac{\partial E}{\partial b^l}\).

or, even,

The first time we see this formula, we might be somewhat surprised to find the presence of weights from layer \( l+1\) in the update of the weights of layer \( l\). And it’s not just the weights. The elements of the vector \( \frac{d E}{d y^{l+1}}\) are there too. The recursive nature of \( \frac{d E}{d y^{l+1}}\), as shown in the second equation of 59, makes us understand that, implicitly, it contains elements from all layers since the last layer \( L\).

This is due to the backpropagation of the error. The derivative of \( E\) starts being calculated with respect to the elements of layer \( L\) and descends, in a chain, to the desired layer \( l\), and in the process, it evokes the precise, appropriate quantities from the other parts of the network.

The formula for updating the bias is, once again, simpler than that for the weights.

that is,

I have added a small continuation to what has been presented in this section in the Appendices, in Section Continuous Learning. In it, I make some observations about what can be provisionally called continuous learning. The interested reader will have it at hand, but it is not indispensable for understanding what we present in this book.

Batch training consists of using batches of several training vectors, \( X\), instead of single vectors, \( x\), at each propagation step. The batch, \( X\), is, in fact, a matrix since the \( x\) are vectors. Let’s keep in mind that a \( j-\text{th}\) vector \( x\) is the column, \( X^j\), of the matrix \( X\).

Here, once again, the form contained in 20 is the recipe for updating the weights, and the rate of change of the error with respect to them has the form contained in 86, that is:

which, according to 75, leads us to:

It is very easy to see that equation 89 uses the form of 75. The only difference is that now we mark \( E\) and \( s\) with the column-vector number, \( j\), to which these elements correspond in the training batch.

The third expression in 88 lets us see that the \( \beta\) instances of the network, calculated in parallel, each on one of the elements of the batch, \( X\), use the same network elements. We see that "concrete" elements like the weights or dynamic elements like the form of the activation changes, for example, are the same for the \( \beta\) terms, although the numerical value, in the case of dynamic variations, may change according to the element of the batch.

Once again, the form of the respective update for the bias is obtained through procedures very analogous to those done above, but starting from equations 64, 77, and 87.

and

Let’s start by recalling that \( W_i=[w_{i1},\dots ,w_{ip}]\) is the vector on the \( i\)-th row of \( W\).

In this way, it also becomes clear why \( \frac{\partial y_i}{\partial W_l}=0\) for \( i\ne l\). This is because \( y_i\) simply does not depend on \( W_l\), and thus, \( \frac{\partial (w_{ij}s_j +b_i)}{\partial w_{lj}}\) must be zero for all \( j\).

Consider the bias vector \( b=[b_{1},\dots ,b_{n}]\). Since \( b\) is a vector, we can, naturally, calculate \( \nabla_b y_I\) for a fixed \( i=I\). However, this calculation is not the most appropriate one to perform.

The Perceptron has a structure, and according to this structure, it has a single bias per neuron. This means that for each vector \( W_I\), on a row of \( W\), there corresponds only one scalar \( b_I\). This produces the interesting result that the non-zero elements of \( \nabla_b y_i\), with \( 1\le i\le n\), are not actually along a row or a column of \( \frac{\partial y}{\partial b}\), but rather along its main diagonal (as shown in the second and third expressions in 19)! Thus,

where \( \frac{\partial y_I}{\partial b_i}=0\) for \( i\ne I\), while \( \frac{\partial y_I}{\partial b_i}=1\) if \( i=I\). In this way, we write \( 1_I\) to indicate that the unit '1' only occurs at position \( I\) and that all other positions are zero.

The way a network’s weights are initialized can strongly impact the success of the training. Inadequate initializations can lead to vanishing (very small gradients) or exploding (very large values) gradients during backpropagation, making learning difficult or impossible. Modern initializations were developed to keep the values of activations and gradients within stable ranges, from the first to the last layers.

Examples:

Normalizing data before feeding it into the network is a critical practice for improving training convergence. Data with very different scales can cause instability or slow down training. Furthermore, normalizing the internal activations of networks (during training) helps maintain stable distributions and accelerates learning.

Examples:

The learning rate is a network’s most sensitive hyperparameter. It determines the size of the steps taken in the parameter space with each update. If it’s too high, the network may never converge; if it’s too low, training can be slow or get stuck in local minima. Modern optimizers improve this process by automatically adapting the steps based on accumulated gradients or past moments.

Examples:

Regularization is the set of techniques that combat overfitting, which occurs when the network memorizes the training data and fails to generalize to new examples. This is especially important in deep networks with a large parameter capacity. Regularization imposes constraints on the weights or the network’s behavior to favor simpler and more robust models.

Examples:

The network’s architecture determines its expressive capacity and its suitability for the task. The choice of architecture directly affects performance, training time, and interpretability. With the advancement of research, specialized models have emerged for different types of data (images, text, sound, time series, etc.), each leveraging specific structures and properties of that data.

Examples:

A network’s performance is directly related to the quality and representativeness of the data it receives. Noisy, incomplete, or biased data can compromise the entire learning process. Preprocessing and enriching the data is an essential step, and often more important than fine-tuning hyperparameters.

Examples:

The way the network is trained strongly influences its ability to converge to a good solution. Effective training techniques help to avoid problems like excessive noise in updates or overfitting to the training data. Some of them are strategies for stopping the training at the right time or splitting the data in a way that makes updates more stable.

Examples:

Monitoring the network’s behavior over time is crucial for diagnosing problems and guiding decisions. The loss and accuracy curves during training and validation reveal signs of overfitting, underfitting, or modeling errors. Furthermore, appropriate metrics help to correctly evaluate the network according to the context of the problem.

Examples:

Although there is an appendix addressing some aspects of Transfer Learning, that content is far from covering the subject and, in truth, represents more of an essay by this author. Transfer learning allows for reusing knowledge from a network trained on a large database and adapting it to a new task with less data. This is extremely useful when the available data is scarce, but the task is similar to another that is already well-explored. It reduces training time and improves generalization, being a common practice today, especially in NLP and computer vision.

Exemplos: