"We will define and train a neural net, and you'll get to see everything that goes on under the hood."
"Backpropagation allows you to efficiently evaluate the gradient of some kind of a loss function with respect to the weights of a neural network."
"Neural networks are just mathematical expressions."
"This backpropagating signal, which is carrying the information of what is the derivative of l with respect to all the intermediate nodes, can be imagined almost like flowing backwards through the graph."
"The loss is a single number that... measures how well the neural net is performing."
"Neural nets are these mathematical expressions that take input as the data and the parameters of the neural net for the forward pass, followed by a loss function that tries to measure the accuracy of the predictions."
"We can use a neural network to learn a function, a curve that fits this data, and if we present it with a candidate point, it may be able to produce a strong estimate of what the corresponding Y value would be."
"Neural networks need to be finely tuned and optimized for the task of interest."
"The first time we can behold the amalgamation of metal and neural nets."
"Non-linearity is what allows deep neural networks to model complex functions."
"Each neuron in a neural network contains an activation function that changes the output of a neuron when given its input."
"Deep learning is a machine learning technique that learns features and tasks directly from data by running inputs through a biologically inspired neural network architecture."
"Understanding how a neural network works is absolutely fundamental, and you have to have some kind of basis on the math behind a neural network before you're really able to properly implement one."
"The goal of a neural network is not to memorize but to generalize from the input data."
"The architecture of our neural network is crucial in determining how effectively it can learn and generalize from the data provided."
"Don't design your own neural networks... use the biggest thing that fits in your GPU and that's mostly the way you choose this."
"So, by combining these three methods, pruning, weight sharing, and also Huffman Coding, we can compress the neural networks, state-of-the-art neural networks, ranging from 10x to 49x without hurting the prediction accuracy."
"A neural network is something called a universal function approximator; it's just a function that receives inputs and generates an output."
"Deep learning goes One Step Beyond this and is a subset of machine learning which focuses explicitly on what are called neural networks."
"Neural network training is essentially adjusting weights until the function represented by the neural network essentially does what you would like it to do."
"The Jetson Nano is capable of running multiple neural networks in parallel, useful for image classification, segmentation, object detection, and can also be used for speech processing."
"The smart money right now is on [neural networks] gonna scale. It scales like nobody's business."
"I think neural nets, data, valuable data right, and feeding neural nets to create value, just seems like that would be something that, in the next 10 or 20 years, whatever you establish now could be massive."
"The key here is that we have this idea of this recurrence relation that captures the cyclic temporal dependency, and indeed, it's this idea that is really the intuitive foundation behind recurrent neural networks or RNNs."
"Activation functions... introduce nonlinearities into the network... because in real life, data is almost always very nonlinear."
"The purpose of activation functions is to introduce nonlinearities into the network."
"The input layer and the output layer are the only two layers that we really concern ourselves with when we look and use a neural network."
"Deep learning takes this idea even further and it's a subset of machine learning that focuses on using neural networks to automatically extract useful patterns in raw data."
"I will need all your brains for this session because today, I want with you to build a neural network."
"We are going to train this neural network to actually figure out the correct weights and biases by itself."
"Neural networks are currently the state of the art and are achieving things that pretty much no other machine learning model is doing right now."
"What's so impressive about neural networks is that...it figures out its own model how to model the logic and all that on its own."
"The thing that really makes neural nets work so well is something really powerful that comes from distributed representations."
"With the incorporation of video modules into our neural network architecture, we significantly improved our ability to understand the dynamics of the driving environment, like estimating the velocity and depth."
"Linear algebra is the backbone of calculations behind neural network algorithms."
"Transformers are the key in GPT, and they're a special type of neural network."
"This is the backpropagation algorithm. It's what's at the heart of deep learning."
"Tesla is one of the more straightforward and significant beneficiaries of the advance in neural network Tech."
"Neural networks simulate how the human brain functions, with neurons representing functions trying to match inputs to outputs."
"Judging what is a collision and what's not a collision and whose fault it is that's a great thing for deep neural nets and we've been applying that."
"Back propagation is the process of taking the error and basically like feeding backwards the error through the network..."
"Neural network art: a whole new kind of art."
"Neural networks can sometimes be a bit of a black box. So we want TensorBoard to be something of a flashlight, something that will let you take the confusing world of TensorFlow and start to dive into it."
"What's interesting about a neural network is that it can learn and adapt to different scenarios."
"Our brains are neural nets... Everything shapes the way you see the world."
"So Evolution figured out very early on that electrical networks are amazing at having memories at integrating information across distance at different kinds of optimization tasks."
"Neural networks, when given a lot of data and computing power, tend to outperform all other models most of the time."
"Neural networks have to figure out the two-dimensional nature of an image. They have to go beyond that to understanding constructs like there are objects in the world and there is space that the objects move through."
"In contrast, the tan-h, or hyperbolic tangent activation function takes any x-axis coordinate and turns it into a y-axis coordinate between -1 and 1."
"In summary, basic vanilla recurrent neural networks are hard to train because the gradients can explode. Kaboom! Or vanish. Poof!"
"The sheer size of a neural network is not enough to give it structure and knowledge but if it's suitably engineered then then why not."
"Neural nets perform better across a wide variety of tasks."
"Neural networks are able to discover features in the control problem, which is very interesting."
"Full self-driving display of version nine will show actual probability distribution of objects, true mind's eye of the neural net."
"We have some little computer vision accelerators and tensor accelerators for some of the neural network work."
"It actually reads all the letters, all the words that we are giving it, it chews and digests them inside of its neural activations."
"Humans are not explicitly programming what's going on inside the neural network, the algorithm programs itself."
"The goal of a neural network is to reliably classify inputs, even when they're not in the training set."
"Neural networks are hungry for data and they do perform well if you feed them enough data."
"Neural networks converge on something useful as they get trained better and better."
"An obvious advantage is that this RNN can process any length of input."
"If it learns a good way to process one input, that is applied to every input in the sequence."
"What are the road networks designed to work with? They're designed to work with a biological neural net to our brains and with Vision our eyes."
"Neural network training: Moving towards replicating the human brain."
"GPT uses large language models which are neural networks."
"Neural networks and biology share the property of being general."
"Neural networks demonstrate dynamic flexibility similar to the brain."
"Expanding neural nets, possibly using transformer networks, is likely next."
"Tesla has developed a very specific narrow use case chip specifically for training Tesla's vision-based neural networks to help solve autonomy."
"Dynamics funnel down pathways in attractor networks."
"The Kendrick K210, a dual core 64-bit unit with an integrated floating point unit, forms something called a KPU, which is useful for neural network development."
"Make a new neural network where the only inputs that matter are what we had for dinner yesterday."
"Machine translation where you input a sentence in one language and train an RNN to output a sentence in a new language."
"It tries to directly learn the policy using our neural network."
"Auto-encoders are really popular for unsupervised learning."
"A neural network isn't just going to learn as if by magic, it needs to be taught."
"Training a neural network is as simple as 1 2 3 4: sample your data, forward it through your network, backpropagate to compute your gradients, and then do a parameter update."
"In every neural network lecture, you need to see a neuron."
"Deep learning is involved with using neural networks which is almost an artificial representation or analog or modeling of what the brain could possibly look like."
"There is actually a very effective way of getting deep large neural networks to perform very well on human-level AI tests."
"End-to-end neural networks is turning out to be the way to go."
"The function that's being computed by the neural net is getting closer to the function we actually wanted."
"Each of these layers in here, this is something known as a neuron."
"With TensorFlow, you can actually do regression, but with a neural net."
"I'm a huge believer in deep neural networks and their power."
"this gave you a more concrete understanding of how neural networks work maybe piqued your interest to dive into the math and do some more yourself"
"Back propagation means that we can take these two things and just multiply them together."
"Anything you can make look like an image may also be suited to convolutional neural networks."
"The neural network revolution has some groundbreaking ramifications."
"By combining many neurons, we can build an ever more intricate function."
"Life is full of patterns, and the neuron network's job is to identify these patterns and reproduce them."
"AI is something based on machine learning and usually implemented with neural networks."
"Deep learning models typically have many layers of neurons which allows them to learn more complex patterns than traditional machine learning models."
"This led to neural networks that could do one kind of thing really well, such as classify images, detect spam, or predict your next YouTube video."
"The essence of intelligence is learning the strengths of connections in a neural network."
"Activation functions apply a linear transformation to the layer output and basically decide whether a neuron should be activated or not."
"If you don't know which function you should use, then just use a ReLU for hidden layers."
"Neural Networks from Scratch teaches everything from a basic forward pass to training and optimizing your model."
"If previously neural nets are special purpose computers designed for specific tasks, GPT is a general purpose computer."
"We backpropagated through a linear layer."
"If you could somehow figure out how learning in neural networks work, then you can program small parallel computers from data."
"Reliability is currently the single biggest obstacle for these neural networks being truly useful."
"This is a neural network without a hidden layer; it just connects the inputs to the outputs directly."
"If you have 100 layers deep, and every second step you put a ReLU, that network's going to learn a lot more things."
"This is a very good starting point for the beginners who want to learn neural networks."
"The approach for solving any problem in neural networks is actually trial and error."
"Current AI is very rigid because of the way the simulated neurons work right now."
"ChatGPT understands context through one neural network and generates responses through another."
"Learning from a neuroscientific standpoint is when neurons in your brain assemble to form thousands of synaptic connections and those connections then assemble into complex three-dimensional neurological networks."
"Our neural networks are not fixed, they are soft, they are malleable, and they are constantly shifting."
"Optimizers of neural networks have the same shape as neural networks themselves."
"The web of connections in neural networks has statistical properties identical to parts of the brain cortex."
"Maybe one very simple representational tool that we can use to allow the model to use the symmetry but also not use the symmetry is the skip connection."
"Geometric deep learning serves two purposes: to provide a common mathematical framework to derive the most successful neural network architectures, and to give a constructive procedure to build future architectures in a principled way."
"One of the very key things that I do on a day-to-day basis is teach graph neural networks to imitate algorithms from perfect data."
"Transformers for example are attentional fully connected graph neural networks which are really good both from the perspective of the alignment to this idea of I'm gonna just find the optimal combination of meaningless data points."
"Most of neural networks are sequential models, meaning that the output of one layer is the input of the next one."
"As soon as they got that up and running, everybody in neural networks realized that this was exactly the right architecture for the problems that we have."
"Activation functions add non-linearity to the networks, representing more complex patterns."
"To train any neural network in a supervised learning task, we first need a data set of samples along with the corresponding labels for those samples."
"The activation function shapes the output of the neuron to be inside a certain range."
"It's called backpropagation because in order to calculate it, we actually need the value at the end."
"Deep learning platforms like TensorFlow enable us to rely on neural networks to predict input shapes."
"Neural networks can only memorize, but they cannot always generalize."
"The results from these neural networks and stable diffusion can be really stunning objects that you could never possibly have in real life."
"...deep learning and deep neural networks are very effective tools for us to address the representation learning problem."
"All the parameter layers in Transformer architecture are feed-forward, locally connected, and have weight sharing across the time dimension."
"Deep recurrent neural networks can be built by stacking many recurrent neural network units together."
"By increasing the depth of neural networks, we can enjoy a huge boost in performance."
"Visualization helps us understand neural network behaviors."
"So to talk to actually address this problem in neural networks and in machine learning in general there's a few different ways that you should be aware of and how to do it because you'll need to apply them as part of your Solutions and your software Labs as well."
"We need a nonlinearity to introduce nonlinearities in our neural networks."
"You can obtain really general insights about what happens when the size of the matrix is called Infinity in such a program."
"The size of matrices in the program correspond to the width of the neural networks."
"All of these different things you do for these different architectures that people were writing about actually can be distilled into this simple form, like this low-level language, which is essentially like tensor programs."
"On the neural network, will evolve or will have very simple behavior in the limit."
"It's amazing what backpropagation can do."
"Transformer was essentially the starting point of a new class of neural network that do not rely on recurrence."
"LSTMs and recurrent neural networks are going to see a lot of use going forward because they can do some really impressive things over time."
"Convolutional neural networks are the workhorse behind a lot of what we call artificial intelligence today."
"We're taking the gradients we compute in one layer and propagating them backwards to help us calculate the gradient of the layer that came before."
"Changing this weight eventually gets propagated over to the l sub i that we care about, but it's not a direct process."
"Even though the back propagation math can seem confusing, there is this pattern going on."
"Multi-head attention computes different representations simultaneously."
"So, as you remember, when we calculated the attention using this formula, softmax of Q multiplied by K, divided by square root of DK..."
"We give it the query, the key, the value, the mask, and the dropout layer."
"A lot of the concepts you learned today like the forward pass and the backward pass are also directly applicable to neural networks."
"Quantization aims to use integer numbers to represent parameters in neural networks while maintaining accuracy."
"Self-attention is an operation on sets rather than sequences."
"Multi-head self-attention enables the network to learn different relationships between the words."
"With multi-head self-attention, operations can now be run in parallel."
"Why does cheap and deep learning work so well? I think Darwinian evolution gave us this particular kind of neural network based computer precisely because it's really well tuned for tapping into the kind of computational needs that our cosmos has dished out to us."
"Neural networks prefer when you're classifying images one of the things you want to do is preprocess or normalize your data."
"The reason is that these neural networks do not carry context. Transformer models, on the other hand, they do carry context."
"We're building neural networks that open the door to deeper understandings."
"You can design your neural network in such a way that classically actually it will be very hard to simulate it but then on a quantum computer you could potentially simulate it very efficiently."
"A pair of neural networks that fight with each other."
"And those gradients are going to be used as weights for the attention matrices to average across them."
"It's no longer the case that maybe you could, so the prevailing view was aspirationally maybe someone could train those things but it's obviously impossible local minima will get you, but no you can train a neural net."
"If you like getting into the more nitty-gritty with neural networks and tinkering, I suggest you check out the Neural Networks from Scratch book."
"Using translational invariance into our model by using convolutional neural nets, we were able to train models much faster with less parameters."
"You obtain a very fine-grained way of controlling the behavior of these neural networks."
"It's a kind of art because if you don't define the exact right or a good architecture for your network, it will never learn what you wanted to learn."
"So in this talk I'm going to explore structures that we see inside of neural network loss functions and some of the implications that has for the design of optimization algorithms."
"Mamba is a new neural net architecture that is better than transformers at language modelling."
"The benefit of LSTM units is that they can mitigate the gradient vanishing problem and also they can facilitate long-range dependencies by having some memory that can persist for longer periods of time."
"How we connect the neural networks is really just cutting-edge. It's almost creative in its nature."
"Neural networks are being applied to help us understand."
"Working with Keras, you just add layers on, remember those hidden layers we're talking about?"
"Training a recurrent neural network is not much more difficult than training a regular feed-forward neural network."
"Neural networks are being applied to so many wonderful things—it's such an infant stage technology. What a wonderful time to jump in."
"It changes your neural networks, it changes your belief systems, it changes your vibrations, it changes your alignment."
"So I hope that clarifies the architecture behind RNN and you understand why you can't use simple neural network here and you have to use specialized neural network called RNN which can memorize for you, which can remember previous state because language is all about sequence."
"Neural networks have multiple layers which capture interactions between features."
"This non-linearity is really the key to neural networks and their power."
"You only need to kind of zoom in and think about this local picture of little computational graph nodes that compute outputs and then multiply the local gradients to compute downstream gradients."
"A recurrent neural network can essentially learn relationships between sequence elements without having separate parameters for each item in the sequence."
"Now it's time to think about our backward pass, and the backward pass of a neural network is when we update the parameters, the weights and biases, to reduce our training error."
"We need to figure out a way to have one neural network process all of these inputs."
"...and in the hidden layer here I have put 4 neurons and these 4 neurons are the size of my embedding vector."
"TANH is an activation function that squishes the input into the range of negative one to one."
"The power and the beauty of neural networks is that it can represent these arbitrarily complex functions."
"The first and the third place winners used RNNs, used LSTMs, recurrent neural networks and map a sequence of images to a sequence of steering angles."
"Neural networks have led to incredible breakthroughs in image recognition, language translation, and media generation."
"Essentially it's a repeated cycle of forward pass to see what the loss is, back propagation to calculate the gradients, and gradient descent to update the weights, repeat it over and over until some criterion is hit."
"At its core, the neural network is searching for transformations that convert the input x to the desired output y."
"Neural networks are often black boxes and gaining some interpretability is an ongoing research area."
"...we can quickly compose neural networks."
"Generally, different learned representations from multiple heads can improve the overall network performance."
"Deep learning is a subset of machine learning and focuses on extracting patterns and building representations from data using neural networks."
"Residual networks introduce direct connections from the beginning to deeper layers to mitigate the vanishing gradient problem."
"LSTMs allow the network to retain key information from the beginning to the end without it vanishing."
"Understanding the vanishing gradient problem is important for anyone implementing neural networks."
"So if you wanted a default choice for a non-linearity use relu that's the current default recommendation."
"The correct way to constrain your neural networks is not by making the network smaller, the correct way to do it is to increase your regularization."
"Residual connections greatly improve training conversions, making models train a lot faster."
"We treat deep neural networks as a black box."
"He wanted to be able to understand neural networks, and to piece them apart, and to build modules so that anyone could start hacking them, like if you were playing with your lego toys."
"So just like you can understand how the parts of a program work, you can understand the parts of a deep network."
"...as fascinating as it is to have these neural networks that are able to learn and to do these amazing capabilities the truly amazing part is the analysis you perform the work that you do and the human intelligence that you applied to these concepts."