# Gradient descent: back to basics with F# and C#

DevFeed: [Gradient descent: back to basics with F# and C#](<https://devfeed.tech/articles/gradient-descent-back-to-basics-with-f-and-c-38430.md>)

Original publisher: [Read original article](<https://khmylov.com/2023/02/gradient-descent-engine/>)

Author: Andrew Khmylov

Published: 2023-02-02T00:00:00Z

Content type: tutorial

Language: en

Sources: [Despite the odds](<https://devfeed.tech/sources/despite-the-odds.md>)

Topics: [C#](<https://devfeed.tech/topics/csharp.md>), [F#](<https://devfeed.tech/topics/fsharp.md>), [implementation](<https://devfeed.tech/topics/implementation.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [c-sharp](<https://devfeed.tech/tags/c-sharp.md>), [code](<https://devfeed.tech/tags/code.md>), [f-sharp](<https://devfeed.tech/tags/f-sharp.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>)

## AI overview

An overview of building a gradient descent engine from scratch, with discussion of using F# and C# and a simplified explanation of neural network training.

## Source excerpt

After dabbling with high-level AI tools recently, I wanted to revisit the basics and look under the hood. As they say, the best way to learn something is to build it. I've been following wonderful "from scratch" videos from Andrej Karpathy, where he starts with the essence of all modern AI architectures: gradient descent engine. So, I decided to keep along and build my own engine at https://github.com/khmylov/dumb-gradient, and this article is an overview of its implementation, and some thoughts on using F# and C# for a task like that. Side note: look back at university I completely forgot that one of my university projects back in 2010 was actually about building an image recognition neural network! It was funny to look back at the source code from that era. Turns out, the core ideas are still relevant, it's just computational advances, new modelling discoveries and infrastructure around them that allowed so much progress in the last decade. I'm not going to recite all the details here, the main (much simplified) ideas to comprehend are as follows (I recommend Deep Learning with PyTorch, fast.ai course, and Huggingface course to get a much deeper dive into this) We can represent most real-world data, like text, images, sounds, as vast arrays of numbers Turns out, we can model a lot of "intelligence" tasks (natural language processing, image recognition, etc.) with sophisticated mathematical functions operating on our numeric representation of real-world data Universal approximation theorem was a break-through postulating that even most sophisticated functions can be actually modeled by combining only primitive linear f(x) = a * x + b and basic non-linear functions such as tanh into some inter-connected network (here come the inevitable neuron models and neural networks), we just need a huge number of them to do that, and some computation power to figure out the arguments for those functions "Training" such networks is figuring out how to tweak randomly initialized