astle dsa (@AstleDsa) / X

astle dsa

4,810 posts

astle dsa

@AstleDsa

Living in complexity, formalism, mathematics and computer science

Joined March 2022

astle dsa
@AstleDsa
Aug 6, 2024
Transformer in C (from scratch) Finally took the time to complete this, the penultimate step for the project. Writing this was easier than I thought, added AddNorm and FFN layers as well. A low hanging fruit would be to parallelize the attention mechanism (model parallelism)
56K
astle dsa
@AstleDsa
Jun 4, 2024
From scratch projects are underrated. Here's a list of things that I want to build from scratch (also making lists are fun): - Operating System/Kernel - Compiler/Interpreter - JS Framework - Database - Vector Database - Deep Learning Framework - Git - LLM (can't run it ofc)
20K
astle dsa
@AstleDsa
Jul 17, 2024
Nearly done with my deep learning framework in C. I have got: - A matrix library - Autograd engine - Batch, Layer and Model abstractions (?blocks) - Parallel/Concurrent Model Training (Data Parallelism) After cleaning up, i'll be getting close to my final goal with this project
20K
astle dsa
@AstleDsa
Jun 26, 2024
Implemented a numpy + autograd engine in C, and trained a simple MLP which learned the inverse function of a matrix. Fun, but much more left to do (which includes cleaning up).
20K
astle dsa
@AstleDsa
Aug 9, 2024
Replying to @izs
I had read somewhere that Tolkien first created a language itself, and than followed it up with the novels ?
55K
astle dsa
@AstleDsa
Jun 25, 2024
While talking to an embedding systems engineer on C being an unsafe language, he mentioned that the way they avoid memory leaks is by simply not using malloc/alloc, which means their code to entirely deterministic at compile time. A reason why C cannot be replaced in embedded sys
37K
astle dsa
@AstleDsa
Jul 1, 2024
Got it to 1.536kB. Turns out I was allocating around 2 million % more memory than needed. Had to really hack around to getting from ~300mb->~7mb->1536B. The total overhead memory usage was finally 0% (3M%->400K%->0%). Tracking memory was fun tbh
astle dsa
@AstleDsa
Jun 28, 2024
Learned a lesson in memory management. My previous code for training a single MLP in my autograd engine in C was taking around ~300Mb of memory. Had to optimise a lot and finally managed to bring it down to ~6Mb. Figuring out how to bring it down further
10K
astle dsa
@AstleDsa
Jun 1, 2024
Replying to @ludwigABAP
The problem is that it only extends to the very basic symbols. A little more abstraction and it's pandemonium
17K
astle dsa
@AstleDsa
Jun 6, 2024
Replying to @zmkzmkz
I did something like this (I think ?) but mathematically:
mathblog.vercel.app
Exploring architectures- Transformers II
A match made in heaven, keys and values
7K
astle dsa
@AstleDsa
Sep 4, 2024
Might build a personal search engine at around 500-1000 blogs
5.5K
astle dsa
@AstleDsa
Oct 28, 2024
Replying to @mu_chrinovic
Due to ML frameworks, it's steered away from the "math intensive" path and requires the very basics to understand Here I'm talking about surface level understanding ofc Also LLMs just output good well known models in zero shot so 🤷
12K
astle dsa
@AstleDsa
Jun 28, 2024
Learned a lesson in memory management. My previous code for training a single MLP in my autograd engine in C was taking around ~300Mb of memory. Had to optimise a lot and finally managed to bring it down to ~6Mb. Figuring out how to bring it down further
12K
astle dsa
@AstleDsa
Jul 5, 2024
Replying to @JoshuaLelon
But that's a very GenZ thing I feel (maybe millennial) due to advent of mobiles and facetime Otherwise I think all the defining moments in my previous generations, at least where I live, happened during broad daylight
25K
astle dsa
@AstleDsa
Aug 6, 2024
Replying to @AstleDsa
Nearly forgot to commit :
github.com
GitHub - astledsa/Deep-Learning-C: An implement of deep learning framework and models in C
An implement of deep learning framework and models in C - astledsa/Deep-Learning-C
2.7K