I was running older , out of date code, without your latest speed up. Here is an updated performance run:
./coli run "Explain how a hash map works." --model /Users/doug/glm52_i4 --ram 80 --ngen 128
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ run
▀
──────────────────────────────────────────────────────────
== Motore C GLM / C GLM engine (glm_moe_dsa), cache=8 expert/layer | expert@8-bit densa/dense@8-bit ==
caricato/loaded in 1.17s | densa residente / resident dense: 9647.10 MB | layers=78 experts=256 | MTP assente/off (draft=0)
[RAM_GB=80.0] cap=8 ok (proiezione picco / projected peak 25.1 GB)
[stop] 3 token di stop / stop tokens: 154820 154827 154829
prompt: 14 token | genero fino a / generating up to 128 (stop EOS=154820) | draft n-gram=0
[gMASK]<|user|>Explain how a hash map works.<|assistant|>At its core, is a hash map (also known as a hash
[t=16 RSS 21.73 GB hit 19% 0.65 tok/s 1.00 tok/fw]
table, dictionary, or associative array) is a data structure used to store information
[t=32 RSS 21.74 GB hit 24% 0.86 tok/s 1.00 tok/fw]
in what programmers call "key-value pairs."**
Imagine you have
[t=48 RSS 21.74 GB hit 24% 0.94 tok/s 1.00 tok/fw]
a giant warehouse full of boxes. You want to store and retrieve items quickly.
[t=64 RSS 21.74 GB hit 23% 0.98 tok/s 1.00 tok/fw]
Instead of searching through every single box one by one (which is how an array
[t=80 RSS 21.74 GB hit 22% 1.01 tok/s 1.00 tok/fw]
or list works), you use a magical filing system.
Here is a step
[t=96 RSS 21.75 GB hit 23% 1.03 tok/s 1.00 tok/fw]
-by-step breakdown of how it works under the hood:
1. The
[t=112 RSS 21.75 GB hit 22% 1.04 tok/s 1.00 tok/fw]
Underlying Array
At a low level, a hash map is essentially just a
[t=128 RSS 21.75 GB hit 23% 1.06 tok/s 1.00 tok/fw]
128 token in 121.33s (1.06 tok/s) | hit-rate expert 22.8% | RSS 21.75 GB
expert caricati (loaded)/token: 660.9 (per-layer 8.81 su/of 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculazione/speculation: 1.01 token/forward (127 fw per/for 128 tok) | MTP acceptance 0% (0/0)
PROFILO/PROFILE: expert-disk 42.7s | expert-matmul 45.9s | attention 23.1s (di cui/incl. kvb 0.1s) | lm_head 0.0s | altro/other 9.6s
I was running older , out of date code, without your latest speed up. Here is an updated performance run:
./coli run "Explain how a hash map works." --model /Users/doug/glm52_i4 --ram 80 --ngen 128
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ run
▀
──────────────────────────────────────────────────────────
== Motore C GLM / C GLM engine (glm_moe_dsa), cache=8 expert/layer | expert@8-bit densa/dense@8-bit ==
caricato/loaded in 1.17s | densa residente / resident dense: 9647.10 MB | layers=78 experts=256 | MTP assente/off (draft=0)
[RAM_GB=80.0] cap=8 ok (proiezione picco / projected peak 25.1 GB)
[stop] 3 token di stop / stop tokens: 154820 154827 154829
prompt: 14 token | genero fino a / generating up to 128 (stop EOS=154820) | draft n-gram=0
[gMASK]<|user|>Explain how a hash map works.<|assistant|>At its core, is a hash map (also known as a hash
[t=16 RSS 21.73 GB hit 19% 0.65 tok/s 1.00 tok/fw]
table, dictionary, or associative array) is a data structure used to store information
[t=32 RSS 21.74 GB hit 24% 0.86 tok/s 1.00 tok/fw]
in what programmers call "key-value pairs."**
Imagine you have
[t=48 RSS 21.74 GB hit 24% 0.94 tok/s 1.00 tok/fw]
a giant warehouse full of boxes. You want to store and retrieve items quickly.
[t=64 RSS 21.74 GB hit 23% 0.98 tok/s 1.00 tok/fw]
Instead of searching through every single box one by one (which is how an array
[t=80 RSS 21.74 GB hit 22% 1.01 tok/s 1.00 tok/fw]
or list works), you use a magical filing system.
Here is a step
[t=96 RSS 21.75 GB hit 23% 1.03 tok/s 1.00 tok/fw]
-by-step breakdown of how it works under the hood:
1. The
[t=112 RSS 21.75 GB hit 22% 1.04 tok/s 1.00 tok/fw]
Underlying Array
At a low level, a hash map is essentially just a
[t=128 RSS 21.75 GB hit 23% 1.06 tok/s 1.00 tok/fw]
128 token in 121.33s (1.06 tok/s) | hit-rate expert 22.8% | RSS 21.75 GB
expert caricati (loaded)/token: 660.9 (per-layer 8.81 su/of 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculazione/speculation: 1.01 token/forward (127 fw per/for 128 tok) | MTP acceptance 0% (0/0)
PROFILO/PROFILE: expert-disk 42.7s | expert-matmul 45.9s | attention 23.1s (di cui/incl. kvb 0.1s) | lm_head 0.0s | altro/other 9.6s