Changes

• Update Makefile to detect AVX512 support and add compiler flags if it's available
• Add AVX512 impl based on existing AVX2 implementation, dot product on one 32-value block of 4-bit quantized ints at a time
• Perform 8 bit -> 16 bit sign extension and multiply+add on 32 values at time instead of 16
• Use built-in AVX512 horizontal reduce add to get sum at the end
• Manual unrolling on inner dot product loop to reduce loop counter overhead

Performance Impact

I'm seeing around 10% speedup on the 4-bit quantized 7B model when running on my AMD 7950x.

Before:

main: mem per token = 14368644 bytes
main:     load time =   923.25 ms
main:   sample time =    85.94 ms
main:  predict time = 23502.37 ms / 92.17 ms per token
main:    total time = 24845.69 ms

After:

main: mem per token = 14368644 bytes
main:     load time =   928.89 ms
main:   sample time =    16.18 ms
main:  predict time =  5720.41 ms / 82.90 ms per token
main:    total time =  6982.89 ms

I was hoping for more, but some other stuff I tried like converting the bytesFromNibbles function to operate on two blocks at a time by using AVX512 were not successful.