hold up, need to fix perplexity.

update: still investigating.

@Green-Sky UB is hard to fix, I really appreciate!

I'll try this PR tomorrow. Before that, let me to make an immature suggestion:

Think about the situation that new segmentation fault occur again, but still take time fix.
Can we add an experimental cli parameter (or env variable), to allow user configure the max memory pool . According to his/her available RAM size (e.g. at least 1 GB). This benefit users who has big RAM and eager to evaluate.

UB is hard to fix, I really appreciate!

it is only UB if you run without address sanitizer 😉

==1743825==ERROR: AddressSanitizer: allocator is out of memory trying to allocate 0x1000000000 bytes
    #0 0x7f6d6c849808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x561e0c966487 in llama_eval_internal /home/green/workspace/github/llama.cpp/llama.cpp:636
    #2 0x561e0c9682bb in llama_eval /home/green/workspace/github/llama.cpp/llama.cpp:1460
    #3 0x561e0c879365 in main /home/green/workspace/github/llama.cpp/main.cpp:229
    #4 0x7f6d6b890082 in __libc_start_main ../csu/libc-start.c:308

==1743825==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: out-of-memory ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 in __interceptor_malloc
==1743825==ABORTING

so, 32GiB are not enough to run perplexity (defaults) on 7B q4_0 . edit: with context 1024
and 64GiB fail to allocate on my machine ...

edit: #407 changes how this works

I decided to wait on #439 and maybe #407, so i don't break perplexity.

for some reason @ggerganov pushed 4870e45 👀

for some reason @ggerganov pushed 4870e45 👀

Unfortunately, this still doesn't fix the memory allocation issues :(

From what I can tell, it pretty much wraps stuff in vectors and adds an assertion to force the code to fail rather than segfaulting.

@ggerganov promised an memory overhaul here #407 (comment)

so i am closing this pr.

Runs smoothly, thanks!

officially replaced by #473