Skip to content

Commit f15d03e

Browse files
committed
docs: fix typo - QSV_MEMORY_HEADROOM_PCT -> QSV_FREEMEMORY_HEADROOM_PCT
[skip ci]
1 parent 502e0ca commit f15d03e

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

docs/PERFORMANCE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -152,15 +152,15 @@ To prevent this, qsv has two memory check heuristics when running "non-streaming
152152
2. subtract a percentage headroom from the total available memory (default: 20%)
153153
3. if this adjusted total available memory is less than the size of the CSV file, qsv will abort with an error
154154

155-
The percentage headroom can be changed by setting the `QSV_MEMORY_HEADROOM_PCT` environment variable to a value between 10 and 90 (default: 20).
155+
The percentage headroom can be changed by setting the `QSV_FREEMEMORY_HEADROOM_PCT` environment variable to a value between 10 and 90 (default: 20).
156156

157157
This CONSERVATIVE heuristic can have false positives however, as modern operating systems can do a fair bit of juggling to handle file sizes larger than what this heuristic will allow, as it dynamically swaps apps to the swapfile, expand the swapfile, compress memory, etc.
158158

159159
For example, on a 16gb Mac mini running several common apps, it only allowed ~3gb csv files, but in practice, it was able to handle files up to 8gb before this heuristic was added.
160160

161161
To apply this CONSERVATIVE heuristic, you can use the command's `--memcheck` option or set the `QSV_MEMORY_CHECK` environment variable.
162162

163-
Otherwise, the default memory check heuristic (NORMAL mode) will only check if the input file's size is larger than the TOTAL memory of the computer minus `QSV_MEMORY_HEADROOM_PCT`. We still do this to prevent OOM panics, but it's not as restrictive as the CONSERVATIVE heuristic. (e.g. if you have a 16gb computer, the maximum input file size is 12.8gb file - 16gb minus 20% headroom).
163+
Otherwise, the default memory check heuristic (NORMAL mode) will only check if the input file's size is larger than the TOTAL memory of the computer minus `QSV_FREEMEMORY_HEADROOM_PCT`. We still do this to prevent OOM panics, but it's not as restrictive as the CONSERVATIVE heuristic. (e.g. if you have a 16gb computer, the maximum input file size is 12.8gb file - 16gb minus 20% headroom).
164164

165165
> NOTE: These memory checks are not invoked when using stdin as input, as the size of the input file is not known. Though `schema` and `tojsonl` will still abort if stdin is too large per this memory check as it creates a temporary file from stdin before inferring the schema.
166166

0 commit comments

Comments
 (0)