Performance: read the response dump line by line instead of loading the whole thing in memory#5
Merged
philnash merged 4 commits intophilnash:masterfrom Mar 12, 2018
kpumuk:dmytro/streamed
Merged
Performance: read the response dump line by line instead of loading the whole thing in memory#5philnash merged 4 commits intophilnash:masterfrom kpumuk:dmytro/streamed
philnash merged 4 commits intophilnash:masterfrom
kpumuk:dmytro/streamed
Conversation
added 4 commits
March 7, 2018 02:34
…he whole thing in memory The response from the service will grow over time. There is no way to get passwords [unpwned](danielmiessler/SecLists#155), so we can safely assume the list will keep growing, adding more an more new hashes. One day it will grow large enough to start taking down servers, when users "DDoS" applications with known "big" pwned password hash prefixes. This PR switches from "load everything to memory and find our hash" to "fetch data in chunks, and process line by line".
In Ruby `start_with?` is heavily optimized compared to regular expressions (more than 2 times faster). This PR replaces regular expressions with `start_with?` ``` 13.103359 0.734251 13.837610 ( 14.620959) 13.238428 0.742140 13.980568 ( 14.506166) 12.836573 0.729563 13.566136 ( 14.191792) 12.408245 0.642944 13.051189 ( 13.333299) ``` P.S. Usually I hate micro-optimizations of this sort, but I was bored, and it does not really decrease readability of the code, so why not
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The response from the service will grow over time. There is no way to get passwords unpwned, so we can safely assume the list will keep growing, adding more new hashes. One day it will grow large enough to start taking down servers when users "DDoS" applications with known "big" pwned password hash prefixes.
This PR switches from "load everything to memory and find our hash" to "fetch data in chunks, and process line by line".
Regular expressions removal
In Ruby
start_with?is heavily optimized compared to regular expressions (more than 2 times faster). This PR replaces regular expressions withstart_with?Before: (1000 requests for "password" with mocked network calls)
After: (1000 requests for "password" with mocked network calls)
P.S. Usually I hate micro-optimizations of this sort, but I was bored, and it does not really decrease readability of the code, so why not