Skip to content

Tests to check if sorted by rarity, if correct regex format,...#109

Merged
bee-san merged 7 commits intomainfrom
tests
Jul 10, 2021
Merged

Tests to check if sorted by rarity, if correct regex format,...#109
bee-san merged 7 commits intomainfrom
tests

Conversation

@amadejpapez
Copy link
Copy Markdown
Collaborator

@amadejpapez amadejpapez commented Jul 8, 2021 •
edited
Loading

  • added 3 bigger tests:
    • check if every regex has a test in test_regex_identifier.py (this requires at least one test with a full regex pattern name because previously a lot of tests asserted just one or two words to match to the name)
    • check if regexes are sorted by rarity in regex.json (fixes Add black + isort checkΒ #106)
    • check if regexes have ^(regex)$ format and fail if there are any in-between boundaries (fixes Add black + isort checkΒ #106)
    • I have also added meaningfull messages, so if one of these tests fail, users will be able to easily see why and how to fix it
  • fixed typo in the Placekey name

@bee-san
Copy link
Copy Markdown
Owner

bee-san commented Jul 8, 2021 •
edited
Loading

Can you write "fixes #106" that way github will auto-close the issue when this is merged? edit your original description!

@bee-san
Copy link
Copy Markdown
Owner

bee-san commented Jul 8, 2021

tests fail!

@ghost
Copy link
Copy Markdown

ghost commented Jul 8, 2021

Can you add black and isort check?

@amadejpapez
Copy link
Copy Markdown
Collaborator Author

amadejpapez commented Jul 8, 2021

Can you add black and isort check?

I am not quite sure how to do that. We have to add them to the workflow file right?

@ghost
Copy link
Copy Markdown

ghost commented Jul 8, 2021

Can you add black and isort check?

I am not quite sure how to do that. We have to add them to the workflow file right?

Yes, you can use this for isort and this one for black.

@amadejpapez
Copy link
Copy Markdown
Collaborator Author

amadejpapez commented Jul 8, 2021 •
edited
Loading

I have reorder the database as some some API keys had rarity of 1 and they were not sorted by rarity which was the reason why my tests failed - so rarity and regex format tests work :)

Here is how regexes are sorted now and what criteria I used:

1 (matching to specific words and characters)

PGP Public Key, PGP Private Key, SSH RSA Public Key, SSH ECDSA Public Key, SSH ED25519 Public Key, Access-Control-Allow-Header, TryHackMe Flag Format, HackTheBox Flag Format, Capture The Flag (CTF) Flag, YouTube Video, Bitcoin Cash (BCH) Wallet Address, Amazon Web Services Simple Storage (AWS S3) URL, Amazon Web Services Simple Storage (AWS S3) Internal URL, Square Application Secret, Square Access Token, Stripe API Key, GitHub Access Token, Amazon Resource Name (ARN)

0.7 (very restricted regex, can still cause false positives)

Internet Protocol (IP) Address Version 6, Uniform Resource Locator (URL), Internet Protocol (IP) Address Version 4, Bitcoin (β‚Ώ) Wallet Address, Latitude & Longitude Coordinates

0.5 (mostly matching to specific characters)

EUI-48 Identifier (Ethernet, WiFi, Bluetooth, etc), Placekey Universal Identifier for Physical Place, Dogecoin (DOGE) Wallet Address, Monero (XMR) Wallet Address, Email Address, Phone Number, American Social Security Number, YouTube Channel ID

0.3 (pretty broad, only a few specific characters)

Monero (XMR) Wallet Address, Litecoin (LTC) Wallet Address, Ripple (XRP) Wallet Address, American Express Card Number, BCGlobal Card Number, Carte Blanche Card Number, Diners Club Card Number, Discover Card Number, MasterCard Number, Maestro Card Number, Visa Card Number, Insta Payment Card Number, JCB Card Number, Korean Local Card Number, Laser Card Number, Solo Card Number, Switch Card Number, Ethereum (ETH) Wallet Address, Amazon Web Services EC2 Security Group identifier, Slack Token, Amazon Web Services Organization identifier, Google API Key, Google OAuth Token, Mailgun API Key, Twilio API Key, Twilio Account SID, Twilio Application SID, Google ReCaptcha API Key, Amazon Standard Identification Number (ASIN)

0.2 (very broad, no specific characters, only length limits)

JSON Web Token (JWT), Amazon Web Services Access Key, Amazon Web Services Secret Access Key, Amazon Web Services EC2 Instance identifier, YouTube Video ID, Recent Unix Timestamp, Recent Unix Millisecond Timestamp, Unix Timestamp, Unix Millisecond Timestamp

0 (madness)

Key:Value Pair

@amadejpapez amadejpapez requested a review from a user July 8, 2021 18:07
SkeletalDemise
SkeletalDemise suggested changes Jul 8, 2021
View reviewed changes
ghost
ghost approved these changes Jul 9, 2021
View reviewed changes
Copy link
Copy Markdown

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I am waiting for black and isort check πŸ˜„

@bee-san bee-san merged commit 3b7be39 into main Jul 10, 2021
@bee-san bee-san deleted the tests branch July 10, 2021 13:05
@bee-san
Copy link
Copy Markdown
Owner

bee-san commented Jul 10, 2021

Looks good, but I am waiting for black and isort check

https://github.com/olance/isort-action
https://github.com/rickstaa/action-black/

For the actions :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bee-san bee-san bee-san approved these changes

+2 more reviewers

@SkeletalDemise SkeletalDemise SkeletalDemise requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add black + isort check

3 participants

@amadejpapez @bee-san @SkeletalDemise