Add builtwith as a subdomain source by ajistrying · Pull Request #122 · edoardottt/scilla

ajistrying · 2023-11-20T04:39:07Z

Attempts to add BuiltWith as a subdomain source for the subdomain command

ajistrying · 2023-11-20T04:42:31Z

+
+	return keys.BuiltWith
+}
+


@edoardottt I have the GetBuiltWithKey() ready for use, but since it shares so much base functionality with GetVirusTotalKey(), I created a GetKey(api string) function below that just returns the needed key based on an api string passed in.

Usage is on lines 369 and 375 of pkg/runner/runner.go. If this is desirable, we can go back to the separate functions, but thought I'd attempt to dry things up here, lmk your thoughts and I can clean this file up based on the direction you'd like.

edoardottt · 2023-11-20T08:30:38Z

@ajistrying golangci-lint is failing, fix that and I'll start reviewing :)

ajistrying · 2023-11-21T03:37:24Z

@ajistrying golangci-lint is failing, fix that and I'll start reviewing :)

Managed to fix the one warning that I had caused to pop up, but there's a couple that popped up where I didn't touch the code, and I don't necessarily think the warnings are worth pursuing, what are your thoughts? Here they are:

edoardottt · 2023-11-21T08:18:53Z

what are your thoughts?

For those ones it's better to build the string using fmt.Sprintf. Then we could check the lint errors again :)

P.S. We should update the readme accordingly as well

ajistrying · 2023-11-22T03:29:06Z

readme updated and linter warnings addressed, hopefully this is what you had in mind!

edoardottt

Have you ever tried the code you've written?
Is it working properly?
Because it looks like you copied the virustotal part here, imo it can't work

ajistrying · 2023-11-22T19:19:39Z

Have you ever tried the code you've written? Is it working properly? Because it looks like you copied the virustotal part here, imo it can't work

Sorry I should've been more clear, I wanted to see if I was on the right track while working through this. I didn't realize that I instinctively moved the PR over to ready for review, but should have given that CI has been running (it's been a long couple of weeks 😅)

The builtwith implementation is taken from virustotal and I'm not done with it, but I was using the virustotal implementation as a base since I'm guessing some of the same functionality will be used.

Sorry for the misunderstanding!

ajistrying · 2023-11-23T02:47:36Z

@edoardottt Now it's really ready for review lol, went through and tested https://api.builtwith.com/domain-api with some test credits and everything looks to be in good shape, here are some screenshots verifying that we're getting back what we need:

(Tested that we were getting back a legit response from Builtwith)

(Cleaned up version)

edoardottt · 2023-11-23T08:33:50Z

Sorry for the misunderstanding!

No worries :)

There's a check failing (go build), I'll start reviewing the pr when all the checks are passing

ajistrying · 2023-11-25T23:27:50Z

Wanted to bump this :)

edoardottt

not working + deprecated method

edoardottt · 2023-11-27T09:05:20Z

+	BuiltWith  string `yaml:"BuiltWith,omitempty"`
 }

 // ReadKeys gets as input a filename (keys.yaml) and returns a Keys object.


Line 47 uses ioutil.ReadFile but it's deprecated, we should use os.ReadFile

edoardottt · 2023-11-27T09:05:49Z

+		return result
+	}
+
+	for _, elem := range bwrWrapper.Results[0].Result.Paths {


Got this error whie trying to use option -bw:

================ SCANNING SUBDOMAINS ================ Pulling data from BuiltWith panic: runtime error: index out of range [0] with length 0 goroutine 1 [running]: github.com/edoardottt/scilla/pkg/opendb.BuiltWithSubdomains({0x7ffdbf2f80f9, 0xa}, {0xc00058c030, 0x24}, 0xf9?) /home/edoardottt/test/scilla-ftr/pkg/opendb/builtwith.go:151 +0x4e8 github.com/edoardottt/scilla/pkg/runner.SubdomainSubcommandHandler({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...) /home/edoardottt/test/scilla-ftr/pkg/runner/runner.go:376 +0xff2 github.com/edoardottt/scilla/pkg/runner.(*Runner).Execute(0xc00031fc70, 0x0?, 0x0?) /home/edoardottt/test/scilla-ftr/pkg/runner/runner.go:84 +0x1e8 main.main() /home/edoardottt/test/scilla-ftr/cmd/scilla/main.go:39 +0xc5 exit status 2

What I think might be happening here is that for some domains entered, the expected Result array we get back from the api in the response is coming up empty, so I think a simple length check is needed to just early return an empty result and move on if that Result array is indeed empty.

Tested this case with a random domain wqeqwd.com, got your error, and after adding in a fix in tested again and the application flowed as expected.

edoardottt

This is the only result I get with a valid API key:

=====================================================
target: google.com
================ SCANNING SUBDOMAINS ================
Pulling data from BuiltWith
google.com.google.com

ajistrying · 2023-11-29T01:15:27Z

This is the only result I get with a valid API key:

=====================================================
target: google.com
================ SCANNING SUBDOMAINS ================
Pulling data from BuiltWith
google.com.google.com

I'm getting the same response from testing the api using the documented call in the docs with google.com as well, so unsure of what your expectation was of what subdomains would be coming back from this service. If that's not what was expected, what was your expectation? Also, if this is the incorrect api that I'm using could you point me to the correct api within Builtwith that you believe needs to be used instead?

edoardottt · 2023-11-29T09:09:02Z

Trying with the domain builtwith.com as input I get more results and it seems to work fine. Due to the fact that for now it's not that reliable I think the best option is to merge this PR adding builtwith as subdomain source but commenting the adding phase in the runner file. This means this source will not be used for now. What do you think?

ajistrying · 2023-11-30T02:47:19Z

Trying with the domain builtwith.com as input I get more results and it seems to work fine. Due to the fact that for now it's not that reliable I think the best option is to merge this PR adding builtwith as subdomain source but commenting the adding phase in the runner file. This means this source will not be used for now. What do you think?

I think that makes sense, unfortunate that it didn't work out as a reliable source, but it happens! I added the comment // Service not fully reliable yet, is that okay or should I use // Service not working like the others?

ajistrying added 2 commits November 19, 2023 23:27

Add builtwith as subdomain source

f511c27

Add builtwith as subdomain source

bf91314

ajistrying commented Nov 20, 2023

View reviewed changes

ajistrying marked this pull request as ready for review November 20, 2023 04:43

auto-assign bot requested a review from edoardottt November 20, 2023 04:43

ajistrying mentioned this pull request Nov 20, 2023

Add builtwith as subdomain source #109

Closed

adjust for linter

4752bcf

ajistrying added 2 commits November 21, 2023 20:59

update readme and adjust for golangci-lint

de5a7c3

forgot ending ://

8b2a581

edoardottt requested changes Nov 22, 2023

View reviewed changes

ajistrying marked this pull request as draft November 22, 2023 19:17

implement builtwith functionality

74e8300

ajistrying marked this pull request as ready for review November 23, 2023 02:44

auto-assign bot requested a review from edoardottt November 23, 2023 02:44

fixing build errors

6df0577

edoardottt requested changes Nov 27, 2023

View reviewed changes

ajistrying added 2 commits November 28, 2023 00:02

use os package over ioutil for ReadFile method

fe9a25d

return result if desired Results array is empty

c4f1473

ajistrying requested a review from edoardottt November 28, 2023 05:13

edoardottt requested changes Nov 28, 2023

View reviewed changes

comment out builtwith runner logic

00ef962

edoardottt changed the base branch from main to devel November 30, 2023 07:35

edoardottt self-assigned this Nov 30, 2023

edoardottt changed the base branch from devel to main November 30, 2023 07:47

edoardottt merged commit dddc0ef into edoardottt:main Nov 30, 2023

chenrui333 mentioned this pull request Apr 7, 2024

scilla 1.3.0 Homebrew/homebrew-core#168270

Merged

Uh oh!

Conversation

ajistrying commented Nov 20, 2023

Uh oh!

ajistrying Nov 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoardottt commented Nov 20, 2023

Uh oh!

ajistrying commented Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edoardottt commented Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajistrying commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edoardottt left a comment

Choose a reason for hiding this comment

Uh oh!

ajistrying commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajistrying commented Nov 23, 2023

Uh oh!

edoardottt commented Nov 23, 2023

Uh oh!

ajistrying commented Nov 25, 2023

Uh oh!

edoardottt left a comment

Choose a reason for hiding this comment

Uh oh!

edoardottt Nov 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoardottt Nov 27, 2023

Choose a reason for hiding this comment

Uh oh!

ajistrying Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoardottt left a comment

Choose a reason for hiding this comment

Uh oh!

ajistrying commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edoardottt commented Nov 29, 2023

Uh oh!

ajistrying commented Nov 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ajistrying Nov 20, 2023 •

edited

Loading

ajistrying commented Nov 21, 2023 •

edited

Loading

edoardottt commented Nov 21, 2023 •

edited

Loading

ajistrying commented Nov 22, 2023 •

edited

Loading

ajistrying commented Nov 22, 2023 •

edited

Loading

edoardottt Nov 27, 2023 •

edited

Loading

ajistrying Nov 28, 2023 •

edited

Loading

ajistrying commented Nov 29, 2023 •

edited

Loading