v2/pkg/resolve/resolve.go (1)

76-88: Wildcard check loop broken

Same issue with maxWildcardChecks (constant int). Use:

- for range maxWildcardChecks {
+ for i := 0; i < maxWildcardChecks; i++ {

v2/pkg/testutils/integration.go (1)

30-35: Non-existent strings.SplitSeq and wrong loop variable

strings.SplitSeq doesn’t exist; additionally range items returns indices, yet you treat them as strings.

- items := strings.SplitSeq(string(data), "\n")
- for i := range items {
-     if i != "" {
-         parts = append(parts, i)
-     }
- }
+ for _, item := range strings.Split(strings.TrimSpace(string(data)), "\n") {
+     if item != "" {
+         parts = append(parts, item)
+     }
+ }

v2/pkg/subscraping/sources/crtsh/crtsh.go (2)

121-129: SplitSeq undefined and loop misuses index

Need standard split and value iteration:

- for subdomain := range strings.SplitSeq(data, "\n") {
+ for _, subdomain := range strings.Split(data, "\n") {

---

155-162: Repeat of previous issue

- for sub := range strings.SplitSeq(subdomain.NameValue, "\n") {
+ for _, sub := range strings.Split(subdomain.NameValue, "\n") {

v2/pkg/subscraping/sources/netlas/netlas.go (1)

100-122: Avoid superfluous string conversion when the payload is already []byte.

jsonRequestBody is a []byte, yet we allocate a new string and wrap it with strings.NewReader.
Use bytes.NewReader(jsonRequestBody) to save an allocation and garbage-copy.

-		resp, err = session.HTTPRequest(ctx, http.MethodPost, apiUrl, "", map[string]string{
-			"accept":       "application/json",
-			"X-API-Key":    randomApiKey,
-			"Content-Type": "application/json"}, strings.NewReader(string(jsonRequestBody)), subscraping.BasicAuth{})
+		resp, err = session.HTTPRequest(ctx, http.MethodPost, apiUrl, "", map[string]string{
+			"accept":       "application/json",
+			"X-API-Key":    randomApiKey,
+			"Content-Type": "application/json"}, bytes.NewReader(jsonRequestBody), subscraping.BasicAuth{})

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

v2/pkg/subscraping/sources/pugrecon/pugrecon.go (1)

81-82: Align response cleanup with the new helper

defer resp.Body.Close() bypasses the new error-aware helper and duplicates logic.
Switching keeps the pattern uniform and catches close errors.

-       defer resp.Body.Close()
+       defer session.DiscardHTTPResponse(resp)

v2/pkg/runner/outputter.go (2)

158-164: Same masking issue as above

See the previous comment; the identical error–flush pattern appears here.

---

233-238: Same masking issue as above

See the first comment for suggested fix/refactor.

v2/pkg/subscraping/sources/pugrecon/pugrecon.go (1)

90-91: Nit: direct string wrap with fmt.Errorf unnecessary

fmt.Errorf("%s", errorMsg) is fine but errors.New(errorMsg) is clearer/cheaper when no formatting is needed.

-       results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("%s", errorMsg)}
+       results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: errors.New(errorMsg)}

(Requires errors import.)

v2/pkg/runner/runner.go (2)

97-99: Defer would guarantee close on early returns

If EnumerateMultipleDomainsWithCtx panics or returns before reaching these lines, the file handle stays open.
Use defer f.Close() with error logging to cover all exits.

---

144-146: Duplicate close-and-log blocks – consider helper

The two identical close/error-log snippets invite DRY violation. A small helper will reduce repetition and future misses.

Example:

func safeClose(path string, c io.Closer) {
    if err := c.Close(); err != nil {
        gologger.Error().Msgf("Error closing file %s: %s", path, err)
    }
}

Then defer safeClose(outputFile, file).

Also applies to: 164-166

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

v2/pkg/subscraping/sources/riddler/riddler.go (2)

32-38: Guard against nil resp before discarding

session.SimpleGet can return a non-nil error together with a nil response.
Calling session.DiscardHTTPResponse(resp) unconditionally risks a panic if resp is nil.

-			s.errors++
-			session.DiscardHTTPResponse(resp)
+			s.errors++
+			if resp != nil { // avoid nil-pointer deref
+				session.DiscardHTTPResponse(resp)
+			}

---

40-50: Check scanner.Err() to surface I/O errors

The loop ignores any scanning error.
If the body is truncated or the connection drops, valuable error information is lost and the source silently mis-reports an empty result set.

@@
 		for scanner.Scan() {
 			line := scanner.Text()
 			if line == "" {
 				continue
 			}
 			for _, subdomain := range session.Extractor.Extract(line) {
 				results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}
 				s.results++
 			}
 		}
+		if scanErr := scanner.Err(); scanErr != nil {
+			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: scanErr}
+			s.errors++
+		}

v2/pkg/subscraping/sources/dnsrepo/dnsrepo.go (1)

67-75: Avoid double-free: remove the second DiscardHTTPResponse call

session.DiscardHTTPResponse(resp) is invoked immediately after the body is read (line 67).
Invoking it again in the JSON-unmarshal error branch (line 73) is redundant and risks a double-close if the helper doesn’t guard against it internally. A single defer registered right after the successful Get call keeps the intent clear and guarantees the response is discarded exactly once on all paths.

-       session.DiscardHTTPResponse(resp)
        return
    }
}

Consider instead:

resp, err := session.Get(...)
if err != nil {
    ...
    return
}
defer session.DiscardHTTPResponse(resp)

v2/pkg/subscraping/sources/anubis/anubis.go (1)

35-40: Possible nil-pointer dereference when request fails

session.SimpleGet can return resp == nil on network/URL errors.
Unconditionally passing that value to session.DiscardHTTPResponse risks a panic if DiscardHTTPResponse dereferences the pointer.

-			session.DiscardHTTPResponse(resp)
+			if resp != nil {
+				session.DiscardHTTPResponse(resp)
+			}

v2/pkg/subscraping/sources/bufferover/bufferover.go (2)

58-66: Potential nil-pointer dereference when response is nil

session.DiscardHTTPResponse(resp) is called even when resp == nil (the branch is guarded by err != nil && resp == nil).
Unless DiscardHTTPResponse guards against a nil argument, this will panic.

 resp, err := session.Get(ctx, sourceURL, "", map[string]string{"x-api-key": apiKey})
-if err != nil && resp == nil {
-    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-    s.errors++
-    session.DiscardHTTPResponse(resp) // resp is definitely nil here
-    return
-}
+if err != nil {
+    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+    s.errors++
+    if resp != nil {                      // guard against nil
+        session.DiscardHTTPResponse(resp)
+    }
+    return
+}

---

68-74: Duplicate cleanup ‑ prefer a single deferred discard

The response body is discarded here on the decode-error path, but also again on the success path (line 76). Repeating the call is unnecessary and easy to miss in future edits. Initialise a defer immediately after Get succeeds instead.

 resp, err := session.Get(ctx, sourceURL, "", map[string]string{"x-api-key": apiKey})
-if resp != nil {
-    defer session.DiscardHTTPResponse(resp)
-}

This guarantees cleanup exactly once and removes the need for explicit calls later.

v2/pkg/subscraping/sources/alienvault/alienvault.go (1)

41-46: Incorrect nil‐check leads to leaked connections and possible nil dereference

err != nil && resp == nil ignores the very common case where err != nil and resp is non-nil (e.g. non-2xx status, redirect loop, TLS handshake error returned after opening the connection).
In that scenario the code:

1. Treats the call as success and continues to decode the body, even though err is non-nil.
2. Never closes the body because execution jumps out before the later DiscardHTTPResponse calls.
3. Risks dereferencing resp inside Decode even though the request failed.

Refactor the guard to handle any error first and always discard the response if available:

 resp, err := session.SimpleGet(ctx, fmt.Sprintf("https://otx.alienvault.com/api/v1/indicators/domain/%s/passive_dns", domain))
-if err != nil && resp == nil {
-    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-    s.errors++
-    session.DiscardHTTPResponse(resp)
-    return
-}
+if err != nil {
+    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+    s.errors++
+    if resp != nil {
+        session.DiscardHTTPResponse(resp)
+    }
+    return
+}
+defer session.DiscardHTTPResponse(resp) // one‐liner guarantees cleanup on every path

v2/pkg/subscraping/sources/hackertarget/hackertarget.go (1)

47-58: Missing scanner.Err() check may swallow upstream read errors

After the scanning loop completes we never inspect scanner.Err(). If the HTTP body is truncated or the connection resets, those errors are silently ignored and the statistics under-report failures.

@@
         }
     }
+
+    // Surface any I/O errors that occurred during scanning.
+    if err := scanner.Err(); err != nil {
+        results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+        s.errors++
+    }

Adding this tiny block keeps error reporting symmetric with the new Close() handling and improves observability.

v2/pkg/subscraping/sources/censys/censys.go (2)

115-120: Guard against nil resp before discarding

session.HTTPRequest can legitimately return a nil response on transport errors. Calling session.DiscardHTTPResponse(resp) unconditionally risks a nil-pointer panic unless that helper already checks for nil internally (not guaranteed from this context).

 if err != nil {
     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
     s.errors++
-    session.DiscardHTTPResponse(resp)
+    if resp != nil {
+        session.DiscardHTTPResponse(resp)
+    }
     return
 }

---

122-129: Validate HTTP status code before attempting JSON decode

The code proceeds to decode the body without checking resp.StatusCode. A non-2xx status (e.g., 401/429) will produce a confusing JSON parse error and hide the real problem. Prefer an explicit check and early error return:

 var censysResponse response
-err = jsoniter.NewDecoder(resp.Body).Decode(&censysResponse)
+if resp.StatusCode != 200 {
+    session.DiscardHTTPResponse(resp)
+    results <- subscraping.Result{
+        Source: s.Name(), Type: subscraping.Error,
+        Error:  fmt.Errorf("unexpected status code %d", resp.StatusCode),
+    }
+    s.errors++
+    return
+}
+
+err = jsoniter.NewDecoder(resp.Body).Decode(&censysResponse)
 if err != nil {
     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
     s.errors++
     session.DiscardHTTPResponse(resp)
     return
 }

v2/pkg/subscraping/sources/threatbook/threatbook.go (1)

55-60: Error-handling path discards a potentially nil response and skips the err != nil && resp != *non-nil* case

The guard clause fires only when err != nil && resp == nil, yet you invoke session.DiscardHTTPResponse(resp) anyway.

1. If resp is nil this call is pointless at best and unsafe at worst (depends on the implementation).
2. If err != nil and resp != nil the current branch is bypassed, leaving the body undiscarded and leaking the connection.

A safer, simpler structure:

- if err != nil && resp == nil {
-     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-     s.errors++
-     session.DiscardHTTPResponse(resp)
-     return
- }
+ if err != nil {
+     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+     s.errors++
+     if resp != nil {
+         session.DiscardHTTPResponse(resp)
+     }
+     return
+ }

This handles every error permutation and never passes a nil pointer into the discard helper.

v2/pkg/subscraping/sources/reconcloud/reconcloud.go (2)

46-52: Wrong error predicate + possible nil-pointer in DiscardHTTPResponse

if err != nil && resp == nil only trips when both are true.
In practice an HTTP client almost always returns nil response together with the error, but you’re still relying on that invariant.
More importantly, when the predicate is true resp is guaranteed to be nil (second operand), yet the code immediately calls session.DiscardHTTPResponse(resp) → potential panic inside that helper.

Fix:

resp, err := session.SimpleGet(ctx, fmt.Sprintf("https://recon.cloud/api/search?domain=%s", domain))
-if err != nil && resp == nil {
+if err != nil {
     results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
     s.errors++
-    session.DiscardHTTPResponse(resp)
+    if resp != nil { // defensive, in case http.Client changes behaviour
+        session.DiscardHTTPResponse(resp)
+    }
     return
 }
+
+// ensure body is always cleaned up on every other exit path
+defer session.DiscardHTTPResponse(resp)

This:

1. Handles any non-nil error.
2. Protects against nil response values.
3. Uses a single defer to guarantee the body is discarded exactly once.

---

55-63: Now redundant calls to DiscardHTTPResponse

With the defer in place (see previous comment) the two explicit calls below are no longer necessary and will double-close the body.

-    session.DiscardHTTPResponse(resp)
-    return
+    return
 }
- session.DiscardHTTPResponse(resp)

Remove them to avoid redundant work and make the control-flow clearer.

v2/pkg/subscraping/sources/waybackarchive/waybackarchive.go (1)

49-65: Missing scanner.Err() check may silently swallow read errors

After the for scanner.Scan() loop finishes, scanner.Err() should be inspected; otherwise network/IO errors are ignored and the source reports zero errors even though parsing failed part-way.

@@
        for scanner.Scan() {
@@
                s.results++
        }
+       if err := scanner.Err(); err != nil {
+           results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+           s.errors++
+       }

This preserves accuracy of statistics and surfaces transient network issues to the caller.

v2/pkg/subscraping/sources/digitorus/digitorus.go (1)

51-64: Handle scanner I/O errors to avoid silent data loss

bufio.Scanner will drop any I/O or token-size error unless scanner.Err() is checked after the loop. Currently such errors are ignored, the scrape silently fails, and s.errors is not incremented.

 for scanner.Scan() {
     …
 }
-if scanner.Err() != nil {
-    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: scanner.Err()}
-    s.errors++
-}
+if err := scanner.Err(); err != nil {
+    results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+    s.errors++
+}

v2/pkg/subscraping/sources/robtex/robtext.go (1)

88-90: Possible nil-dereference when session.Get fails

session.Get may return (nil, err) on transport errors. Passing a nil response into session.DiscardHTTPResponse without a guard assumes that the helper tolerates nil; if it ever dereferences resp, this will panic.

-	if err != nil {
-		session.DiscardHTTPResponse(resp)
-		return results, err
+	if err != nil {
+		if resp != nil {
+			session.DiscardHTTPResponse(resp)
+		}
+		return results, err
 	}

v2/pkg/subscraping/sources/redhuntlabs/redhuntlabs.go (1)

78-82: Page-1 is fetched twice – wastes quota and yields duplicates

The first request already retrieved page=1.
The loop below starts again at page := 1, causing an identical second call and duplicate subdomain emission.

- for page := 1; page <= totalPages; page++ {
+ // page 1 already processed above
+ for page := 2; page <= totalPages; page++ {

v2/pkg/subscraping/sources/chinaz/chinaz.go (1)

41-47: DiscardHTTPResponse must tolerate a nil resp

On the error path resp can legitimately be nil (e.g. network failure or DNS error).
If session.DiscardHTTPResponse blindly dereferences resp.Body this statement will panic and kill the worker goroutine, closing the results channel prematurely.

-            session.DiscardHTTPResponse(resp)
+            if resp != nil {
+                session.DiscardHTTPResponse(resp)
+            }

Alternatively, make DiscardHTTPResponse itself a no-op when resp == nil.

v2/pkg/subscraping/sources/gitlab/gitlab.go (1)

110-123: 404 path still leaks the response body

session.DiscardHTTPResponse(resp) is now correctly used for the happy path (line 122), but if err != nil && resp.StatusCode == http.StatusNotFound (see lines 100-107) the function continues without closing resp.Body, leading to a connection leak.

An easy fix is to always discard/close before the early continue/return, even for the 404 case:

-				if resp == nil || (resp != nil && resp.StatusCode != http.StatusNotFound) {
-					session.DiscardHTTPResponse(resp)
-					…
-				}
+				if resp == nil {
+					…
+				}
+				// Always release the body, even on 404
+				session.DiscardHTTPResponse(resp)
+				if resp != nil && resp.StatusCode != http.StatusNotFound {
+					…
+				}

(or simply defer session.DiscardHTTPResponse(resp) immediately after the GET).

v2/pkg/subscraping/sources/commoncrawl/commoncrawl.go (1)

68-70: Invalid range over integer – won’t compile

for i := range maxYearsBack { … } attempts to iterate over an int.
Replace with an index-based loop:

- for i := range maxYearsBack {
+ for i := 0; i < maxYearsBack; i++ {

v2/pkg/subscraping/sources/digitalyama/digitalyama.go (1)

57-62: Consider reusing session.DiscardHTTPResponse for consistency.

Most other sources updated in this PR switched to the helper session.DiscardHTTPResponse(resp), which both drains and closes the body. Using that helper here would keep the cleanup logic uniform across sources and avoid a tiny bit of duplicated code.

-       defer func() {
-               if err := resp.Body.Close(); err != nil {
-                       results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-                       s.errors++
-               }
-       }()
+       defer session.DiscardHTTPResponse(resp)

Up to you—current code is functionally fine, this is purely a consistency/refactor suggestion.

v2/pkg/subscraping/sources/riddler/riddler.go (1)

40-52: Defer the discard to guarantee cleanup & avoid double-close

Instead of invoking session.DiscardHTTPResponse at the very end of the loop, defer it immediately after the successful SimpleGet.
This (a) guarantees cleanup even on early returns (e.g. context cancel) and
(b) removes the need for the explicit call on line 51.

@@
 		resp, err := session.SimpleGet(ctx, fmt.Sprintf("https://riddler.io/search?q=pld:%s&view_type=data_table", domain))
@@
 		}
 
+		// ensure the response body is always fully discarded/closed
+		defer session.DiscardHTTPResponse(resp)
+
 		scanner := bufio.NewScanner(resp.Body)
@@
-		}
-		session.DiscardHTTPResponse(resp)
+		}

v2/pkg/subscraping/sources/anubis/anubis.go (2)

28-57: Duplicate cleanup calls – consider one defer instead

session.DiscardHTTPResponse(resp) is invoked on every early-return path and once after decoding.
You can simplify control-flow and guarantee cleanup even on future branches by deferring it immediately after the successful SimpleGet.

 		resp, err := session.SimpleGet(ctx, fmt.Sprintf("https://jonlu.ca/anubis/subdomains/%s", domain))
 		if err != nil {
 			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
 			s.errors++
-			if resp != nil {
-				session.DiscardHTTPResponse(resp)
-			}
 			return
 		}
+		// Always close/discard the response when we leave this function.
+		defer session.DiscardHTTPResponse(resp)

This removes repetition and makes future maintenance easier.
(Keep the nil-check inside the defer if DiscardHTTPResponse isn’t nil-safe.)

---

42-45: Lost error information for non-200 responses

When the service replies with a non-OK status you silently return, giving callers no clue what happened.
At minimum, emit an error result so users and metrics capture the failure.

-		if resp.StatusCode != http.StatusOK {
-			session.DiscardHTTPResponse(resp)
-			return
-		}
+		if resp.StatusCode != http.StatusOK {
+			results <- subscraping.Result{
+				Source: s.Name(),
+				Type:   subscraping.Error,
+				Error:  fmt.Errorf("unexpected status code %d", resp.StatusCode),
+			}
+			s.errors++
+			return
+		}

v2/pkg/subscraping/sources/bufferover/bufferover.go (1)

76-77: Unneeded discard after adding a defer

If you adopt the deferred approach suggested above, this explicit session.DiscardHTTPResponse(resp) becomes redundant and can be removed, simplifying the happy-path flow.

v2/pkg/subscraping/sources/hackertarget/hackertarget.go (1)

40-45: Explicit close-error handling is 👍 — consider centralising with session.DiscardHTTPResponse for consistency

The new deferred block correctly captures and reports resp.Body.Close() errors, avoiding silent leaks.
Across the code-base, most sources now delegate response cleanup to session.DiscardHTTPResponse(resp), which also drains the body for connection reuse. To keep behaviour uniform and prevent future drift, you could:

- defer func() {
-     if err := resp.Body.Close(); err != nil {
-         results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-         s.errors++
-     }
- }()
+ defer func() {
+     if err := session.DiscardHTTPResponse(resp); err != nil {
+         results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+         s.errors++
+     }
+ }()

This removes duplication and leverages the helper’s drain/reuse logic.
If DiscardHTTPResponse does not return an error, you can still call it first and fall back to resp.Body.Close() for error capture.

v2/pkg/subscraping/sources/threatbook/threatbook.go (1)

67-71: Consider deferring the discard for single-exit readability

After successful decoding you immediately call session.DiscardHTTPResponse(resp), which is correct.
Using a defer right after the SimpleGet call would eliminate the need to remember manual invocations on every exit path and prevent future leaks when code is refactored.

resp, err := session.SimpleGet(ctx, url)
if resp != nil {
    defer session.DiscardHTTPResponse(resp)
}

Not blocking, but it reduces cognitive load and guards against maintenance slip-ups.

v2/pkg/subscraping/sources/c99/c99.go (1)

59-64: Close-error handling is welcome, but diverges from the new session.DiscardHTTPResponse helper

Good call catching resp.Body.Close() errors and surfacing them through the results channel.
That said, most updated sources in this PR switched to the centralised helper session.DiscardHTTPResponse(resp), which:

• drains & closes the body (keeps the HTTP transport reusable)
• nil-checks resp
• encapsulates the same error handling logic

Using the helper keeps behaviour uniform and avoids duplicating subtle cleanup logic:

-	defer func() {
-		if err := resp.Body.Close(); err != nil {
-			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-			s.errors++
-		}
-	}()
+	defer session.DiscardHTTPResponse(resp)

If pushing the error into results is essential, consider extending DiscardHTTPResponse to optionally report through a callback/channel rather than diverging per source.

v2/pkg/subscraping/sources/intelx/intelx.go (1)

126-134: Remove redundant io.ReadAll before discarding the response

The helper already consumes & closes the body; the explicit io.ReadAll followed by another call introduces an unnecessary second read. Streamline this block:

-           _, err = io.ReadAll(resp.Body)
-           if err != nil {
-               results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-               s.errors++
-               session.DiscardHTTPResponse(resp)
-               return
-           }
-           session.DiscardHTTPResponse(resp)
+           session.DiscardHTTPResponse(resp)

This removes needless I/O while preserving error handling through the helper.

v2/pkg/subscraping/sources/digitorus/digitorus.go (1)

44-49: Consider re-using the common response-cleanup helper for consistency

Every other modernized source now relies on session.DiscardHTTPResponse(resp) so that response closing (and associated error logging) is handled in one place.
Duplicating the logic here means:

1. We lose the single-point behaviour (future fixes/improvements to DiscardHTTPResponse will not reach this file).
2. The per-source errors++ bookkeeping becomes inconsistent – other sources only log, this one also counts the error.

If you want to keep the local error counting, consider extending DiscardHTTPResponse to accept a callback so each caller can decide what to do with the close error, and then replace the block with a single defer.

v2/pkg/subscraping/sources/rapiddns/rapiddns.go (2)

51-52: Guard against nil response before discarding

session.DiscardHTTPResponse(resp) is invoked here after io.ReadAll fails.
While resp is expected to be non-nil in this branch, we already call the same helper in the earlier err != nil path (line 43) where resp can legitimately be nil.
To be defensive and avoid accidental nil-pointer dereferences should DiscardHTTPResponse ever omit an internal nil-check, wrap the call:

-if session.DiscardHTTPResponse(resp)
+if resp != nil {
+    session.DiscardHTTPResponse(resp)
+}

---

55-56: Consider draining once per iteration to avoid duplicate work

You now discard the response body twice per successful iteration:

1. After a read error (lines 51-52).
2. After a successful read (line 55).

The first is needed only on the error path, while the second is always executed.
A clearer pattern is:

resp, err := session.SimpleGet(...)
if err != nil {
    // handle & return
}
defer session.DiscardHTTPResponse(resp) // runs on both success & error below

body, err := io.ReadAll(resp.Body)
if err != nil {
    // handle & return
}

Because the loop exits on each return, the deferred call fires immediately, keeping memory usage low and avoiding repeated identical statements.
Up to you whether the small readability gain outweighs the extra defer allocation per request.

v2/pkg/subscraping/sources/threatcrowd/threatcrowd.go (2)

54-59: Prefer using session.DiscardHTTPResponse to standardise cleanup across sources

Almost every other modernised source now delegates connection reuse + close-error handling to session.DiscardHTTPResponse. Retaining a bespoke resp.Body.Close() block here diverges from that convention and misses the extra drain logic the helper provides.

A minimal, drop-in adjustment:

-       defer func() {
-           if err := resp.Body.Close(); err != nil {
-               results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
-               s.errors++
-           }
-       }()
+       // Drain & close so the connection can be reused.
+       defer func() {
+           if cerr := session.DiscardHTTPResponse(resp); cerr != nil {
+               results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: cerr}
+               s.errors++
+           }
+       }()

Gives identical error reporting while aligning with the rest of the codebase and ensuring HTTP keep-alive.

---

55-56: Minor lint: avoid shadowing err inside the deferred func

if err := … introduces a new shadow variable that can trip static-analysis linters (shadow, gosimple). Using a distinct name keeps the outer err untouched and silences the warning:

-           if err := resp.Body.Close(); err != nil {
-               results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
+           if closeErr := resp.Body.Close(); closeErr != nil {
+               results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: closeErr}

v2/pkg/subscraping/sources/bevigil/bevigil.go (1)

60-67: Prefer a single deferred cleanup instead of multiple explicit calls

session.DiscardHTTPResponse(resp) is now invoked in every exit path (decode failure and success). Replacing these duplicates with one defer immediately after a non-nil resp is obtained shortens the function and eliminates the chance of forgetting a path during future edits.

resp, err := session.Get(ctx, getUrl, "", headers)
if err != nil { … }

+defer session.DiscardHTTPResponse(resp)
 
// remove explicit calls on lines 62 and 66

This mirrors the standard defer resp.Body.Close() idiom and keeps resource-management concerns in one place.

v2/pkg/subscraping/sources/dnsdumpster/dnsdumpster.go (1)

68-68: Remove redundant discard to prevent second close

With the above deferred cleanup in place, this explicit call becomes redundant (and could be nil when the request itself failed). Deleting it simplifies the flow and avoids needless work.

-           session.DiscardHTTPResponse(resp)

v2/pkg/subscraping/sources/sitedossier/sitedossier.go (1)

66-70: Guard against nil response before discarding

session.DiscardHTTPResponse(resp) is invoked unconditionally.
If resp is ever nil (e.g., SimpleGet returned an error and resp was nil), this could panic unless DiscardHTTPResponse is explicitly nil-safe.

-if err != nil {
-    ...
-    session.DiscardHTTPResponse(resp)
+if err != nil {
+    ...
+    if resp != nil {
+        session.DiscardHTTPResponse(resp)
+    }

Please verify that DiscardHTTPResponse gracefully handles nil; otherwise, add the check.

v2/pkg/subscraping/sources/fullhunt/fullhunt.go (1)

62-63: Duplicate call—safe but redundant once defer is applied.

After moving cleanup into a defer, this line becomes unnecessary and can be removed to avoid confusion.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro