Ignore Content-Length header for the purpose of HEAD requests by smashery · Pull Request #17386 · rapid7/metasploit-framework

smashery · 2022-12-15T03:36:50Z

Fixes #3865, by making our parser more compliant with RFC9110, which states:

"...a client MUST retain knowledge of the request when parsing... a corresponding response. For example, responses to the HEAD method look just like the beginning of a response to GET but cannot be parsed in the same manner."

Previously we did not take into account the request's method, and so were parsing a HEAD response like a GET response.
That turned out to be the root cause of this issue, as was suggested in that issue conversation.

As a result of this fix, the auxiliary/scanner/http/http_header module now works, which has been broken for a while, I gather.

Verification

Start msfconsole
use auxiliary/scanner/http/http_header
set rhosts google.com
run
Verify that the headers are returned (previously would just hang then fail)
Verify that other HTTP modules are not affected

adfoster-r7 · 2022-12-15T15:51:06Z

For a future reviewer; We'll need to double check the API isn't used by Metasploit Pro internally

lib/rex/proto/http/client.rb

…d the original request as an opt

gwillcox-r7 · 2023-01-05T19:02:11Z

Before:

msf6 auxiliary(scanner/http/http_header) > show options

Module options (auxiliary/scanner/http/http_header):

   Name         Current Setting                                         Required  Description
   ----         ---------------                                         --------  -----------
   HTTP_METHOD  HEAD                                                    yes       HTTP Method to use, HEAD or GET (Accepted: GET, HEAD)
   IGN_HEADER   Vary,Date,Content-Length,Connection,Etag,Expires,Pragm  yes       List of headers to ignore, separated by comma
                a,Accept-Ranges
   Proxies                                                              no        A proxy chain of format type:host:port[,type:host:port][...]
   RHOSTS       www.google.com                                          yes       The target host(s), see https://github.com/rapid7/metasploit-framework/wiki/Using-Metasploit
   RPORT        80                                                      yes       The target port (TCP)
   SSL          false                                                   no        Negotiate SSL/TLS for outgoing connections
   TARGETURI    /                                                       yes       The URI to use
   THREADS      1                                                       yes       The number of concurrent threads (max one per host)
   VHOST                                                                no        HTTP server virtual host


View the full module info with the info, or info -d command.

msf6 auxiliary(scanner/http/http_header) > set RHOST 142.251.32.164
RHOST => 142.251.32.164
msf6 auxiliary(scanner/http/http_header) > set VHOST www.google.com
VHOST => www.google.com
msf6 auxiliary(scanner/http/http_header) > run
^C
[*] Caught interrupt from the console...
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/http_header) > set SSL true
[!] Changing the SSL option's value may require changing RPORT!
SSL => true
msf6 auxiliary(scanner/http/http_header) > set RPORT 443
RPORT => 443
msf6 auxiliary(scanner/http/http_header) > run

[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/http_header) >

lib/rex/proto/http/packet.rb

gwillcox-r7 · 2023-01-05T19:43:03Z

Been unable to get this working successfully. This is what I get during testing:

msf6 auxiliary(scanner/http/http_header) > show options

Module options (auxiliary/scanner/http/http_header):

   Name         Current Setting             Required  Description
   ----         ---------------             --------  -----------
   HTTP_METHOD  HEAD                        yes       HTTP Method to use, HEAD or GET (Accepted: G
                                                      ET, HEAD)
   IGN_HEADER   Vary,Date,Content-Length,C  yes       List of headers to ignore, separated by comm
                onnection,Etag,Expires,Pra            a
                gma,Accept-Ranges
   Proxies                                  no        A proxy chain of format type:host:port[,type
                                                      :host:port][...]
   RHOSTS       www.google.com              yes       The target host(s), see https://github.com/r
                                                      apid7/metasploit-framework/wiki/Using-Metasp
                                                      loit
   RPORT        80                          yes       The target port (TCP)
   SSL          false                       no        Negotiate SSL/TLS for outgoing connections
   TARGETURI    /                           yes       The URI to use
   THREADS      1                           yes       The number of concurrent threads (max one pe
                                                      r host)
   VHOST                                    no        HTTP server virtual host


View the full module info with the info, or info -d command.

msf6 auxiliary(scanner/http/http_header) > run

[-] Msf::OptionValidateError The following options failed to validate: RHOSTS
msf6 auxiliary(scanner/http/http_header) > set RHOST 142.251.32.164
RHOST => 142.251.32.164
msf6 auxiliary(scanner/http/http_header) > set RPORT 443
RPORT => 443
msf6 auxiliary(scanner/http/http_header) > set SSL true
[!] Changing the SSL option's value may require changing RPORT!
SSL => true
msf6 auxiliary(scanner/http/http_header) > set VHOST www.google.com
VHOST => www.google.com
msf6 auxiliary(scanner/http/http_header) > show options

Module options (auxiliary/scanner/http/http_header):

   Name         Current Setting             Required  Description
   ----         ---------------             --------  -----------
   HTTP_METHOD  HEAD                        yes       HTTP Method to use, HEAD or GET (Accepted: G
                                                      ET, HEAD)
   IGN_HEADER   Vary,Date,Content-Length,C  yes       List of headers to ignore, separated by comm
                onnection,Etag,Expires,Pra            a
                gma,Accept-Ranges
   Proxies                                  no        A proxy chain of format type:host:port[,type
                                                      :host:port][...]
   RHOSTS       142.251.32.164              yes       The target host(s), see https://github.com/r
                                                      apid7/metasploit-framework/wiki/Using-Metasp
                                                      loit
   RPORT        443                         yes       The target port (TCP)
   SSL          true                        no        Negotiate SSL/TLS for outgoing connections
   TARGETURI    /                           yes       The URI to use
   THREADS      1                           yes       The number of concurrent threads (max one pe
                                                      r host)
   VHOST        www.google.com              no        HTTP server virtual host


View the full module info with the info, or info -d command.

msf6 auxiliary(scanner/http/http_header) > run

[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/http_header) > run

[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/http_header) >

gwillcox-r7 · 2023-01-10T16:21:16Z

Sees to be working now 🥳

msf6 > use auxiliary/scanner/http/http_header 
msf6 auxiliary(scanner/http/http_header) > set RHOSTS www.google.com
RHOSTS => www.google.com
msf6 auxiliary(scanner/http/http_header) > run

[+] 142.250.138.103:80   : CACHE-CONTROL: private
[+] 142.250.138.103:80   : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.103:80   : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.103:80   : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.103:80   : SERVER: gws
[+] 142.250.138.103:80   : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsK_O211tiir5GTcgRvDpNPvu9M3TDTDsB-h7zFrjnvUfhgstm8LA-o; expires=Sun, 09-Jul-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.103:80   : TRANSFER-ENCODING: chunked
[+] 142.250.138.103:80   : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.103:80   : X-XSS-PROTECTION: 0
[+] 142.250.138.103:80   : detected 9 headers
[*] Scanned 1 of 7 hosts (14% complete)
[+] 142.250.138.99:80    : CACHE-CONTROL: private
[+] 142.250.138.99:80    : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.99:80    : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.99:80    : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.99:80    : SERVER: gws
[+] 142.250.138.99:80    : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsJChujnYbBEXXzbAAJ7vtuQpWDO0EQACBXoF54iDv320iIpvdZlSg; expires=Sun, 09-Jul-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.99:80    : TRANSFER-ENCODING: chunked
[+] 142.250.138.99:80    : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.99:80    : X-XSS-PROTECTION: 0
[+] 142.250.138.99:80    : detected 9 headers
[*] Scanned 2 of 7 hosts (28% complete)
[+] 142.250.138.104:80   : CACHE-CONTROL: private
[+] 142.250.138.104:80   : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.104:80   : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.104:80   : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.104:80   : SERVER: gws
[+] 142.250.138.104:80   : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsL2Mln-parX8dsgvNM39oaRl-o7xwqm-nk9hl_S_48gHh50iELPEYw; expires=Sun, 09-Jul-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.104:80   : TRANSFER-ENCODING: chunked
[+] 142.250.138.104:80   : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.104:80   : X-XSS-PROTECTION: 0
[+] 142.250.138.104:80   : detected 9 headers
[*] Scanned 3 of 7 hosts (42% complete)
[+] 142.250.138.106:80   : CACHE-CONTROL: private
[+] 142.250.138.106:80   : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.106:80   : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.106:80   : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.106:80   : SERVER: gws
[+] 142.250.138.106:80   : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsJLQ1MYThulA7DhFAQ2Bwo7T8NUbNwB1lnhwEABSKoP5pAx9imXsw; expires=Sun, 09-Jul-2023 16:20:32 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.106:80   : TRANSFER-ENCODING: chunked
[+] 142.250.138.106:80   : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.106:80   : X-XSS-PROTECTION: 0
[+] 142.250.138.106:80   : detected 9 headers
[*] Scanned 4 of 7 hosts (57% complete)
[+] 142.250.138.105:80   : CACHE-CONTROL: private
[+] 142.250.138.105:80   : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.105:80   : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.105:80   : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.105:80   : SERVER: gws
[+] 142.250.138.105:80   : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:33 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsJWKdtYH9q2h_ncu0VaEEKqAaAF3ZZww2Rlsl7kpnl8kfTxa3QYxA; expires=Sun, 09-Jul-2023 16:20:33 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.105:80   : TRANSFER-ENCODING: chunked
[+] 142.250.138.105:80   : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.105:80   : X-XSS-PROTECTION: 0
[+] 142.250.138.105:80   : detected 9 headers
[*] Scanned 5 of 7 hosts (71% complete)
[+] 142.250.138.147:80   : CACHE-CONTROL: private
[+] 142.250.138.147:80   : CONTENT-TYPE: text/html; charset=UTF-8
[+] 142.250.138.147:80   : CROSS-ORIGIN-OPENER-POLICY-REPORT-ONLY: same-origin-allow-popups; report-to="gws"
[+] 142.250.138.147:80   : REPORT-TO: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
[+] 142.250.138.147:80   : SERVER: gws
[+] 142.250.138.147:80   : SET-COOKIE: 1P_JAR=2023-01-10-16; expires=Thu, 09-Feb-2023 16:20:33 GMT; path=/; domain=.google.com; Secure; SameSite=none, AEC=ARSKqsLgXmugY9A6kx4JeE8XgIilOwbCo6cJHkRggX7B4VHBDCmv8N7KBls; expires=Sun, 09-Jul-2023 16:20:33 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
[+] 142.250.138.147:80   : TRANSFER-ENCODING: chunked
[+] 142.250.138.147:80   : X-FRAME-OPTIONS: SAMEORIGIN
[+] 142.250.138.147:80   : X-XSS-PROTECTION: 0
[+] 142.250.138.147:80   : detected 9 headers
[*] Scanned 6 of 7 hosts (85% complete)
[*] Scanned 7 of 7 hosts (100% complete)
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/http_header) >

gwillcox-r7 · 2023-01-10T16:21:43Z

Looks good now @smashery, will land this after the tests pass. Appreciate you fixing this!

gwillcox-r7 · 2023-01-10T17:03:39Z

Release Notes

A bug has been fixed whereby the HTTP library was parsing HTTP HEAD requests like GET requests, which was causing issues due to lack of compliance to RFC9110 standards. By updating the code to be more compliant with these standards, modules such as auxiliary/scanner/http/http_header now work as expected.

smashery added 2 commits December 15, 2022 11:22

Ignore Content-Length header for the purpose of HEAD requests

891ab22

Added unit test to check that HEAD responses don't hang

89dd358

adfoster-r7 reviewed Dec 20, 2022

View reviewed changes

lib/rex/proto/http/client.rb Outdated Show resolved Hide resolved

Keep signature of the read_response function as it was before, and ad…

b05db8b

…d the original request as an opt

gwillcox-r7 self-assigned this Jan 4, 2023

gwillcox-r7 suggested changes Jan 5, 2023

View reviewed changes

lib/rex/proto/http/packet.rb Show resolved Hide resolved

lib/rex/proto/http/packet.rb Show resolved Hide resolved

lib/rex/proto/http/packet.rb Show resolved Hide resolved

smashery and others added 2 commits January 10, 2023 10:57

Fix state machine for HEAD requests

f8b253d

Rubocop fixes and some documentation additions

8b102af

gwillcox-r7 approved these changes Jan 10, 2023

View reviewed changes

gwillcox-r7 added bug rn-fix release notes fix labels Jan 10, 2023

gwillcox-r7 merged commit 87b1f3b into rapid7:master Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore Content-Length header for the purpose of HEAD requests#17386

Ignore Content-Length header for the purpose of HEAD requests#17386
gwillcox-r7 merged 5 commits intorapid7:masterfrom
smashery:http_head_requests

smashery commented Dec 15, 2022 •

edited by gwillcox-r7

Loading

Uh oh!

adfoster-r7 commented Dec 15, 2022

Uh oh!

Uh oh!

gwillcox-r7 commented Jan 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gwillcox-r7 commented Jan 5, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

smashery commented Dec 15, 2022 • edited by gwillcox-r7 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification

Uh oh!

adfoster-r7 commented Dec 15, 2022

Uh oh!

Uh oh!

gwillcox-r7 commented Jan 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gwillcox-r7 commented Jan 5, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Uh oh!

gwillcox-r7 commented Jan 10, 2023

Release Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smashery commented Dec 15, 2022 •

edited by gwillcox-r7

Loading