Add CookieJar support to http_client#14831
Conversation
| end | ||
|
|
||
| def sign_in(username, password) | ||
| http_client.clear_cookies |
There was a problem hiding this comment.
this worries me a little, why do we need to clear cookies here?
are all modules expected to do this?
do cookie values persist between modules?
There was a problem hiding this comment.
You need to clear cookies at this part of the code since you're starting a new communication with the server, and there shouldn't be any cookies in use from previous requests.
Ideally, http_client should be an object instead of a mixin and you'd just create a new http_client for each conversation that needs it's own set of cookies. However, since we haven't done the work to get http_client working as an object yet, clear_cookies has got to stick around as a stopgap.
That being said, I'll double check now to see if cookies remain in the jar between modules, since that could be a problem.
There was a problem hiding this comment.
Maybe this functionality should exist in prepend Msf::Exploit::Remote::AutoCheck, this will continue to bite module developers i'd imagine
There was a problem hiding this comment.
Random internal thought: I'm surprised this hasn't caused issues sooner with other modules that have state stored via mixins between the check and run methods 🤔
There was a problem hiding this comment.
Agreed re: the automatic cookie clear after check, since check should be independent of exploit.
As for why this hasn't happened before, my guess would be that if mixins caused issues moving between check & exploit then it would have been addressed either when the module was first made or when it was landed.
| cj = HTTP::CookieJar.new | ||
|
|
||
| cookie_jar.each do |c| | ||
| next unless c.name == 'experimentation_subject_id' | ||
|
|
||
| payload_cookie = make_default_cookie('experimentation_subject_id', signed_cookie_value) | ||
|
|
||
| payload_cookie.expires = c.expires | ||
| payload_cookie.max_age = c.max_age | ||
| payload_cookie.created_at = c.created_at | ||
| payload_cookie.accessed_at = c.accessed_at | ||
|
|
||
| cj.add(payload_cookie) | ||
| end |
There was a problem hiding this comment.
why is there a need for all this? is there not way to just update the cookies value?
or just create it like was done previously? I don't think all this is necessary
There was a problem hiding this comment.
There are 3 use cases for interacting with the http_client cookie_jar:
- Automatic -
cookie_jarcollects all cookies sent from the server and applies them to all requests going forward, where appropriate. - One Off - What you see above, you want cookie(s) used in a request, so you pass said cookies as a CookieJar
- Global - Use the
http_clientmerge methods to combine a user created CookieJar with thehttp_clientcookie_jar, and said cookies will be applied to all future requests.
Since we're only using these cookies for one request, we don't want to merge them in the http_client cookie_jar, hence the One-Off.
You could make a http_client method that updates cookie values, but it got complicated when you had to account for multiple cookies with the same name but slightly different domains & paths.
| common_cj = HTTP::CookieJar.new | ||
| cookie_jar.each { |c| common_cj.add(c) } |
There was a problem hiding this comment.
Rather than creating a new jar and looping over the old one I think you can use this https://www.rubydoc.info/gems/http-cookie/1.0.3/HTTP/CookieJar#initialize_copy-instance_method
Although I'm genuinely surprised there is no built in merge functionality
There was a problem hiding this comment.
I didn't go with the copy method because "Not all backend store classes support cloning." and I wasn't sure what other backends are used in the code, or what might change in the future. Above code is smellier but IMO more future proof
There was a problem hiding this comment.
Also yes, no merge is annoying. I think it's because HTTP::CookieJar is just a wrapper around a @store object, which can be a few types of collections. So HTTP::CookieJar has to be sparse with the methods it implements to ensure everything works with all the different @stores
|
Before speaking about the specifics of the implementation, I think it would be great to convert over some more modules that are manually keeping track of cookies to see all of the use cases could be handled as well as verifying a few of the recently landed http modules that currently set Let me know your thoughts 👍 |
Makes sense to me 👍 |
|
This is great! Thanks for doing this. It incorporates a new gem, but it's way better than the bespoke approach, which is admittedly kinda shoddy. |
I did try to implement the existing cookie-jar gem we already have added (cookiejar), but it's API is not not not good |
|
|
||
| # Initialize an empty cookie jar to keep cookies | ||
| self.cookie_jar = Set.new | ||
| self.cookie_jar = HTTP::CookieJar.new |
There was a problem hiding this comment.
As discussed on a chat, let's investigate how a wrapper would work:
| self.cookie_jar = HTTP::CookieJar.new | |
| self.cookie_jar = Exploit::Remote::HttpCookieJar.new |
There was a problem hiding this comment.
This will make it easier to unit tests too 🎉
| signed_cookie = sign_payload(secret_key_base, payload) | ||
| signed_cookie_value = sign_payload(secret_key_base, payload) | ||
|
|
||
| cj = HTTP::CookieJar.new |
There was a problem hiding this comment.
cookie_jar is already available:
| cj = HTTP::CookieJar.new | |
| cj = self.cookie_jar |
Which actually makes this line redundant, as we can just use cookie_jar instead of the more cryptic cj 🕵️
There was a problem hiding this comment.
So with the API we're aiming for I think this ends up being:
cookie_jar.clear
cookie_jar.set('experimentation_subject_id', signed_cookie_value)Thining ahead and future proofing an API for a module were we don't have the luxury of clearing the entire cookie jar by itself, we could also use:
cookie_jar = Exploit::Remote::HttpCookieJar.new
cookie_jar.set('experimentation_subject_id', signed_cookie_value)| cookie_jar.each do |c| | ||
| next unless c.name == 'experimentation_subject_id' | ||
|
|
||
| payload_cookie = make_default_cookie('experimentation_subject_id', signed_cookie_value) | ||
|
|
||
| payload_cookie.expires = c.expires | ||
| payload_cookie.max_age = c.max_age | ||
| payload_cookie.created_at = c.created_at | ||
| payload_cookie.accessed_at = c.accessed_at | ||
|
|
||
| cj.add(payload_cookie) | ||
| end |
There was a problem hiding this comment.
This would be a good chunk of code to extract into the http cookie jar wrapper; The python requests API is nice:
https://stackoverflow.com/questions/17224054/how-to-add-a-cookie-to-the-cookiejar-in-python-requests-library
| # Empties the contents of +cookie_jar+, preventing any cookies | ||
| # currently stored in the jar from being used in future requests. | ||
| # | ||
| # @return [void] | ||
| def clear_cookies | ||
| cookie_jar.clear | ||
| end | ||
|
|
||
| # Merges the contents of the +cj+ param into the +cookie_jar+ class var. | ||
| # In the case of Cookies with the same name, domain, and path, the +cj+ cookie will be used. | ||
| # | ||
| # @param cj [HTTP::CookieJar] CookieJar to be merged into +cookie_jar+. | ||
| # | ||
| # @return [HTTP::CookieJar] cookie_jar | ||
| def merge_cookie_jar!(cj) | ||
| cj.cookies.each do |c| | ||
| cookie_jar.add(c) | ||
| end | ||
|
|
||
| cookie_jar | ||
| end | ||
|
|
||
| # Merges the contents of +cj+ & +cookie_jar+ into a new CookieJar. | ||
| # In the case of Cookies with the same name, domain, and path, the +cj+ cookie will be used. | ||
| # | ||
| # @param cj [HTTP::CookieJar] CookieJar to be merged into a new CookieJar with +cookie_jar+. | ||
| # | ||
| # @return [HTTP::CookieJar] | ||
| def merge_cookie_jar(cj) | ||
| common_cj = HTTP::CookieJar.new | ||
| cookie_jar.each { |c| common_cj.add(c) } | ||
|
|
||
| cj.cookies.each do |c| | ||
| common_cj.add(c) | ||
| end | ||
|
|
||
| common_cj | ||
| end | ||
|
|
||
| # Makes a new +HTTP::Cookie+ object with sensible default attributes. | ||
| # | ||
| # @return [HTTP::Cookie] | ||
| def make_default_cookie(cookie_name, cookie_value) | ||
| HTTP::Cookie.new( | ||
| { | ||
| :name => cookie_name, | ||
| :value => cookie_value, | ||
| :domain => rhost, | ||
| :created_at => Time.now, | ||
| :accessed_at => Time.now, | ||
| :origin => "http#{ssl ? 's' : ''}://#{rhost}", | ||
| :secure => ssl, # Are you using HTTPS? | ||
| :path => '/', # Top level of the domain the cookie applies | ||
| :for_domain => false, # If true, send cookie to any host in the domain. | ||
| :httponly => true, # Can the flag be accessed by client side scripts? | ||
| :expires => nil, # At what dateTime (relevant to the client) should the cookie expire | ||
| :max_age => nil, # How old should the cookie be when it expires? (Alternative to +:expires+) | ||
| # If +:expires+ && +:max_age+ are nil, cookie will expire when the current session ends | ||
| } | ||
| ) | ||
| end | ||
|
|
There was a problem hiding this comment.
I think we'd be good to move this entirely into the cookie_jar abstraction class directly
I don't think this violates the law of demeter
6860b83 to
158bb9b
Compare
|
just a note that as a follow on action, the wiki will need to be updated as well. |
ff03afc to
db8cf61
Compare
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
01d74de to
0afbf17
Compare
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
bc5e1da to
3d4ae3b
Compare
3d4ae3b to
88f17c5
Compare
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_jar_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
spec/lib/msf/core/exploit/remote/remote/http/http_cookie_spec.rb
Outdated
Show resolved
Hide resolved
fef5db0 to
1b02344
Compare
|
@agalway-r7 As a pre-requisite to landing this I think we'll want to update the docs to show common workflows for interacting with the cookies/cookie jars etc: https://github.com/rapid7/metasploit-framework/wiki/How-to-Send-an-HTTP-Request-Using-HttpClient |
| # | ||
| # +expire_cookies+ will control if +cleanup+ is called on any passed +Msf::Exploit::Remote::HTTP::HttpCookieJar+ or the client cookiejar |
There was a problem hiding this comment.
I think this documentation needs updated now that expire_cookies isn't a a parameter
There was a problem hiding this comment.
Let's leave this to a follow up PR
|
@agalway-r7 Looks like we'll want to update the exchange module too, as I believe it's broken by this pull request - I haven't ran it, just from a static/eyeballing perspective: Less important, but the spam titan module was recently landed it looks like it uses its own cookie jar as a work around too from what I can see, we could potentially update that - but the above module working is more important. metasploit-framework/modules/exploits/freebsd/webapp/spamtitan_unauth_rce.rb Lines 194 to 202 in b177452 |
| # order. | ||
| def cookies | ||
| @cookie_jar.cookies | ||
| end |
There was a problem hiding this comment.
We might revisit this API in the future, as asking the cookie jar for its cookies seems a bit weird - as it already is a collection of cookies by definition 🤔
There was a problem hiding this comment.
This would pair well with adding support for cookie_jar['cookie_name']
There was a problem hiding this comment.
Let's leave this to a follow up PR
| if domain.nil? || domain.is_a?(DomainName) | ||
| @cookie.domain = domain | ||
| else |
There was a problem hiding this comment.
| if domain.nil? || domain.is_a?(DomainName) | |
| @cookie.domain = domain | |
| else |
I think this is a leaky internal implementation detail, we should probably delete this in a follow up PR
Release NotesUpdated the HttpClient mixin with with a new cookie jar implementation which correctly updates and merges the |
|
I think this is good to land now 🎉 Follow up work includes:
|
|
#14831 (comment) |
| return unless res | ||
|
|
||
| if opts['keep_cookies'] && res.headers['Set-Cookie'].present? | ||
| # XXX: CGI::Cookie (get_cookies_parsed) is hella broken |
There was a problem hiding this comment.
Future us: It's hella broken because Set-Cookie becomes comma deliminated, which messes up the parsing logic later down the line:
metasploit-framework/lib/rex/proto/http/packet/header.rb
Lines 49 to 58 in d9e4aad
get_cookies_parsed was only ever tested with a single Set-Cookie header
Adds the http-cookie gem, and implements
HTTP::CookieJarinhttp_client.http_clientwill now automatically collectSet-Cookieresponses from the server and include them in future requests. Supersedes #14139.Calls to
send_request_cgican now include an instance ofHTTP::CookieJarin theoptsHash. If included, the passed CookieJar will be used in the request instead of the cookies automatically collected from server responses byhttp_client.Calls can also be made to
http_clientto alter the contents of the moduleHTTP::CookieJarinstance by merging in another CookieJar. This will affect the cookies included in all future requests.Verification
List the steps needed to make sure this thing works
msfconsolegitlab_file_read_rceon gitlab docker instance