Skip to content

Add CookieJar support to http_client#14831

Merged
adfoster-r7 merged 5 commits intorapid7:masterfrom
agalway-r7:http-cookie-jar
Apr 30, 2021
Merged

Add CookieJar support to http_client#14831
adfoster-r7 merged 5 commits intorapid7:masterfrom
agalway-r7:http-cookie-jar

Conversation

@agalway-r7
Copy link
Copy Markdown
Contributor

@agalway-r7 agalway-r7 commented Mar 1, 2021

Adds the http-cookie gem, and implements HTTP::CookieJar in http_client. http_client will now automatically collect Set-Cookie responses from the server and include them in future requests. Supersedes #14139.

Calls to send_request_cgi can now include an instance of HTTP::CookieJar in the opts Hash. If included, the passed CookieJar will be used in the request instead of the cookies automatically collected from server responses by http_client.

Calls can also be made to http_client to alter the contents of the module HTTP::CookieJar instance by merging in another CookieJar. This will affect the cookies included in all future requests.

Verification

List the steps needed to make sure this thing works

  • Start msfconsole
  • Set up a gitlab test environment
  • Test execution of gitlab_file_read_rce on gitlab docker instance
  • See if the module gets a session correctly

@agalway-r7 agalway-r7 changed the title Http cookie jar Add CookieJar support to http_client Mar 1, 2021
end

def sign_in(username, password)
http_client.clear_cookies
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this worries me a little, why do we need to clear cookies here?
are all modules expected to do this?
do cookie values persist between modules?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to clear cookies at this part of the code since you're starting a new communication with the server, and there shouldn't be any cookies in use from previous requests.

Ideally, http_client should be an object instead of a mixin and you'd just create a new http_client for each conversation that needs it's own set of cookies. However, since we haven't done the work to get http_client working as an object yet, clear_cookies has got to stick around as a stopgap.

That being said, I'll double check now to see if cookies remain in the jar between modules, since that could be a problem.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this functionality should exist in prepend Msf::Exploit::Remote::AutoCheck, this will continue to bite module developers i'd imagine

Copy link
Copy Markdown
Contributor

@adfoster-r7 adfoster-r7 Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random internal thought: I'm surprised this hasn't caused issues sooner with other modules that have state stored via mixins between the check and run methods 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed re: the automatic cookie clear after check, since check should be independent of exploit.

As for why this hasn't happened before, my guess would be that if mixins caused issues moving between check & exploit then it would have been addressed either when the module was first made or when it was landed.

Comment on lines +544 to +557
cj = HTTP::CookieJar.new

cookie_jar.each do |c|
next unless c.name == 'experimentation_subject_id'

payload_cookie = make_default_cookie('experimentation_subject_id', signed_cookie_value)

payload_cookie.expires = c.expires
payload_cookie.max_age = c.max_age
payload_cookie.created_at = c.created_at
payload_cookie.accessed_at = c.accessed_at

cj.add(payload_cookie)
end
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is there a need for all this? is there not way to just update the cookies value?
or just create it like was done previously? I don't think all this is necessary

Copy link
Copy Markdown
Contributor Author

@agalway-r7 agalway-r7 Mar 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 3 use cases for interacting with the http_client cookie_jar:

  • Automatic - cookie_jar collects all cookies sent from the server and applies them to all requests going forward, where appropriate.
  • One Off - What you see above, you want cookie(s) used in a request, so you pass said cookies as a CookieJar
  • Global - Use the http_client merge methods to combine a user created CookieJar with the http_client cookie_jar, and said cookies will be applied to all future requests.

Since we're only using these cookies for one request, we don't want to merge them in the http_client cookie_jar, hence the One-Off.

You could make a http_client method that updates cookie values, but it got complicated when you had to account for multiple cookies with the same name but slightly different domains & paths.

Comment on lines +435 to +436
common_cj = HTTP::CookieJar.new
cookie_jar.each { |c| common_cj.add(c) }
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than creating a new jar and looping over the old one I think you can use this https://www.rubydoc.info/gems/http-cookie/1.0.3/HTTP/CookieJar#initialize_copy-instance_method

Although I'm genuinely surprised there is no built in merge functionality

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't go with the copy method because "Not all backend store classes support cloning." and I wasn't sure what other backends are used in the code, or what might change in the future. Above code is smellier but IMO more future proof

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also yes, no merge is annoying. I think it's because HTTP::CookieJar is just a wrapper around a @store object, which can be a few types of collections. So HTTP::CookieJar has to be sparse with the methods it implements to ensure everything works with all the different @stores

@adfoster-r7
Copy link
Copy Markdown
Contributor

Before speaking about the specifics of the implementation, I think it would be great to convert over some more modules that are manually keeping track of cookies to see all of the use cases could be handled as well as verifying a few of the recently landed http modules that currently set keep_cookies. The main reason being that for this one module alone we spotted some gremlins already, I imagine there's more that might be a road blocker to a particular approach

Let me know your thoughts 👍

@agalway-r7
Copy link
Copy Markdown
Contributor Author

Before speaking about the specifics of the implementation, I think it would be great to convert over some more modules that are manually keeping track of cookies to see all of the use cases could be handled as well as verifying a few of the recently landed http modules that currently set keep_cookies. The main reason being that for this one module alone we spotted some gremlins already, I imagine there's more that might be a road blocker to a particular approach

Let me know your thoughts 👍

Makes sense to me 👍

@adfoster-r7 adfoster-r7 marked this pull request as draft March 4, 2021 16:06
@wvu
Copy link
Copy Markdown
Contributor

wvu commented Mar 4, 2021

This is great! Thanks for doing this. It incorporates a new gem, but it's way better than the bespoke approach, which is admittedly kinda shoddy.

@agalway-r7
Copy link
Copy Markdown
Contributor Author

agalway-r7 commented Mar 9, 2021

This is great! Thanks for doing this. It incorporates a new gem, but it's way better than the bespoke approach, which is admittedly kinda shoddy.

I did try to implement the existing cookie-jar gem we already have added (cookiejar), but it's API is not not not good


# Initialize an empty cookie jar to keep cookies
self.cookie_jar = Set.new
self.cookie_jar = HTTP::CookieJar.new
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed on a chat, let's investigate how a wrapper would work:

Suggested change
self.cookie_jar = HTTP::CookieJar.new
self.cookie_jar = Exploit::Remote::HttpCookieJar.new

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will make it easier to unit tests too 🎉

signed_cookie = sign_payload(secret_key_base, payload)
signed_cookie_value = sign_payload(secret_key_base, payload)

cj = HTTP::CookieJar.new
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cookie_jar is already available:

Suggested change
cj = HTTP::CookieJar.new
cj = self.cookie_jar

Which actually makes this line redundant, as we can just use cookie_jar instead of the more cryptic cj 🕵️

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So with the API we're aiming for I think this ends up being:

cookie_jar.clear
cookie_jar.set('experimentation_subject_id', signed_cookie_value)

Thining ahead and future proofing an API for a module were we don't have the luxury of clearing the entire cookie jar by itself, we could also use:

cookie_jar = Exploit::Remote::HttpCookieJar.new
cookie_jar.set('experimentation_subject_id', signed_cookie_value)

Comment on lines +546 to +557
cookie_jar.each do |c|
next unless c.name == 'experimentation_subject_id'

payload_cookie = make_default_cookie('experimentation_subject_id', signed_cookie_value)

payload_cookie.expires = c.expires
payload_cookie.max_age = c.max_age
payload_cookie.created_at = c.created_at
payload_cookie.accessed_at = c.accessed_at

cj.add(payload_cookie)
end
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a good chunk of code to extract into the http cookie jar wrapper; The python requests API is nice:
https://stackoverflow.com/questions/17224054/how-to-add-a-cookie-to-the-cookiejar-in-python-requests-library

Comment on lines +404 to +465
# Empties the contents of +cookie_jar+, preventing any cookies
# currently stored in the jar from being used in future requests.
#
# @return [void]
def clear_cookies
cookie_jar.clear
end

# Merges the contents of the +cj+ param into the +cookie_jar+ class var.
# In the case of Cookies with the same name, domain, and path, the +cj+ cookie will be used.
#
# @param cj [HTTP::CookieJar] CookieJar to be merged into +cookie_jar+.
#
# @return [HTTP::CookieJar] cookie_jar
def merge_cookie_jar!(cj)
cj.cookies.each do |c|
cookie_jar.add(c)
end

cookie_jar
end

# Merges the contents of +cj+ & +cookie_jar+ into a new CookieJar.
# In the case of Cookies with the same name, domain, and path, the +cj+ cookie will be used.
#
# @param cj [HTTP::CookieJar] CookieJar to be merged into a new CookieJar with +cookie_jar+.
#
# @return [HTTP::CookieJar]
def merge_cookie_jar(cj)
common_cj = HTTP::CookieJar.new
cookie_jar.each { |c| common_cj.add(c) }

cj.cookies.each do |c|
common_cj.add(c)
end

common_cj
end

# Makes a new +HTTP::Cookie+ object with sensible default attributes.
#
# @return [HTTP::Cookie]
def make_default_cookie(cookie_name, cookie_value)
HTTP::Cookie.new(
{
:name => cookie_name,
:value => cookie_value,
:domain => rhost,
:created_at => Time.now,
:accessed_at => Time.now,
:origin => "http#{ssl ? 's' : ''}://#{rhost}",
:secure => ssl, # Are you using HTTPS?
:path => '/', # Top level of the domain the cookie applies
:for_domain => false, # If true, send cookie to any host in the domain.
:httponly => true, # Can the flag be accessed by client side scripts?
:expires => nil, # At what dateTime (relevant to the client) should the cookie expire
:max_age => nil, # How old should the cookie be when it expires? (Alternative to +:expires+)
# If +:expires+ && +:max_age+ are nil, cookie will expire when the current session ends
}
)
end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd be good to move this entirely into the cookie_jar abstraction class directly

I don't think this violates the law of demeter

@h00die
Copy link
Copy Markdown
Contributor

h00die commented Mar 29, 2021

just a note that as a follow on action, the wiki will need to be updated as well.

@agalway-r7 agalway-r7 force-pushed the http-cookie-jar branch 2 times, most recently from 01d74de to 0afbf17 Compare April 15, 2021 12:39
@agalway-r7 agalway-r7 force-pushed the http-cookie-jar branch 4 times, most recently from bc5e1da to 3d4ae3b Compare April 16, 2021 15:49
@agalway-r7 agalway-r7 marked this pull request as ready for review April 20, 2021 14:41
@adfoster-r7
Copy link
Copy Markdown
Contributor

@agalway-r7 As a pre-requisite to landing this I think we'll want to update the docs to show common workflows for interacting with the cookies/cookie jars etc: https://github.com/rapid7/metasploit-framework/wiki/How-to-Send-an-HTTP-Request-Using-HttpClient

Comment on lines +381 to +382
#
# +expire_cookies+ will control if +cleanup+ is called on any passed +Msf::Exploit::Remote::HTTP::HttpCookieJar+ or the client cookiejar
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this documentation needs updated now that expire_cookies isn't a a parameter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave this to a follow up PR

@adfoster-r7
Copy link
Copy Markdown
Contributor

@agalway-r7 Looks like we'll want to update the exchange module too, as I believe it's broken by this pull request - I haven't ran it, just from a static/eyeballing perspective:

unless res.code == 200 && cookie_jar.grep(/^cadata/).any?

Less important, but the spam titan module was recently landed it looks like it uses its own cookie jar as a work around too from what I can see, we could potentially update that - but the above module working is more important.

# For whatever reason, the web application sometimes returns multiple
# PHPSESSID cookies and only the last one is valid. So, make sure only
# the valid one is part of the cookie_jar.
cookies = res.get_cookies.split(' ')
php_session = cookies.select { |cookie| cookie.starts_with?('PHPSESSID=') }.last
cookie_jar.clear
cookie_jar.add(php_session)
remaining_cookies = cookies.delete_if { |cookie| cookie.starts_with?('PHPSESSID=') }
cookie_jar.merge(remaining_cookies)

# order.
def cookies
@cookie_jar.cookies
end
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might revisit this API in the future, as asking the cookie jar for its cookies seems a bit weird - as it already is a collection of cookies by definition 🤔

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would pair well with adding support for cookie_jar['cookie_name']

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave this to a follow up PR

Comment on lines +105 to +107
if domain.nil? || domain.is_a?(DomainName)
@cookie.domain = domain
else
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if domain.nil? || domain.is_a?(DomainName)
@cookie.domain = domain
else

I think this is a leaky internal implementation detail, we should probably delete this in a follow up PR

@adfoster-r7 adfoster-r7 merged commit 6c6d769 into rapid7:master Apr 30, 2021
@adfoster-r7
Copy link
Copy Markdown
Contributor

adfoster-r7 commented Apr 30, 2021

Release Notes

Updated the HttpClient mixin with with a new cookie jar implementation which correctly updates and merges the Set-Cookie header responses when using the send_request_cgi keep_cookies option.

@adfoster-r7
Copy link
Copy Markdown
Contributor

adfoster-r7 commented Apr 30, 2021

I think this is good to land now 🎉

Follow up work includes:

@h00die
Copy link
Copy Markdown
Contributor

h00die commented Apr 30, 2021

#14831 (comment)
also, the webapp example module could benefit from having this incorporated

return unless res

if opts['keep_cookies'] && res.headers['Set-Cookie'].present?
# XXX: CGI::Cookie (get_cookies_parsed) is hella broken
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Future us: It's hella broken because Set-Cookie becomes comma deliminated, which messes up the parsing logic later down the line:

# Extract each header value pair
header.split(/\r\n/mn).each { |str|
if (md = str.match(/^(.+?)\s*:\s*(.+?)\s*$/))
if (self[md[1]])
self[md[1]] << ", " + md[2]
else
self[md[1]] = md[2]
end
end
}

get_cookies_parsed was only ever tested with a single Set-Cookie header

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement rn-enhancement release notes enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants