Skip to content

Api/tracing#1794

Merged
technoweenie merged 10 commits intoapi-masterfrom
api/tracing
Jan 3, 2017
Merged

Api/tracing#1794
technoweenie merged 10 commits intoapi-masterfrom
api/tracing

Conversation

@technoweenie
Copy link
Contributor

@technoweenie technoweenie commented Dec 21, 2016

This attempts to port over the tracing code from httputil. The main change is relying no tracing/debugging flags on *lfsapi.Client instead of a global *config.Configuration.

TODO

  • Tests
  • HTTP Request Stats

@technoweenie
Copy link
Contributor Author

Ready for review: This PR ports the tracing capabilities from the httputil package.

It's duties, at a high level, are:

func (c *HttpClient) Do(req *http.Request) (*http.Response, error) {
	// trace request details for GIT_CURL_VERBOSE
	traceHttpRequest(c.Config, req)

	// wrap request body for GIT_CURL_VERBOSE
	crc := countingRequest(c.Config, req)
	if req.Body != nil {
		// Only set the body if we have a body, but create the countingRequest
		// anyway to make using zeroed stats easier.
		req.Body = crc
	}

	// track request start time for GIT_LOG_STATS
	start := time.Now()
	
	// make the request
	res, err := c.Client.Do(req)
	if err != nil {
		return res, err
	}

	// trace response details for GIT_CURL_VERBOSE
	traceHttpResponse(c.Config, res)

	// wrap response body for GIT_CURL_VERBOSE and GIT_LOG_STATS
	cresp := countingResponse(c.Config, res)
	res.Body = cresp

	// track http response details for GIT_LOG_STATS
	if c.Config.IsLoggingStats {
		// snip
	}

	return res, err
}

Note that there are three different flags for tracing and debugging:

  1. GIT_TRACE - Handled invisibly through tracerx.Printf()
  2. GIT_CURL_VERBOSE - Normal HTTP tracing, with obscured Authorization Header values
  3. LFS_DEBUG_HTTP - Turns on full Authorization header when tracing HTTP dumps
  4. GIT_LOG_STATS - Dumps details http stats to disk when the command exits

I'm trying to keep the code structure similar, but am interested in other ideas for a future PR. Also, it's impossible to test GIT_TRACE without major changes to the tracerx package :(

@ttaylorr ttaylorr mentioned this pull request Dec 22, 2016
17 tasks
@technoweenie technoweenie merged commit e1677db into api-master Jan 3, 2017
@technoweenie technoweenie deleted the api/tracing branch January 3, 2017 18:33
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Jan 21, 2026
At present, when the GIT_CURL_VERBOSE environment variable is set to
a value such as "1" or "true", Git LFS client outputs some of the details
of each HTTP request and response it makes to its standard error stream.

We gather the HTTP request headers to be logged in the traceRequest()
method of the Client structure from our "lfshttp" package.  To gather
the headers, the traceRequest() method uses the DumpRequest() function
from the from the "net/http/httputil" package in the Go standard library.

Our use of the DumpRequest() function for this purpose dates back to
commit 6fafade of PR git-lfs#366 in 2015,
when it was first introduced into what was then the traceHttpRequest()
function in our "lfs" package.  This function gradually evolved into the
current traceRequest() method through a series of PRs.  First, portions
of the "lfs" package were migrated into a new "httputil" package in
PR git-lfs#1226.  That package was later removed in PR git-lfs#1846 after a new
"lfsapi" package was added in PR git-lfs#1794 with a version of the
traceRequest() method very similar to the current one.  Finally, in
PR git-lfs#3244 we moved the method into the current "lfshttp" package.

Throughout these changes, however, our use of the DumpRequest()
function has remained the same.  However, since the Git LFS client only
acts as an HTTP client and not an HTTP server, we should instead be
using the DumpRequestOut() function from the standard "httputil" package
to gather the headers of an HTTP request for logging purposes.

Replacing our use of the DumpRequest() function with the DumpRequestOut()
function would allow us to output a number of request headers that we
currently fail to log, as was noted with respect to the Accept-Encoding
header in issue git-lfs#6193.

Since Go version 1.6, the "httputil" package's documentation has stated
that the DumpRequest() function "should only be used by servers to debug
client requests", and that the DumpRequestOut() function "is like
DumpRequest but for outgoing client requests":

  https://pkg.go.dev/net/http/httputil@go1.6#DumpRequest
  https://pkg.go.dev/net/http/httputil@go1.6#DumpRequestOut

However, at the time we first adopted the use of the DumpRequest()
function, the Go documentation did not include these directives, and
merely noted that "DumpRequest returns the as-received wire representation
of req, optionally including the request body, for debugging" and that
"DumpRequestOut is like DumpRequest but includes headers that the
standard http.Transport adds, such as User-Agent":

  https://pkg.go.dev/net/http/httputil@go1.4#DumpRequest
  https://pkg.go.dev/net/http/httputil@go1.4#DumpRequestOut

It is reasonable to assume, therefore, that when PR git-lfs#366 was written
its authors simply assumed that the DumpRequest() function was sufficient,
given the state of the Go documentation at that time.

As we would prefer to log a more complete set of client HTTP request
headers, though, particular the Accept-Encoding header which the
Go "net/http" package automatically adds to outgoing requests, we
now update our traceRequest() method to use the DumpRequestOut()
function.

We then also update our Go language TestVerbose*() test functions
to confirm that the Accept-Encoding request header is now captured and
reported when verbose logging is enabled.  Note that without the change
in this PR to the use of the DumpRequestOut() function, these revised
tests would fail, since the Accept-Encoding header would not appear in
headers returned by the DumpRequest() function.
chrisd8088 added a commit to chrisd8088/git-lfs that referenced this pull request Jan 21, 2026
At present, when the GIT_CURL_VERBOSE environment variable is set to
a value such as "1" or "true", the Git LFS client outputs some of the
details of each HTTP request and response it makes to its standard
error stream.

We gather the HTTP request headers to be logged in the traceRequest()
method of the Client structure from our "lfshttp" package.  To gather
the headers, the traceRequest() method uses the DumpRequest() function
from the from the "net/http/httputil" package in the Go standard library.

Our use of the DumpRequest() function for this purpose dates back to
commit 6fafade of PR git-lfs#366 in 2015,
when it was first introduced into what was then the traceHttpRequest()
function in our "lfs" package.  This function gradually evolved into the
current traceRequest() method through a series of PRs.  First, portions
of the "lfs" package were migrated into a new "httputil" package in
PR git-lfs#1226.  That package was later removed in PR git-lfs#1846 after a new
"lfsapi" package was added in PR git-lfs#1794 with a version of the
traceRequest() method very similar to the current one.  Finally, in
PR git-lfs#3244 we moved the method into the current "lfshttp" package.

Throughout these changes, however, our use of the DumpRequest()
function has remained the same.  However, since the Git LFS client only
acts as an HTTP client and not an HTTP server, we should instead be
using the DumpRequestOut() function from the standard "httputil" package
to gather the headers of an HTTP request for logging purposes.

Replacing our use of the DumpRequest() function with the DumpRequestOut()
function would allow us to output a number of request headers that we
currently fail to log, as was noted with respect to the Accept-Encoding
header in issue git-lfs#6193.

Since Go version 1.6, the "httputil" package's documentation has stated
that the DumpRequest() function "should only be used by servers to debug
client requests", and that the DumpRequestOut() function "is like
DumpRequest but for outgoing client requests":

  https://pkg.go.dev/net/http/httputil@go1.6#DumpRequest
  https://pkg.go.dev/net/http/httputil@go1.6#DumpRequestOut

However, at the time we first adopted the use of the DumpRequest()
function, the Go documentation did not include these directives, and
merely noted that "DumpRequest returns the as-received wire representation
of req, optionally including the request body, for debugging" and that
"DumpRequestOut is like DumpRequest but includes headers that the
standard http.Transport adds, such as User-Agent":

  https://pkg.go.dev/net/http/httputil@go1.4#DumpRequest
  https://pkg.go.dev/net/http/httputil@go1.4#DumpRequestOut

It is reasonable to assume, therefore, that when PR git-lfs#366 was written
its authors simply assumed that the DumpRequest() function was sufficient,
given the state of the Go documentation at that time.

As we would prefer to log a more complete set of client HTTP request
headers, though, particularly the Accept-Encoding header which the
Go "net/http" package automatically adds to outgoing requests, we
now update our traceRequest() method to use the DumpRequestOut()
function.

We then also update some of our Go language TestVerbose*() test functions
to confirm that the Accept-Encoding request header is now captured and
reported when verbose logging is enabled.  Note that without the change
in this PR to the use of the DumpRequestOut() function, these revised
tests would fail, since the Accept-Encoding header would not appear in
headers returned by the DumpRequest() function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant