Skip to content

Fix reading config files starting with http:#9275

Merged
akrantz01 merged 11 commits intoinfluxdata:masterfrom
akrantz01:9218-config
May 26, 2021
Merged

Fix reading config files starting with http:#9275
akrantz01 merged 11 commits intoinfluxdata:masterfrom
akrantz01:9218-config

Conversation

@akrantz01
Copy link
Copy Markdown
Contributor

resolves #9218

Fixed an error where telegraf attempted to fetch files starting with http: or https: over HTTP/S rather than read them from the filesystem.

@telegraf-tiger
Copy link
Copy Markdown
Contributor

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label May 17, 2021
@akrantz01
Copy link
Copy Markdown
Contributor Author

!signed-cla

Copy link
Copy Markdown
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This means files take precedence over URLs even though you specified a valid (remote) URI. You can achieve the same with specifying file://http:whateverfileyouneed.

@srebhan srebhan self-assigned this May 18, 2021
Copy link
Copy Markdown
Contributor

@Hipska Hipska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @srebhan, currently the preference is load url and otherwise load file. Now you are first trying to load file and otherwise try to load url. I think the team needs to discuss what preference we want.

@ssoroka
Copy link
Copy Markdown
Contributor

ssoroka commented May 18, 2021

Thanks for the PR!

I don't think this change is unreasonable. Hipska has a good point about handling the error messages, so we need to pay attention to that. Probably the best way to decide if we should parse it as a file or a url is to look for a character sequence that clearly identifies it. Parsing files as urls will still give you a url, which is what caused the original problem. This PR does in fact resolve that problem, it just comes at the cost of checking the disk for matching files first.. Which honestly isn't really a big deal.

If we wanted to stick closely to the intent of the user configuring the plugin, we could match against the regex ^\w+:// and if it matches use url parsing, and if it doesn't, use file parsing. This will remove all ambiguity, and allow for clear error messages that say things like "couldn't read the file %q" or "couldn't parse the url %q".

@srebhan
Copy link
Copy Markdown
Member

srebhan commented May 18, 2021

@ssoroka well my fear is that you, with the change applied, cannot specify a URL if a file with that name exists. Let's make an example. Say a) you want to access a file named http://myfile and b) you want to access a server at http://myserver and let's assume you have (for whatever reason) the files http://myfile and http://myserver on-disk...

With the current implementation you can achieve this by specifying
for a) file://'http://myfile'
for b) http://myserver
and it fill access the file for a) and the server for b).

With the change you specify
for a) http://myfile
for b) http://myserver
while correct for a) it now will access the file http://myserver for b) instead of the server.

Sorry @akrantz01 if my comment was too harsh, but I think this kind of implicit behavior is cumbersome and might lead to unintended effects.

@srebhan
Copy link
Copy Markdown
Member

srebhan commented May 18, 2021

Hmmm and after reading the code again I see that my example is wrong. :-) So currently specifying file://'http://myfile will try to read 'file://http://myfile' instead of 'http://myfile'... So I see your point now @akrantz01...

The team probably should discuss this as @Hipska said, but my preference would be to implement the handling of the file:// scheme such as

func loadConfig(config string) ([]byte, error) {
	if strings.HasPrefix(config, "file://") {
		return ioutil.ReadFile(strings.TrimPrefix(config, "file://"))
	}

	u, err := url.Parse(config)
	if err != nil {
		return nil, err
	}

	switch u.Scheme {
	case "https", "http":
		return fetchConfig(u)
	}

	// If it isn't known scheme, try it as a file.
	return ioutil.ReadFile(config)
}

@Hipska
Copy link
Copy Markdown
Contributor

Hipska commented May 18, 2021

Isn't fetchConfig also able to handle file:// urls? So case scheme is 'file' then do something ;)

@akrantz01
Copy link
Copy Markdown
Contributor Author

To the best of my knowledge, I don't think you can create files with forward slashes in the name on any platform, so I think it'd be best to go with @ssoroka's suggestion. It wouldn't require any changes from the user and would still handle actual URLs properly.

Regarding @Hipska's point, we could still add a case for file:// URLs so users could be explicit if they wanted.

@akrantz01
Copy link
Copy Markdown
Contributor Author

Actually, if we wanted to support file:// URLs through fetchConfig, it would take a bit more work since the default HTTP client doesn't support the file scheme.

Copy link
Copy Markdown
Contributor

@Hipska Hipska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would do something like this: (pseudo code)

if u.Scheme in ["https", "http"] and u.Host not empty { 
    return fetchConfig(u)
}

@srebhan
Copy link
Copy Markdown
Member

srebhan commented May 19, 2021

We will get problems with parameters like file://https:test. Check out https://goplay.space/#SqDP6ybX7Af or
the attached code. That's why I tried to handle the file:// part first. :-)

@akrantz01
Copy link
Copy Markdown
Contributor Author

@ssoroka and I decided that it would be simplest just to use the regex before parsing the URL, since then the implementation isn't dependent on the internals of url.Parse if they ever change. It was also decided to not support file://, since it isn't used anywhere else.

config/config.go Outdated
Comment on lines +918 to +919
default:
// If it isn't a https scheme, try it as a file
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can leave this one out as it does nothing and move the comment next to the last return. Just a minor comment though.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the linter will complain then?

Copy link
Copy Markdown
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akrantz01 thanks for keep going with this persistent issue.
My only (very minor) comment is to remove the default clause, but this is not a blocker.

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label May 20, 2021
@influxdata influxdata deleted a comment from srebhan May 20, 2021
Copy link
Copy Markdown
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect thanks!

Copy link
Copy Markdown
Contributor

@ssoroka ssoroka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. left a suggestion on erroring if the requested scheme doesn't exist.

@akrantz01 akrantz01 merged commit 58479fd into influxdata:master May 26, 2021
arstercz pushed a commit to arstercz/telegraf that referenced this pull request Aug 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix pr to fix corresponding bug ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegraf --config does not read files if files are named http:....

4 participants