Add an option to set a custom user agent to retrieve individual feeds by Jules-Bertholet · Pull Request #3494 · FreshRSS/FreshRSS

Jules-Bertholet · 2021-03-03T06:05:23Z

This allows retrieving the original RSS feed when websites use services like FeedBurner.

Closes #3493

Changes proposed in this pull request:

Add a setting to choose the user agent used when fetching feeds

How to test the feature manually:

Add a feed that is fronted by FeedBurner to FreshRSS. One such is https://www.themoneyillusion.com/feed/atom
Set the user agent to FeedBurner
Ensure the feed location still points to https://www.themoneyillusion.com/feed/atom, and was not changed to the FeedBurner feed because of a 3XX redirect before you added the user-agent
Re-fetch feed content
All feed items should now refer to the original site, not feedproxy.google.com or similar

Pull request checklist:

clear commit messages
code manually tested
unit tests written (optional if too hard)
documentation updated

Additional information can be found in the documentation.

Alkarex · 2021-03-03T09:19:41Z

app/i18n/en/sub.php

 		'ttl' => 'Do not automatically refresh more often than',
 		'url' => 'Feed URL',
+		'useragent' => 'Set the user agent for fetching this feed',
+		'useragent_help' => 'Example: <kbd>FeedBurner</kbd>',


Please use another example, avoiding specific product names, but a more general browser, e.g.

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0)

Alkarex · 2021-03-03T09:22:21Z

Please add a line for your new string in ./cli/i18n/ignore/en-us.php to ignore the fact that it is the same string in en-GB and en-US

aledeg

Could you add the translating strings from en_US in the ignore file?
Run this command to do that: make i18n-ignore-unmodified-keys lang=en-us

Edit This will fix the CI error!

aledeg · 2021-03-03T12:15:22Z

app/Controllers/subscriptionController.php

 			if ($cookie != '') {
 				$opts[CURLOPT_COOKIE] = $cookie;
 			}
+			if ($useragent != '') {


Please, use !== instead of !=.

I know that the code around does not use it, but I think it should. We will fix it when working on it.

This allows retrieving the original RSS feed when websites use services like FeedBurner.

Alkarex · 2021-03-03T22:19:28Z

Works fine. But maybe this option should be made available from the Add feed page (?c=subscription&a=add) as well? Otherwise it might not be possible to add a feed, which requires a specific user agent. Around there https://github.com/FreshRSS/FreshRSS/blob/master/app/views/subscription/add.phtml#L62 and there https://github.com/FreshRSS/FreshRSS/blob/master/app/Controllers/subscriptionController.php#L130

Furthermore, we should either only allow those options for admins, or make them use a user-specific cache, as this might otherwise randomly affect other users of the same FreshRSS instance. See #3367 (comment) This should be in another PR though, but must be done before the release of 1.18.0.

#fix FreshRSS#3367 (comment) #fix FreshRSS#3494 (comment)

Alkarex · 2021-03-07T22:27:52Z

Patch to avoid SimplePie cache pollution when changing cURL options: #3502

* SimplePie prevent cache polution #fix #3367 (comment) #fix #3494 (comment) * Fix bug * Minor improvement * Update cache filename in FreshRSS (1/2) * cacheFilename temp * New SimplePie get_cache_filename() * Fix typos * Update lib/SimplePie/SimplePie.php Typo * Include user-agent and timeout * fix array_merge * Declaration * force_feed was lost in a commit

Alkarex · 2021-03-09T19:14:54Z

@Jules-Bertholet I have added the user-agent (and proxy) option to the subscription page to add a new feed:
f9495f5

Could you please give it a quick test?

Jules-Bertholet · 2021-03-09T19:37:57Z

@Alkarex Works great! (Proxy not tested)

Alkarex · 2021-03-09T19:51:33Z

Thanks for the quick test :-)

Frenzie · 2021-03-09T19:52:50Z

Looks nice :-)

…

On Tue, Mar 9, 2021 at 8:51 PM Alexandre Alapetite ***@***.***> wrote: Merged #3494 <#3494> into master. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3494 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABRQBNYUHXY4HMUUIVGHGTTCZ34DANCNFSM4YQSW22Q> .

Alkarex · 2021-03-09T20:06:11Z

@Jules-Bertholet Could you please add a line for you in https://github.com/FreshRSS/FreshRSS/blob/master/CREDITS.md ?

Alkarex reviewed Mar 3, 2021

View reviewed changes

Alkarex added this to the 1.18.0 milestone Mar 3, 2021

aledeg requested changes Mar 3, 2021

View reviewed changes

Georgelemental added 2 commits March 3, 2021 10:32

Add an option to set a custom user agent to retrieve individual feeds

1320a4c

This allows retrieving the original RSS feed when websites use services like FeedBurner.

Use !== instead of != in subscriptionController.php

b18ea78

aledeg approved these changes Mar 3, 2021

View reviewed changes

Alkarex added a commit to Alkarex/FreshRSS that referenced this pull request Mar 6, 2021

SimplePie prevent cache polution

bd9bc05

#fix FreshRSS#3367 (comment) #fix FreshRSS#3494 (comment)

Alkarex mentioned this pull request Mar 6, 2021

SimplePie prevent cache pollution #3502

Merged

Merge branch 'master' into 3493-custom-ua

552929d

Alkarex closed this in #3502 Mar 9, 2021

Alkarex reopened this Mar 9, 2021

Alkarex added 2 commits March 9, 2021 11:00

Merge branch 'master' into 3493-custom-ua

7ae48ad

Add proxy and user-agent to subscription/add

f9495f5

Alkarex merged commit 800a421 into FreshRSS:master Mar 9, 2021

Alkarex mentioned this pull request Mar 9, 2021

Release candidate FreshRSS 1.18.0 #3512

Closed

Alkarex mentioned this pull request Mar 20, 2021

Fix undefined variable & dead code when adding feed #3546

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add an option to set a custom user agent to retrieve individual feeds#3494

Add an option to set a custom user agent to retrieve individual feeds#3494
Alkarex merged 5 commits intoFreshRSS:masterfrom
Jules-Bertholet:3493-custom-ua

Jules-Bertholet commented Mar 3, 2021 •

edited

Loading

Uh oh!

Alkarex Mar 3, 2021

Uh oh!

Alkarex commented Mar 3, 2021

Uh oh!

aledeg left a comment •

edited

Loading

Uh oh!

aledeg Mar 3, 2021

Uh oh!

aledeg Mar 3, 2021

Uh oh!

Alkarex commented Mar 3, 2021 •

edited

Loading

Uh oh!

Alkarex commented Mar 7, 2021

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Jules-Bertholet commented Mar 9, 2021

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Frenzie commented Mar 9, 2021 via email

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Jules-Bertholet commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alkarex Mar 3, 2021

Choose a reason for hiding this comment

Uh oh!

Alkarex commented Mar 3, 2021

Uh oh!

aledeg left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aledeg Mar 3, 2021

Choose a reason for hiding this comment

Uh oh!

aledeg Mar 3, 2021

Choose a reason for hiding this comment

Uh oh!

Alkarex commented Mar 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alkarex commented Mar 7, 2021

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Jules-Bertholet commented Mar 9, 2021

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Frenzie commented Mar 9, 2021 via email

Uh oh!

Alkarex commented Mar 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jules-Bertholet commented Mar 3, 2021 •

edited

Loading

aledeg left a comment •

edited

Loading

Alkarex commented Mar 3, 2021 •

edited

Loading