When this application was released there was a hardcoded behaviour:
- Process every feed known.
- Sleep 15 minutes.
- Restart
Later it became possible to override the 15-minute delay, to make it longer/shorter. Now that we have caching I don't feel too bad about polling every 30 minutes, but it has to be said that we're probably not being well-behaved.
The specific problem I have is that we cannot say:
- Poll example.com, and example.net every 15 minutes.
- BUT poll example.org every 50 minutes.
I have the feeling this is something that should be supported. We could make some feed skip if we knew our frequency and expected number of times to poll. Or we could count sequential runs so we pretend we have 24 runs and a feed could say:
Or
But this kinda relies on some assumptions and it starts to feel complex. I don't want to start storing "arbitrary state" for feeds, but I do want to too. Help?
I know we have the cache, and that could be used as a proxy. If we assume we're invoked and running "frequently" we could add a timestamp to the URL-cache and then say:
That way if the cached entry was >2hrs we'd poll the feed again (note here I'm using the age of the cache entry as a "last time this feed was fetched", I'm not talking about a full-request vs a cached request being sent. I'm using it as implicit state that goes away upon restart - on the assumption that we're running under deamon mode.)
That might work, but I need to think about it.
TLDR: The default is fine, but I think we should be able to change the frequency on a per-feed basis and a global "sleep then repeat" setting is not the right approach. Some feeds track conversation (reddit, blog-comments, hackernews) and need to be "fast". Other feeds describe youtube videos and only update once every two days at the most frequenty.
When this application was released there was a hardcoded behaviour:
Later it became possible to override the 15-minute delay, to make it longer/shorter. Now that we have caching I don't feel too bad about polling every 30 minutes, but it has to be said that we're probably not being well-behaved.
The specific problem I have is that we cannot say:
I have the feeling this is something that should be supported. We could make some feed skip if we knew our frequency and expected number of times to poll. Or we could count sequential runs so we pretend we have 24 runs and a feed could say:
Or
But this kinda relies on some assumptions and it starts to feel complex. I don't want to start storing "arbitrary state" for feeds, but I do want to too. Help?
I know we have the cache, and that could be used as a proxy. If we assume we're invoked and running "frequently" we could add a timestamp to the URL-cache and then say:
That way if the cached entry was >2hrs we'd poll the feed again (note here I'm using the age of the cache entry as a "last time this feed was fetched", I'm not talking about a full-request vs a cached request being sent. I'm using it as implicit state that goes away upon restart - on the assumption that we're running under deamon mode.)
That might work, but I need to think about it.
TLDR: The default is fine, but I think we should be able to change the frequency on a per-feed basis and a global "sleep then repeat" setting is not the right approach. Some feeds track conversation (reddit, blog-comments, hackernews) and need to be "fast". Other feeds describe youtube videos and only update once every two days at the most frequenty.