The main menu itself? Can't say that feels unresponsive to me.

Sorry, I mean the network menu.
It highly depends on the network state I believe.
The evil function is,
https://github.com/koreader/koreader/blob/master/frontend/ui/network/manager.lua#L62
It calls every time network menu pops up.

I guess I have a fast network (I'd better with my Ubiquiti equipment, even if it's the lowest grade), but that makes more sense. :-P

Though this test is not exactly the same as what we are doing in the isOnline() function, by using mobile network, pinging www.example.com may need 500 milliseconds. I believe the performance would be even worse when mobile signal is weak.

I'm not fond of having a ping being launched every 30 seconds in the background (if I undestand correctly https://github.com/koreader/koreader/pull/3078/files#diff-8a665adb55900a011d1c3928d7447e61R24), and this even when we know wifi is off.
At least, this (the repeated background job, not only the ping) could happen only if wifi has been enabled ?

Btw, just seeing now that my idle emulator is logging every 2/3 seconds:

08/17/17-20:04:39 DEBUG BackgroundRunner: _execute() @  1502993079
08/17/17-20:04:41 WARN  got error waiting for events: ./ffi/SDL2_0.lua:110: Waiting for input failed: timeout

08/17/17-20:04:41 DEBUG BackgroundRunner: _execute() @  1502993081
08/17/17-20:04:44 WARN  got error waiting for events: ./ffi/SDL2_0.lua:110: Waiting for input failed: timeout

08/17/17-20:04:44 DEBUG BackgroundRunner: _execute() @  1502993084
08/17/17-20:04:46 WARN  got error waiting for events: ./ffi/SDL2_0.lua:110: Waiting for input failed: timeout

08/17/17-20:04:46 DEBUG BackgroundRunner: _execute() @  1502993086
08/17/17-20:04:49 WARN  got error waiting for events: ./ffi/SDL2_0.lua:110: Waiting for input failed: timeout

Is that expected that your BackgroundRunner is scheduling itself when there is no job (like it seems to me) scheduled ?

This is probably not the right place to stick this thought, but BackgroundRunner is probably the solution to #2540, isn't is?

RE poire-z: yes, it runs once per two seconds. If there is no job, it quits. You may remove the log if it's annoying. Otherwise, it has no impact. UIManager itself polls much more frequently to receive user input.
I am also struggling how this should be done. Polling only when wifi is on is also not a good solution: on Kindle and Android, we do not control the on / off of the wifi. But we do not have other reliable way to do so. Maybe when the main menu is popped up? Typically this action should finish in milliseconds, so there is enough time to poll the network state before network menu is selected.
I am verifying the battery usage impact of this change. If the usage impact is way to small, I think it does not really matter. We send only 64 bytes per 30 seconds anyway.
I will add a hidden configuration to test this feature before pushing it out.

RE Frenzie: I think network connected event is broken for a while, which may impact the calibre connection. I have not gone deep to the code yet.

on Kindle and Android, we do not control the on / off of the wifi. But we do not have other reliable way to do so.

I see. Nothing among the stuff done in https://github.com/koreader/koreader/blob/master/frontend/device/generic/device.lua#L177 (ifconfig, iwconfig) gives different results whether wifi is off or on or connected to a ssid ? Or something like lsmod | grep -q sdio_wifi_pwr if wifi kernel modules were loaded/unloaded on wifi on/off ?
edit: or the existence or not of stuff in /proc/sys like /proc/sys/net/ipv4/conf/eth0 on kobo ?

UIManager itself polls much more frequently to receive user input

That may be true on the emulator (with SDL, which has no wait with timeout function, and has to poll every 10ms), but it is not the case on kobo/kindle, where:
https://github.com/koreader/koreader/blob/master/frontend/ui/uimanager.lua#L706
makes the process stop and wait on a select syscall:
https://github.com/koreader/koreader-base/blob/master/input/input.c#L240
like any well behaving process should do if it has no other thing to do than wait for some event or some futur date to happen.

I admit that getting out of this every 2 second isnt going to cost much, but it's a bit sad to do that when there is no work to do.
I see that you need it because background runner is a plugin, and can't be notified of new jobs, so it has to check the PluginShare.backgroundJobs table for new stuff every 2 seconds.
But if it's going to be used more and more by core stuff (like in this PR), it may deserve to be put in core, with a proper API to addJob/addJobIfNotThereYet/removeJob() jobs, so that you can, in their code, appropriately scheduleIn/unschedule to the right moment ?
(I get you put a lot of work into that being a plugin, so sorry for suggesting more work or refactoring :)

A simpler alternative could be to just not launch BackgroundRunner:_schedule() on module loading (https://github.com/koreader/koreader/blob/master/plugins/backgroundrunner.koplugin/main.lua#L227), but to have it react to some kind of "ensureBackgroungRunnerRunnig" event, that clients could send after they put some job into PluginShare.backgroundJobs ? And have it stop doing self:_schedule() when there is no more jobs in there?

Using ifconfig can get the wifi state, but it won't solve the problem: we need to poll the state of wifi instead of internet accessibility.

I totally agree that moving background runner to the core would be a good choice, we can have a better control to the job queue without needing PluginShare.

I totally agree that moving background runner to the core would be a good choice, we can have a better control to the job queue without needing PluginShare.

Sounds good!

Maybe when the main menu is popped up? Typically this action should finish in milliseconds, so there is enough time to poll the network state before network menu is selected.

An other possibility for that, not involving BackgroundRunner, would be to extend TouchMenuItem to accept new attributes: delayed_text_func or delayed_checked_func (delayed, or late, or future...), like:

--- a/frontend/ui/widget/touchmenu.lua
+++ b/frontend/ui/widget/touchmenu.lua
@@ function TouchMenuItem:init()
     self.item_frame = FrameContainer:new{
         width = self.dimen.w,
         bordersize = 0,
         color = Blitbuffer.COLOR_BLACK,
         HorizontalGroup:new {
             align = "center",
             CenterContainer:new{
                 dimen = Geom:new{ w = checked_widget:getSize().w },
                 item_checkable and (
                     item_checked and checked_widget
                     or unchecked_widget
                 )
                 or empty_widget
             },
             TextWidget:new{
                 text = getMenuText(self.item),
                 fgcolor = item_enabled ~= false and Blitbuffer.COLOR_BLACK or Blitbuffer.COLOR_GREY,
                 face = self.face,
             },
         },
     }
+    if self.item.delayed_text_func then
+        UIManager:nextTick(function()
+            local new_text = self.item.delayed_text_func()
+            self.item_frame[1][2]:setText(new_text) -- self.item_frame's TextWidget
+            UIManager:setDirty(self.show_parent, function()
+                return "ui", self.item_frame[1][2].dimen
+            end)
+        end)
+    end
     self[1] = self.item_frame
 end

--- a/frontend/ui/network/manager.lua
+++ b/frontend/ui/network/manager.lua
@@ -77,9 +77,10 @@ end

 function NetworkMgr:getWifiMenuTable()
     return {
-        text = _("Wi-Fi connection"),
+        text = _("Wi-Fi connection..."),
         enabled_func = function() return Device:isKindle() or Device:isKobo() end,
-        checked_func = function() return NetworkMgr:isOnline() end,
+        -- checked_func = function() return NetworkMgr:isOnline() end,
+        delayed_text_func = function() require("ffi/util").sleep(2) ; return NetworkMgr:isOnline() and "Wi-Fi connection online" or "Wi-Fi connection not online" end,
         callback = function(menu)
             local wifi_status = NetworkMgr:isOnline()
             local complete_callback = function()

So it would launch (but still block) the ping after menu is drawn, and you would see text being replaced by result of delayed_text_func() (the same could probably to update the checkmark)

edit: and use a quick way (lsmod | grep -q sdio_wifi_pwr on kobo, your ifconfig on kindle) for check_func and whereever isOnline() is used - and only the delayed_text_func that does a ping to update the text of the 1st menu item, for a quick way to see if we are online.

@poire-z I think that won't help to totally remove the blockage: when the delay_text_func is invoking in UIManager loop, any users input will be delayed.

delay_text_func itself is a good idea, we can use it to update the text on menus for delay initialization.

We are adding more features to KOReader, most of them requires IO operations. So moving all IO blocking operations out of UIManager loop to avoid unresponsiveness can significant improve the user experience. Since the delay of these operations are not predictable -- even writing DocSettings may block UI for seconds if a slow SD card is used or free storage is low.
Currently the IO blocking operations happen everywhere, e.g.

• KOSync
• DocSettings
• NetworkMgr:isOnline()
• Update
  The first three make the UI unresponsive if the IO operation takes longer than we expected. The forth one is really a bad UX design, why cannot user continually read the book while we are updating the KOReader?

I don't use KOSync nor OTA update, so can't say much about them - but I guess for an OTA update, I'd like to notice it's happening, get a grasp of the usual time it takes, so i can worry when it takes longer.
For Docsettings saving (even if I don't think i have bad SD now, so i don't notice any slowness), I'd like to somehow notice it when it get worse, so this blocking may be a hint.
But I kinda like the way it is, with lua being monothread, everything is serialized and deterministic/predicatable/undestandable.
Also, it's not like we're playing an action game, we're mostly watching the screen idling :) And I haven't seen issues about unresponsiveness (my only complaint would be about stardict lookup with my 10 dictionaries taking 5 to 15 seconds when fuzzy search is involved.)
Just a hint that I'd like to keep things simple :)

The update thing is only about downloading. Considering you can go on reading for as long as you want before actually exiting it doesn't really make sense to block the UI while downloading. If it does then I reckon there should be a cancel button.

I think most of the users are regular users, they won't care about threading model. But unresponsive UI is something unacceptable: for everything takes an unacceptable longer time, the only thing you can do is to force restart the device and eventually lost everything in DocSettings.
I agree to @Frenzie, and I think this principle applies to most of the IO, especially network tasks. For example, the stardict lookup, even without fuzzy search switch, users should still be able to cancel the lookup if it takes longer then they want. Background runner provides a callback when the task finishes. We can also add a cancel function to it.

@Hzj-jie Btw, we've had some discussion on the network in #3273 (comment) rather than #3160 where it should've been.

Hi Guys, not sure if that's helpful but FYI in android we already have a function to check whether wifi is on or off
https://github.com/koreader/koreader/pull/3396/files#diff-c14ee0e93f7187eeb6f6fe8af52cf0b7R63
(it doesn't use ping)

Btw, BackgroundRunner is about external binaries but I'm not sure why I didn't mention that various libraries exist for making threaded stuff easy such as

https://github.com/torch/threads
https://github.com/LuaLanes/lanes