[4.0] poor man’s cron by alikon · Pull Request #25022 · joomla/joomla-cms

alikon · 2019-05-27T17:03:53Z

porting #22310 to 4.x

Goal

a way to run scheduled "quick" tasks in approximately regular intervals

Summary of Changes

a new system plugin scheduler
two new console command

How it works

In a normal hosting situation that would be job for cron, but if your hosting doesn’t offer cron, then a common solution in these cases is “poor man’s cron:” on every page load (onAfterRespond) the scheduler system plugin check if itself is expected to run and then trigger a new event onExecuteScheduledTask then any job plugin wich listen to that event check if itself is expected to run and if so execute his task.

Testing Instructions (web)

apply PR
System panel -> discover -> you'll find a new system plugin scheduler
install it
set options, publish the plugin and save
for test only
Frequency (in minutes) set = 0 mean run every (onAfterRespond)

install these 2 silly job plugin:
plg_job_one.zip
plg_job_two.zip
job one simply sleep for 1 seconds
job two simply sleep for 3 seconds
set options, publish the plugin and save plugin ONE
Frequency (in unit) set 1
Unit of time set minute

means that this plugin run 1xminute = every minute
if onExecuteScheduledTask has been triggered

set options, publish the plugin and save plugin TWO
maybe use different settings than ONE

Expected result

depending on your (plugins) settings
take a coffe/smoke/whatever 😄
randomly click a couple of frontend pages
check the log file at yoursite/administrator/logs/joomla_web.php
you'll see something like this

#
#<?php die('Forbidden.'); ?>
#Date: 2019-05-30 17:24:39 UTC
#Software: Joomla! 4.0.0-alpha10-dev Development [ Amani ] 16-May-2019 19:22 GMT

#Fields: datetime	priority clientip	category	message
2019-05-30T17:24:39+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2019-05-30T17:24:40+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#221 Processing Time: 1.01 seconds.
2019-05-30T17:24:43+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#166 Processing Time: 3.01 seconds.
2019-05-30T17:24:43+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1564 Total Processing Time: 4.04 seconds.
2019-05-30T17:24:47+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2019-05-30T17:24:47+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1565 Total Processing Time: 0.04 seconds.

1st run
the scheduler plugin (wich runs always frequency=0) trigger the onExecuteScheduledTask event
so plugin ONE and plugin TWO are executed (cause no previous run)

2nd run
the scheduler plugin (wich runs always frequency=0) trigger the onExecuteScheduledTask event
but this time both plugin have an expected scheduled time (for example ONE each 1min TWO each 5min)
the 2nd run was at 17:24:47+00:00
the 1st run was at 17:24:43+00:00
they have only 4 seconds of difference form the previous execution so both plugin are not expected to run

3rd run
it depends on how fast you click & settings

Testing Instructions (webcron)

on scheduler system plugin
set Webcron to yes/enable
set Key to mywebcronactivationkey
then use yoursiteurl/index.php?webcronkey=mywebcronactivationkey as the url for webcron services to be triggred

Testing Instructions (cli/cron)

Go to the cli folder of your website and run
php joomla.php job:run

check the log file at yoursite/administrator/logs/joomla_cli.php

Actual result

N/A

Additional comments

it can be used for tasks like:

deleteExpiredConsents
remindExpiringConsents
logRotation
updateNotifications
sessionGC
etc...

Documentation Changes Required

yes

brianteeman · 2019-05-27T17:32:28Z

Thanks very much for this PR

libraries/src/Console/SchedulerCommand.php

libraries/src/Plugin/PluginHelper.php

libraries/src/Console/SchedulerCommand.php

libraries/src/Plugin/PluginHelper.php

plugins/system/scheduler/scheduler.php

libraries/src/Console/SchedulerCommand.php

libraries/src/Plugin/PluginHelper.php

plugins/system/scheduler/scheduler.xml

richard67 · 2020-02-02T12:47:53Z

@alikon Made alikon#64 to save you some work. Contains all changes in my above reviews, too.

richard67 · 2020-02-02T12:52:10Z

The PHP notice on CLI mentioned in one of my comments above seems really to be an error in the IpHelper. It doesn't check if the array key 'REMOTE_ADDR' exists before checking it in line 488. So this seems not to be a mistake in this PR here.

SharkyKZ · 2020-02-02T12:57:31Z

IMO, changes do not belong in Joomla\CMS\Plugin\PluginHelper.

richard67 · 2020-02-02T13:00:20Z

@SharkyKZ Which ones you mean? The "1918" to "1970" date correction I've suggested above? If you mean that, let me know where else to change them. Or do you mean the PHP notice in CLI? This should already be solved in the utilities package with your PR joomla-framework/utilities#23, I have to check if it has been merged into the 2.0-dev branch there. If so, we maybe just have to update the utilities package in the cms.

Update: The "1918" date correction I've suggested above changes something which will be added to the plugin helper class by this PR here. So if it is wrong to add something there then this PR does it wrong (what I don't believe right now).

Update 2: I just saw the changes in joomla-framework/utilities#23 haven't been merged up yet to the 2.0-dev branch of the utilities package.

SharkyKZ · 2020-02-02T13:08:08Z

I mean any code added to PluginHelper class. At least currently the code is called only by Scheduler plugin so it's more suitable for the code to be contained in the plugin. But I'm not sure what the intention here is.

SharkyKZ · 2020-02-02T13:09:47Z

I'm also not sure how correct is the use of checked_out_time property here.

richard67 · 2020-02-02T13:11:44Z

@SharkyKZ Well I can't say if it's right to add code there or not, but that was then done by this PR and not by my suggested changes. Regarding the "abuse" of the checked out time column: Yes, I agree, it looks a bit like a hack to me. But I could live with that.

[4.0] Fixes for poor man’s cron

richard67 · 2020-02-02T20:43:09Z

Corrected path to logs folder in description.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/25022.}

richard67 · 2020-02-02T21:27:26Z

@alikon So, now I've tested again, and it works for me as described (web, webcron and cli).

What remains to be clarified is the architectural question raised by @SharkyKZ if the PluginHelper class is the right place to add the new stuff, and his and my concern if the usage of the checked_out_time column is suitable for that. For me it works and I'm ok with it, but maybe it can be improved.

I'll wait for some info before I mark my test result with success. Maybe there will be some changes. Just ping me when all is ok or when I shall test again. Now that I know how it works I can do it more quickly.

alikon · 2020-02-02T21:36:04Z

i'm open to hear all the feedback, and i'm really happy that we have some feedback now

for the checked_out_time column usage

yes it seems like an hack, but my goal was that, i didn't want to add another column,
i've written this for j3 and then ported on j4 cause it was too late for j3

for the PluginHelper code

i'm open to listen a better way to do it

richard67 · 2020-02-04T18:38:36Z

@alikon The clean solution would be to create a new class for the new function, if necessary a child class of the plugin helper (@SharkyKZ what do you think?) and to create a new table for the jobs.

@wilsonge Any suggestions?

alikon · 2020-02-13T07:45:50Z

let's give this a nth attempt

astridx · 2020-02-14T14:36:19Z

I have tested this:

git fetch origin pull/25022/head:patch-81
git checkout patch-81
New installation because of database changes
npm ci is not needed but I did it
System panel -> discover -> I found no new system plugin scheduler, but I realize, that it is installed :)

Testing web

I set the options
I installed the two jobs via System extension Upload package and enabled them.

I set the options (In job2 are not language strings but this is not part of this pr)
i wait a short time and see this output in the log file.

#
#<?php die('Forbidden.'); ?>
#Date: 2020-02-14 13:45:14 UTC
#Software: Joomla! 4.0.0-beta1-dev Development [ Mañana ] 17-October-2019 20:21 GMT

#Fields: datetime	priority clientip	category	message
2020-02-14T13:45:14+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:45:14+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1 Total Processing Time: 0.01 seconds.
2020-02-14T13:45:48+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:45:48+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#1 Total Processing Time: 0.01 seconds.
2020-02-14T13:47:43+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:47:43+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#2 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:18+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:18+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#3 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#4 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#5 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#6 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#7 Total Processing Time: 0.03 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#8 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#9 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:24+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:24+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#10 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#11 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#12 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#13 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#14 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#15 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#16 Total Processing Time: 0.01 seconds.
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#17 Total Processing Time: 0.02 seconds.
2020-02-14T13:48:34+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:48:34+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#18 Total Processing Time: 0.01 seconds.
2020-02-14T13:49:02+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:49:02+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#19 Total Processing Time: 0.01 seconds.
2020-02-14T13:49:10+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:49:10+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#20 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:05+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:05+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#21 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#22 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#23 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#24 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#25 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#26 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#27 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:08+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:08+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#28 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:09+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:09+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#29 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:22+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:22+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#30 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:27+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:27+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#31 Total Processing Time: 0.01 seconds.
2020-02-14T13:50:38+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:39+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#1 Processing Time: 1.11 seconds.
2020-02-14T13:50:42+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#1 Processing Time: 3.01 seconds.
2020-02-14T13:50:42+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#32 Total Processing Time: 4.14 seconds.
2020-02-14T13:50:45+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:45+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#33 Total Processing Time: 0.03 seconds.
2020-02-14T13:50:50+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:51+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#34 Total Processing Time: 0.03 seconds.
2020-02-14T13:50:53+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:50:53+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#35 Total Processing Time: 0.02 seconds.
2020-02-14T13:51:36+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:51:36+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#36 Total Processing Time: 0.02 seconds.
2020-02-14T13:51:44+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:51:45+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#2 Processing Time: 1.01 seconds.
2020-02-14T13:51:45+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#37 Total Processing Time: 1.04 seconds.
2020-02-14T13:51:57+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:51:57+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#38 Total Processing Time: 0.02 seconds.
2020-02-14T13:52:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:52:19+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#39 Total Processing Time: 0.03 seconds.
2020-02-14T13:53:02+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T13:53:03+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#3 Processing Time: 1.02 seconds.
2020-02-14T13:53:06+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#2 Processing Time: 3.02 seconds.
2020-02-14T13:53:06+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#40 Total Processing Time: 4.05 seconds.

Testing webcron

I deledted all log files
on scheduler system plugin I set Webcron to yes/enable and set Key to mywebcronactivationkey
In my browser i wrote the address: http://localhost/joomla-cms4/index.php?webcronkey=webcronactivationkey
and I opened it twice.
Log file output:

#
#<?php die('Forbidden.'); ?>
#Date: 2020-02-14 14:20:43 UTC
#Software: Joomla! 4.0.0-beta1-dev Development [ Mañana ] 17-October-2019 20:21 GMT

#Fields: datetime	priority clientip	category	message
2020-02-14T14:20:43+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T14:20:44+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#9 Processing Time: 1.16 seconds.
2020-02-14T14:20:47+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#7 Processing Time: 3.01 seconds.
2020-02-14T14:20:47+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#47 Total Processing Time: 4.46 seconds.
2020-02-14T14:32:28+00:00	INFO 127.0.0.1	scheduler	Scheduler>start
2020-02-14T14:32:29+00:00	INFO 127.0.0.1	scheduler	Job>jobone Task#10 Processing Time: 1.02 seconds.
2020-02-14T14:32:32+00:00	INFO 127.0.0.1	scheduler	Job>jobtwo Task#8 Processing Time: 3.20 seconds.
2020-02-14T14:32:32+00:00	INFO 127.0.0.1	scheduler	Scheduler>stop>task#48 Total Processing Time: 4.23 seconds.

Testing cli/cron

I went to the cli folder of my website and run
php joomla.php job:run
I saw a notice, but the jobs are run.

The output of the log file:

#
#<?php die('Forbidden.'); ?>
#Date: 2020-02-14 13:58:05 UTC
#Software: Joomla! 4.0.0-beta1-dev Development [ Mañana ] 17-October-2019 20:21 GMT

#Fields: datetime	priority clientip	category	message
2020-02-14T13:58:05+00:00	INFO -	scheduler	Starting Scheduler
2020-02-14T13:58:06+00:00	INFO -	scheduler	Job>jobone Task#4 Processing Time: 1.20 seconds.
2020-02-14T13:58:09+00:00	INFO -	scheduler	Job>jobtwo Task#3 Processing Time: 3.01 seconds.
2020-02-14T13:58:09+00:00	INFO -	scheduler	Scheduler tooks:4.218

astridx · 2020-02-14T14:37:27Z

I have tested this item ✅ successfully on 4d0589f

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/25022.}

alikon · 2020-02-14T18:25:17Z

thanks for testing
the PHP notice about the REMOTE_ADDR it has been already solved here :
joomla-framework/utilities#23
but that framework version of the utilities package it is still not in J4

richard67 · 2020-02-14T18:32:23Z

@alikon Meanwhile joomla-framework/utilities#23 is available in the 2.0-dev branch of the framework package, too, so yes, the package can be fetched to J4. 2 weeks ago that wasn't the case yet, there it was only in the master branch there.

wilsonge · 2020-02-16T15:27:29Z

Merged in the utilities update to 4.0-dev. I haven't taken any time to look through this yet - to be honest there's nothing in here b/c breaking so I'm not going to prioritise this either over the other beta blocking issues - if it has to go to 4.1 - it is what it is. But I will try and do an initial review as well over the next few days and give some opinions.

twister65 · 2020-02-22T20:32:32Z

Export database job plugin:
plg_job_exportdb.zip

This job plugin aims to export the entire database (zip archive) to the defined joomla tmp directory / sub-directory at a defined frequency.

twister65 · 2020-02-22T20:32:48Z

I have tested this item ✅ successfully on 4d0589f

with the export database job plugin.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/25022.}

alikon · 2020-06-13T08:31:29Z

redone in #29592

alikon added 8 commits May 27, 2019 18:48

poor cron

0a6708e

poor cron

eba7061

poor cron

af3d9ac

poor cron

3723bb5

poor cron

73833ca

poor cron

9d8a411

poor cron

5b8af0f

poor cron

1037d7c

joomla-cms-bot added Language Change This is for Translators PR-4.0-dev labels May 27, 2019

alikon added 2 commits May 27, 2019 20:08

cs

4a156f8

cs

f3998ea