Codecov Report

Merging #2962 (74db8f4) into develop (4d47c08) will increase coverage by 0.14%.
The diff coverage is 71.18%.

@@             Coverage Diff             @@
##           develop    #2962      +/-   ##
===========================================
+ Coverage    24.00%   24.15%   +0.14%     
===========================================
  Files           49       49              
  Lines        10121    10131      +10     
===========================================
+ Hits          2430     2447      +17     
+ Misses        7691     7684       -7     

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Hmm, works on my machine with node 16.17.1 and 14.20.1 which these checks run...

Yeah, the tests are still somehow fragile.... Sometimes re-running helps, sometimes not...

And now 2 other fails instead, great...

global.page seems to be null sometimes here: https://github.com/MichMich/MagicMirror/blob/4d47c0837ff5bb853664fac599dd182ef1628c37/tests/electron/helpers/global-setup.js#L39

I could see that happen if this if-statement fails: https://github.com/MichMich/MagicMirror/blob/4d47c0837ff5bb853664fac599dd182ef1628c37/tests/electron/helpers/global-setup.js#L15

According to https://playwright.dev/docs/api/class-electronapplication#electron-application-windows windows() doesn't return a promise, so awaiting for it doesn't do anything. I wonder if this if-statement is even needed?

yeah, tests are really fun, you make changes and they all run locally and then they began to fail on github, but not always ...

I wonder if this if-statement is even needed?

try it without, the next statement is trying to get first window so we will see whats happens ...

Trying your branch out, I also got (only once!) some errors in the electron tests where the date is used (like compliments or weather sunset)... But now two runs without errors... SOme weird race conditions?

Most likely since the whole startup was blocking the thread before and NodeJS is single threaded. So the server startup probably blocked electron from receiving any events before that was done.

According to https://playwright.dev/docs/api/class-electronapplication#electron-application-windows windows() doesn't return a promise, so awaiting for it doesn't do anything. I wonder if this if-statement is even needed?

There are two more "await window()" calls in the tests, one in electron/env_spec the other in electron/helpers/global-setup could you get rid of them too?

Think I understand what it was trying to check, when I removed this if-statement it failed once because the test that opens the dev console had the dev console as the first window and not the magicmirror. So probably would need to wait for each window until the correct one appears.

Development console tests › should open browserwindow and dev console

    expect(received).toBe(expected) // Object.is equality

    Expected: "MagicMirror²"
    Received: "DevTools"

That doesn't explain the error on github actions. will have to look into it more tomorrow.

Finally...

Please retry the checks a few times in case this was a fluke :)

(I moved the systemDate-mocking from startApplication() since I've been annoyed by weather_spec.js overriding my config every time it runs and want to try to mock the fetch-request instead later)

Bad news, the tests/electron/modules/calendar_spec.js tests dont fail if change the date (the weather and compliments seem to be fine and fail if I change the dates) :-( Could you look into it?

It's the same in develop-branch, so it's nothing that I broke :)

But I can try to see if I can figure out what's wrong.

The tests will now wait until timeout if the css-classes are missing, not optimal...

Thx for trying to fix it.

Unfortunately the tests in general are still unstable.... :-(

Been running the electron-tests in a loop for over an hour now without any of them failing locally... This is a bit annoying

I spent hours with the tests, running locally means nothing ... every time I thought now they are o.k. I did a run on Github and they failed there ... The timing seems totally different, the stuff on Github is slower and sometimes tests there fails because of the 20 sec. timeout which never happens locally. You can use your develop branch in your fork and push it to your fork on Github to run the tests (codecov will fail but the rest runs) so you can check Github without using this repo (and showing everyone the fails). From the actions tab you can rerun tests without pushing again, thats whats I'm doing after changing tests ...

Well, I'm totally out of ideas so I guess it's time to give up on this PR...

I wont have any time in in the nearby months to continue this so closing the PR.