Added a web backend for deploying apps as a static web server by freakboy3742 · Pull Request #898 · beeware/briefcase

freakboy3742 · 2022-10-05T06:49:41Z

Adds a static web deployment backend.

This uses PyScript to provide client-side Python support.

Removes the placeholder Django backend. The app can be build as an entirely static site.

Also requires:

The new briefcase-web-static-template
The static-web branch of briefcase-template
The pyscript branch of Toga

CI will fail on the web build until (2) is merged.

In the meantime, you can test this by adding:

[tool.briefcase.app.myapp.web]
requires = [
    "../path/to/toga/src/web"
]
style_framework = "Bootstrap v4.6"

to a project that is using a source checkout of Toga on the pyscript branch mentioned above.

Fixes #3.

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

rmartin16

This is awesome. I've only spun pyscript up before just to see it working but being able to run toga makes it a lot more attractive.

A few thoughts:

Should the docs be more forthcoming that this is still early days for all this?
- Mostly thinking of the case where someone imports a package that uses something in the stdlib that stubbed for wasm.
- Maybe linking here or similar.
Python's simple HTTP server and Chrome on Windows
- CTLC+C isn't always working for me....Chrome must be holding open a connection and the web server must be blocking on it. When I close the tab and reopen it, Chome seemingly resets the connection and the web server is unblocked and sees the interrupt and exits.
- Firefox seems to play much nicer with this web server.
- Running the web server like macOS's log streamer would probably avoid this but is a lot more overhead.
Toga Tutorials 0 and 1
- I can't get these to actually render anything. (I've verified i'm using your pyscript toga branch.)
- Not sure what's missing but I'll keep messing with it.

freakboy3742 · 2022-10-05T23:40:45Z

This is awesome. I've only spun pyscript up before just to see it working but being able to run toga makes it a lot more attractive.

A few thoughts:

Should the docs be more forthcoming that this is still early days for all this?

Mostly thinking of the case where someone imports a package that uses something in the stdlib that stubbed for wasm.

Maybe linking here or similar.

At the very least, it worth mentioning that there are limitations on the binary modules that are available, and some runtime complications. I'll add a note.

Python's simple HTTP server and Chrome on Windows

CTLC+C isn't always working for me....Chrome must be holding open a connection and the web server must be blocking on it. When I close the tab and reopen it, Chome seemingly resets the connection and the web server is unblocked and sees the interrupt and exits.

Firefox seems to play much nicer with this web server.

Running the web server like macOS's log streamer would probably avoid this but is a lot more overhead.

Hrm... That's weird. I'm not seeing any problems like that with Chrome on macOS.

As taking a different approach - the comparison with the macOS log streamer isn't really fair, because a macOS streams it's logs to a separate process. In this case, the logs are the stderr output from invoking httpd.serve_forever(). Unless you're suggesting spawning a process to run the web server, and then waiting for that web server process to terminate? Which... I guess it could work, but it seems... overkill.

Toga Tutorials 0 and 1

I can't get these to actually render anything. (I've verified i'm using your pyscript toga branch.)

Not sure what's missing but I'll keep messing with it.

Yeah - I'm aware of those problems. I'm still tinkering with the Toga branch, and I don't think I've pushed all the modifications I've made to date. There's been some changes in PyScript since I started that branch, and I'm trying to get on top of what has changed. I'm hoping to get those issues sorted out today.

rmartin16 · 2022-10-06T00:16:57Z

Hrm... That's weird. I'm not seeing any problems like that with Chrome on macOS.

I'm pretty sure I was only seeing this when pyscript entered an error state; such as when wheel installation was failing because it didn't like the names. Normally, the problem doesn't occur. But given this is a dev web server, we should probably expect problems.

unless you're suggesting spawning a process to run the web server, and then waiting for that web server process to terminate? Which... I guess it could work, but it seems... overkill.

That's what I am talking about...but agreed; this should be a last resort or something.

freakboy3742 · 2022-10-06T06:19:12Z

I've now pushed updates to the pyscript branch.

I can confirm I'm seeing the log lockup issue on Windows - no idea what is causing that.

There's one other edge case that I've observed in testing - if you declare the mainline app as MyApp() (and rely on package metadata), rather than MyApp("Formal name", "org.beeware.bundle"), there's a Heisenbug at app startup - sometimes it works fine, sometimes it doesn't (and it complains about the app needing a formal name). I'm still digging into this one.

rmartin16 · 2022-10-06T15:25:57Z

+                    f"Unable to install dependencies for app {app.app_name!r}"
+                ) from e
+
+        with self.input.wait_bar("Writing Pyscript configuration file..."):


disclosure: i consider this at least borderline nitpicking.

This section of logging is difficult for me to easily parse:

Writing Pyscript configuration file... done Compile static web content from wheels... Processing mywebapp-0.0.1-py3-none-any.whl... Processing toga_core-0.3.0.dev39-py3-none-any.whl... Processing toga_dummy-0.3.0.dev39-py3-none-any.whl... Processing toga_web-0.3.0.dev39-py3-none-any.whl... Found toga_web/static/toga.css Processing travertino-0.1.3-py3-none-any.whl... Compiling static web content from wheels... done

There are three important things happening here but it's hard to separate them out.

There are a lot of ellipsis....but only some say done after them.

Two possible improvements.

Consider the output from pip that directly precedes this:

Processing /home/user/github/freakboy3742/toga/src/core Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Processing /home/user/github/freakboy3742/toga/src/dummy Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Processing /home/user/github/freakboy3742/toga/src/web Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting travertino>=0.1.3 Using cached travertino-0.1.3-py3-none-any.whl (15 kB) Saved ./web/static/mywebapp/www/static/wheels/travertino-0.1.3-py3-none-any.whl Building wheels for collected packages: toga-core, toga-dummy, toga-web Building wheel for toga-core (setup.py): started Building wheel for toga-core (setup.py): finished with status 'done' ...

This use of the indent makes it dramatically easier to quickly skim the output. Using an indent for the Processing <wheel> output really improves readability IMO:

Writing Pyscript configuration file... done Compile static web content from wheels... Processing mywebapp-0.0.1-py3-none-any.whl... Processing toga_core-0.3.0.dev39-py3-none-any.whl... Processing toga_dummy-0.3.0.dev39-py3-none-any.whl... Processing toga_web-0.3.0.dev39-py3-none-any.whl... Found toga_web/static/toga.css Processing travertino-0.1.3-py3-none-any.whl... Compiling static web content from wheels... done

I guess if we really wanted to, we could add an indent argument to Log to manage this...

As for all the ellipsis...

I've been wondering for a while if ellipsis should be exclusively reserved for the Wait Bar. If the indent is added, then I definitely think it makes sense to at least avoid the ellipsis for the Compile static web content from wheels log.

As for the ellipsis for the Processing <wheel> log, I was initially thinking they should be Wait bars....but that would mean the Wait bar would be changing very quickly and that isn't great. So, probably best to get rid of the ellipsis there too and just let the single Wait Bar convey that tasks are ongoing.

This is where this would ultimately end up:

[mywebapp] Building web project... Removing old wheels... done Repacking wheel as /home/user/tmp/beeware/mywebapp/web/static/mywebapp/www/static/wheels/mywebapp-0.0.1-py3-none-any.whl...OK Building app wheel... done Processing /home/user/github/freakboy3742/toga/src/core Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Processing /home/user/github/freakboy3742/toga/src/dummy Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Processing /home/user/github/freakboy3742/toga/src/web Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting travertino>=0.1.3 Using cached travertino-0.1.3-py3-none-any.whl (15 kB) Saved ./web/static/mywebapp/www/static/wheels/travertino-0.1.3-py3-none-any.whl Building wheels for collected packages: toga-core, toga-dummy, toga-web Building wheel for toga-core (setup.py): started Building wheel for toga-core (setup.py): finished with status 'done' Created wheel for toga-core: filename=toga_core-0.3.0.dev39-py3-none-any.whl size=496221 sha256=7d63aaeae0a49ac0b40b5ee57541b8d4f3df935c3e8c5832dae548df32cd7cae Stored in directory: /tmp/pip-ephem-wheel-cache-ox82hom4/wheels/d6/c5/94/ed3559e7d1e227a880e8f68277a009ba9faa83254cdcb5fbce Building wheel for toga-dummy (setup.py): started Building wheel for toga-dummy (setup.py): finished with status 'done' Created wheel for toga-dummy: filename=toga_dummy-0.3.0.dev39-py3-none-any.whl size=27249 sha256=e28701baadd3879cc8530082f8be391c54982dbae3584db35257960c7484ae65 Stored in directory: /tmp/pip-ephem-wheel-cache-ox82hom4/wheels/c0/a2/58/b2501a1be28389106869c459b744fec9e6f5631ada886e2e74 Building wheel for toga-web (setup.py): started Building wheel for toga-web (setup.py): finished with status 'done' Created wheel for toga-web: filename=toga_web-0.3.0.dev39-py3-none-any.whl size=13068 sha256=d369f6eac3c796b1579e223f65f5c0efc17c332aaf868f1071281c979b580f53 Stored in directory: /tmp/pip-ephem-wheel-cache-ox82hom4/wheels/19/e5/bf/617886587698943f953cf16ea7e3623985fedffbc8b187e98c Successfully built toga-core toga-dummy toga-web Installing wheels for dependencies... done Writing Pyscript configuration file... done Compile static web content from wheels Processing mywebapp-0.0.1-py3-none-any.whl Processing toga_core-0.3.0.dev39-py3-none-any.whl Processing toga_dummy-0.3.0.dev39-py3-none-any.whl Processing toga_web-0.3.0.dev39-py3-none-any.whl Found toga_web/static/toga.css Processing travertino-0.1.3-py3-none-any.whl Compiling static web content from wheels... done

Note: pip has a whole infrastructure to manage the current level of indent and incrementing and decrementing it. I'd say limiting ourselves to a single indent level is reasonable.

Thoughts?

Completely agreed that the "wall of text" from that stage isn't great; I went through a couple of iterations of waitbar use before I settled where I did, and I still wasn't happy with the result.

I think the indentation and removing the ellipsis is a good improvement, though; I'll add those in.

More generally, I see 3 generic problems with the output as rendered in the build command:

Firstly, there's no vertical breaks. There's a lot being done, and being done quickly, but no vertical breaks to indicate "phases" in the build. We've been leaning away from empty calls to info(); but the stages are sufficiently short lived (and minor) that they don't really warrant a 'prefix' block. Not sure if there's a good solution here other than those two options.

Secondly, the need to have a "prefix" statement when using a waitbar that produces output. This is easy enough to do with a call to "info", but the "grammar" of those statements isn't clear. I broadly agree with your comment about keeping ellipsis exclusive to waitbars - an ellipsis should always be "closed" with "done" - but if you do that, the grammar of the "What I'm about to do" statement doesn't flow well to me. In this example, we need something to explain what "processing" means, but "Compile static web content" and "Compiling static web content" both feel like the wrong tense. "Compile" might make more sense if it was a prefixed marker, but as currently rendered, it feels like the best of a bad bunch of options, rather than a good option in itself.

Thirdly, there's the indentation issue. I agree indentation is a nice touch; although I do wonder if using indentation here is a crutch for getting over the first two issues. If we're going to start adopting indentation, then I agree having some sort of baked-in API would be a good idea - but the fact that we've been able to survive this long without indentation makes me wonder whether coming up with a good fixes for the first two might be the better (and easier) approach.

Yeah...this is quite reminiscent of my ruminations in #755. I'm hesitant to go too much back to the drawing board given I didn't really find good answers then.

With that, I went through the output of the package command for all the platforms and formats before the apps were created so I could get a good sense of Briefcase output. I couldn't really identify anywhere that a lot of information generated by Briefcase is presented to the user. This may be the first case where particularly relevant information is being shown to the user and quickly. So, this may be the first use-case for something like indentation to increase readability.

In several of the outputs from tools, indentation use was certainly popular (especially with google).

I think indentation is a reasonable addition to logging. It would also be fairly trivial to add an "indent level" param to the APIs; this approach would also inherently limit how broadly this could be used since you wouldn't be able to call arbitrary functions that are "indent-unaware" (for lack of a better term).

rmartin16

I think I've resolved the CTRL-C issue - it appears that on Windows, an explicit server shutdown is required, in addition to the close. It doesn't hurt to do the same on macOS, so we might as well.

I can still create the situation where the web server doesn't respond to CTRL+C.

Add 1/0 to app.py to create an exception.
Start the web server and load the app in chrome on windows.
CTRL+C (most likely) will not work.

This can recreate the situation probably 9 in 10 times for me (refreshing twice before a CTRL-C may help). Refreshing the page in chrome allows the web server to receive the interrupt and exit.

I cannot replicate this with Firefox but I can with Edge (although its chromium now so not surprised). I cannot replicate this on Linux at all; this perhaps suggests this is more a function of Python's interaction with the OS via select on Windows. The stack below seems to reinforce this.

Python stack when stuck on Windows

0x0000000000000000
ntdll.dll!NtWaitForSingleObject+0x14
mswsock.dll+0x80fc
mswsock.dll!Tcpip4_WSHAddressToString+0xa8e
WS2_32.dll!select+0x137
select.pyd!PyInit_select+0x125e
select.pyd!PyInit_select+0x10ab
python310.dll!PyEval_EvalFrameDefault+0x346b
python310.dll!PyEval_EvalFrameDefault+0xf27
python310.dll!PyObject_Hash+0xd25
python310.dll!PyVectorcall_Call+0xb8
python310.dll!PyObject_Call+0x14f
python310.dll!PyEval_EvalFrameDefault+0x5efe
python310.dll!PyFunction_Vectorcall+0x87
python310.dll!PyObject_FastCallDictTstate+0xd3
python310.dll!PyObject_Call_Prepend+0x7f
python310.dll!PyWeakref_NewProxy+0x1e14
python310.dll!PyObject_Call+0x1cb
python310.dll!PyEval_EvalFrameDefault+0x5efe
python310.dll!PyFunction_Vectorcall+0x87
python310.dll!PyEval_EvalFrameDefault+0x5f4
python310.dll!PyMapping_Check+0x1d5
python310.dll!PyEval_EvalCode+0x82
python310.dll!PyArena_Free+0x3f3
python310.dll!PyArena_Free+0x2f3
python310.dll!PyEval_EvalFrameDefault+0x1875
python310.dll!PyFunction_Vectorcall+0x87
python310.dll!PyEval_EvalFrameDefault+0x5f4
python310.dll!PyFunction_Vectorcall+0x87
python310.dll!PyVectorcall_Call+0x5c
python310.dll!PyObject_Call+0x4f
python310.dll!PyMem_SetupAllocators+0x1aa
python310.dll!Py_RunMain+0x13d
python310.dll!Py_RunMain+0x15
python310.dll!Py_Main+0x25
python.exe+0x1230
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

Poking around google, this definitely doesn't look like an unknown problem. It seems to boil down to sockets inherently blocking in python. The underlying abstractions support turning blocking off.

When I added self.socket.setblocking(False) to LocalHTTPServer.__init__(), the server would always respond to CTRL+C...as well as working normally.

Admittedly, I haven't quite sussed out the implications of turning blocking off, though, in this use case. The suggestion seems to be that exceptions may be raised....but I haven't seen this happen.

Do you have any insight here?

[edit]
After further testing, I'm seeing a lot of network errors in the browser console suggesting that the server is abandoning the connection before data is sent or the connection is accepted. idk

freakboy3742 · 2022-10-07T01:08:39Z

I think I've resolved the CTRL-C issue - it appears that on Windows, an explicit server shutdown is required, in addition to the close. It doesn't hurt to do the same on macOS, so we might as well.

I can still create the situation where the web server doesn't respond to CTRL+C.

Ok - that's reproducible for me as well.

The good news - I think I've found a fix. I found an old Python bug https://bugs.python.org/issue31639 that made reference to using ThreadingHTTPServer for http.server for the same reason we're seeing here. The threaded server gives better server performance anyway, so it seems like a good move regardless - and at least in my testing, it addresses the Windows locking bug.

freakboy3742 · 2022-10-07T01:11:15Z

@rhmartin Also - if I could get a review of beeware/briefcase-template#34, the CI failures on the app tests should be resolved.

rmartin16 · 2022-10-07T01:39:04Z

if I could get a review of beeware/briefcase-template#34, the CI failures on the app tests should be resolved.

Oh, good point; I'll review and merge that PR.

Also, did you have any feedback on the additional http server exceptions or the toml thoughts?

freakboy3742 · 2022-10-07T01:43:15Z

Also, did you have any feedback on the additional http server exceptions or the toml thoughts?

I swear Github is hiding things from me... I completely missed those. I'll address them now.

rmartin16 · 2022-10-07T15:58:02Z

There's one other edge case that I've observed in testing - if you declare the mainline app as MyApp() (and rely on package metadata), rather than MyApp("Formal name", "org.beeware.bundle"), there's a Heisenbug at app startup - sometimes it works fine, sometimes it doesn't (and it complains about the app needing a formal name). I'm still digging into this one.

I presume this is that error. Adding for posterity.

PythonError: Traceback (most recent call last):
File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) 
File "/lib/python3.10/site-packages/_pyodide/_base.py", line 506, in eval_code_async await CodeRunner( 
File "/lib/python3.10/site-packages/_pyodide/_base.py", line 357, in run_async coroutine = eval(self.code, globals, locals) 
File "", line 2, in 
File "/lib/python3.10/runpy.py", line 209, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) 
File "/lib/python3.10/runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, 
File "/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) 
File "/lib/python3.10/site-packages/webtemplate/__main__.py", line 4, in main().main_loop() 
File "/lib/python3.10/site-packages/webtemplate/app.py", line 27, in main return webtemplate() 
File "/lib/python3.10/site-packages/toga/app.py", line 234, in __init__ raise RuntimeError('Toga application must have a formal name') 
RuntimeError: Toga application must have a formal name

rmartin16 · 2022-10-07T17:35:05Z

I think I've resolved the CTRL-C issue - it appears that on Windows, an explicit server shutdown is required, in addition to the close. It doesn't hurt to do the same on macOS, so we might as well.

I can still create the situation where the web server doesn't respond to CTRL+C.

Ok - that's reproducible for me as well.

The good news - I think I've found a fix. I found an old Python bug https://bugs.python.org/issue31639 that made reference to using ThreadingHTTPServer for http.server for the same reason we're seeing here. The threaded server gives better server performance anyway, so it seems like a good move regardless - and at least in my testing, it addresses the Windows locking bug.

🦾 Nice. Fixed for me as well.

rmartin16 · 2022-10-07T19:25:10Z

-# briefcase.formats.django =
-#     project = briefcase.platforms.django.project
-#     app = briefcase.platforms.django.app


Throwing this out there while we're here...

Do you want to remove homebrew as well for now?

Oddly, the homebrew case is slightly different. The Django backend never really existed; there was a code module, but it wasn't loaded. The homebrew does exit, and is actually being used by tests. It probably shouldn't be, as it's a bit of a red herring... but it's a little difficult to extract.

Ok - you've successfully nerdsniped me :-)

rmartin16 · 2022-10-07T21:03:04Z

+            self.logger.info(
+                "Web server log output (type CTRL-C to stop log)...",
+                prefix=app.app_name,
+            )
+            self.logger.info("=" * 75)


In lieu of disabling browser caching, I was thinking it might be nice to add a line like this here for folks that aren't perhaps steeped in web dev:

Tip: holding SHIFT while clicking refresh will bypass browser caching

If it's going to cause enough problems to include that message, it's worth including the disable-cache headers. I'll add them.

:) It just kept coming to mind because I was constantly running in to unexpected caching. even if i shutdown the server, made code changes, and start everything back up again.

freakboy3742 added 11 commits October 4, 2022 10:01

Added stub web backend.

100691b

Add code to collect/compile and deploy wheels.

e4ce678

Post-process wheels to extract static content.

104b303

Make wheel references absolute.

68f591b

Add zip packaging for web projects.

694f22d

Add documentation for web platform.

0cd741a

Added Web to the list of supported platforms.

f5cb52b

Add tests for web backend.

8633316

Added change note.

7f2c3ed

Add a web build to CI.

fe1d244

Fixed the CI task definition.

54bd1f7