JSON API and test suite for web UI by nucleogenic · Pull Request #846 · PiSCSI/piscsi

nucleogenic · 2022-09-19T16:04:20Z

Introduction

This change set offers a light take on "API" functionality for the web UI.

An initial suite of tests covering all of the endpoints (except reboot/shutdown) has been provided. These are high-level, happy path tests, which provide some basic assurance that the web UI features work as expected in normal circumstances.

Changes

For convenience, a summary of changes:

`web.py` and `common` module

Requests sent with the Accept: application/json header get a JSON response
All other requests get the existing render_template() HTML response
Calls to flash() are superseded by a single call to response(message=str|list)
Common data (e.g. used in the base template) are now accessed via the env var in templates
A new /env endpoint exposes the above data for API clients
Data previously passed to render_template() that wasn't JSON serializable has been amended at source
Fixed minor bugs encountered that blocked tests passing

Dependencies and Tooling

Added requirements-dev.txt to separate production and dev dependencies
Added pytest and pytest-httpserver as dev dependencies for running tests
Addedflake8 and black as dev dependencies for code style feedback and reformatting
Updated Dockerfiles to allow testing of more features

Try It Out

Build and launch Docker containers:

git clone https://github.com/nucleogenic/RASCSI.git --branch webui-json-responses
cd RASCSI/docker
cp docker-compose.override.yml.example docker-compose.override.yml
docker compose build --no-cache
docker compose up

Install dependencies:
(new terminal, from 'RASCSI' dir)

cd python/web/
python3 -m venv venv_test
source venv_test/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt

Run tests:

pytest -v

Create image via API:

# Login
curl -v --header "Accept: application/json" \
--cookie-jar rascsi_cookies.txt \
--data "username=pi&password=rascsi" \
 http://localhost:8080/login

# Create image
curl -v --header "Accept: application/json" \
--cookie rascsi_cookies.txt \
--data "file_name=foo&type=hds&size=1" \
http://localhost:8080/files/create

Run tests against a real RaSCSI:

Disclaimer: Make a backup first!

Any attached devices will be automatically detached when the test suite starts.
Each test attempts to clean up the resources it creates.

Firewall note: pytest-httpserver binds to a random port on the computer running the tests, and the RaSCSI instance needs to be able to connect in to this. This is required for the following tests:test_download_url_to_afp_dir, test_download_url_to_images_dir, test_download_url_to_iso and test_extract_file. Please adjust your firewall configuration as necessary.

pytest -v --base_url=http://{RASCSI_IP}/ \
--rascsi_username=pi \
--rascsi_password=raspberry \
--httpserver_host={YOUR_COMPUTER_IP} \
--httpserver_listen_address=0.0.0.0

Future Direction

Some other ideas, which don't quite fit in this (already large) PR, and in no particular order:

Run Pytest tests in CI
Add stateless authentication
Update development documentation (workflow with Docker, Pytest, etc)
Create API documentation and guidelines for API stability
Move more data from messages property to individual properties in data property
Reduce cases where multiple messages originate from a single action

Credits

Special thanks to @erichelgeson for the ideas and inspiration from the newweb branch

python/web/src/static/style.css

python/web/src/templates/index.html

python/web/start.sh

python/web/src/web.py

rdmark · 2022-09-23T16:51:47Z

@nucleogenic I haven't used Docker in a long while, but I assume there's an obvious additional dependency here in the Docker executable and libraries themselves? For instance, if I want to run the Docker container on my Windows 10 PC, I basically just download and install a package from their website, right? And then execute the first few bootstrap commands in a Windows shell compatible fashion.

rdmark

This looks great. I only have a few questions and minor suggestions inline. Feel free to merge when you have made any final refinements.

python/common/src/rascsi/file_cmds.py

python/web/src/static/style.css

python/web/src/web.py

python/web/start.sh

python/web/tests/api/test_json_api.py

nucleogenic

Thanks for the review @rdmark! Will provide some updates shortly.

python/common/src/rascsi/file_cmds.py

python/web/src/web.py

python/web/tests/api/test_json_api.py

python/web/src/static/style.css

python/web/src/web.py

nucleogenic · 2022-09-25T11:58:01Z

To run the tests from a Linux Docker host:

pytest -v --httpserver_host=172.17.0.1 --httpserver_listen_address=172.17.0.1

Where 172.17.0.1 is the IP address of docker0 (or relevant interface in your setup).

Supporting updates to web.py and templates

…n standalone web UI installs

commit 20ae4b3 Merge: 7c1ae8e 78bab77 Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Sat Oct 1 10:44:39 2022 +0200 Merge branch 'develop' into feature_memory_management commit 7c1ae8e Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Sat Oct 1 10:35:27 2022 +0200 Merged localizer changes from develop commit cf19ecf Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Sat Oct 1 10:17:50 2022 +0200 Merged changes from feature_more_unit_tests commit 600e5b7 Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Sat Oct 1 10:00:20 2022 +0200 Formatting update commit 303ea88 Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Sat Oct 1 09:57:30 2022 +0200 Improved transparency of geometry data commit 78bab77 Author: Daniel Markstedt <markstedt@gmail.com> Date: Fri Sep 30 19:20:19 2022 -0700 Swedish localization update 2022-09 (#874) * Swedish localization updates * Use LUN for compactness commit 05e0a78 Merge: 71e070b 1938a8b Author: Daniel Markstedt <markstedt@gmail.com> Date: Fri Sep 30 18:57:51 2022 -0700 Merge pull request #851 from akuker/rdmark-rename-host-bridge Rename the Host Bridge INQUIRY product to RASCSI BRIDGE commit dee929f Author: Uwe Seimet <Uwe.Seimet@seimet.de> Date: Fri Sep 30 10:45:12 2022 +0200 Fixed memory issue reported by valgrind commit 71e070b Merge: 3c8e7db 922c3b2 Author: Daniel Markstedt <markstedt@gmail.com> Date: Thu Sep 29 19:31:42 2022 -0700 Merge pull request #853 from akuker/rdmark-issue852 Calculate image sizes for the Product string using MiB/KiB units commit 3c8e7db Author: Daniel Markstedt <markstedt@gmail.com> Date: Tue Sep 27 17:38:34 2022 -0700 Allow the selecting of target dir when uploading or downloading files (#867) Uses a single endpoint for downloading files Adds a select field to pick target dir for both download and upload forms Moves the Macproxy/Netatalk helptext into the helptext blocks, and the related status messages down into the page footer commit 3ac3abb Merge: b04962d 70c073e Author: Daniel Markstedt <markstedt@gmail.com> Date: Tue Sep 27 07:52:24 2022 -0700 Merge pull request #868 from nucleogenic/webui-pytest-warn-on-delete-failures Display a warning when Pytest fixtures fail to delete a file commit 70c073e Author: nucleogenic <nr@nucleogenic.com> Date: Tue Sep 27 15:39:58 2022 +0100 Display a warning when Pytest fixtures fail to delete a file commit b04962d Author: Daniel Markstedt <markstedt@gmail.com> Date: Tue Sep 27 06:09:18 2022 -0700 Swedish translations in localizer.cpp (#865) * Swedish translations * Fix typos commit 5da3d6c Author: Daniel Markstedt <markstedt@gmail.com> Date: Mon Sep 26 17:44:41 2022 -0700 Introduce info.html template and use it to render detailed info (#863) new: - new templates to render structured info contents in - get_diskinfo() class method that calls disktype and returns the results - /diskinfo endpoint in the Flask app that renders the results from get_diskinfo() changed: - /logs/show and /scsi/info endpoints in the Flask app render in templates - Now using the "RaSCSI Reloaded Control Page" header to function as the link back to the homepage (instead of the github project) which is in line with how most webapps work - Removed the center style for "Attached!" to allow the ? button to be placed on the same line - Remove individual device info, and introduced show all device info in a template commit edbaaf6 Author: Daniel Markstedt <markstedt@gmail.com> Date: Sun Sep 25 17:37:28 2022 -0700 Web UI tweaks: Detaching removable devices; SCSI-1 drive profiles (#854) * Display the Detach action for injected removable media * Flag three DEC drives as SCSI-1 commit ed12853 Merge: 016a616 75b0994 Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 26 00:18:06 2022 +0100 Merge pull request #846 from nucleogenic/webui-json-responses JSON API and test suite for web UI commit 75b0994 Author: nucleogenic <nr@nucleogenic.com> Date: Sun Sep 25 18:15:35 2022 +0100 Add missing asserts to API tests commit a142af5 Author: nucleogenic <nr@nucleogenic.com> Date: Sat Sep 24 03:10:01 2022 +0100 Improve organisation of tests commit 6f7e611 Author: nucleogenic <nr@nucleogenic.com> Date: Sat Sep 24 03:10:11 2022 +0100 Add missing dev dependency flake8 commit f8e5870 Author: nucleogenic <nr@nucleogenic.com> Date: Sat Sep 24 00:20:49 2022 +0100 Update comments commit 663de06 Author: nucleogenic <nr@nucleogenic.com> Date: Thu Sep 22 21:41:12 2022 +0100 Added tests for extracting .sit and .7z archive formats commit 65c2286 Author: nucleogenic <nr@nucleogenic.com> Date: Tue Sep 20 01:56:22 2022 +0100 Fix shell exit issue in web/start.sh commit 8062e5f Author: nucleogenic <nr@nucleogenic.com> Date: Tue Sep 20 00:33:47 2022 +0100 Updates to allow tests to run against a remote RaSCSI instance commit 1a15c4c Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 19 17:00:59 2022 +0100 Expose env info to API clients commit 0e6147e Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 19 14:21:31 2022 +0100 Setup pytest, flake8, black + add API tests commit 26aa5eb Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 19 14:20:12 2022 +0100 Update Dockerfiles to allow testing of additional RaSCSI web UI features commit 6ad436c Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 19 14:18:53 2022 +0100 Add --headless option to easyinstall.sh, enable web auth by default on standalone web UI installs commit dd40d71 Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 19 14:16:34 2022 +0100 Fix issue causing stale reservations after loading a config commit fb8f306 Author: nucleogenic <nr@nucleogenic.com> Date: Mon Sep 26 00:00:18 2022 +0100 Implement response generator for HTML and JSON requests Supporting updates to web.py and templates commit 1e9a7d2 Author: nucleogenic <nr@nucleogenic.com> Date: Thu Sep 22 01:20:22 2022 +0100 Move flattening of file type lists to template layer commit 4ef2e20 Author: nucleogenic <nr@nucleogenic.com> Date: Sat Sep 10 02:58:47 2022 +0100 Update get_supported_locales to return a JSON serializable result commit edf65a8 Author: nucleogenic <nr@nucleogenic.com> Date: Sat Sep 10 02:58:35 2022 +0100 Update RaCtlCmds methods to return JSON serializable results commit 922c3b2 Author: Daniel Markstedt <markstedt@gmail.com> Date: Thu Sep 22 11:50:47 2022 -0700 Calculate image sizes for the Product string using MiB/KiB units commit 1938a8b Author: Daniel Markstedt <markstedt@gmail.com> Date: Thu Sep 22 09:03:36 2022 -0700 Rename the Host Bridge INQUIRY product to RASCSI BRIDGE, as expected by the RASETHER.SYS driver.

nucleogenic force-pushed the webui-json-responses branch from 876112b to baf218e Compare September 22, 2022 00:22

nucleogenic commented Sep 22, 2022

View reviewed changes

nucleogenic force-pushed the webui-json-responses branch from baf218e to fc5c804 Compare September 22, 2022 02:05

nucleogenic marked this pull request as ready for review September 22, 2022 02:07

nucleogenic requested review from akuker, erichelgeson and rdmark as code owners September 22, 2022 02:07

rdmark approved these changes Sep 23, 2022

View reviewed changes

nucleogenic commented Sep 23, 2022

View reviewed changes

nucleogenic requested a review from rdmark September 25, 2022 11:54

nucleogenic mentioned this pull request Sep 25, 2022

[Deprecated] SonarCloud updates for Python sources #861

Closed

This was referenced Sep 25, 2022

Test automation for rascsi-web #300

Closed

Run disktype from web UI #836

Closed

rdmark approved these changes Sep 25, 2022

View reviewed changes

nucleogenic added 14 commits September 25, 2022 23:58

Update RaCtlCmds methods to return JSON serializable results

edf65a8

Update get_supported_locales to return a JSON serializable result

4ef2e20

Move flattening of file type lists to template layer

1e9a7d2

Implement response generator for HTML and JSON requests

fb8f306

Supporting updates to web.py and templates

Fix issue causing stale reservations after loading a config

dd40d71

Add --headless option to easyinstall.sh, enable web auth by default o…

6ad436c

…n standalone web UI installs

Update Dockerfiles to allow testing of additional RaSCSI web UI features

26aa5eb

Setup pytest, flake8, black + add API tests

0e6147e

Expose env info to API clients

1a15c4c

Updates to allow tests to run against a remote RaSCSI instance

8062e5f

Fix shell exit issue in web/start.sh

65c2286

Added tests for extracting .sit and .7z archive formats

663de06

Update comments

f8e5870

Add missing dev dependency flake8

6f7e611

nucleogenic added 2 commits September 26, 2022 00:00

Improve organisation of tests

a142af5

Add missing asserts to API tests

75b0994

nucleogenic force-pushed the webui-json-responses branch from 5d09352 to 75b0994 Compare September 25, 2022 23:07

nucleogenic merged commit ed12853 into PiSCSI:develop Sep 25, 2022

Uh oh!

Conversation

nucleogenic commented Sep 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduction

Changes

web.py and common module

Dependencies and Tooling

Try It Out

Future Direction

Credits

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdmark commented Sep 23, 2022

Uh oh!

rdmark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nucleogenic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nucleogenic commented Sep 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nucleogenic commented Sep 19, 2022 •

edited

Loading

`web.py` and `common` module