Keyman for macOS Acceptance Test Procedures

These test procedures are to be run before any beta or stable build is published. Copy these checklists of tests before PRs are merged into the beta and stable branches.

Visual Inspection of master branch

• Verify latest CI build of master is successful

• Verify mac/history.md contains all the current changes

Setup Steps

• Ensure the device is running the latest version of macOS

Big Sur 11.0.1 on Parallels.

• Uninstall previous version of Keyman on the device

• Install latest master build (or whatever version is to be tested) of Keyman on the device

14.0.190

The instructions and steps were clear.
I found that the Keyman menu did not become available immediately after granting Keyman the necessary permissions -- this could be improved. See #3971

Basic smoke test

• Verify that the version displayed in the About box is correct.

• On device, turn Wifi off and/or disconnect Ethernet cable

• Activate Keyman and select Keyman Configuration from the input methods menu. Click Download keyboard

• Verify error message "Could not reach Keyman server!" is displayed (This is currently not happening. Bug reported as: bug(mac): Searching for a Keyboard to download in Keyman for macOS gives no feedback when Internet is unavailable #302)

ISSUE This issue is not yet resolved. However I noted that the Download window does not reset when clicking Download keyboard but retains the previous search. This is a separate issue and is a little problematic for this particular test, because Keyman will not attempt to reload the page after failing here. See #3972.

• Close Keyman configuration windows.

• On device, re-enable Internet connectivity

• Activate Keyman and select Keyman Configuration from the input methods menu. Click Download keyboard.

• Verify that page displays correctly (should have a Keyboard Search box with the prompt: Enter language or keyboard.

• Search for Amharic. Should see several keyboards, including some by The Ge'ez Frontier Foundation and some by SIL.

• Double-click any KMP file (in Finder). Should see a “Do you want the Keyman Input Method to install this Package?“ message. If you click Yes, the package should be installed (or updated) and the configuration window should display, showing that the keyboard(s) in the package are installed and enabled. If you click No, the configuration window should not be displayed (but there is a known issue that the very first time after Keyman loads, if the configuration window has not been shown, it will be displayed even if the user clicks No).

The Keyman Configuration window stays open. This is a minor issue.

Application Compatibility Tests

Amharic

Select the Amharic keyboard. For each of the following applications/contexts, test that

1. typing ta produces ታ;

2. typing t, left-arrow, a produces አት;

3. typing tt, left-arrow, a produces ታት;

4. typing tt, then mouse-clicking between the two ታ characters, and then typing a produces ታት.

   In each of the last three cases, the insertion point should end up before the final ት character.

5. common keyboard shortcuts for the app work as expected. For example: ⌘-F (typically opens a Find dialog or control); ⌘-O (File Open); ⌘-X (Cut); ⌘-C (Copy); ⌘-V (Paste); ⌘-A (Select All). (Note that in Google Docs, these commands correspond to both a Google-Docs command and a browser command. The "normal" behavior in all three browsers is to activate the Google Docs command, not the browser command, with the exception that ⌘-O in Safari opens Safari's standard macOS Open file dialog.)

6. with a few characters displayed, ⌘-A followed by typing t replaces selected characters with ት.

7. typing at, then ⌘-A, followed by a produces አ.

• Chrome browser URL bar

• Chrome browser Facebook search control

• Chrome browser Google Docs document (currently known compatibility issue for steps 3, 4 & 5 because any selection change will cause the context to be reset - Google docs can't report context.)

compatibility issue still present (loss of context acceptable)

• Chrome browser Word Online Note: Not using the Office Extension for Chrome - not sure if that would make a difference
  1. a) Beginning of paragraph (currently known compatibility issues steps 1, 3, 4 & 5)
  2. b) Elsewhere in paragraph

compatibility issue still present

• Atom editor - test in Snippet editor (currently known compatibility issue for steps 3, 4 & 5)

Tested with 1.53.0. Compatibility issue still present (loss of context acceptable). #177 no longer relevant

• Safari browser URL bar

• Safari browser Facebook search control

• Safari browser Google Docs document (currently known compatibility issue for steps 3, 4 & 5 because any selection change will cause the context to be reset - Google docs can't report context.)

compatibility issue still present (loss of context acceptable)

• Safari browser Word Online
  1. a) Beginning of paragraph (currently known compatibility issues steps 1, 3, 4 & 5, 7&8)
  2. b) Elsewhere in paragraph (occasionally steps 4 & 5 deletes a preceding (space?) character, but I haven't figured out when/why. Step 7 is not relevant. Step 8 fails for the same reason it doesn't work at the beginning of the paragraph. There seems to be a certain level of general twitchiness, with extra characters sometimes being displayed - sometimes only briefly - and characters being deleted.)

Tested with 14.0.1 (16610.2.11.51.8) and same compat issues shown

• Firefox browser URL bar

• Firefox browser Facebook search control

• Firefox browser Google Docs document (currently known compatibility issue for steps 3, 4 & 5 because any selection change will cause the context to be reset - Google docs can't report context.)

compatibility issue still present (loss of context acceptable) - earlier ref #171

• Firefox browser Word Online

  1. a) Beginning of paragraph
  2. b) Elsewhere in paragraph

• Messages app

  1. a) New message with nobody filled in in To: line
  2. b) New message in existing conversation, such that the message area has grayed out text: "iMessage")

previous issue appears resolved: (currently known compatibility issue for step 1 if you start with an empty input box. I.e., any sequence that deletes the only character in the editor in an attempt to replace it will leave the editor blank.)

• Notes app

• TextEdit

• Terminal (known compatibility issues: Steps 1 & 3 insert an extra leading space if there are no preceding characters. Skip step 4 - mouse doesn't move insertion point in Terminal. Step 5 can be done using left-arrow instead. Skip steps 7 & 8. If the Terminal window has LOTS of text in it, it won't provide any context, which leads to other compatibility problems.)

ISSUE Document these known compat issues. But this appears resolved in this release: Step 2 results in $ታት.

• LibreOffice 7.0 (currently known compatibility issue for steps 3, 4 & 5 because any selection change will cause the context to be reset - LibreOffice Vanilla can't report context.).

compatibility issue still present (loss of context acceptable)

• Mail (message body)

• MS Word

• MS Powerpoint (see [Mac] Can't switch keyboards when MS PowerPoint text box is active #687 for compat issue)

Passed apart from that compat issue

- [ ] Open Office

removing openoffice testing (see LibreOffice)

• Adobe InDesign ([compatibility issues](keymanapp/keyman: Issue [Mac] Compatibility issues with InDesign on Mac #782) with steps 1-5.)

FAIL: this issue is still present

Optional test of Sentry error reporting

1. In Keyman, turn on verbose logging.
2. Optionally, in Console filter all messages from the Keyman process having the text Sentry
3. Select a keyboard called EnglishSpanish (see /mac/test/EnglishSpanish folder in the source repo)
4. Open an editable file in Xcode.
5. Type Sentry force now

• With each keystroke, Console messages should indicate that the character is being added to the Easter egg string.
• When the final w is typed, a Console message should indicate: Forcing crash now!
• After some delay (you may need to switch Keyman off and on again), the new crash should appear in the console on sentry.keyman.com.

OSK UI/Functionality Tests

• Verify behavior of "Always show on-screen keyboard" option
• Verify behavior and appearance of On-Screen Keyboard and the menu that toggles it on and off.
• With OSK visible, verify that switching keyboards updates the display to show the appropriate layout.
• Verify that switching to system keyboard or other IM turns off Keyman OSK.

What's New Tests

• Refer to the new changes in mac/history.md and verify functionality

• Consolidated crash reporting

• Simpler and Smoother Keyboard Search

• Icon for Keyman app

note that the kmp icon which now shows is a little broken? low priority at present

• Added support for European layouts in OSK

• Specifying app compatibility as a user default: execute the command defaults write keyman.inputmethod.Keyman KMLegacyApps -array com.microsoft.Word in Terminal and then observe Word interactions, including losing context and slower text manipulation (remove by defaults write keyman.inputmethod.Keyman KMLegacyApps -array)

• Improved input reliability with modifier keys and cursor keys