7

For my code I am attempting to get an array of AXMenuItems from an AXMenu (AXUIElementRef). The menu logs successfully, and here is my code:

NSArray *anArray = [NSRunningApplication runningApplicationsWithBundleIdentifier:@"com.apple.dock"];
AXUIElementRef anAXDockApp = AXUIElementCreateApplication([[anArray objectAtIndex:0] processIdentifier]);

CFTypeRef aChildren;
AXUIElementCopyAttributeValue(anAXDockApp, kAXChildrenAttribute, &aChildren);

SafeCFRelease(anAXDockApp);

CFTypeRef aMenu = CFArrayGetValueAtIndex(aChildren, 0);

NSLog(@"aMenu: %@", aMenu);

                    // Get menu items
CFTypeRef aMenuChildren;
AXUIElementCopyAttributeValue(aMenu, kAXVisibleChildrenAttribute, &aMenuChildren);

for (NSInteger i = 0; i < CFArrayGetCount(aMenuChildren); i++) {

    AXUIElementRef aMenuItem = [self copyAXUIElementFrom:aMenu role:kAXMenuItemRole atIndex:i];

    NSLog(@"aMenuItem: %@", aMenuItem); // logs (null)

    CFTypeRef aTitle;
    AXUIElementCopyAttributeValue(aMenuItem, kAXTitleAttribute, &aTitle);


    if ([(__bridge NSString *)aTitle isEqualToString:@"New Window"] || [(__bridge NSString *)aTitle isEqualToString:@"New Finder Window"]) /* Crashes here (i can see why)*/{

        AXUIElementPerformAction(aMenuItem, kAXPressAction);

        [NSThread sleepForTimeInterval:1];

        break;

    }

}

What is the correct way to get the list of AXMenuItems?

Screenshot of the Accessibility Inspector:

inspector

2 Answers 2

20

I have figured out an answer, using @Willeke answer of using AXUIElementCopyElementAtPosition() to get the menu. Since there were multiple dock orientations and hiding, I had to create enums in the .h file as it would be easier to read than 0, 1, or 2.

// .h
typedef enum {
    kDockPositionBottom,
    kDockPositionLeft,
    kDockPositionRight,
    kDockPositionUnknown
} DockPosition;

typedef enum {
    kDockAutohideOn,
    kDockAutohideOff
} DockAutoHideState;

Then, I added the methods to get these states in the .m

// .m
- (DockPosition)dockPosition
{
    NSRect screenRect = [[NSScreen mainScreen] frame];

    NSRect visibleRect = [[NSScreen mainScreen] visibleFrame];

    // Dont need to remove menubar height
    visibleRect.origin.y = 0;

    if (visibleRect.origin.x > screenRect.origin.x) {
        return kDockPositionLeft;
    } else if (visibleRect.size.width < screenRect.size.width) {
        return kDockPositionRight;
    } else if (visibleRect.size.height < screenRect.size.height) {
        return kDockPositionBottom;
    }
    return kDockPositionUnknown;
}

- (DockAutoHideState)dockHidden
{
    NSString *plistPath = [NSHomeDirectory() stringByAppendingPathComponent:@"Library/Preferences/com.apple.dock.plist"];
    NSDictionary *dockDict = [NSDictionary dictionaryWithContentsOfFile:plistPath];

    CFBooleanRef autohide = CFDictionaryGetValue((__bridge CFDictionaryRef)dockDict, @"autohide");
    if (CFBooleanGetValue(autohide) == true) {
        return kDockAutohideOn;
    }
    return kDockAutohideOff;
}

For the dock position unknown, I added in case it was not able to calculate it from the screen positions.

Then, I used a method to get the dock item from the menubar:

- (AXUIElementRef)getDockItemWithName:(NSString *)name
{
    NSArray *anArray = [NSRunningApplication runningApplicationsWithBundleIdentifier:@"com.apple.dock"];
    AXUIElementRef anAXDockApp = AXUIElementCreateApplication([[anArray objectAtIndex:0] processIdentifier]);

    AXUIElementRef aList = [self copyAXUIElementFrom:anAXDockApp role:kAXListRole atIndex:0];

    CFTypeRef aChildren;
    AXUIElementCopyAttributeValue(aList, kAXChildrenAttribute, &aChildren);
    NSInteger itemIndex = -1;

    for (NSInteger i = 0; i < CFArrayGetCount(aChildren); i++) {

        AXUIElementRef anElement = CFArrayGetValueAtIndex(aChildren, i);

        CFTypeRef aResult;

        AXUIElementCopyAttributeValue(anElement, kAXTitleAttribute, &aResult);

        if ([(__bridge NSString *)aResult isEqualToString:name]) {

            itemIndex = i;
        }
    }
    SafeCFRelease(aChildren);

    if (itemIndex == -1) return nil;

    // We have index now do something with it

    AXUIElementRef aReturnItem = [self copyAXUIElementFrom:aList role:kAXDockItemRole atIndex:itemIndex];
    SafeCFRelease(aList);
    return  aReturnItem;
}

This SafeCFRelease() method is a very simple method that checks if the passed value is not nil, then releases (had some issues earlier).

void SafeCFRelease( CFTypeRef cf )
{
    if (cf) CFRelease(cf);
}

And this method [copyAXUIElementFrom: role: atIndex:] is a method from @Willeke answer from another one of my questions:

- (AXUIElementRef)copyAXUIElementFrom:(AXUIElementRef)theContainer role:(CFStringRef)theRole atIndex:(NSInteger)theIndex {
    AXUIElementRef aResultElement = NULL;
    CFTypeRef aChildren;
    AXError anAXError = AXUIElementCopyAttributeValue(theContainer, kAXChildrenAttribute, &aChildren);
    if (anAXError == kAXErrorSuccess) {
        NSUInteger anIndex = -1;
        for (id anElement in (__bridge NSArray *)aChildren) {
            if (theRole) {
                CFTypeRef aRole;
                anAXError = AXUIElementCopyAttributeValue((__bridge AXUIElementRef)anElement, kAXRoleAttribute, &aRole);
                if (anAXError == kAXErrorSuccess) {
                    if (CFStringCompare(aRole, theRole, 0) == kCFCompareEqualTo)
                        anIndex++;
                    SafeCFRelease(aRole);
                }
            }
            else
                anIndex++;
            if (anIndex == theIndex) {
                aResultElement = (AXUIElementRef)CFRetain((__bridge CFTypeRef)(anElement));
                break;
            }
        }
        SafeCFRelease(aChildren);
    }
    return aResultElement;
}

Taking all this code, I put it into one of my methods:

// Check if in dock (otherwise cant do it)

                if ([self isAppOfNameInDock:[appDict objectForKey:@"AppName"]]) {

                    // Get dock item

                    AXUIElementRef aDockItem = [self getDockItemWithName:[appDict objectForKey:@"AppName"]];

                    AXUIElementPerformAction(aDockItem, kAXShowMenuAction);

                    [NSThread sleepForTimeInterval:0.5];

                    CGRect aRect;

                    CFTypeRef aPosition;
                    AXUIElementCopyAttributeValue(aDockItem, kAXPositionAttribute, &aPosition);
                    AXValueGetValue(aPosition, kAXValueCGPointType, &aRect.origin);
                    SafeCFRelease(aPosition);

                    CFTypeRef aSize;
                    AXUIElementCopyAttributeValue(aDockItem, kAXSizeAttribute, &aSize);
                    AXValueGetValue(aSize, kAXValueCGSizeType, &aRect.size);
                    SafeCFRelease(aSize);

                    SafeCFRelease(aDockItem);

                    CGPoint aMenuPoint;

                    if ([self dockHidden] == kDockAutohideOff) {
                        switch ([self dockPosition]) {
                            case kDockPositionRight:
                                aMenuPoint = CGPointMake(aRect.origin.x - 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionLeft:
                                aMenuPoint = CGPointMake(aRect.origin.x + aRect.size.width + 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionBottom:
                                aMenuPoint = CGPointMake(aRect.origin.x + (aRect.size.width / 2), aRect.origin.y - 18);
                                break;
                            case kDockPositionUnknown:
                                aMenuPoint = CGPointMake(0, 0);
                                break;
                        }
                    } else {

                        NSRect screenFrame = [[NSScreen mainScreen] frame];

                        switch ([self dockPosition]) {
                            case kDockPositionRight:
                                aMenuPoint = CGPointMake(screenFrame.size.width - 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionLeft:
                                aMenuPoint = CGPointMake(screenFrame.origin.x + 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionBottom:
                                aMenuPoint = CGPointMake(aRect.origin.x + (aRect.size.width / 2), screenFrame.size.height - 18);
                                break;
                            case kDockPositionUnknown:
                                aMenuPoint = CGPointMake(0, 0);
                                break;
                        }
                    }

                    if ((aMenuPoint.x != 0) && (aMenuPoint.y != 0)) {

                        AXUIElementRef _systemWideElement = AXUIElementCreateSystemWide();

                        AXUIElementRef aMenu;
                        AXUIElementCopyElementAtPosition(_systemWideElement, aMenuPoint.x, aMenuPoint.y, &aMenu);

                        SafeCFRelease(_systemWideElement);

                        // Get menu items
                        CFTypeRef aMenuChildren;
                        AXUIElementCopyAttributeValue(aMenu, kAXVisibleChildrenAttribute, &aMenuChildren);

                        NSRunningApplication *app = [[NSRunningApplication runningApplicationsWithBundleIdentifier:[appDict objectForKey:@"BundleID"]] objectAtIndex:0];

                        for (NSInteger i = 0; i < CFArrayGetCount(aMenuChildren); i++) {

                            AXUIElementRef aMenuItem = [self copyAXUIElementFrom:aMenu role:kAXMenuItemRole atIndex:i];

                            CFTypeRef aTitle;
                            AXUIElementCopyAttributeValue(aMenuItem, kAXTitleAttribute, &aTitle);

                            // Supports chrome, safari, and finder
                            if ([(__bridge NSString *)aTitle isEqualToString:@"New Window"] || [(__bridge NSString *)aTitle isEqualToString:@"New Finder Window"]) {

                                AXUIElementPerformAction(aMenuItem, kAXPressAction);

                                NSInteger numberOfWindows = [self numberOfWindowsOpenFromApplicationWithPID:[app processIdentifier]];
                                // Wait until open
                                while ([self numberOfWindowsOpenFromApplicationWithPID:[app processIdentifier]] <= numberOfWindows) {
                                }
                                break;
                            }
                        }
                        SafeCFRelease(aMenu);
                        SafeCFRelease(aMenuChildren);
                    }
                }

This is pretty complicated, but it works. I probably can't explain it, but I have stress tested this code and it works quite well.

Sign up to request clarification or add additional context in comments.

Comments

1

Answer to the question "How to get an array of AXMenuItems from AXMenu?": aMenuChildren is the list of menu items. To be sure you could filter by role kAXMenuItemRole.

Answer to the question "Why does it not work?": The menu isn't a menu. When you inspect the menu in Axccessibility Inspector, it displays a warning "Parent does not report element as one of its children". The Dock app has one child, a list of dock items.

5 Comments

I use the children list to check if it has a certain title. Then action
Hmmm. Thanks. I'm still very new to accessibility. I'll try to figure it out
I checked out the inspector, and I'm very puzzled. Can you help?
A contextual menus is not a child of its parent. Try to avoid them. If you really want the menu, use AXUIElementCopyElementAtPosition to get a menu item and get its parent.
I'm really get ng to check if the menu has a "new window" option to open a new window or that app (ie Safari)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.