Class elm

Represents a UI element. Clicks, gets properties, etc.

public sealed class elm : IDisposable
Remarks

UI elements are user interface (UI) parts that are accessible through programming interfaces (API). For example buttons, links, list items. This class can find them, get properties, click, etc. Web pages and most other windows support UI elements.

An elm variable contains a COM interface pointer (IAccessible or other) and uses methods of that interface or/and related API.

elm functions that get properties don't throw exception when the COM etc method failed (returned an error code of HRESULT type). Then they return "" (string properties), 0, false, null or empty collection, depending on return type. Applications implement UI elements differently, often with bugs, and their COM interface functions return a variety of error codes. It's impossible to reliably detect whether the error code means an error or the property is merely unavailable. These elm functions also set the last error code of this thread = the return value (HRESULT) of the COM function, and callers can use lastError to get it. If lastError.code returns 1 (S_FALSE), in most cases it's not an error, just the property is unavailable. On error it will probably be a negative error code.

You can dispose elm variables to release the COM object, but it is not necessary (GC will do it later).

An elm variable cannot be used in multiple threads. Only Dispose can be called in any thread.

UI elements are implemented and live in their applications. This class just communicates with them.

Many applications have various problems with their UI elements: bugs, incorrect/nonstandard/partial implementation, or initially disabled. This class implements workarounds for known problems, where possible.

Application Problems
Chrome web browser. Also Edge, Opera and other apps that use Chromium. Window class name is like "Chrome_WidgetWin_1".
  1. Web page UI elements initially are disabled (missing). Workarounds:
    • Functions Find, Exists, Wait and FindAll enable it if used role prefix "web:" or "chrome:". Functions FromXY, FromMouse and Focused enable it if window class name starts with "Chrome". However Chrome does it lazily, therefore shortly after enabling something still may not work. Note: this auto-enabling may fail with future Chrome versions.
    • Start Chrome with command line --force-renderer-accessibility.
  2. Some new web browser versions add new features or bugs that break something.
Firefox web browser.
  1. When Firefox starts, its web page UI elements are unavailable. Creates them only when somebody asks (eg function Find), but does it lazily, and Find at first fails. Workaround: use parameter wait.
  2. Occasionally Firefox briefly turns off its web page UI elements. Workaround: use parameter wait. With other web browsers also it's better to use wait.
  3. Some new web browser versions add new features or bugs that break something.
Applications (or just some windows) that don't have accessible objects but have UI Automation elements.
  1. To find UI elements in these applications, need flag EFFlags.UIA.
Java applications that use AWT/Swing. Window class name starts with "SunAwt".
  1. Must be enabled Java Access Bridge (JAB).
    If JAB is missing/disabled/broken, the Find UI element tool shows an "enable" link when you try to capture something in a Java window. Or you can enable JAB in Options > OS or in Control Panel > Ease of Access Center. Or use jabswitch.exe. Then restart Java apps. Also may need to restart apps that tried to use Java UI elements.
  2. JAB is part of Java. Install Java 64-bit x64 or ARM64 (as your OS).
  3. If you'll use JAB in 32-bit script processes (unlikely), also install Java 32-bit. If you'll use JAB in x64 script processes on Windows ARM64 (unlikely), also install Java x64.
  4. Not supported on 32-bit OS.
  5. JAB bug: briefly shows a console window at PC startup, after every sleep, etc. Or opens an invalid Windows Terminal window; workaround: in Terminal settings set Default Terminal Application = Windows Console Host.
Some controls.
  1. UI elements of some controls are not connected to the UI element of the parent control. Then cannot find them if searching in whole window.
    Workaround: search only in that control. For example, use prop"class" or "id". Or find the control (wnd.Child etc) and search in it.
Some controls with flag EFFlags.NotInProc. UI elements of many standard Windows controls have bugs when they are retrieved without loading dll into the target process (see EFFlags.NotInProc). Known bugs:
  1. Toolbar buttons don't have Name in some cases.
  2. elm.Focus and elm.Select often don't work properly.
Workarounds: Don't use EFFlags.NotInProc. Or use EFFlags.UIA.
When cannot load dll into the target process. For example Windows Store apps.
  1. Function Find is much slower, and uses much more CPU when waiting. More info: EFFlags.NotInProc.
Processes of a different 32/64 bitness than this process. Also ARM64 processes.
  1. To load the dll is used Au.DllHost.exe, which makes slower by about 50 ms first time.
DPI-scaled windows (see Dpi.IsWindowVirtualized).
  1. In some cases "element from point" and "get rectangle" functions may not work correctly with such windows. This process must be per-monitor-DPI-aware.
Examples

Click link "Example" in Chrome.

var w = wnd.find(0, "* Chrome");
var e = w.Elm["web:LINK", "Example"].Find(5);
e.Invoke();

Click a link, wait for new web page, click a link in it.

var w = wnd.find(0, "* Chrome");
var e = w.Elm["web:LINK", "Link 1"].Find(5);
e.WebInvoke();
w.Elm["web:LINK", "Link 2"].Find(5).WebInvoke();
Namespace: Au
Assembly: Au.dll
Inheritance
object
elm

Properties

Name Description
ChildCount

Gets the number of direct child UI elements.
DefaultAction

Gets default action of elm.Invoke.
Description

Gets description.
Elm

Gets an elmFinder for finding UI elements in this UI element. Example: var e2 = e1.Elm["ROLE", "Name"].Find();. Example: var e2 = e1.Elm["ROLE1", "Name1"]["ROLE2", "Name2"]["ROLE3", "Name3"].Find();. Example: print.it(e.Elm.FindAll());.
Help

Gets help text.
IsChecked

Calls elm.State and returns true if has state CHECKED.
IsChecked2

Calls elm.State and returns true if has state CHECKED, null if has state MIXED, else false. Use this function with 3-state checkboxes.
IsDisabled

Calls elm.State and returns true if has state UNAVAILABLE.
IsFocused

Calls elm.State and returns true if has state FOCUSED.
IsInvisible

Calls elm.State and returns true if has state INVISIBLE and does not have state OFFSCREEN.
IsOffscreen

Calls elm.State and returns true if has state OFFSCREEN.
IsPassword

Calls elm.State and returns true if has state PROTECTED.
IsPressed

Calls elm.State and returns true if has state PRESSED.
IsReadonly

Calls elm.State and returns true if has state READONLY.
IsSelected

Calls elm.State and returns true if has state SELECTED.
Item

Gets or changes simple element id, also known as child id.
KeyboardShortcut

Gets keyboard shortcut.
Level

Gets or sets indentation level for elm.ToString.
MiscFlags

Returns some additional info about this variable, such as how the UI element was retrieved (inproc, UIA, Java).
Name

Gets name.
Parent

Gets parent element. Same as elm.Navigate with argument "pa".
Rect

Gets location of this UI element in screen.
Role

Gets standard or custom role, as string.
RoleInt

Gets role as enum ERole.
SelectedChildren

Gets selected direct child items.
State

Gets UI element state (flags).
UiaCN

Gets the UI Automation ClassName property.
UiaId

Gets the UI Automation AutomationId property.
Value

Gets or sets value.
WndContainer

Gets the container window or control of this UI element.
WndTopLevel

Gets the top-level window that contains this UI element.
path

Gets an elmFinder for finding UI elements in a window or UI element that can be set later with elmFinder.In. Example: var e = elm.path["ROLE", "Name"].In(w).Find();. Same as var e = w.Elm["ROLE", "Name"].Find();. Example: var e = elm.path["ROLE1", "Name1"]["ROLE2", "Name2"]["ROLE3", "Name3"].In(w).Find();.

Methods

Name Description
Check(bool, Action<elm>)

Checks or unchecks this checkbox or toggle-button, or selects this radio button. To check/uncheck calls callback function.
Check(bool, string)

Checks or unchecks this checkbox or toggle-button, or selects this radio button. Uses elm.Invoke or elm.SendKeys.
ComboSelect(string, string, double)

Finds and selects an item in the drop-down list of this combo box or drop-down button.
Dispose()

Releases COM object and clears this variable.
Expand(Strings, Action<elm>, double, bool)

Expands multiple treeview control items using a path string.
Expand(Strings, string, double, bool)

Expands multiple treeview control items using a path string.
Expand(bool, Action<elm>, double, bool)

Expands or collapse this expandable UI element (tree item, combo box, expander, dropdown button).
Expand(bool, string, double, bool)

Expands or collapse this expandable UI element (tree item, combo box, expander, dropdown button).
Focus(bool)

Makes this UI element focused for keyboard input.
GetProperties(string, out EProperties)

Gets multiple properties.
GetRect(out RECT, wnd, bool)

Gets location of this UI element in the client area of window w.
GetRect(out RECT, bool)

Gets location of this UI element in screen.
Html(bool)

Gets HTML.
HtmlAttribute(string)

Gets a HTML attribute.
HtmlAttributes()

Gets all HTML attributes.
Invoke()

Performs the UI element's default action (see elm.DefaultAction). Usually it is "click", "press" or similar.
JavaInvoke(string)

Performs one of actions supported by this Java UI element.
MouseClick(Coord, Coord, MButton, int)

Clicks this UI element.
MouseClickD(Coord, Coord, int)

Double-clicks this UI element.
MouseClickR(Coord, Coord, int)

Right-clicks this UI element.
MouseMove(Coord, Coord, int)

Moves the cursor (mouse pointer) to this UI element.
Navigate(string, double)

Gets an adjacent or related UI element - next, child, parent, etc.
PostClick(Coord, Coord, MButton, int)

Posts mouse-click messages to the container window, using coordinates in this UI element.
PostClickD(Coord, Coord)

Posts mouse-double-click messages to the container window, using coordinates in this UI element.
PostClickR(Coord, Coord)

Posts mouse-right-click messages to the container window, using coordinates in this UI element.
ScrollTo()

Scrolls this UI element into view.
Select(ESelect)

Selects or deselects.
SendKeys(params KKeysEtc[])

Makes this UI element focused (elm.Focus) and calls keys.send.
ToString()

Formats string from main properties of this UI element.
WaitFor<T>(Seconds, Func<elm, T>)

Waits for a user-defined state/condition of this UI element. For example enabled, checked, changed name.
WebInvoke(Seconds?, Action<elm>)

Calls elm.Invoke or action and waits until changes the web page (window name and page name).
focused(EFocusedFlags)

Gets the keyboard-focused UI element.
fromEvent(wnd, EObjid, int)

Gets the UI element that generated the event that is currently being processed by the callback function used with API SetWinEventHook or WinEventHook.
fromMouse(EXYFlags)

Gets UI element from mouse cursor (pointer) position.
fromWindow(wnd, EObjid, EWFlags)

Gets UI element of window or control. Or some its standard part - client area, titlebar etc.
fromXY(POINT, EXYFlags)

Gets UI element from point.
fromXY(int, int, EXYFlags)

Gets UI element from point.