Open
Conversation
- Add /api/a11y/native-tree endpoint returning elements in screen reader traversal order - Add NativeScreenReaderEntry model with label, hint, value, role, traits, isHeading, bounds - Implement WalkIosA11yTree using UIAccessibilityElement/UIAccessibilityContainer (VoiceOver ground truth) - Implement WalkAndroidA11yTree using AccessibilityNodeInfo per-view (TalkBack ground truth): - Use Clickable/LongClickable/Checkable/hasLabel as focusability predicate instead of heuristics - Remove VisibleToUser filter — TalkBack navigates to off-screen elements in scrollable containers - Use NoHideDescendants check to correctly prune entire subtrees - Use object identity for cycle detection instead of view.Id (multiple views can share same ID) - Implement WalkWindowsA11yTree using AutomationPeer (Narrator ground truth) - Implement WalkMacA11yTree using NSAccessibility Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- FrameworkElement does not have Background in WinUI; cast to Control or Panel before accessing the property (CS1061) - AutomationPeer.IsContentElementCore() is protected; use the public IsContentElement() wrapper instead (CS0122) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant new accessibility and UI inspection features to the DevFlow agent, including new API endpoints for accessibility tree inspection and element highlighting, as well as enhancements to the element information model. The changes enable clients to retrieve both a semantic accessibility tree and the native screen reader traversal order, and to visually highlight elements (with auto-clear and scroll-into-view support). The codebase is also extended to collect and expose detailed accessibility and color information for UI elements.
Accessibility inspection and API enhancements:
/api/accessibilityand/api/a11y/native-treeto expose the accessibility tree and the native screen reader order, respectively. The/api/accessibilityendpoint returns a flattened list of elements with accessibility information, while/api/a11y/native-treereturns the exact order in which a platform screen reader would visit elements. [1] [2]ElementInfomodel to includeAccessibilityInfo,EffectiveTextColor, andEffectiveBackgroundColorproperties, enabling richer inspection of UI and accessibility attributes. [1] [2]AccessibilityInfoandNativeScreenReaderEntryclasses to represent native accessibility properties and screen reader traversal entries.VisualTreeWalkerto populate accessibility info using both MAUI semantic properties and platform-specific details, and to provide the native accessibility tree and mapping between native views and MAUI element IDs. [1] [2] [3]Element highlight and UI interaction:
/api/action/highlightendpoint and supporting logic for visually highlighting elements, with support for custom color, auto-clear duration, scroll-into-view, and fallback bounds for native-only elements. The highlight overlay is managed with cancellation and cleanup logic. [1] [2] [3]These changes provide a foundation for advanced accessibility testing and visual debugging workflows in the DevFlow agent.
Most important changes:
Accessibility APIs and models
/api/accessibilityand/api/a11y/native-treeendpoints for retrieving accessibility information and native screen reader traversal order, including logic for flattening and counting accessibility elements. [1] [2]AccessibilityInfoandNativeScreenReaderEntryclasses, and extendedElementInfowith accessibility and effective color properties. [1] [2]VisualTreeWalkerto populate accessibility info from MAUI and native APIs, and to provide the native accessibility tree and view-to-ID mapping. [1] [2] [3]Highlighting and UI feedback
/api/action/highlightendpoint andHighlightRequestclass, supporting visual highlighting of elements with configurable color, duration, scroll-into-view, and fallback to raw bounds. Includes logic for auto-clear and cancellation of highlights. [1] [2] [3]General improvements
RegisterRoutes()to include new endpoints for accessibility and highlighting actions.