1# Automatic clicks (for developers) 2 3Automatic clicks is a Chrome OS feature to automatically generate mouse events 4when the cursor dwells in one location. 5 6 7Dwell control supports users with motor impairments. A user who is unable to 8use a mouse or unable to click with a mouse, but who is able to control the 9cursor position (often using an alternative input method) will need to use 10dwell control to perform mouse or trackpad actions, including left-click, 11right-click, double-click, click-and-drag and scroll. 12 13## Using automatic clicks 14 15Go to Chrome settings, Accessibility settings, “Manage accessibility Features”, 16and in the “mouse and input” section enable “Automatically click when the 17cursor stops”. You can adjust timing, radius, stabilization, and whether to 18revert to a left-click after another type of action has been taken from the 19settings page. 20 21 22A on-screen menu bubble will appear in the corner. Dwell over this menu to 23change the action taken, or pause the feature. There is also a button to 24re-position the menu to another corner of the screen. 25 26## Reporting bugs 27 28Use bugs.chromium.org, filing bugs under the component UI>Accessibility with 29the label “autoclick” (or, use 30[this template](https://bugs.chromium.org/p/chromium/issues/entry?summary=Autoclick%20-%20&status=Available&cc=katie%40chromium.org%2C%20qqwangxin%40google.com&labels=Pri-3%2C%20autoclick%2C&components=UI>Accessibility)). 31 32 33Open bugs have the label 34“[autoclick](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=label%3Aautoclick)”. 35 36## Developing 37 38### Code location 39 40Automatic clicks code lives mainly in three places: 41 42- A controller, event-rewriter, and widgets to draw around click locations, in 43ash/autoclick/ 44 45- UI through menu bubbles and their controllers, in 46ash/system/accessibility/autoclick* 47 48- A component extension to provide Accessibility tree information, in 49chrome/browser/resources/chromeos/accessibility/autoclick/ 50 51In addition, there are settings for automatic clicks in 52chrome/browser/resources/settings/a11y_page/manage_a11y_page.* 53 54 55### Tests 56 57Tests are in ash_unittests and in browser_tests: 58 59``` 60out/Release/ash_unittests --gtest_filter=”Autoclick*” 61out/Release/browser_tests --gtest_filter=”Autoclick*” 62``` 63 64### Debugging 65 66Developers can add log lines to any of the autoclick C++ files and see output 67in the console. To debug the Autoclick extension, the easiest way is from an 68external browser. Start Chrome OS on Linux with this command-line flag: 69 70``` 71out/Release/chrome --remote-debugging-port=9222 72``` 73 74Now open http://localhost:9222 in a separate instance of the browser, and debug 75the Autoclick extension background page from there. 76 77## How it works 78 79AutoclickController is a pre-target EventHandler with very high priority, 80which means it can receive and act on mouse events before other parts of 81Chrome OS get them. 82 83 84AutoclickController::OnMouseEvent receives mouse events and checks whether 85the event is close to the current dwell target (in which case dwell count-down 86should continue), or far enough away to clear the target and set a new one. 87 88 89There is a small delay before the user sees the count-down timer ring UI appear 90(AutoclickRingHandler) show up, which is controlled by the start_gesture_timer_. 91Performing the click is controlled by the autoclick_timer_. 92 93 94When the autoclick_timer_ completes it calls 95AutoclickController::DoAutoclickAction. This function first checks if the 96target point is over either of the autoclick bubbles, the menu or the scroll 97bubble. If it is over a bubble the gesture will not be handled with a synthetic 98event, but instead sent directly to that menu. This keeps focus from shifting 99and things like dialogs or context menus from closing when the user interacts 100with an autoclick bubble. But, if the target was not over the bubble, a 101synthetic event is generated as follows: 102 103### Left-click, right-click and double-click 104 105Synthetic mouse events for ui::ET_MOUSE_PRESSED and ui::ET_MOUSE_RELEASED are 106created with the appropriate mouse button flags, and sent to the WindowTreeHost 107under the target point for processing. For double-click, a second press and 108release pair are also sent. 109 110### Click-and-drag 111 112A synthetic mouse event for ui::ET_MOUSE_PRESSED is created at the first dwell. 113An AutoclickDragEventRewriter is enabled and begins re-writing all MOUSE_MOVED 114events to MOUSE_DRAGGED events to create the illusion of a drag. This occurs 115in AutoclickDragEventRewriter::RewriteEvent. 116 117A final synthetic mouse event for ui::ET_MOUSE_RELEASED is created at the 118second dwell, and the AutoclickDragEventRewriter is disabled. 119 120### Scroll 121 122On a dwell during scroll, the scroll target point is changed. No scroll events 123are generated from AutoclickController until the user hovers over the scroll 124pad buttons (AutoclickScrollButton class). The AutoclickScrollButtons track 125whether they are currently hovered, and while hovered they fire a repeating 126timer to ask the AutoclickController to do a scroll event. 127 128#### Scroll location 129 130By default, the scroll position is at the center of the active screen. Users 131may dwell anywhere on the screen to change the scroll location. 132 133 134When the scroll location is changed, the AutoclickController will request the 135bounds of the nearest scrollable view from the Autoclick component extension 136via the AccessibilityPrivate API. The Autoclick component extension has access 137to accessibility tree information, and using a HitTest is able to find the view 138at the scroll location, then walks up the tree to find the first view which can 139scroll, or stops at the nearest window or dialog bounds. This logic takes place 140in autoclick.js, onAutomationHitTestResult_. When the scrolling location is 141found, the bounds of the scrollable area are highlighted with a focus ring. 142In addition, the bounds are sent back through the AccessibilityPrivate API, 143routed to the AutoclickController, which passes it via the 144AutoclickMenuBubbleController to the AutoclickScrollBubbleController, which 145does layout accordingly. 146 147### Bubble Menus: interface and positioning 148 149The AutoclickController owns the AutoclickMenuBubbleController, which controls 150the widget and AutoclickMenuView view for the autoclick bubble menu. 151AutoclickMenuView inherits from TrayBubbleView for styling, and contains all 152the buttons to change click types, pause, and update the menu position. 153 154 155Similarly, AutoclickMenuBubbleController also owns 156AutoclickScrollBubbleController so that it can pass on messages about 157positioning and activation from AutoclickController. 158AutoclickScrollBubbleController owns the widgets and AutoclickScrollView for 159the autoclick scroll bubble. AutoclickScrollView inherits from TrayBubbleView 160for styling, and contains the scroll pad buttons, including all the logic to 161draw the custom shape buttons for left/right/up/down scrolling. 162 163#### Menu positioning 164 165The autoclick bubble menu can be positioned in the four corners of the screen 166and defaults to the same location as the volume widget (which depends on 167LTR/RTL language). AutoclickMenuBubbleController takes a preferred 168AutoclickMenuPosition enum and uses that to determine the best position for 169the menu in AutoclickMenuBubbleController::SetPosition. This function finds 170the ideal corner of the screen, then uses CollisionDetectionUtils (also used 171by Picture-in-Picture) to refine the position to avoid collisions with system 172UI. 173 174#### Scroll bubble positioning 175 176The scroll bubble starts out anchored to the automatic clicks bubble menu, but 177if the user selects a new scroll point it will move. When a scroll point is 178selected, if the scrollable region found by the Autoclick component extension 179is large enough, the scroll bubble will be anchored near the scroll point 180itself, similarly to the way the context menu is anchored near the cursor on 181a right click. When the scrollable region is small, the scroll bubble will be 182anchored to the closest side of the scrollable region to the scroll point, as 183long as there is space for it on that side. 184 185#### Clicking on the bubble menus 186 187The AutomaticController cannot generate synthetic click events over the 188bubbles, because that would cause context and focus changes. For example, if 189the user has a drop-down menu open, clicking the autoclick menu bubble will 190cause the drop-down to close. Instead, the AutoclickController must check to 191see if an event will take place over a bubble menu, and if so, request that 192AutoclickMenuBubbleController forward the event to the bubble via 193AutoclickMenuBubbleController::ClickOnBubble. This generates a synthetic mouse 194event which does not propagate through the system, so there is no focus or 195context change, allowing users to continue to interact with whatever was on 196screen. 197 198## For Googlers 199 200Googlers could check out the automatic clicks feature design docs for more 201details on design as well as autoclick’s UMA. 202 203- Overall product design, [go/chromeos-dwell-design](go/chromeos-dwell-design) 204 205- On-screen menu design, 206[go/chromeos-dwell-menu-design](go/chromeos-dwell-menu-design) 207 208- Scrolling design, 209[go/chromeos-dwell-scroll-design](go/chromeos-dwell-scroll-design) 210 211- UX mocks, [go/cros-dwell-ux](go/cros-dwell-ux) 212