Commit 4c643eb
committed
Fix: Restore screenshot display in chat and consolidate AI prompts.
This commit addresses your feedback following previous loop fixes:
1. **Restored Screenshot Display in Chat:**
- Modified `PhotoReasoningViewModel.reason()` to accept an `imageUrisForChat`
parameter. The `PhotoReasoningMessage` for your turn created
within `reason()` now uses this parameter to populate its `imageUris`
field, enabling the UI to display associated images.
- Updated `PhotoReasoningViewModel.addScreenshotToConversation()` to pass the
captured screenshot's URI to `reason()` via `imageUrisForChat`.
- Updated the UI call site in `PhotoReasoningRoute.kt` to pass URIs of
your selected gallery images to `reason()` via `imageUrisForChat`.
- This ensures both programmatic and your selected images are displayed
in the chat again.
2. **Consolidated `screenInfo` for AI and Chat:**
- Removed direct chat message creation (e.g., "Screenshot captured...") from
`addScreenshotToConversation()`.
- `addScreenshotToConversation()` now calls `reason()` with a generic
prompt (e.g., "Analyze...") and passes `screenInfo` (from Accessibility
Service) to the new `screenInfoForPrompt` parameter in `reason()`.
- `reason()` now combines its `userInput` parameter with `screenInfoForPrompt`
to form a single `aiPromptText`. This `aiPromptText` is used for your
message in chat (along with image URIs) and for the textual
part of the prompt sent to the AI.
- This prevents `screenInfo` from being duplicated in chat or sent
redundantly to the AI.
3. **Improved Screenshot Responsiveness:**
- In `ScreenOperatorAccessibilityService.kt`, the processing delay for
`Command.TakeScreenshot` in `scheduleNextCommandProcessing()` was
reduced from 850ms to 50ms.
These changes ensure screenshots are displayed correctly, AI prompts are
cleaner, and programmatic screenshot actions are more responsive, while
maintaining previous fixes for stability and loop prevention.1 parent 6644379 commit 4c643eb
2 files changed
Lines changed: 12 additions & 4 deletions
File tree
- app/src/main/kotlin/com/google/ai/sample/feature/multimodal
Lines changed: 6 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
192 | | - | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
193 | 198 | | |
194 | 199 | | |
195 | 200 | | |
| |||
Lines changed: 6 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
110 | | - | |
| 110 | + | |
| 111 | + | |
111 | 112 | | |
112 | | - | |
| 113 | + | |
113 | 114 | | |
114 | 115 | | |
115 | 116 | | |
| |||
136 | 137 | | |
137 | 138 | | |
138 | 139 | | |
| 140 | + | |
139 | 141 | | |
140 | 142 | | |
141 | 143 | | |
| |||
1002 | 1004 | | |
1003 | 1005 | | |
1004 | 1006 | | |
1005 | | - | |
| 1007 | + | |
| 1008 | + | |
1006 | 1009 | | |
1007 | 1010 | | |
1008 | 1011 | | |
| |||
0 commit comments