Try to keep tests going

[tmeschter]

In our integration tests it is possible that the agent will pause and ask for permission to continue before completing the operation. E.g. it may create a deployment plan, then pause and ask for permission to carry it out. Alternatively it may ask a clarifying question, such as what Azure region, services, or subscription to use for the deployment.

Since these are meant to be automated tests we generally provide a single follow up response that is provided regardless of what question is being asked: "Go with recommended options." This works in that it allows the test to proceed. However, it is entirely possible that the agent will pause for input more than once. If that happens the test will hang waiting on input that will never come, and will eventually be killed when it hits a timeout.

Here we try a small change, updating the follow up from "Go with recommended options" to "Continue with recommended options until complete.". The intent is to have the agent continue until the operation is complete and to use its best judgement rather than asking the user for input.

Fixes #1481.

[Copilot]

Pull request overview

Updates integration test follow-up prompts to encourage the Copilot agent to continue autonomously when it asks clarifying questions during non-interactive runs.

Changes:

• Replaces "Go with recommended options." with "Continue with recommended options until complete." in several integration test suites.
• Applies the new follow-up wording across validate/prepare/upgrade/resource-visualizer/deploy tests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/azure-validate/integration.test.ts	Updates follow-up prompt used in validation-focused scenarios.
tests/azure-upgrade/integration.test.ts	Updates non-interactive follow-up prompt for upgrade skill invocation tests.
tests/azure-resource-visualizer/integration.test.ts	Updates follow-up prompt used in non-interactive, end-to-end diagram generation tests.
tests/azure-prepare/integration.test.ts	Updates the shared follow-up prompt used across many prepare prompts.
tests/azure-deploy/integration.test.ts	Updates follow-up prompt used in deploy skill-invocation tests.

Comments suppressed due to low confidence (2)

tests/azure-validate/integration.test.ts:129

• This follow-up prompt encourages the agent to proceed “until complete”, which can include actual deployment. In these deployment-validation tests you explicitly assert that azure-deploy must NOT be invoked, so this wording can make the test flakier by nudging the agent past validation. Consider a follow-up that explicitly scopes to validation/planning only (e.g., proceed with recommended options for validation and stop before deployment).

    const FOLLOW_UP_PROMPT = ["Continue with recommended options until complete."];

    test("terminates at validation for static whiteboard web app", () => withTestResult(async () => {
      const agentMetadata = await agent.run({
        prompt: "Create a static whiteboard web app and deploy to Azure.",
        nonInteractive: true,
        followUp: FOLLOW_UP_PROMPT,
        shouldEarlyTerminate: (metadata) =>
          hasValidationCommand(metadata) || isSkillInvoked(metadata, "azure-deploy"),
      });

tests/azure-resource-visualizer/integration.test.ts:90

• This follow-up prompt is still only sent once. If the agent pauses for clarification multiple times, there won’t be another automatic reply and the run can stall until timeout. Consider providing several repeated follow-up responses (or adjusting the runner to auto-repeat the last response).

  const FOLLOW_UP_PROMPT = ["Continue with recommended options until complete."];

  describe("resource-group-visualization", () => {
    test("generates architecture diagram for a resource group", () => withTestResult(async () => {
      let workspacePath: string | undefined;

      const agentMetadata = await agent.run({
        prompt: "Generate a Mermaid diagram showing my Azure resource group architecture. Save the diagram to an architecture.md file in the current working directory.",
        nonInteractive: true,
        followUp: FOLLOW_UP_PROMPT,
        setup: async (workspace: string) => {