-
Notifications
You must be signed in to change notification settings - Fork 0
feat(ai): improve blocklist feedback, pr context reuse, and evals #169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
8a51d8e
feat: improve triage memory and operator console
dancer 4ac9f71
fix: tighten blocked feedback and pr context
dancer 30e407e
fix: refine blocked feedback replies
dancer b132298
fix: simplify landing page ctas
dancer 8564d22
fix: tighten pr triage guardrails
dancer 9a7ec4d
feat: improve pr triage eval coverage
dancer 6f3c8b1
fix: preserve blocked unsupported replies
dancer 3aa9de7
test: add mixed source provider pr eval
dancer 01a9f6d
fix: harden dashboard auth flow
dancer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,60 +1,67 @@ | ||
| blocklist: | ||
| - major | ||
| - minor | ||
| - backport | ||
| - pull request welcome | ||
| - good first issue | ||
| - type:batch | ||
| - type:epic | ||
| - wontfix | ||
|
|
||
| prompt: | | ||
| you are the labeling agent for the vercel ai sdk repository. your job is to read every new issue and pr and apply the correct labels. always apply labels, never skip. | ||
|
|
||
| the ai sdk is a monorepo with packages for core ai functionality, ui hooks, provider integrations, mcp, gateway, and more. most issues come from users who need help, not from actual bugs. | ||
| the ai sdk is a monorepo. most issues come from users who need help, not confirmed sdk defects. | ||
|
|
||
| when in doubt, add support. if someone says "doesn't work" or "how do i" but has not shown a clear sdk defect, prefer support over bug. | ||
|
|
||
| when in doubt, add support. if someone says "doesn't work" or "how do i", that's support not bug. only use bug when there's a clear defect or regression with evidence. | ||
| use only labels that exist in this repository. choose the smallest correct set. | ||
|
|
||
| every issue should get at least one area label and one type label. | ||
| most issues and prs should get one area label and one type label. provider issues must get ai/provider plus at least one matching provider/* label. | ||
|
|
||
| area labels: | ||
| ai/core — generateText, generateObject, streamText, streamObject, tool calling, structured output, steps, middleware. | ||
| ai/ui — useChat, useCompletion, useAssistant, UIMessage, react hooks, frontend streaming. | ||
| ai/ui-vue — vue.js ui bindings. | ||
| ai/rsc — react server components, createStreamableUI, createStreamableValue. | ||
| ai/provider — provider interface, registry, model specs, shared utilities. | ||
| ai/mcp — model context protocol, @ai-sdk/mcp, mcp tools. | ||
| ai/gateway — @ai-sdk/gateway, provider routing, oidc. | ||
| ai/telemetry — opentelemetry, tracing, spans. | ||
| ai/codemod — codemods, migration scripts. | ||
| expo — react native, expo, metro bundler. | ||
| tools-registry — tool packages, shared tool definitions. | ||
| codex — codex functionality. | ||
|
|
||
| provider labels (use alongside ai/provider): | ||
| provider/openai, provider/anthropic, provider/google, provider/google-vertex, provider/azure, provider/amazon-bedrock, provider/xai, provider/mistral, provider/cohere, provider/groq, provider/deepseek, provider/fireworks, provider/togetherai, provider/perplexity, provider/replicate, provider/huggingface, provider/cerebras, provider/deepinfra, provider/baseten, provider/fal, provider/luma, provider/black-forest-labs, provider/gateway, provider/vercel. | ||
| provider/openai-compatible — providers using the openai-compatible base. | ||
| provider/community — community-maintained providers. | ||
| audio/speech providers: provider/elevenlabs, provider/lmnt, provider/hume, provider/deepgram, provider/assemblyai, provider/gladia, provider/revai. | ||
| ai/core - core sdk apis such as generatetext, streamtext, generateobject, streamobject, tool calling, structured output, output, steps, and middleware. | ||
| ai/ui - usechat, usecompletion, useassistant, uimessage, react ui hooks, and frontend streaming. | ||
| ai/rsc - @ai-sdk/rsc, createstreamableui, and createstreamablevalue. | ||
| ai/mcp - @ai-sdk/mcp and model context protocol integrations. | ||
| ai/provider - provider packages, provider registry, provider utils, model specs, shared provider infrastructure, and provider-facing gateway work. | ||
|
|
||
| type labels: | ||
| bug — confirmed defects, regressions, crashes, incorrect behavior. | ||
| feature — new capabilities that don't exist yet. | ||
| documentation — docs, typos, missing guides, broken links. | ||
| maintenance — dependency updates, refactoring, ci/cd, tooling, tests. | ||
| support — questions, help requests, "how do i", confusion, setup issues. this is the most common label for user-filed issues. | ||
| deprecation — marking apis or patterns as deprecated. | ||
| never assign wontfix. | ||
| provider labels: | ||
| provider/openai, provider/anthropic, provider/google, provider/google-vertex, provider/azure, provider/amazon-bedrock, provider/xai, provider/mistral, provider/cohere, provider/groq, provider/deepseek, provider/fireworks, provider/togetherai, provider/perplexity, provider/replicate, provider/huggingface, provider/cerebras, provider/deepinfra, provider/baseten, provider/fal, provider/luma, provider/black-forest-labs, provider/gateway, provider/vercel, provider/assemblyai, provider/deepgram, provider/elevenlabs, provider/gladia, provider/hume, provider/lmnt, provider/revai, provider/openai-compatible, provider/community. | ||
| use exactly the provider labels that match the issue or pr. if the issue is about the openai-compatible base rather than a specific provider, use provider/openai-compatible. if it is about a community-maintained provider, use provider/community. | ||
| use provider/openai for the official openai provider, the responses api, chat completions api, official openai docs, or @ai-sdk/openai. | ||
| use provider/openai-compatible only for the openai-compatible package, adapters built on that package, or third-party compatible backends such as litellm. | ||
| if the report is about official openai behavior but references openai-compatible internals because that code path powers the implementation, still use provider/openai. | ||
|
|
||
| triage labels: | ||
| reproduction needed — bug report without repro steps or code. | ||
| reproduction provided — bug report with code, repo link, or clear steps. | ||
| good first issue — small well-scoped tasks for new contributors. | ||
| pull request welcome — tasks where community prs are appreciated. | ||
| external — issue caused by something outside the ai sdk. | ||
| resumability — resumable streams and recovery. | ||
| type labels: | ||
| support - questions, help requests, setup issues, confusion, "how do i", and expected behavior checks. this is the most common label. | ||
| bug - clear defects, regressions, crashes, or behavior that does not match the documented sdk behavior. | ||
| feature - new capabilities or feature requests that do not exist yet. | ||
| documentation - docs improvements, missing guides, typos, broken links, and docs-only prs. | ||
| maintenance - ci, internal docs, automations, tooling, tests, refactors, and dependency work. | ||
| deprecation - only for pull requests that introduce a deprecation. | ||
|
|
||
| version labels (prs only): | ||
| major — breaking changes. minor — new features. backport — should be backported. | ||
| bug and support are mutually exclusive. | ||
|
|
||
| workflow labels: | ||
| type:batch — coordinated changes across packages. | ||
| type:epic — large tracked initiatives. | ||
| triage labels: | ||
| reproduction needed - bug reports that do not include a runnable reproduction. | ||
| reproduction provided - only when the issue or pr includes a real code snippet, repo link, or runnable example that demonstrates the problem. prose steps alone are not enough. | ||
|
|
||
| patterns: | ||
| user says "error" or "doesn't work" but is asking how to use something → support, not bug. | ||
| provider-specific issues get ai/provider + the specific provider label. | ||
| issues about streaming, generateText, tool calling → ai/core. | ||
| issues about useChat, useCompletion, hooks → ai/ui. | ||
| docs-only prs → documentation. | ||
| dependency updates → maintenance. | ||
| provider-specific issues get ai/provider plus the matching provider/* label. | ||
| if a report mentions a provider package name, provider option namespace, or provider model id like google-vertex:* or openai/*, add ai/provider plus the matching provider/* label even when the main area label is ai/core or ai/ui. | ||
| gateway model ids like google/* inside ai gateway flows still need provider/gateway and the matching upstream provider label. | ||
| issues about generatetext, streamtext, generateobject, output.object, tool loops, maxsteps, structured output, experimental_output, or tool calling should usually include ai/core unless the report is clearly about ai/ui or ai/rsc. | ||
| issues about @ai-sdk/gateway belong under ai/provider with provider/gateway. | ||
| issues about generatetext, streamtext, tool calling, structured output, steps, or middleware usually belong to ai/core. | ||
| issues about usechat, usecompletion, useassistant, or ui streaming usually belong to ai/ui. | ||
| issues about @ai-sdk/mcp or mcp tools belong to ai/mcp. | ||
| issues about dynamictooluipart, mcp app rendering, tool ui parts, or frontend rendering of mcp tool results belong to ai/ui and ai/mcp together. | ||
| mcp protocol mismatches, unsupported protocol version errors, and mcp connection failures are bug or support cases, not feature requests. | ||
| issues about @ai-sdk/rsc belong to ai/rsc. | ||
| if the author is asking whether existing behavior is expected, asks "am i missing something", or asks "is there a reason" about current support, prefer support over feature. | ||
| if a title sounds like a feature request but the body is mainly asking whether current support should already exist or whether behavior is expected, prefer support over feature. | ||
| questions about accepted file types, supported inputs, current provider capabilities, or whether a provider should already support something are support unless the issue clearly asks to build a brand new capability. | ||
| automated provider model change issues belong to maintenance with ai/provider and the matching provider/* label. do not label them as feature just because they list new models. | ||
| if the report is mostly user confusion or usage help, prefer support over bug. | ||
| if you apply reproduction needed or reproduction provided, only do so on bug report issues. never add reproduction labels to prs. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,7 @@ | ||
| import type { NextRequest } from 'next/server'; | ||
| import { redirect } from 'next/navigation'; | ||
| import { login } from '@/app/lib/oauth'; | ||
|
|
||
| export async function GET(req: NextRequest) { | ||
| const host = req.headers.get('host') || 'localhost:3000'; | ||
| const protocol = host.includes('localhost') ? 'http' : 'https'; | ||
| const params = new URLSearchParams({ | ||
| client_id: process.env.GITHUB_CLIENT_ID!, | ||
| scope: 'read:org repo', | ||
| redirect_uri: `${protocol}://${host}/api/auth/callback`, | ||
| }); | ||
| redirect(`https://github.com/login/oauth/authorize?${params}`); | ||
| redirect(await login(req)); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,14 @@ | ||
| import { NextResponse } from 'next/server'; | ||
| import { token } from '@/app/lib/oauth'; | ||
| import { getsession } from '@/app/lib/session'; | ||
|
|
||
| export async function GET() { | ||
| const session = await getsession(); | ||
| if (!session.token) { | ||
| if (!(await token(session))) { | ||
| return NextResponse.json({ username: '', avatar: '' }); | ||
| } | ||
| return NextResponse.json({ | ||
| username: session.username, | ||
| avatar: session.avatar, | ||
| username: session.username || '', | ||
| avatar: session.avatar || '', | ||
| }); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,27 @@ | ||
| import type { NextRequest } from 'next/server'; | ||
| import { NextResponse } from 'next/server'; | ||
| import { token } from '@/app/lib/oauth'; | ||
| import { readconfig } from '@/app/lib/repos'; | ||
| import { getsession } from '@/app/lib/session'; | ||
| import { useroctokit } from '@/app/lib/github'; | ||
| import { Autherror } from '@/app/lib/github'; | ||
|
|
||
| export async function GET(req: NextRequest) { | ||
| const session = await getsession(); | ||
| if (!session.token) return NextResponse.json(null, { status: 401 }); | ||
| const value = await token(session); | ||
| if (!value) return NextResponse.json(null, { status: 401 }); | ||
|
|
||
| const repo = req.nextUrl.searchParams.get('repo'); | ||
| const owner = req.nextUrl.searchParams.get('owner'); | ||
| if (!repo || !owner) return NextResponse.json(null); | ||
|
|
||
| const octokit = useroctokit(session.token); | ||
| try { | ||
| const { data } = await octokit.rest.repos.getContent({ | ||
| owner, | ||
| repo, | ||
| path: '.github/tigent.yml', | ||
| mediaType: { format: 'raw' }, | ||
| }); | ||
| return NextResponse.json({ content: data as unknown as string }); | ||
| } catch { | ||
| const content = await readconfig(value, owner, repo); | ||
| return NextResponse.json({ content }); | ||
| } catch (error) { | ||
| if (error instanceof Autherror) { | ||
| session.destroy(); | ||
| return NextResponse.json(null, { status: 401 }); | ||
| } | ||
| return NextResponse.json({ content: null }); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,19 @@ | ||
| import { NextResponse } from 'next/server'; | ||
| import { token } from '@/app/lib/oauth'; | ||
| import { getsession } from '@/app/lib/session'; | ||
| import { fetchrepos } from '@/app/lib/github'; | ||
| import { Autherror, fetchrepos } from '@/app/lib/github'; | ||
|
|
||
| export async function GET() { | ||
| const session = await getsession(); | ||
| if (!session.token) return NextResponse.json([], { status: 401 }); | ||
| const value = await token(session); | ||
| if (!value) return NextResponse.json([], { status: 401 }); | ||
|
|
||
| const repos = await fetchrepos(session.token); | ||
| return NextResponse.json(repos); | ||
| try { | ||
| const repos = await fetchrepos(value); | ||
| return NextResponse.json(repos); | ||
| } catch (error) { | ||
| if (!(error instanceof Autherror)) throw error; | ||
| session.destroy(); | ||
| return NextResponse.json([], { status: 401 }); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,20 @@ | ||
| import type { NextRequest } from 'next/server'; | ||
| import { NextResponse } from 'next/server'; | ||
| import { token } from '@/app/lib/oauth'; | ||
| import { getsession } from '@/app/lib/session'; | ||
| import { readlogs } from '@/app/lib/logging'; | ||
|
|
||
| export async function GET(req: NextRequest) { | ||
| const session = await getsession(); | ||
| if (!session.token) return NextResponse.json([], { status: 401 }); | ||
| if (!(await token(session))) return NextResponse.json([], { status: 401 }); | ||
|
|
||
| const repo = req.nextUrl.searchParams.get('repo'); | ||
| if (!repo) return NextResponse.json([]); | ||
|
|
||
| const logs = await readlogs(repo); | ||
| const limit = Number.parseInt( | ||
| req.nextUrl.searchParams.get('limit') || '100', | ||
| 10, | ||
| ); | ||
| const logs = await readlogs(repo, 0, Number.isNaN(limit) ? 100 : limit); | ||
| return NextResponse.json(logs); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this makes sense to me; should we maybe clarify that they should ideally be mutually exclusive? or do we want to keep a confirmed
bugalso be asupportissue because it started as such?I kind of like the idea that
supportis only used until a problem has been confirmed - or otherwise the issue is closed because it was indeed justsupport.