test: rework live tests to use vcr#50
Merged
Merged
Conversation
![managerbot-app[bot]](https://avatars.githubusercontent.com/in/853044?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed Changes
Managerbot reviewed 15 out of 72 changed files in this pull request and generated 9 comments.
Files not reviewed (57)
- test/fixtures/anthropic_stream/text_events.json
- test/fixtures/anthropic_stream/thinking_events.json
- test/fixtures/anthropic_stream/tool_events.json
- test/fixtures/openai_codex_stream/reasoning_events.json
- test/fixtures/openai_codex_stream/text_events.json
- test/fixtures/openai_codex_stream/tool_events.json
- test/fixtures/openai_responses_stream/reasoning_events.json
- test/fixtures/openai_responses_stream/text_events.json
- test/fixtures/openai_responses_stream/tool_events.json
- test/fixtures/openai_stream/reasoning_events.json
- test/fixtures/openai_stream/text_events.json
- test/fixtures/openai_stream/tool_events.json
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_anthropic_apikey_messages_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_anthropic_apikey_messages_none_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_openai_apikey_completions_none_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_openai_apikey_completions_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_openai_apikey_responses_none_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_openai_apikey_responses_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/cache_test_rb/test_live_cache_read_tokens_on_second_turn_openai_oauth_codex_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_media_test_rb/test_live_handoff_media_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_media_test_rb/test_live_handoff_media_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_media_test_rb/test_live_handoff_media_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_media_test_rb/test_live_handoff_media_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_test_rb/test_live_handoff_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_test_rb/test_live_handoff_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_test_rb/test_live_handoff_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/handoff_test_rb/test_live_handoff_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/prompt_too_long_test_rb/test_live_prompt_too_long_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/prompt_too_long_test_rb/test_live_prompt_too_long_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/prompt_too_long_test_rb/test_live_prompt_too_long_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/prompt_too_long_test_rb/test_live_prompt_too_long_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/prompt_too_long_test_rb/test_live_prompt_too_long_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_image_test_rb/test_live_image_streaming_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_image_test_rb/test_live_image_streaming_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_image_test_rb/test_live_image_streaming_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/stream_image_test_rb/test_live_image_streaming_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_image_test_rb/test_live_image_streaming_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_reasoning_test_rb/test_live_basic_thinking_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_reasoning_test_rb/test_live_basic_thinking_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_reasoning_test_rb/test_live_basic_thinking_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/stream_reasoning_test_rb/test_live_basic_thinking_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_reasoning_test_rb/test_live_basic_thinking_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_basic_tool_call_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_basic_tool_call_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_basic_tool_call_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_basic_tool_call_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_basic_tool_call_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_multi_turn_tool_streaming_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_multi_turn_tool_streaming_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_multi_turn_tool_streaming_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_multi_turn_tool_streaming_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_multi_turn_tool_streaming_openai_oauth_codex_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_text_streaming_anthropic_apikey_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_text_streaming_anthropic_oauth_messages_claude-sonnet-4-20250514.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_text_streaming_openai_apikey_completions_gpt-5_1.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_text_streaming_openai_apikey_responses_gpt-5_4.yml
- test/fixtures/vcr_cassettes/integration/live/stream_test_rb/test_live_text_streaming_openai_oauth_codex_gpt-5_4.yml
Comments suppressed due to low confidence (1)
-
test/integration/live/stream_test.rb:276
- (change_propagated) The `define_stream_tests_for` method previously accepted a `name:` keyword argument that was unused in the test name construction (provider and model were used instead). Removing it is correct, but verify that any external callers have been updated — the call sites in this file have been updated, which is good.
| end | ||
|
|
||
| def vcr_cassette_path(cassette_name) | ||
| VCR::Cassette.new(cassette_name).file |
There was a problem hiding this comment.
VCR::Cassette.new(cassette_name).file instantiates a new cassette object just to get the file path. This is an implementation detail of VCR that may break across versions. Consider using File.join(VCR.configuration.cassette_library_dir, "#{cassette_name}.yml") instead for a more stable approach.
| without_vcr do | ||
| run_handoff_for(provider:, model:) | ||
| end | ||
| with_vcr_adapter(provider:, model:,) do |adapter| |
There was a problem hiding this comment.
There is a trailing comma after model: in with_vcr_adapter(provider:, model:,). While Ruby allows trailing commas in method calls, it's unconventional and inconsistent with the rest of the codebase. Consider removing it.
| without_vcr do | ||
| run_handoff_for(provider:, model:) | ||
| end | ||
| with_vcr_adapter(provider:, model:,) do |adapter| |
There was a problem hiding this comment.
There is a trailing comma after model: in with_vcr_adapter(provider:, model:,). While Ruby allows trailing commas in method calls, it's unconventional and inconsistent with the rest of the codebase. Consider removing it.
| thinking_started = false | ||
| thinking_chunks = "" | ||
| thinking_completed = false | ||
| response = adapter.stream(prompt, reasoning:,) do |event| |
There was a problem hiding this comment.
There is a trailing comma in adapter.stream(prompt, reasoning:,). This is inconsistent style. Consider removing the trailing comma.
| ANTHROPIC_STREAM_TOOL_EVENTS_FIXTURE = JSON.parse(File.read(File.expand_path("../fixtures/anthropic_stream/tool_events.json", __dir__)), symbolize_names: true) | ||
| ANTHROPIC_STREAM_THINKING_EVENTS_FIXTURE = JSON.parse(File.read(File.expand_path("../fixtures/anthropic_stream/thinking_events.json", __dir__)), symbolize_names: true) | ||
|
|
||
| class StreamTest < Test |
There was a problem hiding this comment.
This file defines a class named StreamTest which conflicts with any other StreamTest class in the test suite. It's being deleted here, but it's worth noting that the naming collision was a pre-existing bug — the new stream_reasoning_test.rb uses StreamReasoningTest which avoids this issue.
we are using vcr for live tests so we can use that as part of the debugging process is my hunch vs all this random recording crap
trying to create a good base line of features that all providers would support
billybonks
force-pushed
the
test/add-fixtures
branch
from
907660b to
05e513a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
it's a pain to have them live using vcr is better we can always regenerate if we want to do live test again, kept the handoff fixutre creators for now