Missing Chunks in Streamed Responses When Using the SDK

[frankjony17]

Description:

I conducted several tests and can confirm that the issue arises only when using the SDK.

1. Direct API Calls via Postman:

• All requests were successful.

2. SDK Calls with Debugging: During SDK calls, errors were identified:

• Occasionally, all chunks are returned correctly.
• A significant percentage of the time, the last 3–4 chunks are not returned, and no error is displayed.
• The latest version of the SDK was tested, but the issue remains unresolved.

Direct calls were made to the Mistral model deployed on Azure, following the official SDK documentation; however, the issue persists.

Azure Test Code:

import { MistralAzure } from "@mistralai/mistralai-azure";

const mistral = new MistralAzure({
  apiKey: "some-key",
  endpoint: "https://mistral-small-2409-some-url.ai.azure.com",
});

export async function runMistralSmall() {
  const result = await mistral.chat.stream({
    model: "mistral-small-2409",
    stream: true,
    messages: [
      {
        content:
          "Who is the best French painter? Answer in one short sentence.",
        role: "user",
      },
    ],
  });

  for await (const event of result) {
    // Handle the event
    console.log(event.data.choices[0]);
  }
}

Responses:

• Missing Chunks (the last chunks are not returned):

{
  index: 0,
  delta: { role: 'assistant', content: '' },
  finishReason: null
}
{ index: 0, delta: { content: 'Claude' }, finishReason: null }
{ index: 0, delta: { content: ' Mon' }, finishReason: null }
{ index: 0, delta: { content: 'et' }, finishReason: null }
{ index: 0, delta: { content: ' is' }, finishReason: null }
{ index: 0, delta: { content: ' often' }, finishReason: null }
{ index: 0, delta: { content: ' considered' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' best' }, finishReason: null }
{ index: 0, delta: { content: ' French' }, finishReason: null }
{ index: 0, delta: { content: ' painter' }, finishReason: null }
{ index: 0, delta: { content: ' due' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' his' }, finishReason: null }
{ index: 0, delta: { content: ' significant' }, finishReason: null }
{ index: 0, delta: { content: ' contributions' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' Im' }, finishReason: null }
{ index: 0, delta: { content: 'pression' }, finishReason: null }
{ index: 0, delta: { content: 'ist' }, finishReason: null }
{ index: 0, delta: { content: ' movement' }, finishReason: null }

All Chunks Returned Correctly (occasionally, all chunks are returned):

{
  index: 0,
  delta: { role: 'assistant', content: '' },
  finishReason: null
}
{ index: 0, delta: { content: 'Claude' }, finishReason: null }
{ index: 0, delta: { content: ' Mon' }, finishReason: null }
{ index: 0, delta: { content: 'et' }, finishReason: null }
{ index: 0, delta: { content: ' is' }, finishReason: null }
{ index: 0, delta: { content: ' often' }, finishReason: null }
{ index: 0, delta: { content: ' considered' }, finishReason: null }
{ index: 0, delta: { content: ' one' }, finishReason: null }
{ index: 0, delta: { content: ' of' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' best' }, finishReason: null }
{ index: 0, delta: { content: ' French' }, finishReason: null }
{ index: 0, delta: { content: ' pain' }, finishReason: null }
{ index: 0, delta: { content: 'ters' }, finishReason: null }
{ index: 0, delta: { content: ' due' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' his' }, finishReason: null }
{ index: 0, delta: { content: ' significant' }, finishReason: null }
{ index: 0, delta: { content: ' contributions' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' Im' }, finishReason: null }
{ index: 0, delta: { content: 'pression' }, finishReason: null }
{ index: 0, delta: { content: 'ist' }, finishReason: null }
{ index: 0, delta: { content: ' movement' }, finishReason: null }
{ index: 0, delta: { content: '.' }, finishReason: null }
{ index: 0, delta: { content: '' }, finishReason: 'stop' }

Please investigate this issue as it significantly impacts the reliability of SDK-based integrations.

[petod108]

I got the same random error. To get stream not finished.

[jean-malo]

Hi,
Thanks for opening this issue. I will look into it and get back to you when I have more information.
To speed up the investigation on our side, did you observe this consistently for specific prompts or is it mostly random ?

Thanks!

[jean-malo]

@petod108 are you using Azure too ?

[frankjony17]

Hi, Thanks for opening this issue. I will look into it and get back to you when I have more information. To speed up the investigation on our side, did you observe this consistently for specific prompts or is it mostly random ?

Thanks!

I tested with more than 5 random prompts, small and large ones and the behavior is the same.

[franher]

@jean-malo have you been able to reproduce it? any idea if this is an actual bug on your SDK? Thank you

[franher]

Hello @jean-malo could you share if the issue may be prioritized?

[santiago-perez-axa]

It happens the same for me with the latest version 1.5.0. The last chunks with the usage details are missing most of the times.

import { MistralAzure } from "@mistralai/mistralai-azure";


async function run(){

  const mistral = new MistralAzure({
    endpoint: "...",
    apiKey:'...,
    debugLogger: console
  });

  async function run() {
    const result = await mistral.chat.stream({
      model: "mistral-large-2411",
      messages: [
        {
          content:
            "Hello! talk about the meaning of life?",
          role: "user",
        },
      ],
    });

    let usage;
    for await (const event of result) {
      // Handle the event
      console.log(JSON.stringify(event));
      if(event.data.usage){
        usage= event.data.usage;
      }
    }
    if(!usage){
      console.error('Usage is missing');
    }
  }

  run();
}

(async()=> {
  await run()
})();

{"data":{"id":"574144f5f706403aacf9ecc1ae3d79ed","object":"chat.completion.chunk","created":1750257525,"model":"mistral-large-2411","choices":[{"index":0,"delta":{"content":" and"},"finishReason":null}]}}
{"data":{"id":"574144f5f706403aacf9ecc1ae3d79ed","object":"chat.completion.chunk","created":1750257525,"model":"mistral-large-2411","choices":[{"index":0,"delta":{"content":" what"},"finishReason":null}]}}
Usage is missing

[Andy-d-g]

The issue is still here... in my case with vibe cli