Android: speech rate and backend switching are not reflected properly

[diamondStar35]

Hello.

First of all, I'm sorry if it seems like I'm opening too many discussions or issues.

I have encountered 2 behavioral problems when integrating Prism into an Android project, and not really sure if they are bugs or that is the intended behavior.

The first issue is that, when using the TTS backend and not the Android screen-reader one, I have an issue where speech rate is not being reflected properly in Prism. It doesn't even depend on the system's default speech rate because:

At first when I used that backend, it was so slow. I tried increasing the system's speech rate, it did not help. The only thing that really helped a little, is increasing speech rate through the TTS engine itself, but even so, I think Prism does not map this correctly.

Normally when I use the TTS engine, setting speech rate to 50% is not slow. However, in Prism it's very slow, even when setting speech rate to 1.0 in Prism itself. I do not know what is the problem, but I can provide any needed info.

The other problem, which only happens on Android: when switching to another speech backend, Prism completely stops speech and I have to restart my application in order for the new backend to take effect. Is this a known bug or something wrong with my code?

The only reason I mentioned this here is that it never happened to me neither on Windows or Linux.

Regards.

[ethindp]

Hi there,

Can you provide more info?

Specifically, the speech rate thing is a bit weird. In Android, there is no upper limit to speech rate. So, in theory, the upper bound could be infinity. This is obviously a huge problem for Prism, since we would like "1.0" to map to the maximum rate supported by the backend. To alleviate this, I did a lot of digging and decided to use what the settings app uses as a hard limit, which is 600 percent. However, the rate may be slow because I am using the wrong numeric transform; at the moment we use a linear piecewise transformation, but it may need to be an exponential or logarithmic one. So that may need some fiddling with to get right. It doesn't help that testing on Android is such a spurious/difficult thing to do, so I gave up on eventually trying, which wasn't what I wanted to do but testing across API levels 30-36 proved to be a non-starter when I tried. This definitely falls under a bug, so please open an issue so we can track it. You can just reference this discussion for context, or re-describe the problem there in more detail and provide an MRE.

With respect to your second problem, can you clarify exactly what's happening? Can you provide the code your using? You must also keep in mind the multi-threading requirements of the docs as well if you are using Prism in a multi-threaded context.

[diamondStar35]

@ethindp Here is the code I use for backend switching and speech rate.

  private void SetSpeechBackend(ulong? backendId)
  {
      _settings.SpeechBackendId = backendId;
      _speech.PreferredBackendId = backendId;
      _menuRegistry.RefreshSpeechSettingsMenu();
      SaveSettings();
  }

  public ulong? PreferredBackendId
  {
      get => _screenReader.PreferredBackendId;
      set
      {
          if (_screenReader.PreferredBackendId == value)
              return;

          _screenReader.PreferredBackendId = value;
          _screenReaderReady = InitializeScreenReader();
      }
  }

  private bool InitializeScreenReader()
  {
      try
      {
          try
          {
              _screenReader.Close();
          }
          catch
          {
          }

          _screenReader.TrySAPI(true);
          _screenReader.PreferSAPI(false);
          var initialized = _screenReader.Initialize();
          if (initialized)
              ApplySpeechRate();
          return initialized;
      }
      catch
      {
          return false;
      }
  }

  public bool Initialize()
  {
      Close();

      try
      {
          _context = new Context();
          _backend = OpenBackend(_context);
          return true;
      }
      catch
      {
          Close();
          return false;
      }
  }

  private void SetSpeechRate(float rate)
  {
      _settings.SpeechRate = rate;
      _speech.SpeechRate = rate;
      SaveSettings();
  }

  public float SpeechRate
  {
      get => _speechRate;
      set
      {
          var clamped = Math.Max(0f, Math.Min(1f, value));
          _speechRate = clamped;
          ApplySpeechRate();
      }
  }

  private void ApplySpeechRate()
  {
      if (!_screenReaderReady)
          return;

      try
      {
          _screenReader.SetRate(_speechRate);
      }
      catch
      {
      }
  }

  public void SetRate(float rate)
  {
      try
      {
          if (_backend != null)
              _backend.Rate = rate;
      }
      catch
      {
      }
  }

  public float? Rate
  {
      get => Supports(Features.GetRate) && Native.GetRate(_handle, out var value) == Error.Ok ? value : null;
      set
      {
          if (value.HasValue && Supports(Features.SetRate))
              ThrowIfError(Native.SetRate(_handle, value.Value));
      }
  }

  public static Error SetRate(IntPtr backend, float rate) => Methods.SetRate(backend, rate);

  [DllImport(Library, CallingConvention = CallingConvention.Cdecl)]
  private static extern Error prism_backend_set_rate(IntPtr backend, float rate);

  public Error SetRate(IntPtr backend, float rate) => prism_backend_set_rate(backend, rate);

  Background-thread proof (this is a real background call site):

  Task.Run(async () =>
  {
      await Task.Delay(delayMs).ConfigureAwait(false);
      if (token != Volatile.Read(ref _hintToken))
          return;
      if (_disposed || _index != index)
          return;
      var delayedHint = item.GetHintText();
      if (!_usageHintsEnabled() || string.IsNullOrWhiteSpace(delayedHint))
          return;
      _speech.Speak(delayedHint!, SpeechService.SpeakFlag.NoInterrupt);
  });

  That _speech.Speak(...) path reaches Prism:

  private bool TrySpeakWithScreenReader(string text, bool interrupt)
  {
      ...
      default:
          if (_screenReader.Speak(text, interrupt))
              return true;

          return _screenReader.Output(text, interrupt);
  }

  public bool Speak(string text, bool interrupt = true)
  {
      ...
      if (_backend.Supports(Features.Speak))
      {
          _backend.Speak(text, interrupt);
          return true;
      }
      ...
  }

  public void Speak(string text, bool interrupt)
  {
      ThrowIfClosed();
      ThrowIfError(Native.Speak(_handle, text, interrupt));
  }

  public static Error Speak(IntPtr backend, string text, bool interrupt) => Methods.Speak(backend, text, interrupt);

  [DllImport(Library, CallingConvention = CallingConvention.Cdecl)]
  private static extern Error prism_backend_speak(IntPtr backend, IntPtr text, [MarshalAs(UnmanagedType.I1)] bool interrupt);

  public Error Speak(IntPtr backend, string text, bool interrupt) => Strings.WithUtf8(text, ptr => prism_backend_speak(backend, ptr,
  interrupt));

Here, Prism is also called from a background thread: "Task.Run(... _speech.Speak(...))"

[ethindp]

Here, Prism is also called from a background thread: "Task.Run(... _speech.Speak(...))"

Then you need to synchronize access between both threads. Please thoroughly read the documentation around multi-threaded requirements.

[diamondStar35]

Then you need to synchronize access between both threads. Please thoroughly read the documentation around multi-threaded requirements.

I have already done so, and the result was the same.

I am doing this in DotNet:

        private readonly object _sync = new object();

All methods such as Speak, Init, backend selection and queries use "lock (_sync)".
An example is:

        public bool Initialize()
        {
            lock (_sync)
            {
                CloseUnlocked();

                try
                {
                    _context = new Context();
                    _backend = OpenBackend(_context);
                    return true;
                }
                catch
                {
                    CloseUnlocked();
                    return false;
                }
            }
        }

        public bool Speak(string text, bool interrupt = true)
        {
            lock (_sync)
            {
                if (string.IsNullOrWhiteSpace(text) || _backend == null)
                    return false;

                try
                {
                    if (ShouldUseMemorySpeechLocked())
                        return SpeakToPlayerLocked(text, interrupt);

                    if (_backend.Supports(Features.Speak))
                    {
                        _backend.Speak(text, interrupt);
                        return true;
                    }

                    if (_backend.Supports(Features.Output))
                    {
                        _backend.Output(text, interrupt);
                        return true;
                    }

                    return false;
                }
                catch
                {
                    return false;
                }
            }
        }

        private void CloseUnlocked()
        {
            _player?.Stop();
            _backend?.Dispose();
            _backend = null;
            _activeBackendId = null;
            _context?.Dispose();
            _context = null;
        }
    }
}

        public bool Initialize()
        {
            lock (_sync)
            {
                CloseUnlocked();

                try
                {
                    _context = new Context();
                    _backend = OpenBackend(_context);
                    return true;
                }
                catch
                {
                    CloseUnlocked();
                    return false;
                }
            }
        }

So not really sure then what is the problem.

[ethindp]

Ah okay. I will have a test binary soon which in theory should solve both of these but we will see

[diamondStar35]

I've mentioned that the problem of speech rate has been fixed. I just want to add something here which is that switching backends on Android still fails, and I haven't been able to identify the problem yet.
Since this is not related to the previous issue, I didn't mention it.

[ethindp]

Understandable. I'm not entirely sure why the backend switching issue is occurring. Can you walk me through what exactly happens? Like, does the app crash outright? I don't have an android test device, and Prism is extremely unreliable on the Google-provided Android emulator images due to problems outside my control. So testing these backends to ensure that they are as stable as I would like them to be is quite challenging. I'm not at all happy about it and would like to dramatically improve things in future.

[ethindp]

@diamondStar35 Even better, could you provide me with a test app that reproduces this other problem? I can try to get emulator stuff to work on my machine to locally debug the problem.

[diamondStar35]

Thanks for at least trying to help.

No, the app does not crash. What happens is that whenever I switch to another backend, Prism gets completely stuck even if I do that and if there was no speech running.

The app continues to run normally, but because Prism is stuck, all backends are unavailable and thus they do not show. I have to restart the app for the new backend to take effect.

Like I mentioned, none of this happens at all on other platforms, not even Linux, even before synchronizing threads honestly.

I have tried looking at the adb logcat to see if there are any exceptions but there was nothing related to Prism.

[diamondStar35]

@ethindp Will create a test app in a few.

[diamondStar35]

@ethindp Thanks for your idea, it actually helped me a lot to isolate the problem.

Not sure if this was stated somewhere explicitly, but Android speech backends need an Android looper context with a looper-backed thread.

I tried different solutions on that test app I've created, and the first thing I did was trying a single worker thread for all Prism calls.

It worked for everything except backend switching.

I switched to a looper-backed thread with a looper context, backend switching behaved normally.

Sorry if this was stated somewhere. If you still want the source of the test app to see it, I could upload it here too.