What's happening?
For some reason, in very rare cases, when communicating with NPCs, the game mod may incorrectly process Russian characters (as I understand it, specifically the first character), which is why the encoding of the data that ends up in memory is completely broken, and LLM starts receiving broken strings at the input.
The response from LLM itself, in turn, contains a normal response without broken encoding.
Steps to reproduce
Simple NPC communication causes in rare cases
How often does this happen?
It happens very rarely.
Expected behavior
If the behavior is correct, all characters should be returned in normal format
What area is affected?
NPC dialogue / conversation
TTS Provider
XTTS
LLM Model
deepseek/deepseek-v4-flash
STT Provider (if relevant)
Disabled / not using STT
Game Version
AE (1.6.1170+)
Modlist
The cleanest version of SkyrimNet without additional mods, only those required for SkyrimNet/SeverAction to work
SkyrimNet version
Beta 20.2
Did this work before?
Yes, it stopped working (regression)
What changed? (if it used to work)
No response
Log file
Unfortunately, I don't have the mod log file at the moment, but I'll definitely attach it here with more details if I catch this error again.
Anything else?
The encoding issue occurs for most users who communicate in Russian, but I've noticed that there are several different ways similar issues with broken encoding can occur, all for different reasons and most likely different problems altogether. The main issue (which causes broken encoding to be stored in memory and transmitted in the future) is related to this.
When I tried to solve this problem myself by setting up a proxy server that would repair user input requests before sending them to LLM itself, I achieved the result that in 100% of cases, LLM returned broken text (although in the service API logs, everything looked normal).
When discussing how to solve this problem, we were told that it is possible to enable the function of using the Beta version of utf8 Unicode in the Regional Standards settings, but this did not bring any progress, the text is still broken and transmitted in a problematic way.
Also, to make it easier to fix broken characters, I created an application that is pinned in my GitHub repositories, and after sending it several databases of some users (who complained about problems), I noticed that the problem occurs not only with Russian letters, but also with long dashes, for example.
Also, when processing through the application, I noticed that my program cannot always restore one character (and instead of it, the "�" character is returned), perhaps this will somehow help in finding a solution to the problem, but as I wrote at the very beginning, the problem arises precisely because one of the characters is lost.
Ошибка в get_table_data(memories): Could not decode to UTF-8 column 'content' with text '���������� ����������������������, ����������������, ������ ���������� ��� ���� ���������� �������������������������� ������������������ ���� ��
Traceback (most recent call last):
File "C:\Users\Selenter\Desktop\SkyrimNetDBFixer\main.py", line 154, in get_table_data
rows = db_manager.get_all_rows(table_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Selenter\Desktop\SkyrimNetDBFixer\core\db_manager.py", line 137, in get_all_rows
rows = cursor3.fetchall()
^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: Could not decode to UTF-8 column 'content' with text '���������� ����������������������, ����������������, ������ ���������� ��� ���� ���������� �������������������������� ������������������ ���� ��I solved it by simply adding the removal of this bad symbol:
fixed = fixed.replace('�', ' ')
What's happening?
For some reason, in very rare cases, when communicating with NPCs, the game mod may incorrectly process Russian characters (as I understand it, specifically the first character), which is why the encoding of the data that ends up in memory is completely broken, and LLM starts receiving broken strings at the input.
The response from LLM itself, in turn, contains a normal response without broken encoding.
Steps to reproduce
Simple NPC communication causes in rare cases
How often does this happen?
It happens very rarely.
Expected behavior
If the behavior is correct, all characters should be returned in normal format
What area is affected?
NPC dialogue / conversation
TTS Provider
XTTS
LLM Model
deepseek/deepseek-v4-flash
STT Provider (if relevant)
Disabled / not using STT
Game Version
AE (1.6.1170+)
Modlist
The cleanest version of SkyrimNet without additional mods, only those required for SkyrimNet/SeverAction to work
SkyrimNet version
Beta 20.2
Did this work before?
Yes, it stopped working (regression)
What changed? (if it used to work)
No response
Log file
Unfortunately, I don't have the mod log file at the moment, but I'll definitely attach it here with more details if I catch this error again.
Anything else?
The encoding issue occurs for most users who communicate in Russian, but I've noticed that there are several different ways similar issues with broken encoding can occur, all for different reasons and most likely different problems altogether. The main issue (which causes broken encoding to be stored in memory and transmitted in the future) is related to this.
When I tried to solve this problem myself by setting up a proxy server that would repair user input requests before sending them to LLM itself, I achieved the result that in 100% of cases, LLM returned broken text (although in the service API logs, everything looked normal).
When discussing how to solve this problem, we were told that it is possible to enable the function of using the Beta version of utf8 Unicode in the Regional Standards settings, but this did not bring any progress, the text is still broken and transmitted in a problematic way.
Also, to make it easier to fix broken characters, I created an application that is pinned in my GitHub repositories, and after sending it several databases of some users (who complained about problems), I noticed that the problem occurs not only with Russian letters, but also with long dashes, for example.
Also, when processing through the application, I noticed that my program cannot always restore one character (and instead of it, the "�" character is returned), perhaps this will somehow help in finding a solution to the problem, but as I wrote at the very beginning, the problem arises precisely because one of the characters is lost.
I solved it by simply adding the removal of this bad symbol: