Skip to content

Commit 80cfccb

Browse files
vsc55vsc55
committed
update README
1 parent 32f5892 commit 80cfccb

1 file changed

Lines changed: 95 additions & 51 deletions

File tree

‎README.md‎

Lines changed: 95 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Migration from SQLite to MySQL/MariaDB
2+
13
---
24

35
## ⚠️ Important: Table name case-sensitivity in MySQL/MariaDB
@@ -17,7 +19,7 @@ If you encounter "table not found" errors after migration, review your server se
1719

1820
If you encounter an error like the following during migration:
1921

20-
```
22+
```text
2123
* MySQL Error [1064]: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ...
2224
* Error Query: SET group_concat_max_len = 1024 * 1024 * 100;SELECT CONCAT('SELECT * FROM (',GROUP_CONCAT(...)
2325
@@ -35,10 +37,12 @@ TypeError: 'NoneType' object is not iterable
3537
```
3638

3739
### What does this mean?
40+
3841
- The script attempted to run a dynamic SQL query that failed due to a syntax error (error 1064).
3942
- After the SQL error, the script tried to process the result, but since the query failed, the result was `None`, causing a Python `TypeError`.
4043

4144
### How to troubleshoot
45+
4246
- Carefully check the SQL error message and the problematic query shown after `* Error Query:`.
4347
- Review your MySQL/MariaDB version and compatibility with the generated SQL.
4448
- Check the value of `lower_case_table_names` and your table names for case-sensitivity issues (see section above).
@@ -48,28 +52,28 @@ If you need help, include the full error output (including the SQL and Python tr
4852

4953
---
5054

51-
# Migration procedure
55+
## Migration procedure
5256

53-
* 1\. [Requirements](#1-requirements)
54-
* 2\. [Create database and user in the server MySql/MariaDB](#2-create-database-and-user-in-the-server-mysqlmariadb)
55-
* 2.1\. [A Single Database](#21-a-single-database)
56-
* 2.2\. [In Multiple DataBases or Servers MySql/MariaDB](#22-in-multiple-databases-or-servers-mysqlmariadb)
57-
* 3\. [Download Script and install dependencies](#3-download-script-and-install-dependencies)
58-
* 4\. [Create and prepare tables](#4-create-and-prepare-tables)
59-
* 5\. [Data Migration](#5-data-migration)
60-
* 5.1\. [Data Migration (*Single Database*)](#51-data-migration-single-database)
61-
* 5.2\. [Data Migration (*Multiple DataBases or Servers MySql/MariaDB*)](#52-data-migration-multiple-databases-or-servers-mysqlmariadb)
62-
* 6\. [Help](#6-help)
63-
* 7\. [F.A.Q.](#7-faq)
57+
- 1 [Requirements](#1-requirements)
58+
- 2 [Create database and user in the server MySql/MariaDB](#2-create-database-and-user-in-the-server-mysqlmariadb)
59+
- 2.1 [A Single Database](#21-a-single-database)
60+
- 2.2 [In Multiple DataBases or Servers MySql/MariaDB](#22-in-multiple-databases-or-servers-mysqlmariadb)
61+
- 3 [Download Script and install dependencies](#3-download-script-and-install-dependencies)
62+
- 4 [Create and prepare tables](#4-create-and-prepare-tables)
63+
- 5 [Data Migration](#5-data-migration)
64+
- 5.1 [Data Migration (*Single Database*)](#51-data-migration-single-database)
65+
- 5.2 [Data Migration (*Multiple DataBases or Servers MySql/MariaDB*)](#52-data-migration-multiple-databases-or-servers-mysqlmariadb)
66+
- 6 [Help](#6-help)
67+
- 7 [F.A.Q. (Frequently Asked Questions)](#7-faq-frequently-asked-questions)
6468

6569
> This would be the procedure to migrate the Ombi databases from SQLite to a MySQL/MariaDB server.
66-
>
67-
> If there is an error you can contact Discour or you can open an incident [here](https://github.com/vsc55/ombi_sqlite_mysql/issues).
68-
70+
>
71+
> If there is an error you can contact Discour or you can open an incident on [GitHub Issues](https://github.com/vsc55/ombi_sqlite_mysql/issues).
6972
7073
## 1. Requirements
71-
* Python3
72-
* Ombi version 4.0.728 or higher
74+
75+
- Python3
76+
- Ombi version 4.0.728 or higher
7377

7478
## 2. Create database and user in the server MySql/MariaDB
7579

@@ -78,13 +82,15 @@ On the MySQL/MariaDB server we will create the database and the user that we wil
7882
> **NOTE: Just follow one of the following two points. Point 2.1 if we want to use a single database, or point 2.2 if we want to separate the databases.**
7983
8084
### 2.1. A Single Database
85+
8186
```mysql
8287
CREATE DATABASE IF NOT EXISTS `Ombi` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
8388
CREATE USER 'ombi'@'%' IDENTIFIED BY 'ombi';
8489
GRANT ALL PRIVILEGES ON `Ombi`.* TO 'ombi'@'%' WITH GRANT OPTION;
8590
```
8691

8792
### 2.2. In Multiple DataBases or Servers MySql/MariaDB
93+
8894
```mysql
8995
CREATE DATABASE IF NOT EXISTS `Ombi` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
9096
CREATE DATABASE IF NOT EXISTS `Ombi_Settings` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
@@ -94,50 +100,61 @@ GRANT ALL PRIVILEGES ON `Ombi`.* TO 'ombi'@'%' WITH GRANT OPTION;
94100
GRANT ALL PRIVILEGES ON `Ombi_Settings`.* TO 'ombi'@'%' WITH GRANT OPTION;
95101
GRANT ALL PRIVILEGES ON `Ombi_External`.* TO 'ombi'@'%' WITH GRANT OPTION;
96102
```
97-
> _You need to GRANT ALL PRIVILEGES for every database you create._
103+
104+
> *You need to GRANT ALL PRIVILEGES for every database you create.*
98105
99106
[Go Up](#migration-procedure)
100107

101108
## 3. Download Script and install dependencies
109+
102110
1. Download the script.
111+
103112
```bash
104-
$ git clone https://github.com/vsc55/ombi_sqlite_mysql.git ombi_sqlite_mysql
105-
$ cd ombi_sqlite_mysql
106-
$ chmod +x *.py
113+
git clone https://github.com/vsc55/ombi_sqlite_mysql.git ombi_sqlite_mysql
114+
cd ombi_sqlite_mysql
115+
chmod +x *.py
107116
```
117+
108118
2. Install the dependencies according to the operating system we use.
109-
1. Lib python-mysqldb:
119+
2.1. Lib python-mysqldb:
120+
110121
```bash
111-
$ emerge -va dev-python/mysqlclient # Gentoo
112-
$ pip3 install mysqlclient # Python Pip
113-
$ python3 -m pip install mysqlclient # Python Pip
122+
emerge -va dev-python/mysqlclient # Gentoo
123+
pip3 install mysqlclient # Python Pip
124+
python3 -m pip install mysqlclient # Python Pip
114125
```
126+
115127
> WARNING: It is not recommended to use the python3-mysqldb package on Debian/Ubutu systems as the version it installs fails with modern versions of MariaDB/MySQL.
116-
2. Lib dev-python/packaging:
128+
129+
2.2. Lib dev-python/packaging:
130+
117131
```bash
118-
$ apt-get install python3-packaging # Debian/Ubuntu
119-
$ emerge -va dev-python/packaging # Gentoo
120-
$ pip3 install packaging # Python Pip
121-
$ python3 -m pip install packaging # Python Pip
132+
apt-get install python3-packaging # Debian/Ubuntu
133+
emerge -va dev-python/packaging # Gentoo
134+
pip3 install packaging # Python Pip
135+
python3 -m pip install packaging # Python Pip
122136
```
123137

124-
125138
[Go Up](#migration-procedure)
126139

127140
## 4. Create and prepare tables
141+
128142
1. Update to the latest version of ombi.
129143
2. Stop ombi
130144
3. Create or Modify **database.json** to use mysql.
145+
131146
```bash
132147
$ python ombi_sqlite2mysql.py -c /etc/Ombi --only_db_json --host 192.168.1.100 --db Ombi --user ombi --passwd ombi
133148
Migration tool from SQLite to MySql/MariaDB for ombi (3.0.4) By VSC55
134149
135150
Generate file "database.json":
136151
- Saving in (/etc/Ombi/database.json)... [✓]
137152
```
153+
138154
4. **Only if we are going to use *Multiple DataBases* or *Multiple Servers*.**
139155

140156
To be able to use multiple servers or databases we will need to manually edit **database.json**.
157+
141158
```json
142159
$ vi database.json
143160
{
@@ -155,17 +172,20 @@ GRANT ALL PRIVILEGES ON `Ombi_External`.* TO 'ombi'@'%' WITH GRANT OPTION;
155172
}
156173
}
157174
```
175+
158176
> The example above will export the **"OmbiDatabase"** and **"SettingsDatabase"** databases to server **"192.168.0.100"** but to different databases on the same server, while the **"ExternalDatabase"** database will be sent to server **"192.168.1.200"**.
159177

160178
5. Run the following command:
161-
```
162-
$ /opt/ombi/Ombi --migrate
179+
180+
```bash
181+
/opt/ombi/Ombi --migrate
163182
```
164183

165184
> **Note:**
166185
If our sqlite database and json configuration files are stored in a different location than the one where we have Ombi installed, we will have to add the --storage argument.
186+
167187
```bash
168-
$ /opt/ombi/Ombi --migrate --storage /etc/Ombi
188+
/opt/ombi/Ombi --migrate --storage /etc/Ombi
169189
```
170190

171191
> **Note:**
@@ -174,20 +194,21 @@ GRANT ALL PRIVILEGES ON `Ombi_External`.* TO 'ombi'@'%' WITH GRANT OPTION;
174194

175195
[Go Up](#migration-procedure)
176196

177-
178197
## 5. Data Migration
179198

180199
When it comes to migrating the data, we have several different ways of doing it.
181200
We can export everything to a single database (step 5.1), to different databases or to different mysql servers (step 5.2).
182201

183202
### 5.1. Data Migration (*Single Database*)
203+
184204
> For data migration we will need the file **"migration.json"** that contains the locations of the SQLite databases.
185-
>
205+
>
186206
> If this file does not exist, it will be created and will search the databases in the folder specified with the parameter **"--config"**.
187207
>
188208
>If we don't want to migrate all the data, we can generate the file **"migration.json"** with the parameter **"--only_manager_json"** and then edit it by deleting the databases we don't want to migrate.
189-
190-
> ### If we do not want to export OmbiExternal.
209+
>
210+
> ### If we do not want to export OmbiExternal
211+
>
191212
> ```bash
192213
> $ python ombi_sqlite2mysql.py -c /etc/Ombi --only_manager_json
193214
> Migration tool from SQLite to MySql/MariaDB for ombi (3.0.4) By VSC55
@@ -197,7 +218,9 @@ We can export everything to a single database (step 5.1), to different databases
197218
>
198219
> $ vi /etc/Ombi/migration.json
199220
> ```
221+
>
200222
> Content "migration.json":
223+
>
201224
> ```json
202225
> {
203226
> "OmbiDatabase": {
@@ -212,7 +235,7 @@ We can export everything to a single database (step 5.1), to different databases
212235
> ```
213236
214237
1. Start data migration.
215-
> _The script will **empty the tables** from the MySQL/MariaDB database and automatically migrate the data from SQLite to MySQL/MariaDB._
238+
> *The script will **empty the tables** from the MySQL/MariaDB database and automatically migrate the data from SQLite to MySQL/MariaDB.*
216239
217240
```bash
218241
$ python ombi_sqlite2mysql.py -c /etc/Ombi --host 192.168.1.100 --db Ombi --user ombi --passwd ombi
@@ -255,15 +278,17 @@ We can export everything to a single database (step 5.1), to different databases
255278
256279
MySQL > Disconnecting... [✓]
257280
```
281+
258282
2. Start ombi and test if everything works fine.
259283
260284
[Go Up](#migration-procedure)
261285
262-
263286
### 5.2. Data Migration (*Multiple DataBases or Servers MySql/MariaDB*)
287+
264288
> For data migration to multiple databases or servers we will need the file **"database_multi.json"** that contains the locations of the servers where we are going to export the data.
265289
266290
1. To create the file **"database_multi.json"** we will use the file **"database.json"** so we will only have to rename it.
291+
267292
```json
268293
$ vi database_multi.json
269294
{
@@ -284,6 +309,7 @@ We can export everything to a single database (step 5.1), to different databases
284309
285310
> We can omit the export of a database by removing it from **"database_multi.json"** or adding the property **"Skip"**.
286311
> The example next will export the databases **"OmbiDatabase"** and **"SettingsDatabase"** but omit **"ExternalDatabase"**.
312+
287313
```json
288314
$ vi database_multi.json
289315
{
@@ -305,6 +331,7 @@ We can export everything to a single database (step 5.1), to different databases
305331
306332
> We can also send the same database to different servers with the following configuration.
307333
> The example next sends databases "OmbiDatabase", "SettingsDatabase" and "ExternalDatabase" to servers 192.168.1.100 and 192.168.1.200.
334+
308335
```json
309336
$ vi database_multi.json
310337
{
@@ -334,10 +361,13 @@ We can export everything to a single database (step 5.1), to different databases
334361
}
335362
}
336363
```
337-
> ### NOTE: If you want to export all the content to several servers we will have to repeat the point "Create and prepare tables" with the different servers so that all the tables are created. You will also have to modify the file **database.json** at the end of the export process before running ombi to leave a single server for each database.
364+
365+
> [!IMPORTANT]
366+
> If you want to export all the content to several servers we will have to repeat the point "Create and prepare tables" with the different servers so that all the tables are created. You will also have to modify the file **database.json** at the end of the export process before running ombi to leave a single server for each database.
338367
339368
2. Start data migration.
340369
> The script will empty the tables from the MySQL/MariaDB database and automatically migrate the data from SQLite to MySQL/MariaDB.
370+
341371
```bash
342372
$ python ombi_sqlite2mysql_multi.py -c /etc/Ombi
343373
Migration tool from SQLite to Multi MySql/MariaDB for ombi (1.0.0) By VSC55
@@ -489,8 +519,8 @@ We can export everything to a single database (step 5.1), to different databases
489519
490520
[Go Up](#migration-procedure)
491521
492-
493522
## 6. Help
523+
494524
```bash
495525
$ python ombi_sqlite2mysql.py -h
496526
Migration tool from SQLite to MySql/MariaDB for ombi (3.0.4) By VSC55
@@ -535,10 +565,10 @@ Options:
535565
536566
[Go Up](#migration-procedure)
537567
568+
## 7. F.A.Q. (Frequently Asked Questions)
538569
539-
## 7. F.A.Q.
570+
**P:** Errors appear in the verification of the migrated data saying that there is more data in SQLite than in MySQL or vice versa.
540571
541-
**P: Errors appear in the verification of the migrated data saying that there is more data in SQLite than in MySQL or vice versa.**
542572
```bash
543573
- Running [############################################################] 9242/9242
544574
- [!!] -> __efmigrationshistory -> [SQLite (0) / MySQL (41)] = -41
@@ -572,17 +602,23 @@ Options:
572602
- [!!] -> userqualityprofiles -> [SQLite (0) / MySQL (20)] = -20
573603
- Checking [############################################################] 43/43
574604
```
605+
575606
S: We will have to force the elimination of the data in all the tables with the parameter `--force` as follows.
607+
576608
```bash
577609
# Single Database:
578610
$ python ombi_sqlite2mysql.py -c /etc/Ombi --force --host 192.168.1.100 --db Ombi --user ombi --passwd ombi
579611
```
612+
580613
```bash
581614
# Multiple DataBases or Servers MySql/MariaDB:
582615
$ python ombi_sqlite2mysql_multi.py -c /etc/Ombi --force
583616
```
617+
584618
---
585-
**P: A syntax error occurs when you are about to start cleaning the tables**
619+
620+
**P:** A syntax error occurs when you are about to start cleaning the tables
621+
586622
```bash
587623
Migration tool from SQLite to MySql/MariaDB for ombi (3.0.8) By VSC55
588624
@@ -623,14 +659,19 @@ Traceback (most recent call last):
623659
for table, count in return_query[1]:
624660
TypeError: 'NoneType' object is not iterable
625661
```
662+
626663
S: This error has been detected in debian when using the python3-mysqldb library installed from apt.
627-
The version installed with apt (1.3.10-2) is old and produces the error with newer databases. The solution is to install the library with pip since the version (2.1.0) installed with pip works correctly.
664+
The version installed with apt (1.3.10-2) is old and produces the error with newer databases. The solution is to install the library with pip since the version (2.1.0) installed with pip works correctly.
665+
628666
```bash
629-
$ apt-get remove python3-mysqldb
630-
$ pip3 install mysqlclient
667+
apt-get remove python3-mysqldb
668+
pip3 install mysqlclient
631669
```
670+
632671
---
633-
**P: Table "XXX" requiered is not exist in the server MySQL**
672+
673+
**P:** Table "XXX" requiered is not exist in the server MySQL
674+
634675
```bash
635676
$ python3 ombi_sqlite2mysql.py -c /etc/Ombi --host 127.0.0.1 --db Ombi --user ombi --passwd ombi
636677
Migration tool from SQLite to MySql/MariaDB for ombi (3.0.8) By VSC55
@@ -645,12 +686,15 @@ Read tables [!!]
645686
646687
MySQL > Disconnecting... [✓]
647688
```
689+
648690
S: This error typically occurs when tables were not successfully created with the --migrate argument to ombi. This may be because the configuration (database.json file) and the databases are not in the same folder where we have installed ombi. The solution is to add the --storage argument when we run the migration.
649691
650692
In the following example, both the database.json file and the databases are stored in /etc/Ombi:
693+
651694
```bash
652-
$ /opt/ombi/Ombi --migrate --storage /etc/Ombi
695+
/opt/ombi/Ombi --migrate --storage /etc/Ombi
653696
```
697+
654698
---
655699
656700
[Go Up](#migration-procedure)

0 commit comments

Comments
 (0)