Feature/webdav sync

[jdkruzr]

Outstanding Issues:

• Needs image/other attachment hashing solution to avoid duplication (requires messing with database, tabling for now)
• Needs folder deletion/cascade solution (also requires messing with database)

[Ethran]

Initial review, I only glanced at the code. Letter I'll actually look at code in folder sync.

Please clean up the stuff left behind by AI.
In setting syncing setting should be implemented in separate file. Probably other setting also should be done this way, but don't change it in this PR.

[Ethran]

Also, as I understand it should work with nextcloud?

If so, please give a short instruction how to setup it, etc. I will test it on nextcloud probably (if I have still access to university instance).

[jdkruzr]

Also, as I understand it should work with nextcloud?

If so, please give a short instruction how to setup it, etc. I will test it on nextcloud probably (if I have still access to university instance).

Do you mean how to set up Nextcloud? Or something else? And, do you want me to just create a "how to use sync" .md in /docs? That's probably easiest.

[Ethran]

It would be best to have two files:

1. How to use sync, with Nextcloud as the example, to be linked in the README. If user needs to change default settings of nextcloud include it. Just how to connect app to the server, what to have in mind etc.

2. overview of how sync works for developers.
   If AI is used to generate it, ensure the information is accurate and as compact as possible, without unnecessary details. Something similar to what already is written in docs.

[Ethran]

Also, how do you handle conflicts in stroke data?

I feel like there should be some function that decide how to resolve conflicts, and it should be reusable for imports, if possible.

https://github.com/Ethran/notable/blob/main/app/src/main/java/com/ethran/notable/io/ImportEngine.kt

Whenever possible reuse functions, it makes code easier to maintain.

[Ethran]

I would say that first we should settle on the format for notable pages. Make proper documentation explaining how its handle etc. Add ability for app to import export it, handle marge conflicts. Then syncing should reuse this format.

It should be internal to notable, but it has to be well thought out. It also should be easily modifiable as db schema changes. Now you putted all the data in json, its really bad approach for space, speed, and possibly future modifications. The strokes points should be encoded in SB1 -- see docs.

I think it would be best to implement the basic sync format in a separate PR, then update the WebDAV sync to use that format. The format doesn't need to be perfect initially, but it should be flexible and easy to modify. Labeling it "experimental" for the first few releases is a sensible approach: it lets us make breaking changes without requiring migrations. That said, it would be good to start from a solid foundation.

I will be going on vacation for couple of days soon, so I won't be answering probably.

[jdkruzr]

OK just for clarification: what do you mean by conflicts in stroke data? Do you mean two different devices making changes? Because since this isn't a multiuser system, my inclination is just to go with "most recent page wins."

[Ethran]

Multiple devices making changes, changes done on server while device was offline, etc. A lot of things might get wrong, there might be a lot of edge cases:
Example: One device removes page, other changes something on this page. What should happen when both devices go online and try to sync with server?

I was thinking that there should be some strategies for merging predefined:

notable/app/src/main/java/com/ethran/notable/io/ImportEngine.kt

Line 32 in 9783721

enum class ImportConflictStrategy {

So in questionable situations we know what to do (ie always prioritise device, ask case by case, etc) I didn't implemented it yet, so you can change it a little bit, if you have better idea. But I would expect some functions that will take data from device and database, and merging settings, and return merged data. Try to make it as much generic as possible without too much hassle, so it can be latter reused.

Also, for now one strategy will be enough, ie take newest changes, but leave space for other to be added later.

And, I really think that debugging will be much easier if you add first possibility to import new format, then to merge changes from the linked file, and then to reuse those for syncing. Just testing one thing at the time.

Also, I'm will have quite busy two weeks, so I might not look into code, but I will try to answer questions.

[jdkruzr]

Multiple devices making changes, changes done on server while device was offline, etc. A lot of things might get wrong, there might be a lot of edge cases: Example: One device removes page, other changes something on this page. What should happen when both devices go online and try to sync with server?

I was thinking that there should be some strategies for merging predefined:

notable/app/src/main/java/com/ethran/notable/io/ImportEngine.kt

Line 32 in 9783721

enum class ImportConflictStrategy {

So in questionable situations we know what to do (ie always prioritise device, ask case by case, etc) I didn't implemented it yet, so you can change it a little bit, if you have better idea. But I would expect some functions that will take data from device and database, and merging settings, and return merged data. Try to make it as much generic as possible without too much hassle, so it can be latter reused.
Also, for now one strategy will be enough, ie take newest changes, but leave space for other to be added later.

And, I really think that debugging will be much easier if you add first possibility to import new format, then to merge changes from the linked file, and then to reuse those for syncing. Just testing one thing at the time.

Also, I'm will have quite busy two weeks, so I might not look into code, but I will try to answer questions.

OK, got you I think. I don't think we need a new format at all; I've gone ahead and switched to SB1 and it works quite well imo. For now at least I've defaulted to "last writer wins" including preventing changes to notes saved after deletion from being lost. I don't get the sense there's going to be huge demand for multiple conflict resolution strategies as long as it works consistently.

[Ethran]

I think that it would be best to get the some rough sketch on how the sync is working, I saw that you added documentation for users, that's great.

When you add it I will review this, so you you will know if concept is alright. It will also be needed for latter development. The purpose for this file should be to get an idea how the sync works, how its structured, without looking at the code itself.

Proper review of the code will have to wait a little bit, in February I should be able to do it.

[Ethran]

Please make sure that all the information on different providers are correct, or mark [UNTESTED] if you cant.

Try to get more data from refused connection, give user more information what want wrong.

Give more troubleshooting information, I spend a lot of time trying to find what is wrong. URL: https://your-nextcloud.com/remote.php/dav/files/ it claimed that is correct, but on trying to create folder it failed, with 403.

For now I tested only with my phone, after reviewing code I will check on my main device.

[Ethran]

it's a gear wheel

[Ethran]

there should not be a username in the link, at list in my case.
in my case this worked:
https://your-nextcloud.com/remote.php/webdav/

You should probably add instructions "how to know that my url is correct?" in troubleshooting section.
ie for next cloud going to the url should result in blank page or
"This is the WebDAV interface. It can only be accessed by WebDAV clients such as the Nextcloud desktop sync client."

[Ethran]

The error message should be more descriptive, try showing:
something like:
"url does not exists"
"wrong password/username"

[Ethran]

I'm not sure how you check which file was modified, I suppose that you aren't using ETag, which would require db changes, and hashes I think are not available over WebDav. I think later (NOT in this PR) we should save ETag to db.

Either way, I think that it is very important to make sure that server time, and device time are the same. If there would be two devices with different times set, the merging base on modification time will break. I even think that we should not allow syncing if time is messed up (or at list require explicite action to sync with broken time).

[Ethran]

please add switch to explicitly allow syncing on mobile data. It shouldn't be syncing by default when it's not connected to wifi.

[Ethran]

Also, please provide instructions how to setup syncing if user have 2fa, ie:
Log in via browser
Go to Settings → Security
Under Devices & sessions
Click Create new app password
Give it a name (e.g. notable),
You will then see a username and password created for this app. Use them to setup nextcloud.

[Ethran]

Also, the ui doesn't look great with dark mode:

And there should be button to show password that was entered, but only if user just entered it, not the saved password. For saved password I don't see a point in showing anything else then "change password".