[Bug] Book analysis is slow for webp images

Komga environment

• OS: Debian 10
• Komga version: 0.57.0
• I am running Komga with Docker
  • Docker image tag [e.g. latest, beta]:
• I am running Komga from the jar
  • Java version: 11.0.8
• I have a problem in the web interface
  • Browser (with version):
• I have a problem with an OPDS client application
  • OPDS Application (with version):
• I have a problem with the Tachiyomi extension
  • Tachiyomi version:
  • Tachiyomi extension version:

Describe the bug

After noticing that a library scan on a fresh installation of Komga had been stuck on counting the number of pages inside my CBZ files for several hours earlier today, I aborted the scan and set up a new installation of Komga for this bug report. I continued by creating a new library containing only a single 10MB/35 pages CBZ file, started a library scan and observed that it took Komga almost two minutes to complete the "AnalyzeBook" task. The thumbnail generation task, meanwhile, finished in just a couple of seconds.

Steps to reproduce

1. Create a fresh installation of Komga v0.57.0.
2. Download this .zip file, rename it to .cbz and create a new folder for it: Locke & Key - Welcome to Lovecraft v01.zip
3. Create a new library containing only this CBZ file.
4. Wait a few minutes for Komga to complete the library scan.
5. Check the log files.

Expected behavior

Komga only takes a fraction of a second to finish counting the number of pages inside the CBZ file.

Actual behavior

Komga takes almost two minutes to finish counting the number of pages inside the CBZ file. While testing different CBZ files in my collection, I observed that this time increases linearly with the number of pages. For example, it took Komga almost 20 minutes to finish scanning a 70MB/221 pages CBZ file.

Additional context

I think this issue is at least tangentially related to #278. I've been using Komga with the same media collection on the same hardware since about December 2019 and never had any issues with these files. In older versions, it only took Komga a fraction of a second each to finish the "AnalyzeBook" task for the CBZ files in my collection.

Edit: It looks like this issue does not affect PDF files. Komga only takes 270ms to finish scanning a 50+ pages PDF file for me.

Log file

I've noticed that Komga prints the error message ERROR 30813 --- [DefaultMessageListenerContainer-1] unknown.jul.logger: TODO exactly once for each corresponding page in the CBZ file, for a total of 35 times.

Spoiler

2020-08-20 15:03:37.867  INFO 30813 --- [http-nio-7264-exec-7] o.g.k.domain.service.LibraryLifecycle    : Adding new library: Comics with root folder: file:/mnt/storage/media/literature/Comics/
2020-08-20 15:03:37.899  INFO 30813 --- [http-nio-7264-exec-7] o.g.k.application.tasks.TaskReceiver     : Sending task: ScanLibrary(libraryId=02AV46FH6BCM3)
2020-08-20 15:03:37.995  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: ScanLibrary(libraryId=02AV46FH6BCM3)
2020-08-20 15:03:38.008  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.LibraryScanner  : Updating library: Library(name=Comics, root=file:/mnt/storage/media/literature/Comics/, importComicInfoBook=true, importComicInfoSeries=true, importComicInfoCollection=true, importComicInfoReadList=true, importEpubBook=true, importEpubSeries=true, importLocalArtwork=true, scanForceModifiedTime=false, scanDeep=false, id=02AV46FH6BCM3, createdDate=2020-08-20T15:03:37, lastModifiedDate=2020-08-20T15:03:37)
2020-08-20 15:03:38.009  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.FileSystemScanner   : Scanning folder: /mnt/storage/media/literature/Comics
2020-08-20 15:03:38.010  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.FileSystemScanner   : Supported extensions: [cbz, zip, cbr, rar, pdf, epub]
2020-08-20 15:03:38.012  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.FileSystemScanner   : Excluded patterns: [#recycle, @eaDir]
2020-08-20 15:03:38.012  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.FileSystemScanner   : Force directory modified time: false
2020-08-20 15:03:38.038  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.FileSystemScanner   : Scanned 1 series and 1 books in 20.9ms
2020-08-20 15:03:38.044  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.LibraryScanner  : Adding new series: Series(name=Locke & Key, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/, fileLastModified=2020-08-20T15:01:09.700904, id=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38.033306, lastModifiedDate=2020-08-20T15:03:38.033311)
2020-08-20 15:03:38.133  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: RefreshBookMetadata(bookId=02AV46G62B30H)
2020-08-20 15:03:38.139  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.LibraryScanner  : Library updated in 130ms
2020-08-20 15:03:38.144  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: AnalyzeBook(bookId=02AV46G62B30H)
2020-08-20 15:03:38.148  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task ScanLibrary(libraryId=02AV46FH6BCM3) executed in 147ms
2020-08-20 15:03:38.159  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: RefreshBookMetadata(bookId=02AV46G62B30H)
2020-08-20 15:03:38.162  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.MetadataLifecycle   : Refresh metadata for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:03:38.176  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.i.m.l.LocalArtworkProvider         : Looking for local thumbnails for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:03:38.183  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: RefreshSeriesMetadata(seriesId=02AV46G66B9VF)
2020-08-20 15:03:38.186  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task RefreshBookMetadata(bookId=02AV46G62B30H) executed in 26.9ms
2020-08-20 15:03:38.191  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: AnalyzeBook(bookId=02AV46G62B30H)
2020-08-20 15:03:38.195  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookLifecycle   : Analyze and persist book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:03:38.196  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookAnalyzer    : Trying to analyze book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:03:38.252  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookAnalyzer    : Detected media type: application/zip
2020-08-20 15:03:38.581 ERROR 30813 --- [DefaultMessageListenerContainer-1] unknown.jul.logger                       : TODO [note: this error message is printed exactly 35 times]
2020-08-20 15:05:30.078  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookAnalyzer    : Book has 35 pages
2020-08-20 15:05:30.095  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: GenerateBookThumbnail(bookId=02AV46G62B30H)
2020-08-20 15:05:30.102  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: RefreshBookMetadata(bookId=02AV46G62B30H)
2020-08-20 15:05:30.106  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task AnalyzeBook(bookId=02AV46G62B30H) executed in 112s
2020-08-20 15:05:30.122  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: RefreshSeriesMetadata(seriesId=02AV46G66B9VF)
2020-08-20 15:05:30.124  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.MetadataLifecycle   : Refresh metadata for series: Series(name=Locke & Key, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/, fileLastModified=2020-08-20T15:01:09.700, id=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38)
2020-08-20 15:05:30.171  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.i.m.l.LocalArtworkProvider         : Looking for local thumbnails for series: Series(name=Locke & Key, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/, fileLastModified=2020-08-20T15:01:09.700, id=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38)
2020-08-20 15:05:30.176  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task RefreshSeriesMetadata(seriesId=02AV46G66B9VF) executed in 53.8ms
2020-08-20 15:05:30.180  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: GenerateBookThumbnail(bookId=02AV46G62B30H)
2020-08-20 15:05:30.182  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookLifecycle   : Generate thumbnail and persist for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:05:30.183  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookAnalyzer    : Generate thumbnail for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:05:30.259 ERROR 30813 --- [DefaultMessageListenerContainer-1] unknown.jul.logger                       : TODO
2020-08-20 15:05:32.613  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookLifecycle   : House keeping thumbnails for book: 02AV46G62B30H
2020-08-20 15:05:32.616  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.domain.service.BookLifecycle   : Book has bo selected thumbnail, choosing one automatically
2020-08-20 15:05:32.620  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task GenerateBookThumbnail(bookId=02AV46G62B30H) executed in 2.44s
2020-08-20 15:05:32.623  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: RefreshBookMetadata(bookId=02AV46G62B30H)
2020-08-20 15:05:32.625  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.MetadataLifecycle   : Refresh metadata for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:05:32.632  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.i.m.l.LocalArtworkProvider         : Looking for local thumbnails for book: Book(name=Locke & Key - Welcome to Lovecraft v01, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/Locke%20&%20Key%20-%20Welcome%20to%20Lovecraft%20v01.cbz, fileLastModified=2020-03-23T10:36:42, fileSize=10309532, number=1, id=02AV46G62B30H, seriesId=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38.118)
2020-08-20 15:05:32.632  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.application.tasks.TaskReceiver     : Sending task: RefreshSeriesMetadata(seriesId=02AV46G66B9VF)
2020-08-20 15:05:32.633  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task RefreshBookMetadata(bookId=02AV46G62B30H) executed in 9.73ms
2020-08-20 15:05:32.638  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Executing task: RefreshSeriesMetadata(seriesId=02AV46G66B9VF)
2020-08-20 15:05:32.639  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.domain.service.MetadataLifecycle   : Refresh metadata for series: Series(name=Locke & Key, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/, fileLastModified=2020-08-20T15:01:09.700, id=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38)
2020-08-20 15:05:32.663  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.k.i.m.l.LocalArtworkProvider         : Looking for local thumbnails for series: Series(name=Locke & Key, url=file:/mnt/storage/media/literature/Comics/Locke%20&%20Key/, fileLastModified=2020-08-20T15:01:09.700, id=02AV46G66B9VF, libraryId=02AV46FH6BCM3, createdDate=2020-08-20T15:03:38, lastModifiedDate=2020-08-20T15:03:38)
2020-08-20 15:05:32.664  INFO 30813 --- [DefaultMessageListenerContainer-1] o.g.komga.application.tasks.TaskHandler  : Task RefreshSeriesMetadata(seriesId=02AV46G66B9VF) executed in 25.6ms

[gotson]

Komga doesn't count pages during the analysis, it does much more than that.

The analysis for pdf is faster, because the pages are always rendered in JPEG at the same dimensions, so the analysis doesn't need to analyze each page to detect the media type and size of each page.

[gotson]

The message ERROR 30813 --- [DefaultMessageListenerContainer-1] unknown.jul.logger: TODO is just a log from the webp library.

The webp library used in Komga is quite beta and not very fast, which would explain why your book is long to analyze.

Analysis time will also depend on your hardware (cpu) and disk or network access speed to access the books.

@gotson It only took Komga a fraction of the time to scan the identical files in previous versions, though. Just for reference, this is a colocated server with NVMe SSDs, two 12c/24t CPUs and 128GB of RAM, so I don't think a lack of processing power is causing Komga to take 20 minutes to scan a single file.

[gotson]

It's doing more in recent versions than it did before.

Where are your files located and how are they accessed?

@gotson The files are stored on the same server; they're not being accessed via NFS or another network file system. I run Komga as a regular systemd service, so it's accessing the files like any other application would.

Are you able to reproduce the issue with my test file? Does it also take several minutes for Komga to finish scanning it on your machine?

I have about 2,200 files in my collection, most of which are tankobons. Based on my tests, it takes Komga around three seconds to process one page. Assuming an average page count of 100 pages, that would work out to an initial scan time of 7 days, 15 hours and 20 minutes, not including thumbnail generation.

Even taking the fact that the "AnalyzeBook" task does more than it did in previous versions into account, that figure seems extremely high to me. Again, just for reference, the initial library scan in previous versions only took a few hours to finish and that's including the thumbnail generation process (which accounted for the vast majority of that time).

[gotson]

I'm on my phone so cannot test now. A cbz with 100 pages is analyzed in 5s on my system.

[gotson]

I managed ti reproduce, it only happens with webp images. The latest versions of Komga perform an image analysis on each image to get the dimensions. This takes around 3s per image on my system for a webp.

There are no other webp libraries for Java. There's an open issue at twelve monkeys (a java image library) to have a proper one, but I discussed with the author and the performance would be most likely similar.

Another solution would be to use native libraries, and to that end I did submit a PR some time ago on a project to properly package native libraries for multiple architectures, but it's still pending (sejda-pdf/webp-imageio#6).

To come back to your issue, it's only gonna impact webp files. I don't know how many you have, I found them to be pretty rare in the western comics and European bds, and only saw a few for mangas.

@gotson Thanks for the explanation. I sadly converted my entire library to WEBP months ago due to the significant space savings and improved page loading times compared to JPEG. Oh well, at least I have confirmation that this is expected behavior now. 😅

[gotson]

It is really not wise to convert from a lossy format like JPEG to another format. You should only ever convert from lossless to lossy, never from lossy to anything else!

@gotson I would agree if we were talking about music, but lossless sources for almost all other types of media simply don't exist. I personally think the almost imperceptible quality loss is well worth the significant space and bandwidth savings.