Extremely slow performance

[DanielBiegler]

How long does it take for you to compress a couple of images?

I tried compressing a 7,8MB JPG with --quality 84 and it took nearly 20 minutes.

I also tried a 1,4MB JPG with --quality 85 and it took nearly 10 minutes.

I must assume that this is not normal - is something wrong with my binary?

I am on Ubuntu 16.04 LTS, intel core i7-4790K CPU @ 4.00GHz
I installed gflag via sudo apt-get install libgflags-dev and got libpng via sudo apt-get install libpng16-dev. After that I make with no errors.

convert -quality 85 src.jpg dst.jpg runs in under 1 second, if that is any help.

Anyone else experience this?

[robryk]

It's true that Guetzli is really slow to compress. As a rough order of magnitude estimate, it takes ~ 1 minute / MPixel on a 2.6 GHz Xeon. The time consumed should increase a bit more quickly than linearly with image size.

How large in pixel count are your images?

[iron-udjin]

Are you going to implement multi-threaded image processing? It would be really helpful while guetzli doesn't have CPU/RAM usage optimizations for now.

[jan-wassenberg]

Would it be an option to invoke the binary multiple times in parallel? That's much easier than adding threading everywhere inside Guetzli+butteraugli, and should work unless you need to compress really large images.

[iron-udjin]

Yes, I use this option. But it will be great to have multi-threaded processing for large images in the future. That's why I'm asking about plans for implementation this feature.

[DanielBiegler]

@robryk the big one is around 16MP (5300x3000) so your estimate is roughly in that ballpark.

I was just testing guetzli and didn't dive into its specifics yet, will it be possible to multithread its workload?

For example I wanted to compress around 200 pictures each one is roughly 7-8mb in size. This would take forever. If I hyperthread with 8 threads and each picture takes roughly 20 minutes, the compression would take over 8 hours to complete.

[robryk]

@DanielBiegler If you have 200 pictures, I'd echo @jan-wassenberg's suggestion: run multiple instances of Guetzli and thus process multiple pictures in parallel. This will be more effective parallelization than anything that can be done inside Guetzli.

[kornelski]

@robryk I presume large part of the slowness and memory use is because it's the first release and Guetzli hasn't been optimized yet. How much of the slowness is inherent to the algorithm and unavoidable, and how much can be done to improve the speed?

[robryk]

@pornel We didn't try to optimize Guetzli in ways that could make it harder to modify. That means that there's likely both some speedup available by just optimizing single routines and, more significantly, speedup available by restructuring parts of Guetzli (e.g. attempting to reuse more computation results between iterations).

That said, I believe that much more can be done for memory consumption, which we didn't optimize nearly at all.

[DanielBiegler]

@robryk my estimate already took that into account.

Later, I'll compare the lower end around --quality 30 and if the results are good, maybe using guetzli in the background for a couple of days will be worth it(?) - we'll see.

Thanks for the quick answers.

[slatted]

I've been surprised overall with the performance. I have a 13MB image of Beef Slouvaki, and its been running for 25 min (using between 3 - 5gb of RAM), and it still isn't finished.

I was able to resize a 350kb~ image relatively quick (<2 min)

[robryk]

@slatted The runtime grows slightly faster than linearly with image size. If you want some inkling into what's happening, you might wish to pass --verbose commandline flag.

[slatted]

@robryk thanks, the 13MB image just finished up (right around 30min) and comes in at 2.9MB (@ 84 quality). Visually can't notice a difference. Very cool

[clouless]

is it normal, that a 4MB file needs more than 8GB of RAM inside Docker?

I have guetzli running inside docker and the docker process has 8GB of RAM.
Guetzli process gets killed.

I also tried to give the Docker Daemon 12GB but guetzli still got killed.

you can test it with:

docker run -i -t codeclou/kartoffelstampf:1.1.1 bash
# docker container bash opens

wget -O hires.jpg https://codeclou.github.io/kartoffelstampf/test-images/test-affinity-photo-600dpi.jpg

guetzli hires.jpg hires_comp.jpg

Any hints of what I can do to lower RAM consumption?
Is guetzli not suited to run dockerized?

dockerhub: https://hub.docker.com/r/codeclou/kartoffelstampf/

[robryk]

@clouless The image you're trying to compress has ~70MPix (its size is 8333x8333). According to the readme Guetzli uses ~300MB per MPix, so you should expect it to use ~21GB on that image.

This is obviously far from ideal (both the high constant and the inability to process the image in tiles), but this is within the current expectations. #11 is the issue for reducing memory consumption.