Page 1 of 1
convert is toooooo slow with many images
Posted: 2015-06-13T17:47:06-07:00
by saulortega
Hello.
Version: ImageMagick 6.6.9-7 2014-03-06 Q16
http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP
I'm running this command:
Code: Select all
convert *.JPG -morph 2 -monitor -normalize mov/%05d.jpg
With 10 images, it takes
15 seconds. With 30 images, it takes
1 hour!.
With 10 images, it do not writes on disk. With 30 images (~1.2 MB) it writes 400 MB in temp disk. With 50 images 3 GB!
Why???!!!
Re: convert is toooooo slow with many images
Posted: 2015-06-13T18:51:22-07:00
by fmw42
You are reading every image into memory. That will slow things down. See
http://www.imagemagick.org/Usage/files/#massive for processing large image(s).
Re: convert is toooooo slow with many images
Posted: 2015-06-14T05:58:08-07:00
by saulortega
Code: Select all
However your system may reject large memory requests and still force IM to cache the image to disk (about 1000 times slower).
That explains it.
Is not there some way to tell IM not load all images in memory at once but in twos?
Re: convert is toooooo slow with many images
Posted: 2015-06-14T06:29:29-07:00
by magick
You could write a shell script to call the convert utility for every two images. You might get some improvement by upgrading to the latest ImageMagick release @ 6.9.1-5. You might get some improvement by forcing all pixel access to disk (to prevent swap thrashing) by adding "-limit memory 2mb -limit map 2mb" to your command line. You might get some improvement by setting the MAGICK_TMPDIR environment variable to a partition other than /tmp, instead use /data/tmp, for example.
Re: convert is toooooo slow with many images
Posted: 2015-06-20T15:40:07-07:00
by saulortega
Well, I programmed my own script. Now I takes 1 hour processing 1000 images.
Re: convert is toooooo slow with many images
Posted: 2015-10-05T07:49:31-07:00
by Nemo_bis
Another experience: to create a PDF from 24 JPG files, convert eats 3 GiB RAM and 4 GiB swap, bringing the machine to swapdeath, and is eventually killed by Linux.
This makes sense given
viewtopic.php?f=1&t=28438&p=126258 says we need 8 bytes per pixel and these are 35 MPx images: 35*10**6*24*8/1024**3 = 6 GiB memory needed. And I'd need to convert ~300 such images, not just 24!
Changing the resource limits makes it die faster but not complete the job, it just can't take this much memory on my 8 GB RAM machine.
Code: Select all
$ identify -version
Version: ImageMagick 6.8.8-10 Q16 x86_64 2015-03-10 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fftw fontconfig freetype gslib jng jpeg lcms ltdl lzma openexr pangocairo png ps rsvg tiff webp wmf x xml zlib
$ convert -list resource
File Area Memory Map Disk Thread Throttle Time
--------------------------------------------------------------------------------
38400 15.948GB 7.4266GiB 14.853GiB unlimited 2 0 unlimited
$ time convert 000*jpg output.fromjpeg75.pdf
Ucciso
real 7m23.713s
user 0m21.734s
sys 0m13.826s
$ time convert -limit memory 7GiB 000*jpg output.fromjpeg75.pdf
Ucciso
real 0m36.269s
user 0m22.991s
sys 0m10.424s
In contrast mogrify completes the job in a reasonable time, but then I have to merge the individual PDF files. As magick suggested in another thread about large collections of TIFF, I tried tiffcp + tiff2pdf instead and it's indeed way faster: 1 min instead of 4 for the same PDF.
Code: Select all
$ time mogrify -format jpg -quality 75 000*tif
real 1m46.553s
user 1m6.503s
sys 0m17.276s
$ time mogrify -format pdf 000*jpg
real 4m10.036s
user 3m19.764s
sys 0m23.655s
$ time tiffcp -c lzw 000*tif out.lzw.tiff
real 0m57.404s
user 0m35.480s
sys 0m4.750s
$ time tiff2pdf -o output.jpeg75.pdf -j -q 75 -u m -F out.lzw.tiff
real 0m55.016s
user 0m49.864s
sys 0m4.687s
$ time tiff2pdf -o output.lossless.pdf -z -u m -F out.lzw.tiff
real 4m15.793s
user 4m1.839s
sys 0m9.028s
Re: convert is toooooo slow with many images
Posted: 2015-10-05T15:44:32-07:00
by magick
Note, ImageMagick resource limits are enforced for ImageMagick, not its delegate libraries. We cannot control, for example, how much resources Ghostscript consumes or the JPEG library.