Legacy ImageMagick Discussions Archive

I'm trying to produce tiles of a very large (~2GB) TIFF using the following command:

convert huge_file.tiff -verbose -limit file 10000 -limit area 4000000000 -limit memory 4000000000 -limit map 4000000000 -crop 512x512\! +repage +gravity -background black -flatten +adjoin -set filename:tile "%[fx:page.x/20+1]_%[fx.page.y/20+1]" "tiles/output_%[filename:tile].tiff"

It's way too slow. It takes over six hours. I've tried it without using the -limit settings, still no joy. The CPU usage never goes above 8%, and after a time drops to 2.4% with 2 threads. I want to see that baby maxxed! Memory usage increases from 20.7MB after about 10 minutes to a Gigabyte or so. I have 8GB of RAM and 4 cores. How do I get this thing to take advantage of my resources?

When I sample the process it looks like it's spending a lot of time locking and unlocking mutexes and waiting for semaphores. Am I encountering some kind of deadlock?

Using OS X Snow Leopard on a 2010 Macbook Pro.

Followup:

I've turned on -debug cache. When the memory usage starts increasing after about ten minutes the debug output looks as follows (note the switch from WritePixelCachePixels to ReadPixelCachePixels):

....
huge_file.tiff[0][28554x1+0+80784]
2010-11-10T11:25:59-05:00 5:58.900 25.350u 6.6.5 Cache convert[57286]: cache.c/WritePixelCachePixels/5517/Cache
huge_file.tiff[0][28554x1+0+81600]
2010-11-10T11:25:59-05:00 5:59.030 25.350u 6.6.5 Cache convert[57286]: cache.c/WritePixelCachePixels/5517/Cache
huge_file.tiff[0][28554x1+0+81642]
2010-11-10T11:26:00-05:00 5:59.130 25.370u 6.6.5 Cache convert[57286]: cache.c/OpenPixelCache/4054/Cache
open huge_file.tiff[0] (heap memory, 512x512 2MiBB)
2010-11-10T11:26:00-05:00 5:59.130 25.370u 6.6.5 Cache convert[57286]: cache.c/ReadPixelCachePixels/4781/Cache
huge_file.tiff[0][512x1+0+0]
2010-11-10T11:26:02-05:00 6:01.320 25.410u 6.6.5 Cache convert[57286]: cache.c/ReadPixelCachePixels/4781/Cache
huge_file.tiff[0][512x1+512+0]
2010-11-10T11:26:02-05:00 6:01.320 25.410u 6.6.5 Cache convert[57286]: cache.c/OpenPixelCache/4054/Cache
open huge_file.tiff[0] (heap memory, 512x512 2MiBB)
2010-11-10T11:26:04-05:00 6:03.610 25.460u 6.6.5 Cache convert[57286]: cache.c/ReadPixelCachePixels/4781/Cache
huge_file.tiff[0][512x1+1024+0]
2010-11-10T11:26:04-05:00 6:03.610 25.460u 6.6.5 Cache convert[57286]: cache.c/OpenPixelCache/4054/Cache
open huge_file.tiff[0] (heap memory, 512x512 2MiBB)
...

Any help is much appreciated.

If its a huge TIFF file, most likely its pixels are cached from disk even if the cropped versions are processed in memory. Disk I/O is slow on the standard 5400 RPM SATA drives. The other problem could be OpenMP. We'll assume you have a modern version of ImageMagick. The current release is 6.6.5-7. Turn off multi-core processing like this:

• export MAGICK_THREAD_LIMIT=1

Verify with

• identify -list resource

Now type this command

• convert huge_file.tiff -verbose -crop 512x512\! +repage +gravity -background black -flatten +adjoin -set filename:tile "%[fx:page.x/20+1]_%[fx.page.y/20+1]" "tiles/output_%[filename:tile].tiff"

Now try again with

• convert -limit area 0 huge_file.tiff -verbose -crop 512x512\! +repage +gravity -background black -flatten +adjoin -set filename:tile "%[fx:page.x/20+1]_%[fx.page.y/20+1]" "tiles/output_%[filename:tile].tiff"

Post a URL to your image. We'll download it and report back timing on our quad-core Linux node.

Thanks for the quick response. Set the thread limit to 1 and tried both of your suggested commands. No obvious improvement, although I haven't let them run to completion. Still ~2% CPU after a few minutes. Uploading the TIFF will have to wait as I'm on a 3G internet connection.

I decided to write my own tiling code using the latest libtiff beta (4.0). It completes the tiling process on my 2GB+ and 4GB+ images in about 3 minutes. A bit better than overnight