Speed up -unique-colors on a large image

[JohnsonZA]

I'm using -unique-colors on fairly large (6400x25500) TIFF files that are LZW compressed and I'm struggling with both memory requirements and execution time. Each image should have around 10 unique colors.

Since the unique algorithm is O(n log n) complexity, I was wondering if it would shorten execution time by splitting the image up to calculate the unique colours for each cropped region and then combining the results.

My first attempt looks like this:

Code: Select all

convert test.tif -crop x2000 -unique-colors -append -unique-colors temp.png

This approach obviously uses the most memory because it's loading the TIFF as well as all the cropped regions into memory, but it seems to be the fastest. I've toyed around with the size of the cropped regions but can't decide where the overheads become a disadvantage.

Another approach is to use MPC files, which obviously slows down the process, but halves the memory usage.

I've noticed that only one core does all the work when -unique-colors is running. Is there no way to parallelise this function?

Does anyone have any insights as to how to get fast results, while at the same time trying to limit memory use or is this as good as it gets?

[fmw42]

If your images have only about 10 TRULY unique colors (and not just visually appearing that way), then you could try simply reducing the size of your input images, say by 10 using a non-interpolating reduction such as -sample and then get the unique colors.

convert image.tif -sample 10% -unique-colors ...

[snibgo]

I suppose you are using Q-8?

[JohnsonZA]

If your images have only about 10 TRULY unique colors (and not just visually appearing that way), then you could try simply reducing the size of your input images, say by 10 using a non-interpolating reduction such as -sample and then get the unique colors.

convert image.tif -sample 10% -unique-colors ...

That works well, but I'm worried about small regions of colour disappearing when using a nearest neighbour sampling. Thanks for the suggestion.

I suppose you are using Q-8?

Yes.