Converting Log DPX to sRGB

[timurhai]

Hello.

I am trying to convert 10bit Log DPX image to sRGB TGA (or JPEG).
I read documentaion and try several variants with no success.

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace Log -colorspace sRGB colorbars.tga

Code: Select all

Version: ImageMagick 6.8.8-7 Q16 x86_64 2014-02-19 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC OpenMP
Delegates: fontconfig freetype jng jpeg openexr png tiff x zlib

colorbars.dpx DPX 1280x720 1280x720+0+0 10-bit TrueColor RGB 3.695MB 0.030u 0:00.030
colorbars.dpx=>colorbars.tga DPX 1280x720 1280x720+0+0 10-bit sRGB 2.765MB 0.160u 0:00.029

The result was a Linear (RGB as you call) image.

I read forum and founded that there are lots of threads on this theme.
But some are not suitable for me, some are not complete (both sides did not got each other).
Also there is a mess with RGB, sRGB and ImageMagick version, so I dowloaded and build the latest version (6.8.8-7).

It seems that if DPX file is Log (or you specify it by "-set colorspace Log") it converts it to linear (RGB).
- checked it in other softwate (Nuke), linear exactly,
if convert this linear to sRGB the result is the same as if convert log in srgb in Nuke directly

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace Log -colorspace RGB colorbars.tga

- the result even darker than Linear.

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace sRGB -colorspace sRGB colorbars.tga

- the result is Log (no change)

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace RGB -colorspace sRGB colorbars.tga

- the result even lighter than Log

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace RGB -colorspace RGB colorbars.tga

- the result is Log (no change)

Code: Select all

convert -verbose -version colorbars.dpx -set colorspace Log -colorspace sRGB -colorspace sRGB colorbars.tga
convert -verbose -version colorbars.dpx -set colorspace Log -colorspace sRGB -colorspace RGB colorbars.tga
convert -verbose -version colorbars.dpx -set colorspace Log -colorspace RGB -colorspace sRGB colorbars.tga
convert -verbose -version colorbars.dpx -set colorspace Log -colorspace RGB -colorspace RGB colorbars.tga

- gived incorrect results too

So it seems that i did't got how to convert Log to sRGB right, and what "-set colorspace" and "-colorspace" exactly do.
Could you help me, please.

[magick]

We can reproduce the problem you posted and have a patch in ImageMagick 6.8.8-8 Beta available by sometime tomorrow. Thanks.

If you post a URL to your colorbars.dpx image, we'll verify our patch against this image.

[timurhai]

Thank you for a quick reply!

Here i generated 10,12 and 16 bit Log DPXes:
http://cgru.info/test/imagemagick/
colorbars.jpg file shows how sRGB image should look

Also i placed a real film scan and a jpeg to show it.

ps
The film is enough old, you can place/show scan anywhere.

[magick]

The colorbars_log1{0,2,6}bit.dpx image files include metadata that declare the images as RGB. Do you think they should be in the Log colorspace instead? As you know you can override the colorspace with the -set colorspace option. Is that a reasonable thing to do? Ok, let's try that with ImageMagick 6.8.8-8 Beta from http://www.imagemagick.org/download/beta:

• convert colorbars_log10bit.dpx -set colorspace Log -colorspace sRGB -depth 8 colorbars.png

Yup, the patch in ImageMagick 6.8.8-8 Beta works, colorbars.png matches the colors of the reference image colorbars.jpg.

[timurhai]

Hi! Great! It works!
All DPXes are converted right!
DPX metadata is not always correct, so flag "-set colorspace" is very useful.
( it's a pity, but some programs can modify colorspace and does not modify metadata, Nuke for example )

But now the same issuse with OpenEXR
EXR stores data as a float (half by default). And it is enough precision to not to use any colorspace (better not to modify color data at all). So default EXR colorspace is Linear (no colorspace).

Code: Select all

convert -verbose -version some_film.exr -set colorspace RGB some_film.exr.tga
convert -verbose -version some_film.exr -set colorspace sRGB some_film.exr.tga

- gives the same result - Linear.

EXR images (Linear colorbars) are here too:
http://cgru.info/test/imagemagick/

ps
Non floating precision formats as tga, jpg and most others, can't keep original colors - not enought precision for it and there will be artefacts - as monitors (and most display devices) has gamma (usually 2.2 (1.8 for apple)). And we will see 256 (8 bits per channel) intensity gradation in shadows (this is called "banding effect"). So for such formats sRGB colorspace is used (sRGB is about gamma 2.2 - but it is not - especially in shadows ).

pps
i just trying to explain that we (films post-production) are using log DPX and linear EXRs
- this formats can handle high dynamic range images that can be displayed in cinema
(sRGB formats for previews on a common monitor only)

btw
i am writing a free (and open-source) program too:
http://cgru.info/rules/#/CG_PROJECT/SCENES
web gui (javascript->php->python->imagemagick->ffmpeg) to generate previews
that is why i am so interested in right conversion to sRGB

[magick]

Unless an image format specification requires sRGB, such as PNG, we write the format in the colorspace of the source image. So a linear EXR RGB images is written as linear RGB to TGA, for example. We leave it to the user to decide which colorspace to save the image. In your example, use:

• convert -verbose -version some_film.exr -colorspace sRGB some_film.exr.tga

If the TGA specification requires sRGB as its colorspace, let us know and we can add a patch to ImageMagick to automagically convert linear RGB to sRGB before we write the image.

[timurhai]

Almost all image formats requires sRGB. Only DPX, EXR (float TIFF and some rare others) not use sRGB - one use Log and others can handle any colors without any colorspace at all.
For example all JPEG files we shot by photo has sRGB, any image on any web site and anything that designed to be shown on a common monitor has sRGB. If it was not so we all will see banding anywhere.
Our human eye http://www.telescope-optics.net/eye_int ... sponse.htm is about Log colorspace, and we can't see 256 intensity levels without banding. To smooth gradient monitors use gamma 2.2 ( and any tv-set)
- this came long ago from CRT devices that has about gamma 2.2 function between voltage and result luminance - this occasional historical reason help us not to see banding by Log eye when we split intensity on 256 levels when we came into the digital age.
Now TFT (and all not CRT) devices repeats that CRT gamma 2.2 specially for humans not to see banding.

So, do you understand that we can't store an image with no colorspace and just 256 intensity levels for a human eye?
You can open any graphics redactor (Gimp for example), make 3 colors: (0,0,0) (127,127,127) and (255,255,255) and you will see by eye that gray (127) is much closer to 255 and do not have middle intensity.
- i just draw a gradient in a Gimp, in the middle was 127, but eye see that gradient is not linear and shifted to white. Only when i applied levels and set gamma to .4545 (1/2.2) i see a linear gradient (middle intensity in the middle of the gradient) and a banding, of course.

So when you save a Linear image to tga (jpeg,png,any not float or log), you loose information in shadows. Images that operates all common people are sRGB, most people does not know about it - and there is no need to know it for most common image operations. But who knows prefer to perform operations in a linear colorspace (without sRGB colorspace). You should not be a professional to notice how defocus and motion blur works in linear and sRGB colorspaces when highlights and shadows are near - in linear it is much more natural ( so does any operation that mix high and low intensity colors together ).

There is lots of explanations on this theme in inet.
- i just searched an internet (but haven't read )
http://www.w3.org/Graphics/Color/sRGB.html
http://www.comp-fu.com/2012/03/linear-g ... on-part-2/
http://www.comp-fu.com/wp-content/uploa ... mation.gif
http://en.wikipedia.org/wiki/SRGB

Most common people shoot a photo (sRGB) open it in a gimp, photoshop or any other soft that does not change colorspace (sRGB), perform levels, crop, and save an image (in a common sRGB format). It seems that the same workflow in imagemagick too. And this is normal when everywhere is sRGB ( and you are not defocus gourmand ).
Professional soft opens an image and immediately translate it from some colorspace (from Log or sRGB) to a physical intensity values (linear colorscpace) - the exception is EXR that has natural colors already. Show user a viewer with sRGB LUT. Perform operations with a physically correct color values (defocus gourmands are very glad here), than on save it applies colorspace back (if format is not EXR) - log for DPX and sRGB for others.
But in this case soft must have floating point internal engine - or it loose colors.
If imagemagick engine operates with integers and does not have floating point math, it can (professional color correction gourmands use other soft, usually designed for a special hardware). But you should know, that you always operate in sRGB colorspace.

I hope it is helpful for you and sorry if you does not need any lessons )

Back to our concrete images and commands:

Code: Select all

convert -verbose -version some_film.exr -colorspace sRGB some_film.exr.tga

- works!

Unless an image format specification requires sRGB, such as PNG, we write the format in the colorspace of the source image. So a linear EXR RGB images is written as linear RGB to TGA, for example. We leave it to the user to decide which colorspace to save the image

- i think this is right, colorspace should be kept with no change, if user does not manually specify it for an input and\or output image.

If the TGA specification requires sRGB as its colorspace, let us know and we can add a patch to ImageMagick to automagically convert linear RGB to sRGB before we write the image.

- yes tga and all other common formats (not floating point) reqires sRGB, and there should be a conversion if colorspace is not sRGB.

And now there is some mess with the different ways we convert Log to sRGB and Linear to sRGB.

ps
professional photographers ( and lots of people that just think so ) prefer to operate raw camera formats too - the same reason ( or they just heard that something should be better )

pps
photoshop and gimp has an integer color math and it is more than enough for lots of people that use them for profession reasons

[magick]

• yes tga and all other common formats (not floating point) reqires sRGB

Can you point to a reference? sRGB is mostly a defacto-standard. Not a standard embedded into many image format specifications. ImageMagick is designed such that the user decides the workflow-- such as which colorspace transforms to apply (typically with -colorspace or -profile). We are reluctant to force a transform from linear to non-linear RGB unless the image format standard clearly states that sRGB is a required colorspace. The PNG specification, for example, requires a sRGB chunk to distinguish between linear and non-linear RGB. The chunk specifies the rendering intent.

[fmw42]

Most common people shoot a photo (sRGB) open it in a gimp, photoshop or any other soft that does not change colorspace (sRGB)

Both PS and GIMP have a color control panel that often opens when an image is loaded and asks which colorspace to use. So it is not really automatic, but up to the user to decide.

[timurhai]

Sorry, i did not got you enough.

You say that "sRGB is mostly a defacto-standard" - so all formats that does not define their own standard are sRGB.
DPX defines standard - Log (Cineon)
EXR says that they are linear and we need to convert it to sRGB to see on a common monitor.

http://www.w3.org/Graphics/Color/sRGB.html
- can it be the reference, or what do you expect?

We can write any color values in any format, format can not to define any colorspace. We just assuming that jpeg is sRGB, and no matter what chunk can format have, we definitely can't write linear data to jpeg - banding + compression will produce huge artifacts. You can open any dark JPEG (not matter what colorspace it is - it just a convention) and lets assume that it is linear. So you can simple apply gamma 2.2 to see it real colors on a common monitor (gamma 2.2 too) - and you will see what shadows JPEG file has.
Do you understand what i am trying to say?

You can think that imagemagick is enough far from this theme, but EXR files needs conversion to be viewed - that is a fact. And no matter what is the history and reasons of such conversion.
Just assume that "to view file" (on a common device) is the same as "to write tga" or jpg, or any other format that does not need any conversion to view.
http://www.openexr.com/photoshop_plugin.html
- lets imagine that photoshop is common device to view images, in that docs they just apply gamma 2.2 to EXR - this is just simplified sRGB conversion.

ps
the same was for DPX - need conversion for view on a monitor
- but double: at first from Log ( not floating point method to store high intensity - apply log and loose some precision in highlights - human eye is log too and so we have less precision in highlights too )
and than to sRGB to view on common device (to white jpeg).

[magick]

Understood but there are articles like http://rizzy3d.wordpress.com/2012/06/27 ... w-in-maya/ that say:

Note: The textures we import in Maya are in sRGB if its a JPEG, PNG or TIFF format. Files like EXR, HDR, TARGA etc are in Linear space.

Or see http://www.sidefx.com/docs/houdini13.0/render/linear which says:

• OpenEXR, Targa Linear (colorspace)

[timurhai]

Ok. Houdini and Maya and may be some other 3d soft assume that TGA is linear, may be they also assume that it has at least 16 bit per channel (but as far as i know it can't). Because there is no way to store linear image with just 8 bit per channel (by the reasons that i told before and can be simple founded in internet too).

We are not using TGA in 3d textures (never heard about it) favorite texture artists format is tif, i saw many other formats as textures, but not tga.
2d software opens TGA with no conversion (so they assuming that it is sRGB). Gimp, Photoshop, After Effects, Digital Fusion, Apple Shake are not converting TGA, Nuke converting it from sRGB t linear.
- my opinion that 2d soft such as Nuke and Adobe After Effects and Photoshop know about TGA better than 3d maya-houdini (and for textures - with a such simple care about sRGB <-> RGB by a simple gamma function).
It is a television format (and even there it is not much popular). Several times we get sources and put results back in TGA for TV - all was in sRGB.

Later, just after TGA:
http://www.sidefx.com/docs/houdini13.0/render/linear

Other - If the data is stored in 8 bit fixed format, it is assumed to be sRGB, otherwise linear.

- they knows too, that it is impossible to store linear colors in 8 bits!

I Searched inet too, founded nothing about TGA colorspace (and its specifications pdf too). If not specified - it is sRGB! ( no conversion means sRGB and not linear ( linear should be shown with a conversion) ). It has some "color map" in specifications, but i think it is not usually used.

Even if you prefer houdini-maya side - it is your choice (but better to count software that assume that TGA is linear and soft that not).
Was is with a JPEG? Even houdini-maya assume that it is sRGB.

================================================================================================================================

btw
I use TGA for temporary sequence only (to encode it later). At first i used more popular format PNG - but once i occasionally have a deal with imagemagick<->TGA and noticed that your program writes it much faster. This is the only reason i use TGA
May be i should use some other format, i should repeat tests ( that was about 3-5 years ago ).
- later i generate previews from a special sequences that was already designed for preview and were in sRGB (usually simple JPEG sequence), and now i have a task to generate previews from production sequences, they are only DPX and EXR, may be TIFF ( that can be and RGB(16 bit) and sRGB and float and Log(16 bit))

ps
Do you really understand that we can't store linear colorspace in 8 bits per channel? ( linear colorspace means without any colorspace )

pps
My opinion that TGA is a dead format: bad compression, max 8 bits per channel, max 4 channels, no layers.
It still exists only because some TV studios hardware still works and still waits TGA on input.
- sorry if i am bothering you about TGA, we can forget about it. Lets focus on JPEG.

ppps
As soon as it will be possible, on my work i'll try to check other formats speed and convert tool.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
May be houdini and maya developers read TGA specification and mixed up pixel and channel )) and they think that it can handle 32 bits per channel and not per pixel ))

[fmw42]

linear colorspace means without any colorspace

No it does not. It usually means that the data is either linear (gamma=1) RGB or non-linear (gamma=0.4545) sRGB. If linear, the normal sRGB gamma and data have been corrected to 1 from 0.4545 which is normal for most image formats. That usually makes the image darker than the non-linear sRGB.

According to http://www.dca.fee.unicamp.br/~martino/ ... tgaffs.pdf, there is a field that stores the gamma value.

On quick review, I see no field for colorspace. So software will have to rely upon the gamma value to know whether to think of it as RGB or sRGB. The problem is that without a colorspace field, one cannot tell whether the image is sRGB with a modified gamma=1 or whether the data has actually been modified from gamma=0.4545 to conform with gamma=1.

In IM that difference is whether one uses -set colorspace RGB or -colorspace RGB. But without a colorspace field, one would not know whether the data is actually changed or just the gamma field modified without changing the pixel data. Therefore assumptions must be made which way to interpret the gamma=1 linear value.

[snibgo]

Do you really understand that we can't store linear colorspace in 8 bits per channel?

This is not true. Any number of bits may represent linear, or sRGB, or anything else.

But I would agree with: "If data has to be represented in only 8 bits, sRGB is usually a better choice than linear RGB."

[timurhai]

Hi!
I did not said that you can not write linear colors to a 8 bit format. Of cource, technically you can write any 0-255 number in any 8bit format. But if it will be linear, you will se banding when you will apply gamma 2.2 to see it. Especially jpeg compression artefacts are very visible after it.

I always agree that user should choose and specify colorspace.

Back to our files and commands:

Why we convertiong Log DPX to sRGB JPEG by "-set colorspace Log" and Linear EXR to sRGB JPEG by "-colorspace sRGB" (logically it should be "-set colorspace RGB" ?

But i can think that it just a program particularity. I've already got methods to operate and DPX and EXR.

Just want you to understand that we always operate sRGB and always look at it through devices with gamma 2,2. Only few professional programs operate linear colors (and this is not gimp, photoshop). And any JPEG is sRGB as we can't see it without compression artefacts after gamma 2,2.

ps
Any way, thank you for cooperation )