[OC] How to: Custom RAW editor.

I always wanted to know how RAW files work and how editors interpret the data. So I created a small RAW editor that loads Canon CR2 raw files and has some minor editing tools. I'll lay out the few steps necessary which each RAW editor does to get your image to look as it should. I tried to be as simple as possible so still interesting for those who have no idea about programming or computer science.

CR2 file type

I wrote my own file reader but I suggest anyone who wants to try this, use libraries, much easier but of course less fun ;p

The Canon raw file is the one I focused on since I only own Canon cameras. The file is essentially a TIFF file format containing the RAW data encoded as lossless JPEG. There is a lot of meta data contained like EXIF data and other information for decoding purposes. But most of it is not necessary at all and I skipped most of it.

The RAW data is stored as a Lossless JPEG. Decoding the data was a challenge since it is compressed using a rather (at first) confusing encoding technique. What it stores is essentially the difference between one pixel to the next (this also explains why the more noise the larger the files). (Huffman coding)

Luckily Canon decided to have the data uninterrupted by any potential encoding flags, which makes decoding it a breeze.

RAW data

The RAW data looks like this. What you see here is each line of decoded data in grey scale. This might look confusing at first and trust me the indexing of each pixel is pain to handle especially for different cameras. The sensor in this case is read out in two slices which have to be arranged like this:

|-------------------------|-------------------------| | | | | Slice #1 | Slice #2 | | | | |-------------------------|-------------------------| 

But each encoded line combines two sensor lines RGRGRG....GBGBGBG and so what you see on that image is this:

|-------------------------|-------------------------| | Line #1 Slice #1 RGRG | Line #2 Slice #1 GBGB | | Line #3 Slice #1 RGRG | Line #4 Slice #1 GBGB | | ... | ... | |-------------------------|-------------------------| | Line #1 Slice #2 RGRG | Line #2 Slice #2 GBGB | | Line #3 Slice #2 RGRG | Line #4 Slice #2 GBGB | | ... | ... | |-------------------------|-------------------------| 

You may also notice the black border this is sensor data and is cut off from the final image BUT still used for development purposes which I'll talk about in the next section.

This is how it looks like assembled. Notice the grid like pattern on a pixel basis, this is how the RAW image looks like without any demosaicing and white balancing, which we'll discuss next.

Why slices though? My guess is this allows some type of multithreading and thus the camera can encode the data faster.

RAW data manipulation

Canon stores their 14bit data not exactly as 14bit. First of all they don't set "black" to 0 but have it offset. For my files it was for each color channel [R,G,G,B] -> [2107, 2179, 2180, 2125]. This is calculated using the margin I previously talked about. That margin is not "exposed" thus it is "black". This offset has to be subtracted from each and every pixel.

Now since this is done we have to manipulate each RGB channel with a white balancing multiplier. The RAW file has some WB settings within it but if you use auto WB it will not store the exact settings (or at least I couldn't find them). That is usually done automatically by RAW editors but is essentially the Temp and Tint slider in lightroom. What happens is it applies a multiplier to each channel. Green is usually kept at 1.0 and Red and Blue is adjusted.

After we applied whitebalancing we can finally recreate lost information. What I mean by that is since each pixel only records Red, Green or Blue we have to "interpret" the other colors for each pixel. The sensor array looks like this:

RGRGRGRGRGRG... GBGBGBGBGBGB... RGRGRGRGRGRG... GBGBGBGBGBGB... 

A very naive approach is to simple calculate the mean from the neighboring pixels. Works ok but of course for sharp edges it will look blurred out. The more complex the algorithm the better results you will get. I use something in between which tries to detect edges but it's very very basic, so I won't be expecting good 1:1 results nor will small thin lines look anywhere as good as with professional RAW editors. And last but not least since the image is recorded linearly and our eyes more or less see light in a logarithmic scale. I applied a gamma curve (2.2) before any other image manipulation. Otherwise it just looks dark.

Image Manipulation/Editor

This is how my editor looks like. Very straight forward. From top to bottom:

• Output Histogram

• Input Histogram (makes adjusting the levels easier)

• Levels

• Temperature white balancing

• Tint white balancing

• Saturation

• Curve

• Red, Green, Blue sensor calibration

Most people know the RGB format but a more approachable color space is HSV. HSV encodes color as hue (all colors arranged from 0 to 360 degrees), Saturation and Value (brightness). You can adjust brightness, saturation very easily in this color space.

Most editors have the same tools: Curves, Levels or White, Black, Shadow sliders etc. What they do is essentially the same: They map the brightness value of each pixel to a new one. This is all pretty straight forward if you know how each of these tools map colors. But the more interesting one is the sensor calibration sliders at the bottom.

Each sensor has it's own "characteristics" in how each pixel records its designated color spectrum. They don't just record red, green or blue. They record the amount of red, green and blue. This means that all colors in between will be recorded differently for each sensor. These sliders help me adjust the color bias. Lightroom allows you to shift hue but Canon is pretty dead on with their red green and blue so I only allow a change in contribution. I used a normal distribution at Red, Green and Blue and scale each color using the slider values. I'm most likely wrong with that approach but it works surprisingly well and you can recreate pictures profiles with these sliders as well.

Notice the render button, I can't just update the image each time I modify the input because it takes roughly 200ms to render the full image and another half a second to display it (java sucks at displaying large image files) so I prefer doing it manually instead of waiting every single time. This could be improved upon easily but not worth it ;p

Problems

Demosaicing is the main difficulty with interpreting RAWs, so here some problems that occur when you use a simple/bad algorithm:

Fine details

Different colored edges

High contrast edges

And of course all the nice features like lens distortion, CA, noise reduction and other fancy tools are way harder to implement.

Comparison

Quick Lightroom edit vs Mine (Screenshot of my editor for this result)

As you can see my color profile fix is going wrong on that branch. But overall I think it's not that bad, right? ;p

But even for some very demanding scenes (had to do a lot of shadow pushing here) my editor does surprisingly well when scaled to web use: http://bit.ly/2swxYAc far from perfect and I will not stop using lightroom but still I'm happy with the results.

Ask any question, I'll probably won't be around for the next few hours but I'll get back to them first thing I'm back on reddit.

submitted by /u/photenth to r/photography
[link] [comments]