Liquid Rescale help page

The Liquid Rescale plugin is an implementation of the content-aware resizing by seam carving algorithm by Shai Avidan and Ariel Shamir.

It aims at resizing pictures non uniformly while preserving the features of the picture, i.e. avoiding distortion of the important parts of the picture. It can also be used to remove portions of the picture in a consistent way.

It works both ways, but enlarging gives better results if done in successive steps. It can use extra layers as masks to select which features of the image should be preserved and which should be discarded.

See also Appendix: The seam carving technique in brief at the end of this page for an introduction to seam carving concepts.

The plugin works on the active layer or floating selection. It can also operate on multiple layers through the "Filter all layers" plugin and using the GIMP Animation Package facilities for video editing.

The plugin only works on whole layers, if a selection is present it is saved to a channel and removed. If the layer has a transparency mask, an option is given to select the behaviour (apply/discard).

User interface description

Dialog example

Select new width and height

In this section it is possible to choose the size and the mode of operation. It is advisable to rescale always in one direction at a time. If both the width and the height are changed, rescaling is performed by default on the width first, then on the height. This can be changed in the "Advanced" tab.
The two small buttons on the right can be used to reset the size parameters to the image size (top) or set them to the last used values (bottom).
The default mode is just liquid rescaling to the desired size. The other options are to apply liquid rescaling or the standard scaling back to the original size once the first liquid rescaling is done. This can be useful for example for object removal and content enhancement.

Note that the final size can be set automatically when removing objects, see below

Feature masks

Quick guide

The easiest way to manually select the features of the image that you want to protect or discard is the following:

  • Select the layer you want to resize (not a floating selection) and invoke the plugin
  • Press the "New" button in the "Feature preservation mask" section (this will create a transparent layer, with 50% opacity, and pick an appropriate colour).
  • Put the plugin dialog aside and paint on the areas of the image you want to preserve
  • Go back to the plugin dialog and press the refresh button to see the mask you painted
  • Repeat everything for the features to discard

If you want to remove portions of the image, you can choose the scaling direction (horizontal/vertical) and finally press the "Auto size" button to determine the final size automatically (this is only reliable when dealing with very simple masks).

Note #1: object removal is only possible when shrinking. By default, feature discard masks are ignored when enlarging, because in that case the masked areas would be inflated rather then removed. If you actually want to get this effect, you need to unset the corresponding option in the "Advanced" tab.

Note #2: preservation of features is not possible if enlarging too much, because the inflation process is the exact reverse of the shrinking process, so the maximum amount of pixels you can add to a layer corresponds to the amount of pixels which are not protected. For example, if you have a 1000 pixel wide image and you have marked a 800 pixel wide area for protection, the final width should be less than 1200 pixels. If you want to enlarge more, do it in steps.

Full description

In the "Feature preservation mask" section it is possible to specify if there are regions of the image which should be preserved from modification, by checking the "Activate feature preservation" box, and by choosing a layer among the ones which belong to the current image (excluding the active one) in the "Available layers" menu. The "New" button creates a transparent layer with 50% opacity and an appropriate name, and sets it as the selected layer mask (note that this fails if the plugin is invoked on a floating selection, because GIMP does not allow to work on other layers in such case). It also sets the foreground colour to green; the previous colour is restored when the dialog is closed.
The intensity of the effect of the mask on each pixel is obtained as the average on the pixel's colour channels, multiplied by the alpha channel. Thus, for an RGB layer with an alpha channel, the maximum value is achieved on white pixels, while black ones and transparent ones are ignored. In general, it is advisable to use a single colour for each mask, and, if needed, fine-tune the strenght of the mask for different areas by using the transparency value, instead of lightness or saturation.
The size and position of the selected layer can be arbitrary, the plugin will only use those portions which overlap with the active layer. The "Strength" scale can be used as a global parameter to modulate the feature preservation. Note that thinner selection areas (in the sense of the resizing) need higher strength values, and vice-versa.

The Feature discard selection section is nearly identical to the previous one, but the effect is reversed, and the default colour for the "New" button is red instead of green.
Furthermore, there's an additional button, "Auto size", which can be used for "quick & dirty" object removals. When this button is pressed, the plug-in tries to compute the exact amount of rescaling which is needed in order to remove the areas of the image marked by the current discard mask, and sets the final size accordingly (in the "Select new width and height" section). It will always produce a single-direction rescale, either horizontal (it changes just the width and sets the height to its original value) or vertical (it changes just the height and sets the width to its original value); which direction to use must be chosen in the menu nearby the "Auto size" button, before using it.
Note that this option is only suitable for naïve object removals, and it may fail if the masks are non-trivial.
[Technical information: the final size is determined by simply finding out the maximum number of pixels in a line for which the mask value is above a given threshold, which is set to the 50% of a full RGB colour (e.g. red in the default setting).]

Output

Output

This tab has a number of options related to the output of the plugin.

Output on a new layer
Specifies if the resized image should be written on a new layer. By default, it is off.
Resize image canvas
Specifies whether the image canvas should be resized to meet the new size of the active layer. By default, it is on.
Resize auxiliary layers
Specifies whether the layers used to manually select the features of the image or to specify a rigidity mask will be resized along with the active layer. If this option is checked, those layers will be first cropped to the active layer's size, then they will undergo the same seam carving and inserting process as the active layer. By default, it is on, but it is only active when some layers are selected in the "Feature masks" tab or if a rigidity mask is being used (see below).
Output the seams on a new layer
Specifies whether to create a new layer with the seams map. If this option is checked, two buttons appear on the right of it to let the user choose the seams colour range. The seams are computed on the original layer as far as they are needed for the new size (e.g. there will be 100 seams if reducing or enlarging by 100 pixels), the remaining areas are transparent. In order to read the result, use this option together with "Output on a new layer", and overlay the seams map to the original layer. If you rescale in two directions at once, you will obtain two maps, but the second one will be computed over an invisible intermediate image, so this is not very useful. The same applies in the case you choose to rescale back to the original size with LqR.

Advanced

Advanced

This tab has some options which can be used to tweak the plugin behavior.

Seams control
Max transversal step
This option lets you choose the maximum transversal step that the pixels in the seams can take. In the standard algorithm, corresponding to the default value step = 1, each pixel in a seam can be shifted by at most one pixel with respect to its neighbors. This implies that the seams can form an angle of at most 45 degrees with respect to their base line.
Increasing the step value lets you overcome this limit, but may lead to the introduction of artifacts. In order to balance the situation, you can use the rigidity setting.
Overall rigidity
Use this value to give a negative bias to the seams which are not straight. May be useful to prevent distortions in some situations, or to avoid artifacts from pixel skipping (it is better to use low values in such case). This setting applies to the whole selected layer if no rigidity mask is used.
[Technical information: the bias is proportional to the difference in the transversal coordinate between each two successive points, elevated to the power of 1.5, and summed up for the whole seam.]
Use a rigidity mask
It is possible to specify different rigidity values for different areas of the image, using an auxiliary layer, by following the same procedure used to specify the feature masks (the default colour for newly created rigidity masks is blue). If a rigidity mask is specified, its value at each pixel (computed in the same way as for the feature masks) will be used as a rigidity coefficient, to be multiplied with the overall one.
Note #1: transparent (or black) areas in the rigidity mask always correspond to 0 rigidity, no matter what the overall value is.
Note #2: rigid areas have a slight repulsive effect on seams, similar to that of a preservation mask. In order to compensate for this, you may use a discard mask.
Gradient function
It is possible to choose which function is to be used to calculate the energy map of the image. For each pixel in the image, the x,y components of the gradient are computed from the pixel's four nearest neighbors. The exact form of the energy map depends on the choice of the gradient function, and this in turn determines which elements of the image are more important, and should thus be preserved, and which are not, and can be removed (if shrinking) or inflated (if enlarging). Using the Null function in conjunction with the feature preservation mask can be used to override completely the automatic feature detection, thus allowing the user to generate his own energy map (e.g. by another plugin).
Rescale order
When rescaling in both directions at the same time, it can be chosen whether to resize first in the horizontal and then in the vertical direction, or vice versa.
Ignore dicard mask when enlarging
The effect of the discard mask is reversed when the plugin is used for enlarging an image; as this is normally undesirable, they are ignored by default when the first scaling direction is to be enlarged (in such cases, a small warning icon will appear in the "Feature discard mask" section). Unset this option if you want to override this behaviour.

Using the plugin for video and animation editing

Using the plugin with GAP

The plugin can take advantage of the GIMP Animation Package (GAP) for video editing, making it possible to apply liquid rescaling to all the frames of a video (please refer to the GAP documentation for instructions about video editing in GIMP).

Note however that the GAP facilities only allow to select the working layer, but the liquid rescale plugin needs auxiliary layers as well to work at its full potential. In order to work around this problem, auxiliary layers are set according to their names: therefore, if you want to use a preservation mask in each frame, for example, you should take care that all the layers which you intend to use as such have the same name in all of the frames. The same applies to discard masks and rigidity masks.

In order to use liquid rescale on all frames, you need select "Frame Modify" in the "Video" menu; this will open a dialog: in the "Function" section select "Apply filter on layer(s)", then set appropriate rules for layer selection and frame range and click OK. Another dialog will appear, asking you to choose a filter: select plug-in-lqr, then choose either "Apply constant" or "Apply varying".
In the latter case, you will be presented the liquidrescale plugin dialog twice, once for the first frame and once for the last one: these will set the extreme values for all the graded parameters (such as sizes, strengths, rigidity etc.), and intermediate values will be used in the intermediate frames (for example, if you set the width to 100% for the first frame and 110% for the last one, and there are 11 frames overall, intermediate frames will have widths of 101%, 102%, 103% and so on). However, all the on/off options (such as the resize order, the auxiliary layer names etc.) will be taken from the last dialog for all the intermediate frames.

Applying the plugin to all layers (e.g. for animations)

It is also possible to apply the plugin to all the layers in a given image (for example when editing an animation), using the option "Filter all layers..." from the "Filters" menu and choosing "plug-in-lqr" in the dialog which appears. Operations follow the rules explained in the GAP section; however, you will run into problems if using auxiliary layers as masks, since the plugin will be applied to those as well.

Appendix: The seam carving technique in brief

The plugin works by finding so-called "seams" over an image, i.e. continuous, zig-zagged lines transversing the image from top to bottom ("vertical" seams), or from left to right ("horizontal" seams). When one such seam, say a vertical one, is removed from an image (the "carving" operation), the width of the image is reduced by one pixel. Removing horizontal seams reduces the height. Iterating such operations (find a seam and remove it), one can reduce the image size at will. Collecting together all the seams which were carved from an image, in their respective order, constitutes what is called a "seams map".

Mirroring the carving process, by inserting additional seams besides the ones which are found by the algorithm, instead of removing them, image enlargement can be obtained, too. The meaning of the seam map is reversed in this case.

In order to get good results form this technique, the main issue is finding which are the most suitable seams to carve or to insert. If the aim is simply to change the proportions of the image without affecting too much the content, for example, "good" seams will be those which don't cross important features of the image, and instead pass through a background landscape.

By default, the plugin is tries to find the seams which cross the lowest-contrast areas (how this happens exactly is specified by the gradient function choice in the "Advanced" tab). Therefore, each pixel of the image is assigned a so-called "energy value": the higher the contrast, the higher the energy, and seams are less likely to cross high-energy areas.

Since in many cases this simple contrast-based method is not optimal, the energy function can be "biased" by the user, who can decide that some areas should have a higher energy (using a preservation masks) or, on the contrary, that they should have a lower energy (using a discard mask), and therefore effectivily "drive" the seams and the whole process.

Rigidity masks work in a similar way, but the energy bias that they add does not only depend on the mask value over each pixel, but also on how much the seams which try to cross that pixel are straight.

One intuitive picture is as such: the seams try to get from one side to the other of the image keeping on a path with an energy as low as possible; preservation masks have the effect of putting a barrier on their path, which, if high enough, deviates the seams away from it; discard mask, on the other hand, are like valleys which attract seams. Rigidity masks have the effect of putting barriers besides each seam, which tend to make it straighter.