There is a video that explains the technique in mode detail from the Siggraph presentation, and several image processing applications have implemented it. GIMP happens to have an implementation of this algorithm (called Liquid Rescale), so I gave it a try to see how well it works.

Below is our test image, which is a shot taken at the summit of Little Si. Using Liquid resize, I first tried to reduce the image to about half the width. The output was a decent image (regarding the inpainting job). The blending of the background (which does not have clear unique shapes) was good, but when it came to the subjects of the photo, it is clear that the algorithm can’t really determine which areas are good to remove and which are not.

I tried next to provide feedback regarding areas to preserve. I decided to keep myself in the picture and make any other area eligible for removal:

Providing feedback definitely helped, though it ended up with a distorted view of my friend. That is where the opposite feedback of areas to remove comes into play:

Now I have a decent image. There are some artifacts still in the front, which are not uncommon with inpainting techniques, but in general I think the output is good. I tried to upscale the image to twice the size with no hinting first. Again, not indicating areas to preserve is not a good idea.

After providing a hint to preserve both my friend and I, the results were pretty good.

For modifying a 1600 x 1200 photo, the tool took about 4 seconds per operation, which is fine for an app, but not there yet for an interactive control for a web page. However, the biggest blocking problem I see is the need of feedback to get back useful results. I think that using Seam carving combined with a simple foreground/background detection algorithm can go a long way into automating the process.

The GIMP implementation seems pretty useful for doing photo and other edits, and Photoshop also has an implementation. To close on this, I tried using an insomart image. I liked the results, as it let me emphasize some of the image details while keeping the full background.

The way the feature works, is that you need to select a couple images, and then are given the option to replace one patch from patches in other images. What the tool asks you is for images to be “similar and with the same point of view”, and also let’s you use at most 10 images. My guess is that it is trying to do a similar matching as Photosynth, where it tries to identify features on the image, and based on those features align the photos. On all my in-painting work, I generally considered the whole image, but this might help in narrowing down the area from which to take content from.

I gave the tool a couple tests. First, something easy with objects on top of an uniform texture:

Here I am replacing one person against a water background. The tool seems to be doing it pretty well:

Now it is time to try out a more structured background. I struggled on this scenario a bit more, as the tool is not very flexible at letting the user correct it’s mistakes. It also seems that any change in depth can really throw it off. It seems that a possible improvement to the technology is to better consider scale factors when suggesting patches to replace. I ended up with a bit of a weird composition:

Finally, an example I liked of another fuse, again with a texturized background and 2 photos with similar depth.

In general, I think that the Photo fuse feature is a cool use of In-painting. I will try to take some photos with some of my observations in mind (try with textures and with similar depth)

After some time away from image processing, I want to go and discover what is new about it, and what new ideas are taking place. This blog is about me spending some of my time playing with new image processing tools, algorithms and more to see what it is new that we can do with images today.