Insomart

One of the interesting applications of image processing on the web is what is called “Seam Carving”. The idea is that it is possible to identify areas of an image that can be added/removed as an image is resized. This means that when downscaling, instead of reducing the size of the full image, regions are taken out. On the flip side, when upscaling, either regions are added, or inpainting is used to extend areas.

There is a video that explains the technique in mode detail from the Siggraph presentation, and several image processing applications have implemented it. GIMP happens to have an implementation of this algorithm (called Liquid Rescale), so I gave it a try to see how well it works.

Below is our test image, which is a shot taken at the summit of Little Si. Using Liquid resize, I first tried to reduce the image to about half the width. The output was a decent image (regarding the inpainting job). The blending of the background (which does not have clear unique shapes) was good, but when it came to the subjects of the photo, it is clear that the algorithm can’t really determine which areas are good to remove and which are not.

I tried next to provide feedback regarding areas to preserve. I decided to keep myself in the picture and make any other area eligible for removal:

Providing feedback definitely helped, though it ended up with a distorted view of my friend. That is where the opposite feedback of areas to remove comes into play:

Now I have a decent image. There are some artifacts still in the front, which are not uncommon with inpainting techniques, but in general I think the output is good. I tried to upscale the image to twice the size with no hinting first. Again, not indicating areas to preserve is not a good idea.

After providing a hint to preserve both my friend and I, the results were pretty good.

For modifying a 1600 x 1200 photo, the tool took about 4 seconds per operation, which is fine for an app, but not there yet for an interactive control for a web page. However, the biggest blocking problem I see is the need of feedback to get back useful results. I think that using Seam carving combined with a simple foreground/background detection algorithm can go a long way into automating the process.

The GIMP implementation seems pretty useful for doing photo and other edits, and Photoshop also has an implementation. To close on this, I tried using an insomart image. I liked the results, as it let me emphasize some of the image details while keeping the full background.

One of the projects I worked in on college was an image smart erase. It was my final project at the Computer Vision course in Brown University, and I decided to implement the tool based on the description in a paper by Antonio Criminisi from Microsoft Research. My implementation (on Matlab of course) diverged a little from the paper, but gave me decent results, which made me wonder what was missing for that kind of technology to make it into a more mainstream product. A couple years later, the new Windows Live Photo gallery shipped a feature called Photo Fuse, which is using in-painting to replace a patch of one image from another image.

The way the feature works, is that you need to select a couple images, and then are given the option to replace one patch from patches in other images. What the tool asks you is for images to be “similar and with the same point of view”, and also let’s you use at most 10 images. My guess is that it is trying to do a similar matching as Photosynth, where it tries to identify features on the image, and based on those features align the photos. On all my in-painting work, I generally considered the whole image, but this might help in narrowing down the area from which to take content from.

I gave the tool a couple tests. First, something easy with objects on top of an uniform texture:

Here I am replacing one person against a water background. The tool seems to be doing it pretty well:

Now it is time to try out a more structured background. I struggled on this scenario a bit more, as the tool is not very flexible at letting the user correct it’s mistakes. It also seems that any change in depth can really throw it off. It seems that a possible improvement to the technology is to better consider scale factors when suggesting patches to replace. I ended up with a bit of a weird composition:

Finally, an example I liked of another fuse, again with a texturized background and 2 photos with similar depth.

In general, I think that the Photo fuse feature is a cool use of In-painting. I will try to take some photos with some of my observations in mind (try with textures and with similar depth)

One of the areas that I am really like in technology is image processing. It all comes back to my early days at INAOE, where I assisted at the image science lab. Part of my work was to write code to automate lab processes, or to do analysis on experiments, which often came back as images. It was during this time that I learned about topics like FFT, image filters, edge detection, etc. It was also when I started to experiment with a combination of algorithms to come up with the images in this site, which I called Insomart.

After some time away from image processing, I want to go and discover what is new about it, and what new ideas are taking place. This blog is about me spending some of my time playing with new image processing tools, algorithms and more to see what it is new that we can do with images today.