Images, Overlords, and More

From the mind of a Chitika engineer…

A LOT has been happening in the past week. I am excited to say there will be many improvements coming down the pipeline. One interesting project is that of web capture. There are times when we want to expand our inventory for some of our ad units like eMiniMalls to ads that do not come with images. We cannot just slap those new ads on without images because this will be an eye sore on your pages. There are two solutions to this problem: (1) find an image for a similar product, or (2) illustrate where the user is about to go. The first is done some times, and sometimes does fairly well. Other times, we just don’t have an image, period. So now we can illustrate where the user goes. How do we do this? Well, if you are like me and you have browsed the internet enough, you already know the answer— take a picture of the web site they are about to go. This has already been done, by countless people, so it’s not that exciting. Still, though, it is fun taking your own approaches to designing a system to take images for people. In designing this system, there are a few rules that come to mind:

• If someone asks for an image of a web site, and we don’t have an image, we should not try to get it.
• If someone asks for an image of a particular size, and we have another size, we should have the raw image on store to resize.
• We should always be able to take images of web sites
• The system should be massively scalable (yes, this process is embarrassingly parallel-ize-able) The solution: A master-slave* system. With the right URL, you will— after going through a huge akamai-style cache system —ask a master system for an image of a web site. If this image does not exist, it will add it to a queue. Meanwhile, many machines around the world will ask that master system for requests to answer. These machines will promise a small number of responses within a small amount of time. Within that time, the slaves will work tirelessly trying to get the screenshots required within the promised time. If it fails, it asks the master for more work and alerts me. This process continues indefinitely. The result:  This system will allow us to push a lot more revenue through the Chitika Network, due to expanding to other Advertising Partners. *My apologies to Los Angeles County.