Mystic Nights freebie search is still running. Maybe he/she might be able to give you some information. 

http://mystic-nights.com/directory/index.php

I don't do web programming so I can only guess. I'd assume you would have a list of sites. IIRC Google has a web service API you can use, but shooting off real time queries for a long list of sites doesn't seem too practical e.g. what would you query?

I'd think that you'd have a 'bot' of some kind that would periodically check each site's free stuff pages. There are free and commercial web crawler apps that would probably make it easier to automate. You would have a database to store the results. Assuming you have a web host then it would probably be easiest and least expensive to use whatever database they provide. You may find some that already have a simplke template set up for basic database query. You could have just the site or page URL and the item name like Mystic. Adding descriptions and thumbnails would be more complex. As long as you don't hotlink to the downloads, I'm guessing you're OK, though it wouldn't hurt to ask permission to crawl the sites. People can and do have a robots.txt file to prevent their sites from being crawled.

That's just my very off the top of the head guess :-) 

"Democracy is a pathetic belief in the collective wisdom of individual ignorance." - H. L. Mencken