Renderosity folx: Here's an algorithm you might consider. Monday midnight: Recheck last week's missing file. Any that still cannot be reached, dump them. Then check all remaining URL's, compiling a list of those that don't answer up. Wednesday 8AM: Use Monday's list to compile a (smaller) list of those that still don't answer. Friday 4PM: Use Wednesday morning's list to compile the final list (to be deleted Monday if still offline). This would catch all the sites that are down for any long term but would probably not eliminate many (or any) truly available sites. It will avoid "catching" sites that do daily maintenance at a specific time and it will make sure that the outage is at least quasi-permanent. You should also consider PhilC's suggestion of having the 'bot email those who end up on the list rather than dumping them. It's a pretty good bet that the site is a goner if you don't get any reply and it's still down after a week of no show and another week for email turnaround.