Secret Metafilter highlights discussions that are still active on older Metafilter posts. It shows recent comments (made in the past 4 days) on posts that are between a week and a month old. (Metafilter posts are closed to comments after a month.)
The best non-public way to contact me is via email: firstname.lastname@example.org.
Secret Metafilter scrapes the past month's worth of posts on metafilter.com every two hours.
The scraper advertises itself with the User-agent "secretmefibot", and it obeys Disallow rules in robots.txt. It is hardcoded not to scrape more than 10 pages/minute.
The scraper tries hard not to scrape a page unless it has to by scraping post metadata from index pages and then deciding if it needs to scrape each post page to get new comments. During the day, while Metafilter is active, the scraper will typically scrape about 25 pages every two hours (an average of one page every five minutes). When Metafilter is less active, fewer pages will need to be scraped.
Yes. See this list.
That's great! All the code is available at https://github.com/wiseman/secretmetafilter.