Secret Metafilter

What is Secret Metafilter?

Secret Metafilter highlights discussions that are still active on older Metafilter posts. It shows recent comments (made in the past 4 days) on posts that are between a week and a month old. (Metafilter posts are closed to comments after a month.)

The idea comes from Metafilter user painquale, who wished there was a way to easily find older, active threads, and called the body of such threads "Secret Metafilter."

Who is responsible for this?

I am John Wiseman, AKA jjwiseman and lemonodor. There is no official link between this project and Metafilter.

The best non-public way to contact me is via email: jjwiseman@gmail.com.

Where does the data come from?

Secret Metafilter scrapes the past month's worth of posts on metafilter.com every two hours.

The scraper advertises itself with the User-agent "secretmefibot", and it obeys Disallow rules in robots.txt. It is hardcoded not to scrape more than 10 pages/minute.

The scraper tries hard not to scrape a page unless it has to by scraping post metadata from index pages and then deciding if it needs to scrape each post page to get new comments. During the day, while Metafilter is active, the scraper will typically scrape about 25 pages every two hours (an average of one page every five minutes). When Metafilter is less active, fewer pages will need to be scraped.

Are there bugs?

Yes. See this list.

I want to see the code or run it myself

That's great! All the code is available at https://github.com/wiseman/secretmetafilter.

Powered by Google App Engine