I once tried to archive this subreddit, but was foiled by my lack of technical knowledge.
Does anyone out there know how to download an entire subreddit, but exclude pages that aren't actually part of the sub itself?
In any case, it would be good if more than one person archived the whole subreddit for posterity. That would allow reconstruction, should anything bad happen to reddit itself. Since it's almost all text, it should take up around a gigabyte, or even less.
Reddit limits queries to two per second from the same IP, but this shouldn't present a problem since the free archiving software I found can be set to meet that limit. Where I ran into trouble was limiting the amount downloaded to just what was on the subreddit itself, while preserving hyperlinks to external material.
[–]MRSPArchiver 1 point2 points3 points (0 children) | Copy Link
[–]Imnotmrabut 1 point2 points3 points (1 child) | Copy Link
[–]EvilPundit[S] 0 points1 point2 points (0 children) | Copy Link
[–]ZimbaZumba 1 point2 points3 points (1 child) | Copy Link
[–]EvilPundit[S] 0 points1 point2 points (0 children) | Copy Link
[–]EvilPundit[S] 0 points1 point2 points (0 children) | Copy Link
[+]aussiehank [recovered] (9 children) | Copy Link
one bot, data mining, at rough guess 90 million dynamic ( not static ) pages, at 43200 queries per 24 hours ?
5 years 259 days.
[–]EvilPundit[S] 1 point2 points3 points (8 children) | Copy Link
[+]aussiehank [recovered] (7 children) | Copy Link
there is nil facility within the reddit API to filter out subreddits. the bot would need to crawl the whole of reddit, some 90 million posts, since the creation of MR, March 19, 2008.
[–]EvilPundit[S] 0 points1 point2 points (6 children) | Copy Link
[+]aussiehank [recovered] (5 children) | Copy Link
yes, a bot can be created. to be clear, we are discussing dumping offline the whole MR sub, since day one. reddit displays each sub with a limit of 1000 posts. you can't access any post beyond this '1000 post cache'; you need to have the direct URL and to each post.
to start this sub dump project, you need the URL of the first ever official, unofficial post on MR ?
next, provide the URL of the second, third, fourth... post on MR ?
[–]EvilPundit[S] 0 points1 point2 points (4 children) | Copy Link
[+]aussiehank [recovered] (3 children) | Copy Link
the admins would have this info. but the admins have not ever provided a facility that allows access to a sub, since day one. reddit being a business, i doubt they ever will allow access to a direct dump of a sub.
maybe in the way distant future, the admins will gift official dumps, to archive dot org.
[–]EvilPundit[S] 0 points1 point2 points (2 children) | Copy Link
[+]aussiehank [recovered] (1 child) | Copy Link
plenty of good quality forums, blogs, youtube videos related to MR and the manosphere have just disappeared, or been deleted. For example, Marc Rudov's 'the no-nonsense man' blog, podcast, and videos, anyone have that archived.
i've been dumping the URL of posts on MR i find of value into a text file. at some point in the future, i will write a bot to download the URL's and archive as a PDF. best to understand how to archive 'stuff' off the internet, and start archiving, offline, today ;)
[–]EvilPundit[S] 0 points1 point2 points (0 children) | Copy Link
[+]o9JOhY6iPn [recovered] (2 children) | Copy Link
Leave an opt-out for people who don't want to be archived.
[–]EvilPundit[S] 0 points1 point2 points (1 child) | Copy Link
[–]o9JOhY6iPn0 points [recovered] | Copy Link