RuneScape Community Forums (since 2001)

The Zybez Forums ( http://forums.zybez.net ) should now be closed to new posts. There's no telling how long the forums will be kept online beyond this. For the immediate foreseeable future, it seems they will stay online (as a read-only archive). But beyond a year, I'd say it looks bleak. For that eventuality, I've gone ahead and done something... but first let me tell you some facts:

1. Archive.org only saves a few pages of the website over years. We can manually request it to archive individual pages but saving the whole site will take forever and Archive.org probably has limits in place for people attempting this sort of thing. It's safe to say that Archive.org will stay online forever. So, copies (of certain pages) of Zybez Forums will always be online. These copies will probably outlast any other archives of Zybez Forums in the long run.

2. I was not given the Zybez Forums database. Had I been given it, I could have put it online and the site would remain open and online. 

3. So, the only other thing I could think of to ensure that we have the most complete possible archive of the site is to scrape it. 

 Scraping means that a crawler (like a bot with a web-browser) is gonna try to browse Zybez Forums, clicking links randomly, and saving the pages one by one along with whatever pictures or javascript files or whatever it can save. Here's the problem: Zybez Forums have over a decade's worth of posts. Forget the posts, just the memberlist is so long that scraping each of the user profile pages will take several forevers.

Nevertheless, I decided to give it a try anyways.

As of me posting this, the scraper has already scraped over 2 GB worth of content but I suspect that's not even a small fraction of the whole thing. It's going slow as not to overload any servers/networks - but it's going steady. I'll keep it running so the archive will get more and more complete.

Fun fact: the scraper is saving EVERYTHING, including remotely hosted pictures. That means those screenshots or signatures or whatever in the posts are also being archived.

If you're interested to see it, here it is:

https://zybez.runescapecommunity.com/forums.zybez.net/index.html

That's pretty fantastic, I'm assuming there's no way for you to crawl the private boards? It'd be awesome if you had those backed up even if they can't be posted publicly.

Good to hear there's at least an attempt at archiving our clanning history. Even better that its saving screenshots/avatars/sigs.

Major props.

(09-17-2018, 05:49 PM)Dox17 Wrote: [ -> ]Good to hear there's at least an attempt at archiving our clanning history. Even better that its saving screenshots/avatars/sigs.

Major props.

Where's my most biased runner-up sig...

e: sent my 400 USD payment

(09-17-2018, 05:46 PM)Wee Man Wrote: [ -> ]That's pretty fantastic, I'm assuming there's no way for you to crawl the private boards? It'd be awesome if you had those backed up even if they can't be posted publicly.

I tried (to make Httrack play nice with a proxy and use my Chrome browser) to crawl the private boards, but... I failed. Maybe I'll try again. But if not, I can always take page-long screenshots, but that's just crumbs of the pie.

Godspeed crawler. Sucks losing all that history, but any attempt at all to archive them is great. As Teddy said, we were taught that once on the internet always on the internet, but even the internet is hosted on a physical server than can be lost.

Is there no way at all that Curse would relinquish control of the boards?

---

Also, might be worth creating an archive forum in clan discussion so that we can at least copy over some of the recaps of the big / historic fights?

It's going to be weird for people that haven't checked back in a year or two to find that it's read-only or eventually, gone entirely.

That's really cool to see, if people want to look stuff up and search previous posts and all then it's there forever. Shame about curse not handing over the database, have they given you a flat no or just not replied?

That's one of the biggest myths out there; once on the internet, always on the internet.