Failure to Archive RI, Feb 2019: My Surrender
Posted: Wed Feb 13, 2019 4:30 pm.
Since late last year it has been my serious plan to create a new archive of the entire board, much like the one provided by Joao back in November 2016.
viewtopic.php?f=33&t=40198
(Also, to spend less time here after providing that.)
Sorry everyone, I don't know what I've been doing wrong. Here's the report of my failure:
Using two different versions of A1WD by Microsys, both the most current as well as the one used by Joao in 2016 (7.7.0), I made several major efforts. But they all failed.
The first couple of attempts ran in the background for several days in Dec and Jan. I made the insane attempt to download the site along with all of the displayed external graphics as resident. This was stupid! It was never going to work. I stopped trying that.
But even without it, it became clear I'd have to wait until I had access to faster Internet.
Once I had a faster connection, I tried again several times, using various settings offered by the program.
On the most recent try, I let the scan run for more than a week. It still wasn't done when I finally stopped it. The stored files had reached 18GB and more than 300,000 thread pages! This is wrong, given the board's current total of 39,000 threads in total and 646,000 individual posts, with no more than 15 posts per page. There might be about 100K thread pages, no more. In Nov 2016 the site had 36,000 threads and 82,000 pages. Two years ago, Joao's archive was under 4GB, just 1.2GB as a zip file.
Anyway, despite this awesome size, the results lacked a great many files that can be found on the site and especially lacked many index pages, unlike Joao's perfect archive of 2016. But there were also countless other identical pages in triplicate, quadruplicate, or ten times. Which is how it turned into 18 GB and was still growing.
So whatever I was doing, or whatever the program was doing, it was a fuck up and I have no idea why. I suppose it's also possible that it's neither me nor the program, but that the problem is caused by some change in the board settings (to do with robots or some such).
At this point I must surrender. This was a lot of work for nothing. (The program may run in the background, but of course one constantly checks it and has a slower machine because of it.)
Regulars, I invite any of you to try, maybe with another downloader, or with superior skills and experience, and see if you can manage it.
It seemed so easy!
.
Since late last year it has been my serious plan to create a new archive of the entire board, much like the one provided by Joao back in November 2016.
viewtopic.php?f=33&t=40198
(Also, to spend less time here after providing that.)
Sorry everyone, I don't know what I've been doing wrong. Here's the report of my failure:
Using two different versions of A1WD by Microsys, both the most current as well as the one used by Joao in 2016 (7.7.0), I made several major efforts. But they all failed.
The first couple of attempts ran in the background for several days in Dec and Jan. I made the insane attempt to download the site along with all of the displayed external graphics as resident. This was stupid! It was never going to work. I stopped trying that.
But even without it, it became clear I'd have to wait until I had access to faster Internet.
Once I had a faster connection, I tried again several times, using various settings offered by the program.
On the most recent try, I let the scan run for more than a week. It still wasn't done when I finally stopped it. The stored files had reached 18GB and more than 300,000 thread pages! This is wrong, given the board's current total of 39,000 threads in total and 646,000 individual posts, with no more than 15 posts per page. There might be about 100K thread pages, no more. In Nov 2016 the site had 36,000 threads and 82,000 pages. Two years ago, Joao's archive was under 4GB, just 1.2GB as a zip file.
Anyway, despite this awesome size, the results lacked a great many files that can be found on the site and especially lacked many index pages, unlike Joao's perfect archive of 2016. But there were also countless other identical pages in triplicate, quadruplicate, or ten times. Which is how it turned into 18 GB and was still growing.
So whatever I was doing, or whatever the program was doing, it was a fuck up and I have no idea why. I suppose it's also possible that it's neither me nor the program, but that the problem is caused by some change in the board settings (to do with robots or some such).
At this point I must surrender. This was a lot of work for nothing. (The program may run in the background, but of course one constantly checks it and has a slower machine because of it.)
Regulars, I invite any of you to try, maybe with another downloader, or with superior skills and experience, and see if you can manage it.
It seemed so easy!
.
