PunkDownload.com - New file hosting server for APN

We *think* we have FINALLY found the solution for the problem of "server responded with 0 code"

This fucking bug has been haunting us for at least 2 years with a small fraction of forum users reporting intermittent issues that we were never able to reproduce. We had zero clue what could be causing this.

But with the help of @Sergx81 and patient forum users who have reported the issues, we have finally been able to track it down.

The upload form looks the same, but in the backend everything was rewritten into a new version with a fork of the original Dropzone2 javascript we use. But the major game-changer was chunking, once correctly configured, should solve all upload issues.

Thanks everyone for your patience with this issue!

 

I've encountered a problem with the upload form that I haven't before: when it's uploaded (the upload speed is a bit lower than before, isn't it?) it updates to the green "Upload Successful!" text, but then nothing populates. It just sits there indefinitely. After stubbornly waiting it out it finally went to "A timeout occurred - Error code 524 - Visit cloudflare.com for more information."

 

Sorry, just saw this. The tech team is slower than usual this summer, half of the team is on vacation.

I tested the upload form and could not reproduce the issue, so it looks like it's another one of those intermittent issues. However since the beginning of July we had a lot of issues with Server 6 which is hosting the uploads. We get alerts almost every day that the server goes down for a few minutes. So it's possible that it is the root cause of the intermittent issues and slow upload speed (it was fast when I tested)

We have recently added a debugging log to the upload form, so every time an upload fails it saves some hints for the team to figure out what happened. If you give the exact file name of your upload, I can search in the log to see what happened

I bumped the issue with the team and someone made some modifications to the upload form (improved chunk assembly with timeout protection and improved Dropzone error handling). Maybe it helps, maybe it doesn't.

 

Oh wait, I might have overlooked an important detail.

@Spike one of many said this:

@Starfighter, could you please clarify where do you see this error? Is it on PunkDownload.com or on Anarcho-Punk.net?

If I understand correctly, Spike got a timeout on Anarcho-Punk.net and not on PunkDownload.com. This is an important detail that changes everything about the diagnosis

Here is how the upload flow works:
1) You create a thread on Anarcho-punk.net (on Server 3) but the upload form is embedded in an iframe hosted on PunkDownload.com (Server 6)
2) The file is uploaded on PunkDownload.com (Server 6)
3) After the upload, you are redirected to the forum (Server 3)
4) When you're back on the forum, the script needs Server 3 to get the uploaded file from Server 6 (this is the URL sent by Spike)

So it looks like it's failing at Step 4, which would mean that the upload form is fine and the problem is somewhere AFTER the redirect to the forum (so at this point we're on Server 3, not Server 6)

Sorry for getting into the technical details, but the TLDR is we need to know if it times out on PunkDownload.com or on Anarcho-Punk.net (a screenshot of the full Cloudflare error page would show it)

A temporary workaround could be to refresh the page when you see the Cloudflare error on Anarcho-Punk.net

Edit: I modified Step 4 to bypass Cloudflare when Servers 3 and 6 are communicating with each other. Added fallback if it fails, and verbose debugging.

 

Still not fixed... forum users of Pirate Punk reported the same problem

Working on it...

Edit: fixed

 

Sorry for late response, I've been away from my PC for large amounts of time this summer (kidney stone trouble for a few weeks and a vacation in another part of the country). It was on APN and I only saw it once, but I read that it's fixed now so yay progress!

 

I've been seeing more and more broken image links around this place, what's that about? Here's an example, I noticed one of my uploads have a broken image, even though the image is in the zip file itself. But when I go to edit in a third party image link (I thought at least I can patch things up that way) it displays just fine in the editor. W... what....


Thread view:



Editor view:

 

Ok, this one is really strange. I spent an hour on this, but I don't have any clues. I need more data. Could you please give me other examples of broken images? Trying to see if there are correlations or common points between them, like strange headers or weird characters in filenames.

How images work on this forum: when you post an image URL, the forum will try to import this image locally and save a cached version, to make sure the image is always displayed even if the remote source goes offline. But sometimes the import process fails, and the result is a broken image.

When viewing a thread, it shows the locally cached image (on Server 3). When editing a thread, it shows the remote image (on Server 6). So the fact that the broken image is displaying correctly when editing proves that the bug is with the local cache on Server 3, and that the original uploaded image on Server 6 works fine.

 

This could actually be the case, because it has (from what I remember right now) only happened to the few uploads where I haven't scanned the cover myself. Meaning those images are from the internet and could contain all sorts of weirdness. I'm gonna provide only my own scans from now on and see if the issue re-surfaces.

 

If I'm not mistaken that was a cache problem where the workaround involved forcing a new call of the image, but I'm not known for good memory!

 

The first disk was replaced today by the data center technicians.

It's the same story as last time: this server has four disks, and the RAID 10 array must be rebuilt every time a disk is replaced.

If the server loses more than one disk, all data would be lost. Currently, we're already missing one disk, which was just replaced, so the server is running with three out of four disks, the critical limit. The other faulty disk identified by Sergio is still at risk of sudden death, and rebuilding the RAID 10 array puts enormous pressure on the server's disks, increasing the risk of total loss.

So to limit the risk, we've decided to take Server 6 offline while we safely rebuild the disks. The first step will take at least 24 hours to integrate the new disk, then we'll have to start all over again to replace the second, dying disk.

So during this time, downloads AND uploads will be temporarily unavailable. It's hard to estimate how long, I'd say around 3 days given the exceptional situation.

 

I have tested the workaround the last time I tried debugging this issue. I found one of your uploads with a broken image, took the URL, and added a random variable to the end of the URL, but it didn't fix it.
(changing the URL of the image forces a cache refresh because the URL looks new, but the random variable at the end is ignored and the image still displays)

If the images end up being fixed by themselves over time, indeed, that would confirm that the issue is not about Content-Type but rather some caching weirdness.

We will get back to this bug after working out the HDD issues of Server 6.