BIG EDIT: I wrote a tool that took the invalid character sequences (which were actually 5-8 bytes long, each), and replaced them with real characters. All posts should be good now. If you see any posts with invalid characters still, and you know what those characters are supposed to be, please let me know.
Well, to those unaware, I upgraded the forum to the latest version the other day, and all hell broke loose.
But, after that was tamed, one thing remained broken... That's right, some posts were totally blank!
... Ok, fine, maybe that's not a "that's right, " thing, but that's beside the point.
Anyway, after performing some voodoo on the database, I coaxed it to regurgitate the missing posts. Like many things regurgitated, however, they were not in their original form.
Specifically, any character invented by Microsoft Word, MSN or any other Microsoft product that isn't a real character (i.e., the curly quotes and the single character ellipsis (...)) now looks like a bunch of garbled symbols.
New posts with those fake characters in them should be A-OK from now on. But, the old ones -- gone.
The reasons for this are arcane, but:
The fake characters were implemented as higher-numbered ascii characters (those in the 128 - 255) range, which is fine in the ISO-8859-1 codepage which is default on Windows. However, at some point (probably during the upgrade), the database went stupid, and decided that it was not holding ISO-whatever-1 data, and that it was holding UTF-8 data.
Now, converting from ISO- to UTF-8 is very possible, and the forum even includes a big, shiny button to do so. But, the data is not interchangeable. Conversion is necessary. So, the forum (thinking it was dealing with UTF-8) would go along, and then puke when it saw the invalid characters. Somewhere in the process, the invalid string would be turfed, in favour of "".
The method of fixing it was easy. Just run the big conversion button that I mentioned. However, because the forum thought it was converting FROM UTF-8 (since that's what the database thought it had), TO UTF-8, but pretending it was dealing with ISO-whatever... Anyway, I hardly understand this any more than you do.
Long story short, curly quotes and their ilk are gone. Short story long, anything by Charles Dickens.
