bobpeers -> blog

Living in Denmark I always have to develop sites that support the extra characters found in Danish names (ø,å,æ plus Swedish, Norwegian and all the other European characters). For this reason I always use UTF-8 encoding to avoid and problems with displaying the characters.

Just recently I had a major headache getting Firefox and IE6 to insert the same data (sent via a XMLHttp GET request to an ASP page) into a database, the two browsers seemed to treat the same data differently so with one page Firefox would garble the characters while IE worked fine, while on another page the opposite was true.

This lead me to investigate exactly what is required to correctly send UTF-8 data to an ASP page and insert the characters into a database, fortunately I found the solution.

1. In the Javascript code you use to send the GET request make sure you add the correct headers as shown here.

   xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
   xmlhttp.setRequestHeader("Content-Type", "text/html;charset=UTF-8");

2. Encode the querystring using encodeURIComponent() so a request looks like this.

   xmlhttp.open("GET", "yourpage.asp?e="+encodeURIComponent(data));

3. At the start of your ASP page add the following lines (the CodePage=’65001′ refers to UTF-8).

   Response.CodePage = "65001"
   Response.AddHeader "Content-Type", "text/html;charset=UTF-8"

4. Finally make sure you save all your pages as UTF-8 (without the Byte-order mark, or BOM) and not ANSI. For this reason I recommend you do not use Notepad as it will add the BOM if used to save UTF-8 files. Use Notepad++ or some other program where you can choose the format to save.

I’ve been writing some new Intranet pages at work that include a searchable phone book that uses the XMLHttp object to return data in the background.

Many of the names in the database include Scandinavian Characters like å, ø and æ for example. IE always returned the correct results but if I searched for the Scandinavian characters using Firefox the search would return no results. It was clear that the SQL string being built in the ASP script was somehow mangling the characters so the query returned no data.

I checked the charcodes using JavaScript (using CharCode()) before the GET request was sent and IE and Firefox gave the same result. But then if I returned what the ASP script saw (using the asc function) it was clear that IE and Firefox disagreed at this point.

I assumed this was because the XMLHttp object in Firefox was using the wrong encoding so I added the following lines to force UTF-8 encoding (as used by the MS Access database phone book).

xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
xmlhttp.setRequestHeader("Content-Type", "text/html;charset=uft-8")

Unfortunately this did nothing, but after a lot of testing and head scratching I finally discovered that my ASP script was saved in ANSI format. As soon as I converted the script to UTF-8 and saved it everything worked perfectly in Firefox.