XML Special Character Gotchas

You know you've got past the beginner tutorial and you're doing the real thing with XML when you start to get encoding and special character issues. Here's what I mean. You get some code which takes some XML, perhaps produced by a web page, and passes it to an XML parser. Now, just like other formats like JSON and HTML, XML requires that certain 'special characters' be 'escaped' (replaced with sequences like '&' which is the escape sequence for an ampersand character). So if your XML contains some of these characters then they have to be replaced or the XML is not right. This introduces a catch-22 gotcha for XML parsers (at least it does for the ubiquitous one I use and probably does for others too) in that you might need to parse the XML as a first step in replacing the special characters (or how can you tell the character is in element or attribute content rather than a comment, namespace string or element or attribute name, say?). Trying to parse XML containing these characters might cause the parser to throw an error. That, apparently, is because the XML standard specs seem to give the impression this is the correct behavior of an XML parser (though I'm told on authority that this isn't strictly what the specs intended).

The way I had to solve this with my C# code and .NET XML parser was to try parsing the XML, then catch any 'XML exception' errors, then do some things with the exception message and parsed data which aren't exactly advisable (since you don't know what state the parser data will be in after the exception) but seem to be unavoidable for an ordinary developer like myself. I have to get the offending bit of data from the parser using the line number and line position of the error from the error / exception message. Then do some intelligent replacing of a special character while avoiding replacing special characters which are part of the escape sequence of an already replaced special character! Phew! I just wish the parser would do all this for me but there are so few XML parsers available to me in my development environment (two I think) and neither handle this scenario the way I'd like, it seems.

Here's a rendition of a general function to do all this in C# with .NET 2 or 3. I'm not necessarily proficient enough to say anyone else should use this code - it probably needs some better error handling and optimization, besides the fact it makes some assumptions which might not always be safe. It show the kind of issues a developer may face when parsing XML. Actually, I couldn't find much code anywhere on the Internet to handle this issue. In fact found very little written about this issue at all apart from an aged link here which helped as a starting point http://support.microsoft.com/kb/316063 :

public static string EscapeXmlSpecialCharacters(string XmlString)

{

string resultString = "";

//Create and load the XML document.

XmlDocument doc = new XmlDocument();

try

{

doc.LoadXml(XmlString);

resultString = XmlString;

}

catch (XmlException ex)

{

StringReader str = new StringReader(XmlString);

StringWriter stw = new StringWriter(new StringBuilder(resultString));

string output = "";

long i = 0;

string strline = "";

long linenumber = (int)ex.LineNumber;

long lineposition = (int)ex.LinePosition;

while (i < linenumber - 1)

{

strline = str.ReadLine();

stw.WriteLine(strline);

i = i + 1;

}

strline = str.ReadLine();

string strOffendingCharacter = strline.ToString().Substring((int)lineposition - 2, 1);

string strOffendingCharacterAndFollowing5 = strline.ToString().Substring((int)lineposition - 2, 5);

switch (strOffendingCharacter)

{

case "<":

strline = strline.Substring(0, (int)lineposition - 2) + "&lt;" + strline.Substring((int)lineposition - 1);

break;

case "&":

// ensure we are not replacing the ampersand in an already escaped special character (&lt;, &gt;, &apos;, &quot; or &amp;)

switch (strOffendingCharacterAndFollowing5.Substring(1, 3))

{

case "lt;":

break;

case "gt;":

break;

default:

switch (strOffendingCharacterAndFollowing5.Substring(1, 4))

{

case "amp;":

break;

default:

switch (strOffendingCharacterAndFollowing5)

{

case "apos;":

break;

case "quot;":

break;

default:

strline = strline.Substring(0, (int)lineposition - 2) + "&amp;" + strline.Substring((int)lineposition - 1);

break;

}

break;

}

break;

}

break;

}

stw.WriteLine(strline);

strline = str.ReadToEnd();

stw.WriteLine(strline);

output = stw.ToString();

str.Close();

str = null;

stw.Flush();

stw.Close();

stw = null;

resultString = EscapeXmlSpecialCharacters(output);

}

return resultString;

}

Then you can put in a step between receiving some XML from, say, a web control like a grid and sending that XML as a string to an XML parser like XmlReader so that you can read the string into a .NET dataset, say. No idea what the equivalent issues and solution are like in Java, sorry. If you write your own XML parser though this is one of the issues you will have to bear in mind and handle, along with similar XML-related issues like handling the illegal characters and characters which are not encoded with the encoding declared in the XML declaration (usually UTF-8). Targeting such a parser at supporting just MicroXML, say (see earlier blogs on MicroXML and MicroXSD) might help to keep these issues to something manageable for people writing their own parsers; we'll see perhaps.