Where did that JSON field go? Serializing IHtmlString to JSON.

TL;DR

If your brain consumes Stack Overflow questions better than blog posts, go see “How do I serialize IHtmlString to JSON with Json.NET?” over there.

IHtmlString doesn’t play nicely with JSON serialization

If you have an IHtmlString in your JSON (regardless of arguments against putting raw HTML in JSON), you will probably need to customize the serialization to get the HTML out of that variable. In fact, the serialization will probably be invalid compared to what you expect; it does make sense if you think about how the serialization process works.

Fortunately, putting together a quick Json.NET JsonConverter will make the issue go away.

What you might expect from some object containing an IHtmlString variable named x:

{ x = "some <span>text</span>" }

What default .NET JavaScript serialization and default Json.NET serialization will give you for said object:

{ x = { } }

How to fix it with Json.NET:

public class IHtmlStringConverter : Newtonsoft.Json.JsonConverter {
    public override bool CanConvert(Type objectType) {
        return typeof(IHtmlString).IsAssignableFrom(objectType);
    }
    ...
    public override void WriteJson(Newtonsoft.Json.JsonWriter writer, object value, Newtonsoft.Json.JsonSerializer serializer) {
        IHtmlString source = value as IHtmlString;
        if (source == null) {
            return;
        }
        writer.WriteValue(source.ToString());
    }
}

Background

While working on some random API, we noticed one of our JSON fields went from being a long string containing raw HTML (legacy baggage) to an empty object: SomeFieldWithHtml: {}.

While the API JSON-generating code hadn’t changed, that field was pulled from code shared by our MVC website. It seems that field was converted directly to an IHtmlString to avoid doubly-encoding it in an MVC view. If you ever have the same issue, you can avoid this issue entirely by leaving the source as a string and doing a quick MvcHtmlString.Create(x) on it before sending it to your view/view-model.

Code

For the full source used in this post, including an example MVC project where a controller has action methods to test all the variations here, head over to this post’s GitHub repository.

Initial setup for a new ASP.NET MVC site in IIS7

Background

Over the years, I have spent far too many hours running the same set of commands against ASP.NET and ASP.NET MVC sites to bring them up to what I consider a starting point. Most of the time, I have to refresh myself about how to do at least one of them. This is a list of those commands in a central location for myself and anyone else to use. As with any good instructions, there is no guarantee you won’t completely destroy your server, site, or soul using these commands. I can only say they worked for me once.

  1. Remove unnecessary HTTP headers
    1. Server
    2. X-Powered-By
    3. X-AspNet-Version
    4. X-AspNetMvc-Version
  2. Adding dynamic content compression (e.g., GZIP)
  3. GZIP for JSON and JSONP content types (e.g., “application/json”)
  4. Complete appcmd command-line list

Remove unnecessary HTTP headers

By default, IIS and ASP.NET give you a few headers in each and every HTTP response sent out.

Server: Microsoft-IIS/7.5 (or 7.0, or whatever IIS version you have)
X-Powered-By: ASP.NET
X-AspNet-Version: 4.0.303319 (or 2.0.50727, or 1.1.4322, or whatever version you have)
X-AspNetMvc-Version: 3.0 (or 2.0, or 1.0, or whatever version you have)

People will rant and rave about removing these headers for security and whether that helps. I am firmly on the side of “not-why-I-care-at-all”. I always figure if I am a target, obscurity doesn’t buy me anything when someone with very little computing power can just throw packets exploiting all the major frameworks at me and see which stick (which happens even when you aren’t a real “target”). The reason I do care is that removing these headers strips out even more data that has to be sent between you and the end user. Even though the size difference is small, it applies to every single HTTP response handled by your site. For such simple one-time changes, I will gladly take the savings.

Server (the annoying one)

This HTTP header appears to be the most annoying to remove. In fact, short of an HTTP module, the common method for doing so is a new line in a [probably new] event handler in global.asax.cs (or global.asax.vb).

protected void Application_PreSendRequestHeaders() {
    HttpContext.Current.Response.Headers.Remove("Server");
}

It’s not as pretty as the rest of the removal stuff below, but it gets the job done. In fact, they could all be solved this way, but I tend to lean on IIS to do the heavy lifting when I can.

X-Powered-By (the more annoying web.config/IIS one)

Many places that offer suggestions on removing the X-Powered-By: ASP.NET header talk about using IIS. You can certainly do this if you have permission to do so on your hosting provider and like the IIS GUI. I prefer things that I can push out with source control, so I go the web.config route.

<system.webServer>
    <httpProtocol>
        <customHeaders>
            <remove name="X-Powered-By" />
        </customHeaders>
    </httpProtocol>
</system.webServer>

Command line happiness:

%windir%\System32\inetsrv\appcmd.exe set config "{your-site-name}" /section:system.webServer/httpProtocol /-"customHeaders.[name='X-Powered-By']"

The root source of this setting is actually found in your server’s applicationHost.config (%windir%\system32\inetsrv\config\applicationHost.config). If you have the ability to modify that file, you can remove it there to make it the default for all IIS sites.

<system.webServer>
    <httpProtocol>
        <customHeaders>
            <clear />
            <add name="X-Powered-By" value="ASP.NET" /> <!--Remove this line.-->
        </customHeaders>
        <redirectHeaders>
            <clear />
        </redirectHeaders>
    </httpProtocol>
</system.webServer>

Command line happiness:

%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpProtocol /-"customHeaders.[name='X-Powered-By']" /commit:apphost

X-AspNet-Version (the easy web.config one)

To make X-AspNet-Version: {your-aspnet-version} go away, you just need to tweak your site’s web.config file. In the <system.web> section, under <httpRuntime>, which is probably already there, just make sure the enableVersionHeader="false" attribute is added to it.

<system.web>
    <httpRuntime enableVersionHeader="false" />
</system.web>

Command line happiness:

%windir%\System32\inetsrv\appcmd.exe set config "{your-site-name}" /section:system.web/httpRuntime /enableVersionHeader:false

I’m guessing this could be done at the applicationHost.config level, though I have never tried it (no real excuse, just too scared to mess with such a vital config entry too much).

X-AspNetMvc-Version (the global.asax code one)

Obviously, if you are tweaking a non-MVC ASP.NET site, you can ignore this one.

To make X-AspNetMvc-Version: {your-aspnetmvc-version} go away, you need to add some code and redeploy your site. In your global.asax.cs (or global.asax.vb) file, you will need to add a single line to the application start event.

protected void Application_Start() {
    MvcHandler.DisableMvcResponseHeader = true;
    // ...
}

Adding dynamic content compression (e.g., GZIP)

Reducing the amount of data you send over the wire is pretty good idea, especially if you are targeting low-bandwidth users like mobile browsers. While GZIP has some performance impact on the server and client, it is negligible on most systems. To add this system to IIS, add in the “Dynamic Content Compression” role service under the “Web Server (IIS)” role in “Server Manager”. If I am doing it visually, I typically just follow along with this iis.net article on the process.

Command line happiness:

%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/globalModules /+[name='DynamicCompressionModule',image='%windir%\System32\inetsrv\compdyn.dll'] /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/modules /+[name='DynamicCompressionModule',lockItem='true'] /commit:apphost

If this has already been installed, this will error out with a message about adding a duplication “DynamicCompressionModule” entry.

GZIP for JSON and JSONP content types (e.g., “application/json”)

[Update 2012-02-02] NOTE: If you have a load-balancer, you may need to figure out how to get it to allow these content types through as GZIP. In one situation where I added this setting to a series of servers behind F5′s BIG-IP load balancer, it still wouldn’t get GZIP out to the requester. BIG-IP was either stripping the request of Accept-Encoding: gzip or getting GZIP from IIS and decompressing it before letting it return to the original requester (I didn’t confirm what was happening, but I would assume the former since it is less CPU-intensive).

While getting GZIP compression is great for HTML, JavaScript, CSS, and most others with “Dynamic Content Compression” is great, it drops the ball on content types it doesn’t have explicit orders to manipulate. One such content type that should get compression is dynamically-generated JSON and JSONP.

While working on the [Sierra Trading Post API](http://dev.sierratradingpost.com/] I noticed that IIS with dynamic compression enabled doesn’t do anything to JSON content types, “application/json” (JSON) and “application/javascript” (JSONP). Compression is decided by the content types listed in applicationHost.config (%windir%\system32\inetsrv\config\applicationHost.config). By default, you get most of the types served by the average ASP.NET site: text/* (covering text/html). If you need to add more to this list, add them to the config file. It does not matter if they come before or after the <add mimeType="*/*" enabled="false" /> wildcard entry that is there by default.

<system.webServer>
    <httpCompression ... >
        ...
        <dynamicTypes>
            ...
            <add mimeType="application/json; charset=utf-8" enabled="true" />
            <add mimeType="application/javascript; charset=utf-8" enabled="true" />
        </dynamicTypes>
    </httpCompression>
</system.webServer>

Command line happiness:

%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/javascript; charset=utf-8',enabled='True']" /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost

For some reason, these are not recognized without the charset=utf-8 part. As well, if you don’t notice a change right away, that’s normal; you must recycle the app pool first (TechNet has a nice article with info about app pool recycling and why it doesn’t happen here).

 %windir%\System32\inetsrv\appcmd recycle apppool /apppool.name:"{your-app-pool-name}"

Again, some people would suggest an HTTP module for handling this, but it seems like IIS would be better equiped to handle this since it is already doing so for the rest of my outbound content.

Complete appcmd command-line happiness

If you are following along at home, most of this stuff can be poked into server config files, either applicationHost.config or your site’s web.config (all but the Server: Microsoft-IIS/7.5 HTTP header). If you want the fire-and-forget solution, here are all the commands smashed together for your batch file use.

%windir%\System32\inetsrv\appcmd.exe set config "{your-site-name}" /section:system.webServer/httpProtocol /-"customHeaders.[name='X-Powered-By']"
%windir%\System32\inetsrv\appcmd.exe set config "{your-site-name}" /section:system.web/httpRuntime /enableVersionHeader:false
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/globalModules /+[name='DynamicCompressionModule',image='%windir%\System32\inetsrv\compdyn.dll'] /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/modules /+[name='DynamicCompressionModule',lockItem='true'] /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/javascript; charset=utf-8',enabled='True']" /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost
%windir%\System32\inetsrv\appcmd recycle apppool /apppool.name:"{your-app-pool-name}"

If you want to do all this to applicationHost.config alone, you just swap out the first two lines from hitting “{your-site-name}” to pointing to the apphost config. Here they are all smashed together for that variant. NOTE: I have not confirmed that setting enableVersionHeader in applicationHost.config will work.

%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpProtocol /-"customHeaders.[name='X-Powered-By']" /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config /section:system.web/httpRuntime /enableVersionHeader:false /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/globalModules /+[name='DynamicCompressionModule',image='%windir%\System32\inetsrv\compdyn.dll'] /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/modules /+[name='DynamicCompressionModule',lockItem='true'] /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/javascript; charset=utf-8',enabled='True']" /commit:apphost
%windir%\System32\inetsrv\appcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/json; charset=utf-8',enabled='True']" /commit:apphost
%windir%\System32\inetsrv\appcmd recycle apppool /apppool.name:"{your-app-pool-name}"

Getting dynamic ExpandoObject to serialize to JSON as expected

Serializing ExpandoObjects

I am currently creating a JSON API for a handful of upcoming Sierra Trading Post projects. When I found myself generating JSON for a stripped-down representation of a number of domain classes, all wrapped with some metadata, I turned to dynamic and things have been going quite well. Unfortunately, there was a hurdle to getting the JSON to look the way I wanted.

If you start playing around with serializing ExpandoObject to JSON, you will probably start finding a number of options. The easiest solution to find is the one that comes with .NET, JavaScriptSerializer under System.Web.Script.Serialization. It will happily serialize an ExpandoObject for you, but it probably won’t look the way you expect. Your searches will probably vary, but I found Newtonsoft’s JSON.NET, which handled ExpandoObject right out of the NuGet box. Then I stumbled on ServiceStack.Text (also “NuGettable”). While it does even weirder things than the .NET serializer with ExpandoObjects, it supposedly does them very fast.

Test code

dynamic test = new ExpandoObject();
test.x = "xvalue";
test.y = DateTime.Now;

BCL JavaScriptSerializer (using System.Web.Script.Serialization;)

JavaScriptSerializer javaScriptSerializer = new JavaScriptSerializer();
string jsonOfTest = javaScriptSerializer.Serialize(test);
// [{"Key":"x","Value":"xvalue"},{"Key":"y","Value":"\/Date(1314108923000)\/"}]

Not quite what I was looking for but it makes sense if you realize that ExpandoObject plays very nicely with IDictionary<string, object>. By using some code borrowed from StackOverflow (not the accepted answer, but I like it) and theburningmonk.com, you put together a custom serializer for ExpandoObject and you can get something more typical of what went into assembling the object.

/// <summary>
/// Allows JSON serialization of Expando objects into expected results (e.g., "x: 1, y: 2") instead of the default dictionary serialization.
/// </summary>
public class ExpandoJsonConverter : JavaScriptConverter {
    public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer) {
        // See source code link for this extension method at the bottom of this post (/Helpers/IDictionaryExtensions.cs)
        return dictionary.ToExpando();
    }
    public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer) {
        var result = new Dictionary<string, object>();
        var dictionary = obj as IDictionary<string, object>;
        foreach (var item in dictionary)
            result.Add(item.Key, item.Value);
        return result;
    }
    public override IEnumerable<Type> SupportedTypes {
        get {
            return new ReadOnlyCollection<Type>(new Type[] { typeof(ExpandoObject) });
        }
    }
}

JavaScriptSerializer javaScriptSerializer = new JavaScriptSerializer();
javaScriptSerializer.RegisterConverters(new JavaScriptConverter[] { new ExpandoJsonConverter() });
jsonOfTest = javaScriptSerializer.Serialize(test);
// {"x":"xvalue","y":"\/Date(1314108923000)\/"}

Newtonsoft Json.NET

string jsonOfTest = Newtonsoft.Json.JsonConvert.SerializeObject(test);
// {"x":"xvalue","y":"\/Date(1314108923000-0600)\/"}

That worked exactly as I expected. If I can get JSON.NET to work consuming JSON under MonoTouch, it will make my life quite easy; more to come on that.

ServiceStack.Text

string jsonOfTest = JsonSerializer.SerializeToString(test);
// ["[x, xvalue]","[y, 8/23/2011 08:15:23 AM]"]

ServiceStack’s JSON serialization system does something similar to the .NET JavaScriptSerializer, but not quite the same. I haven’t spent enough time with ServiceStack to know how this syntax will work out on consumption by another deserializing system, but I suspect this may be something that only ServiceStack would handle correctly.

Unfortunately, the author of the project was nice enough to confirm that ServiceStack.Text does not currently afford the same extensibility as the .NET JavaScriptSerializer for overriding its default behavior in this situation. He did welcome a pull request which I will look into.

ServiceStack.Text also doesn’t appear to support deserializing into an ExpandoObject as this resulted in an empty object.

dynamic testDeserialization = ServiceStack.Text.JsonSerializer.DeserializeFromString<ExpandoObject>(jsonOfTest);

I haven’t confirmed if ServiceStack.Text deserializing works under MonoTouch yet. If it does, it would be worthwhile to have it running API JSON generation as well as the client-side JSON consumption since there is evidence it performs quite nicely.

UPDATE

I slapped together a new MonoTouch project in MonoDevelop and tossed in ServiceStack.Text’s DLLs with a few bits of code and confirmed it works great for a deserializing JSON into a pre-defined object.

public class TestClass {
    public int x { get; set; }
    public string x { get; set; }
}
TestClass result = ServiceStack.Text.JsonSerializer.DeserializeFromString<TestClass>("{\"x\":3,\"y\":\"something\"}");
Console.WriteLine("result: x={0}, y={1}", result.x, result.y);
// result: 3, something

UPDATE (2012-04-05)

I missed a blog entry on the author’s blog describing how to get Json.NET to output DateTime in different ways. For example, if you prefer the ISO-8601 output, you would be able to tell it to use the IsoDateTimeConverter. When I went to update the test project, the latest version of Json.NET (4.5 Release 1) now defaults to ISO-8601. Since I don’t want to risk break an existing API, I tweaked the DateFormatHandling to make the output match the old default. Examples are hard to find since this is such a new release, so I slapped one together and submitted it to the Json.NET docs (now on GitHub).

Newtonsoft.Json.JsonSerializerSettings settingsWithMicrosoftDateFormat = new Newtonsoft.Json.JsonSerializerSettings() { DateFormatHandling = Newtonsoft.Json.DateFormatHandling.MicrosoftDateFormat };
string jsonOfTest = Newtonsoft.Json.JsonConvert.SerializeObject(test, settingsWithMicrosoftDateFormat);
// {"x":"xvalue","y":"1333640263042-0600"}

Unfortunately, even Json.NET 4.5 is appending a timezone offset to the DateTime serialization that isn’t found in the .NET implementation. I’ll look into a custom implementation of DateTimeConverterBase and I have posted this as a question on StackOverflow.

Additional Notes

I haven’t played much with what problems may arise with the various representations of DateTime objects on the consumption side, but they definitely all handled it differently here.

Source Code

To get all the code in an ASP.NET MVC project (download, load solution, hit F5 [I hope]), check out the bitbucket.org repository for this post.