I recently had the need to switch a robots.txt file dynamically, depending on whether a site was deployed to live or beta servers. The solution was very simple, ditch the real file and replace it with a controller action. This is a brief note on how I did this. I don’t pretend this is an ideal solution, it was something I knocked together before I went out for dinner.

The solution I went for was in three parts: a web.config transform, routing and the controller itself.

The web config transform for my live site looks like this:

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="SiteStatus" value="live" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/>
    </appSettings>
</configuration>
view raw Web.live.config This Gist brought to you by GitHub.

The route is equally simple:

routes.MapRoute(
    "Robots.txt",
    "Robots.txt",
    new 
    { 
        controller = "Robots", 
        action = "RobotsText" 
    }
);

view raw routing.cs This Gist brought to you by GitHub.

And finally the actual controller:

public class RobotsController : Controller
{
    public FileContentResult RobotsText()
    {
        var content = "User-agent: *" + Environment.NewLine;

        if (string.Equals(ConfigurationManager.AppSettings["SiteStatus"], "live", StringComparison.InvariantCultureIgnoreCase))
        {
            content += "Disallow: /elmah.axd" + Environment.NewLine;
            content += "Sitemap: http://www.jacquelinewhite.co.uk/sitemap.xml";
        }
        else
        {
            content += "Disallow: /" + Environment.NewLine; 
        }

        return File(
                Encoding.UTF8.GetBytes(content),
                "text/plain"); 
    }
}
view raw RobotsController.cs This Gist brought to you by GitHub.

And there we have it, my robots.txt file now is customised on a per deployment basis.