Blog Articles and information on C# and .NET development topics

When running Sitemap Creator on the development version of cyotek.com, we found all links pointing to articles returned a 404 status code when crawling was attempted. But if same URL was copied into a browser, it would load correctly.

This surprised us, as cyotek.com is the main site we test Sitemap Creator and WebCopy on and they've always worked in the past. Next, we tried it directly on cyotek.com, and got the same result. However, this being the release version of the web, we started receiving error emails from the website (these are not sent from the debug builds).

The exception being reported was this:

System.Web.HttpException: A public action method 'display' could not be found on controller 'Cyotek.Web.Controllers.ArticleController'.
  at System.Web.Mvc.Controller.HandleUnknownAction(String actionName)
  at System.Web.Mvc.Controller.ExecuteCore()
  at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
  at System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(RequestContext requestContext)
  at System.Web.Mvc.MvcHandler.ProcessRequest(HttpContextBase httpContext)
  at System.Web.Mvc.MvcHandler.ProcessRequest(HttpContext httpContext)
  at System.Web.Mvc.MvcHandler.System.Web.IHttpHandler.ProcessRequest(HttpContext httpContext)
  at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
  at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

This error message certainly raised eyebrows, as of course, this action does exist.

This is the current definition of the display article action:

[OutputCache(CacheProfile = "Short")]
[AcceptVerbs(HttpVerbs.Get)]
public ActionResult Display(string id, bool? posted)
{
}

As soon as we looked at the code, we realised what had happened. By default both Sitemap Creator and WebCopy make HEAD requests to obtain the headers for a given URL, such as the content type. They use these headers to determine if they should go ahead and download the entire file - Sitemap Creator won't download anything that isn't text/html for example.

And this is the problem - in the last update to cyotek.com, we changed a few site settings to stop the number of error emails occurring due to spammer activity. For some reason the AcceptVerbs attribute was applied to the Display action method at this point. And as it is only set to accept GET, it means our HEAD calls automatically fail.

One changing the attribute, everything started working nicely again.

[AcceptVerbs(HttpVerbs.Get | HttpVerbs.Head)]

For once, a nice and simple mystery to solve, and a nice little tip which will hopefully help anyone else who has a similar issue.

• 2010-11-20 - First published
• 2020-11-21 - Updated formatting

While we appreciate comments from our users, please follow our posting guidelines. Have you tried the Cyotek Forums for support from Cyotek and the community?