You can extend the functionality of the Sitemap Creator by writing your own addins using C# or VB.NET.
This tutorial will walk you though creating a simple addin for import and exporting a robots.txt file. The source code for this addin can be downloaded from the link at the end.
First, create a new class library project, and add the following references:
- Cyotek.Core
- Cyotek.ApplicationServices
- Cyotek.SitemapCreator
- Cyotek.Web.Crawler
- Cyotek.Web.Cralwer.Resources
We will be working on juggling the inheritance levels to avoid the number of required references as the beta progresses.
Add a new class, and inherit it from AddinBase. This abstract class provides the base functionality needed to create your addin. Alternatively, you could fully implement the IAddin interface.
When using AddinBase, as a minimal you must override the Name property and Initialize method, however other properties are available.
public class RobotsAddin : AddinBase
{
public override string Name
{ get { return "Robots"; } }
public override string Title
{ get { return "Robots.txt Addin"; } }
public override void Initialize(ServiceContainer serviceContainer)
{
}
}
The addin framework is built around the concepts of "commands", blocks of code which you can then attach to the user interface without knowing or caring how the UI is implemented.
In this tutorial, we will create two commands, one for importing an existing robots.txt file, and another for exporting.
Add another class, this time inheriting from CrawlerCommand. This command class is for simple "click to execute" type commands with no additional elements such as dropdown lists or sub menus.
As with AddinBase, CrawlerCommand is abstract. The constructor of your command must call the base constructor, passing in the name of your command. You can name the command anything you want, but it must be unique.
You must override the Execute method, which will be called when your command is to be actioned. You also need to override the Text property to specify the text displayed in the user interface.
public class ImportRobotsCommand : CrawlerCommand
{
public ImportRobotsCommand() :
base("File.ImportRobots")
{ }
public override void Execute()
{
}
public override string Text
{ get { return "&Import robots.txt..."; } }
}
The CrawlerCommand class has a property named Settings which returns a reference to the active project file.
For the purposes of this tutorial, we'll be working with the ExclusionCollection collection accessed via the UriExclusions property. The next step is to populate our Execute method with the code to read a list of exclusions from a robots.txt file and update the exclusions collection accordingly.
string fileName;
using (OpenFileDialog dialog = new OpenFileDialog())
{
dialog.Title = "Import robots.txt";
dialog.Filter = Strings.Filter;
dialog.DefaultExt = Strings.DefaultExtension;
if (dialog.ShowDialog(this.Application.MainForm) == DialogResult.OK)
fileName = dialog.FileName;
else
fileName = null;
}
if (!string.IsNullOrEmpty(fileName))
{
int skipCount;
int addCount;
skipCount = 0;
addCount = 0;
foreach (string line in File.ReadAllLines(fileName))
{
if (line.TrimStart().ToLower().StartsWith(Strings.DisallowPrefix))
{
string url;
url = line.Substring(Strings.DisallowPrefix.Length).Trim();
if (!string.IsNullOrEmpty(url) && !this.Settings.UriExclusions.Contains(url))
{
addCount++;
this.Settings.UriExclusions.Add(url, ExclusionType.All);
}
else
skipCount++;
}
}
MessageBox.Show(string.Format("Import complete. {0} URL's were added to the exclusion list, {1} URL's were already present and ignored.", addCount, skipCount), this.Text, MessageBoxButtons.OK, MessageBoxIcon.Information);
}
Now that the Import command is ready, it's time to do the export command. Create a new command class in the same way you did for importing, but this time we'll be generating a robots.txt from any enabled URI exclusions in the project
string fileName;
using (SaveFileDialog dialog = new SaveFileDialog())
{
dialog.Title = "Export robots.txt";
dialog.Filter = Strings.Filter;
dialog.DefaultExt = Strings.DefaultExtension;
if (dialog.ShowDialog(this.Application.MainForm) == DialogResult.OK)
fileName = dialog.FileName;
else
fileName = null;
}
if (!string.IsNullOrEmpty(fileName))
{
int saveCount;
int skipCount;
saveCount = 0;
skipCount = 0;
using (FileStream stream = new FileStream(fileName, FileMode.Create))
{
using (StreamWriter writer = new StreamWriter(stream))
{
writer.WriteLine(Strings.DefaultUserAgent);
foreach (Exclusion exclusion in this.Settings.UriExclusions)
{
if (exclusion.Enabled && exclusion.Pattern.StartsWith("/"))
{
saveCount++;
writer.WriteLine("{0} {1}", Strings.DisallowPrefix, exclusion.Pattern);
}
else
skipCount++;
}
}
}
MessageBox.Show(string.Format("Import complete. {0} URL's were exported, {1} URL's were skipped due to either being disabled or not supported.", saveCount, skipCount), this.Text, MessageBoxButtons.OK, MessageBoxIcon.Information);
}
Now that you have your two commands, it's time to register them with the Sitemap Creator and add them to the user interface. To do this, we need to go back to the Initialize method in the addin class created at the start of the tutorial.
The ICommandManager interface allows you to register your commands with the host. You can access this via the CommandManager property of the ServiceManager class. You can either add commands individually via the Add method, or load all supported commands at once via the RegisterAssembly method. As all of the commands in this tutorial have parameterless constructors, and have the AutoLoad attribute specified (this is inherited from the CrawlerCommand class) then we can use RegisterAssembly - we just pass in the assembly we want to load:
commandManager.RegisterAssembly(this.GetType().Assembly);
With these commands now registered, they can be accessed and executed by other commands or addins. But in order for the user to access them, we'll need to add them to the front end.
Another property of the ServiceManager class is Application. The ISitemapCreatorApplication interface provides properties and methods for directly interacting with the host application.
In this case, we want to add our two commands to the File menu, which we can do by calling methods on the MainMenu property.
IMenu fileMenu;
int insertionIndex;
fileMenu = ServiceManager.Current.Application.MainMenu.MenuItems["File"];
insertionIndex = fileMenu.GetIndexOf("File.SaveAs") + 1; // plus one due to the separator
fileMenu.InsertMenuItem(insertionIndex, commandManager[typeof(ExportRobotsCommand)]);
fileMenu.InsertMenuItem(insertionIndex, commandManager[typeof(ImportRobotsCommand)]);
fileMenu.InsertSeparator(insertionIndex);
Using the functionality exposed via the IMenu interface, we can get the position of an existing command, and insert our own commands after this. When the end user clicks the menu item the command will be automatically executed. There's no need to hook into any specific user interface events and you don't have to worry about future versions of the product breaking your addin - unless of course any commands you are expecting to be present (File and File.SaveAs in this case) are removed.
Once you've created your functionality, just compile your assembly and copy the output DLL to the addins subfolder of your Sitemap Creator installation folder. When you restart the program, it will automatically detect and load your addin.
Hopefully this tutorial shows you just how easy it can be to extend the Sitemap Creator for your own purposes. You can download the sample project for this addin by clicking the link below.
