Cyotek Sitemap Creator Revision History

Please report any errors in how websites are crawled so that we can continue to improve Cyotek Sitemap Creator. The more information you can provide, the better we can make the product.

Changes and new features

• Added a Replace section to the Regular Expression dialog to make it easier to test replacement expressions
• Various performance enhancements
• The Errors tab no longer lists "Unknown Response" for non-200 HTTP codes, but instead includes the code description
• Added the ability to run user defined custom tools from within the application
• Attempting to open a recent file which no longer exists now prompts to remove the missing file from the recent files list
• Removed status bar animations

Bug fixes

• Fixed a crash when crawling if a rule was created with an invalid regular expression
• Reworked application mutex to avoid silent startup and shutdown exceptions
• Fixed regular expression cache not being thread safe
• Fixed status bar messages from occasionally not appearing
• Fixed rare object disposed crash when generating site maps using the FTP plugin

Please report any errors in how websites are crawled so that we can continue to improve Cyotek Sitemap Creator. The more information you can provide, the better we can make the product.

Changes and new features

• Added the ability to enable the "multi line" option in the Regular Expression editor to easier test patterns using ^ on $ on lists of URL's
• Added a Test URL option for Forms, allowing you to test that your forms can be successfully POSTed prior to running a full crawl
• Changed settings dialogs to use a tabbed interface

Bug fixes

• Fixed a large number of issues with the application services libraries and components
• Fixed an issue where attributes of posted URL's were not correctly loaded if encountered at a later point during the crawl
• Fixed a crash which could occur when using the title replacement options and a page had a null title
• Fixed a crash which could occur when scanning a HTML tag containing a malformed URL
• Fixed an issue where email addresses were stripped if they contained the # character and the "strip fragments" option was enabled

Please report any errors in how websites are crawled so that we can continue to improve Cyotek Sitemap Creator. The more information you can provide, the better we can make the product.

Changes and new features

• The Regular Expression editor can now list all URL's in the current link map to help with testing expressions
• Text editor dialog now supports find and replace functionality and other minor tweaks
• Reworked the Rule Editor to work around problems with rules which just manipulated content as opposed to basic inclusion or exclusion

Bug fixes

• Fixed a crash which occurred when choosing "View Link Map" from list view context menus
• Fixed a crash which occurred when saving a project and no files were present in any build results
• Fixed an issue where sitemaps would fail to build if the active project was untitled
• Corrected typos in flag enumerations
• Fixed an issue where font based window scaling wasn't working correctly
• Fixed an issue where the first item in a list view would be hidden when enabling the filter bar if it was previously hidden
• Fixed an issue where attempting to build sitemaps after cancelling a crawl would always return that the build was cancelled
• Text editor now correctly sets default folders when selecting "Save As"
• Fixed tab character not working in the text editor

Please report any errors in how websites are crawled so that we can continue to improve Cyotek Sitemap Creator. The more information you can provide, the better we can make the product.

Bug fixes

• Fixed a crash introduced in build 1.0.0.12 when saving a new project without performing a crawl
• Fixed a possible shutdown crash when not disposing of event handlers correctly

Changes and new features

• The URI control for selecting the website to analyze is now tied to the system URI history

Bug fixes

• Fixed a crash introduced in build 1.0.0.11 when trying to view link properties from any list control
• The "Analyze Successful" message is no longer displayed when the analyze has been performed during the building of sitemaps
• Fixed a problem where tooltips weren't large enough to accommodate their contents
• Opening or creating a project didn't clear the contents of the Build Results tab
• Fixed issues building a sitemap that had only an output filename specified without including a path
• Fixed the & character from not appearing correctly in the status bar
• Fixed issue with application window being sent behind other top level windows when cancelling a crawl
• Corrected typo's in Robots.txt add-in text

Changes and new features

• The last-modified meta tag is now supported. If found on matching documents, it will be preserved in the crawl map and used to sort the sitemap.
• The Link Map window now remembers its size and position

Bug fixes

• Generating the site map tree no longer marks the project has having changed
• Dates are now saved as UTC
• The list of incoming URI's for any given URI were being incorrectly populated
• When using the host replacement setting, commands now use the original URI as appropriate
• Fixed an issue where if a URI was referred to in multiple locations, after the first time it was encountered the outgoing and incoming URI links would not be updated correctly for future encounters.
• When reloading a project, the link map is no longer crawled looking for pages directly matching the root element, but all non-excluded URI's are formed into the map, resolving a problem where the crawl map generating from reloading a project may not match the crawl map generating from analyzing a website.

Changes and new features

• A new addin is available which allows you to automatically upload sitemaps to a FTP server after successful generation
• A new rule option has been added that disables a sitemap link being created for a given URL, but still allows the contents of the URL to be crawled
• A new rule option has been added that can be used to prevent a rule from matching a child URI
• A new rule option has been added that allows a URL which would normally be skipped to be added to the sitemap without actively crawling them, for example a RSS feed.
• Redirects are no longer listed in the Errors tab and do not trigger the "page errors found" message
• Rules which enable the inclusion of content now appear with green icons in list views
• Redirect processing now honors 303 and 307 response codes
• Report lists now display tooltips
• If a link redirects to another, the destination is now stored with the original link
• Link properties dialog now shows redirect information

Bug Fixes

• Exception reports were using the file version instead of the product version
• Fixed a rare XML crash when saving a project
• Fixed a crash which would sometimes occur when editing a rule or a form
• Fixed a crash opening a project if build result information was present for an unavailable sitemap provider
• External URI processor no longer transforms the URL if the "Host Replacement" option has been specified
• HTML Site Map plugin now correctly includes all items in the crawl map
• Fixed a crash which would occur if the referring URL was not available
• Fixed a crash which would occur if the "content-type" header wasn't present when pre-processing a URL
• Buttons in the main window now correctly follow the colors of the main theme

Changes and Updates

• Meta refresh redirects are now crawled
• Changed how redirects are handled, these will now appear in the main report lists
• Skipped list now displays the content type of entries
• Added new Not Found and Redirect exclusion reasons, redirects and missing files will no long appear as "None" in skip lists.

Bug Fixes

• When using URL replacement, under certain conditions the replaced URL's would be garbled.
• Two URL's with the same host bar the www prefix (e.g. http://cyotek.com/ and http://www.cyotek.com/) are now treat the same when determining if a URL is external.
• URI's were not correctly combined on pages being crawled as a result of a redirect.
• Reloading a sitemap which contained redirects did not display a map for any content discovered after the redirect
• No longer attempts to download content for redirected responses
• Project's weren't always being correctly marked as changed

Changes and Updates

• Substantial performance improvements have been made when loading large projects containing many links.
• Updated to use Html Agility Pack 1.4
• A new option to control if headers should be saved in the project file has been added. This option is disabled by default.
• The log tab has been removed.
• Cut, copy and paste commands are now available from the main window. However, lists and trees currently only support copy.

Bug Fixes

• Titles and Descriptions were attempted to be obtained from all files, causing a rare crash.
• The Accept GZip Compression option was never correctly read from the project file.
• The Sitemaps section in the Project Properties dialog was broken, preventing providers from being selected or unselected.
• The Results view wasn't being cleared when a new site analysis was started if previous information was present.
• Title Replacements with a blank pattern could behave very oddly

Changes and Updates

• Add-ins can now be enabled and disabled.
• Appearance themes are now enabled.

Bug Fixes

• Relative paths weren't being saved in project files correctly
• Fixed a problem where running on XP either didn't display disabled images or crashed.

Changes and additions

• 401 authentication requests are now supported, either via predefined credentials or during the crawl via a password dialog.
• The default buffer size has been increased to a larger value, allowing for faster downloads. In addition, the buffer size is now configurable.
• Gzip compression is now supported.
• Deflate compression is now supported.
• Crawling is now performed on a separate thread, resolving sluggish behaviour with the user interface. Disabled for this build
• The Link Map Viewer now has a tab for displaying all links found. All lists in this dialog have had new columns added with more details on the links.
• The project properties dialog now provides access to properties which could not be changed in previous builds.
• Object model simplified, some confusing class inheritance has been removed.
• Added the ability for additional content type handlers to be used.
• Added the ability to specify multiple seed URI's.
• A new configuration section has been added allowing you to store authentication credentials in a project file and to disable the password dialog when crawling.
• Added a new viewer extensibility options allowing new tabs to be added to the interface.
• Major refactoring of the base IApplication implementation.
• Response headers are now stored in the link map. The Link Properties dialog now displays these headers.
• The Link Properties dialog now displays local path information and the ability to open, open the containing folder, or edit the local file. This change is more for Cyotek WebCopy than for the Sitemap Creator.
• Scanning of subdomains is now supported.
• You can now select from a common list of user agents.
• Crawling will no longer occur above the root level by default. A new option has been added to toggle this behaviour.

Bug Fixes

• Redirects were not followed for 301 or 307 status codes.
• Page links found in an IFrame or Frameset were not scanned.
• Image links saved into a Google sitemap weren't updated if the project was using host replacement.
• Cancelling a crawl now also correctly aborts the current transfer instead of waiting for it to complete.
• If a list was scrolled horizontally, the content menu displayed from the filter bar wasn't positioned correctly.
• Fixed a bug where response headers were not available if the request was not an expected response code.
• The results list in the regular expression editor didn't resize with the window.
• The result expression editor no longer displays results for a blank expression.
• Duplicate keyboard accelerators have been fixed.
• The Sorted property of a crawl map now correctly defaults to false.
• Fixed a problem where it was possible for the CommandManager to try and load classes it had no business loading, causing error messages to be displayed on startup.
• Fixed a problem where command interface elements were not always given a name, leading to a problem where items could not be accessed unless the full text was known.
• The failure to load an image resource for a command interface element will no longer cause the application to fail to initialize.
• Fixed some layout problems in Windows XP.

Changes and additions

• Added the ability to include an URL pattern with a title replacement option. This allows you to include titles for non-HTML pages, such as an RSS feed or to replace titles on a single page where more or page pages have the same title.
• Exclusions have been renamed to Rules to reflect their changing nature in this build and future planned enhancements.
• Added a new option for Rules which allows you to include links to images in the sitemap. Any matched images will be be saved into the Google sitemap.
• Titles are now stored with the link map for non html documents if the title element is specified for the link. The new Google image option will uses these titles for generated image sitemap entries.
• When using the Add Rule context menu item from a result list, the editing dialog is now displaying allowing the entire rule to be configured.
• The Add Rule command now includes any applicable query string in the URL for the rule.
• The Build Results view now contains additional text if page errors were detected.
• Build results are now saved with the project
• The Build Results tab is now automatically repopulated when opening a project if previous build information is present.
• A new Edit option has been added to files listed in the Build Results tab. This allows you to view, edit, save and print plain text files within the Sitemap Creator.
• Added an option to sort pages and folders rather than just as found.
• A basic Regular Expression Editor is available and can be accessed via the Function button displayed next to supported fields.
• Error text associated with a page error is now stored in the link map.
• The page errors list will now be regenerated on loading a project with a saved link map.
• The Link Map Viewer now displays link titles and error text.

Bug Fixes

• Fixed a problem where URL's contained spaces were incorrectly encoded in the ASP.NET sitemap.
• The Add Rule and Add Form dialog's caused a crash when being used to create rather than modify items.
• If a link to child of a page which has been matched to a rule with the DisableCrawl option is detected, the entire link will now be excluded.
• Fixed some selection inconsistencies in rules and forms editors.
• The Add Rule command now automatically escapes regular expression elements within the URL, such as the ? of a query string.

Changes and additions

• Command line arguments added.
• The Exclusion Type functionality has been removed and replaced with a series of flags to allow more control over exclusions.
• A new "Use Full URI" option has been added for exclusions. If this flag is set, the entire URI including the domain is used for exclusion matching, otherwise only the path and query is used. This change makes it easier to match URL's such as "/sitemap" but not "subfolder/sitemap".
• A new "Use Modified URI" option has been added for exclusions. If this flag is set, the modified URL as used by the sitemap is used, otherwise the original URL is used. This flag is mainly used when you are using domain aliases to remap URL's.
• The Exclusions and Forms lists and editors have been recoded to work in a less hacky fashion.
• Added the ability to turn on filtering for the remaining lists.
• Additional options added to the context menu for filter columns.
• The link map now stores the skip reason, if applicable.
• The Skipped Pages list is now automatically repopulated when opening a project.
• The Pages list is now automatically repopulated when opening a project.
• Some optimisation made to the crawl process to make it a little faster.
• Redirects are now followed.
• The popup progress dialog no longer appears when an operation is being performing from the UI.
• Added the ability to reorder exclusions.
• The last used folder is now remembered when displaying the Open File dialog.

Bug Fixes

• Fixed a crash when reordering forms.
• The skipped pages list is now correctly populated during the analyze process after getting broke in build 1.0.0.3
• Fixed a bug in ObservableCollection which resulted in duplicate entries appearing in lists such as Exclusions when editing an existing item.
• Title replacement regular expression patterns are now case insensitive to be consistent with other pattern matching used in the software.
• The Source column has been removed from the Analyze Results as it's not applicable for Sitemap Creator projects.
• A status of "Skipped" instead of "Failed" could be returned in some circumstances.

Changes and additions

• The prompt which appears if you attempt to generate a sitemap using an old link map now only appears once per project session.
• Holding down shift when clicking the Build Sitemap or Build All Sitemaps commands will automatically analyze the website first. This will also suppress the displaying of the old link map prompt.
• Filter functionality has now been build into list views, removing the clunky interface from the View Link Map dialog.
• Added a new option to control how URL's are combined if a link starts with a forward slash but doesn't match the absolute path of the website being crawled.
• The Site and DefaultDocuments properties has been moved to the core CrawlerSettings object along with all related functionality. The core object is now fully responsible for creating the map based on the configured content types. This change won't affect the Sitemap Creator directly but makes it easier for other products to display a sitemap.
• Changed how temporary file names are generated for crawling with a specified cache folder.
• Added Title and Description properties to the LinkInfo object
• The sitemap is no longer saved in a project file. Instead, it will be automatically regenerated from the link map information.
• Added the ability to view the properties of link map entries.
• Various user interface tweaks

Bug Fixes

• The wrong status was shown when attempting to post a form and the attempt failed.
• The last crawled date wasn't being read from a project.
• The wrong icons would get displayed in the column header when sorting a list
• Fixed a crash which could occur when right clicking an empty or filtered list in the Link Map dialog.
• Fixed an issue where certain link info properties were not getting persisted
• The progress displayed in the dialog didn't always reflect the progress of the results view.
• Temporary files created when saving a project file were not being removed after the save was complete
• A warning would be displayed for any sitemaps that could not be generated even if that sitemap was not selected for use with the Create All command.
• Fixed an crash which would occur if exclusions were defined and a URI did not have a content type
• Setting some LinkInfo properties did not mark the object as changed

Changes and additions

• Added file association to setup
• Projects are now added to the shell Recent Documents list and, in Windows 7, the jumplist.
• Document crawling has been refactored and separated to allow CSS, script, images, anchors etc to be independently scanned.

Bug Fixes

• Fixed an issue where a header check on a page which did not exist caused a crash
• Fixed a crash which would occur when trying to determine the default extension of an unspecified content type.
• Fixed an issue where an URI which should have been skipped is still processed.
• Fixed an issue when downloading a content type included due to a global inclusion setting but which did not support crawling.
• Fixed various layout glitches.
• Fixed an issue where the wrong status icon was displayed for a "failed" operation

Changes and additions

• The XML schema for the project file has changed, loss of some settings is possible. Please contact us for details on upgrading your projects.
• The sitemap is now located in the Project object rather than the SitemapCreator object. This means it can be persisted in a project file and reloaded without having to re-analyze a site.
• When opening a project containing a previously saved sitemap, if you try to build a sitemap without analyzing the site first, you are prompted if you want to use the existing sitemap or analyze the site from scratch.
• The Page object was being used to represent both forms and sitemap pages. This has now been seperated into Page and FormPage.
• Added a new Html Page sitemap type.
• Build results is now displayed as tab in the main window rather than via the separate popup dialog.
• Added the ability to disable exclusions and posted forms without having to remove them.
• The "Posted pages" functionality has been renamed to "Forms"
• Lists can now be sorted on any column
• The Sitemap Creator now utilizes Cyotek's Application Services, allowing easy integration of additional commands and functionality and making it much simpler to write add-ons. Existing support for the old plugin framework has been dropped.
• Addins are now operational.
• Domain Aliases, Exclusions, Forms and Title Replacements editors all now allow in-line editing of items. The original double click to remove, edit and add behaviour is still present.
• Moved some options around in the Project Properties dialog, and additional options have been added.
• Added additional URI checks
• You can now add exclusions by context clicking an URL in the Results pane.
• Added the ability to disable the removal of fragment information (bookmarks) from URL's.
• Added status icons to all lists and replaced several button glyphs.
• The crawler now generates a map of all URL's in a site. This can optionally be saved into a Sitemap Project file.
• Added the ability to view the link map from within the Sitemap Creator. If you have saved the link-map into your project, you can reopen the project and view the link-map without having to re-analyze your website.
• Added the "Open in browser" option to the context menu for Page Error entries.
• The DontLeaveSite property of the crawler object has been removed, this behaviour is now implicit and cannot be disabled on a global scale.
• Added progress support to the crawler and front end when downloading files.
• Added additional display options for the sitemap tree

Bug Fixes

• Editors which displayed a relative URL caused a crash if the base URI for the project was invalid.
• Page events always returned "Bad Request" for the HttpStatus property when no status was available.
• Corrected a problem where the crawler downloaded binary files incorrectly.
• Corrected a problem where URL's containing the hash character could be incorrectly parsed
• The page errors tab wasn't always being activated if one or more page errors were found.
• Page errors are now correctly created if the response code of a request is not a success code
• Corrected a problem where query string values were not encoded.
• Fixed a crash which could occur when modifying exclusions in the main window after opening the popup editor.

• Initial Release