Validate links

In Optimizely Content Management System (CMS), you can track broken links for a website using the link validator scheduled job. The Link Validation scheduled job checks the links in tblContentSoftLink, performs a head request against each one, and saves the status of the links back to tblContentSoftLink. The result of the validation job is available as a report called Link Validation report in the Report Center.

The scheduled job first gets a batch of up to 1000 links from tblContentSoftLink. The job returns only unchecked links or links that were checked earlier than when the job started. The job uses the date the link was last checked and the re-check interval to determine if the link should be checked again.

Each of the links in the batch is checked using a head request if the servers' robots.txt allows for this. No host is checked more than once every five seconds. If a link exists on a host that was checked in the last five seconds, the job waits five seconds and then checks the link.

The job saves the status of the link and the date the link was checked, and includes the HTTP status code if possible, to tblContentSoftLink. The job saves information about when a link was first found broken. After the first batch of links is checked, a new batch is fetched from the database.

The job continues until it cannot get any more unchecked links form the database, or the job's runtime has exceeded the value set in maximumRunTime. The job stops if many consecutive errors are found on external links, in case of a general network problem with the server running the site.

Configure the link validation

None of the settings are required, but you can use them to customize the behavior of the link validation job. You can configure programmatically or with configuration files as described in Configure CMS. Here is an example

services.Configure<LinkValidatorOptions>(options => {
                options.ExternalLinkMinimumRequestInterval = TimeSpan.FromSeconds(5);
                options.MaximumRunTime = TimeSpan.Zero;
                options.RecheckInterval = TimeSpan.FromHours(7);
                options.UserAgent = "EPiServer Link Checker";
                options.ProxyAddress = new Uri("http://myproxy.mysite.com");
                options.ProxyUser = "myUserName";
                options.ProxyPassword = "secretPassword";
                options.ProxyDomain = ".mysite.com";
                options.InternalLinkValidation = ValidationType.Api;
                options.ExcludePatterns.Add(".*doc");
                options.ExcludePatterns.Add(".*pdf");
            });

The available settings are described at Link Validation.

Known limitations

The link validator does not handle private resources except for pages. This includes documents and images stored on a local file system which does not allow anonymous access. If you use forms authentication, these links are not validated and do not display in the link report. If you use basic or Windows authentication, links to these resources result in 401 (access denied) in the link report. This may be true for an intranet site with Windows authentication and anonymous access disabled.