To overcome this issue you could write your one rewrite provider like this:
protected override bool IsHtmlUrlValidForRewrite(UrlBuilder contextUrl, UrlBuilder url)
if (url.Port > 0 && (url.Scheme == System.Uri.UriSchemeHttp || url.Scheme == System.Uri.UriSchemeHttps))
Settings settings = Settings.MapHostToSettings(url.Host, false);
if ((settings != null) && settings.SiteUrl.Port > 0 && settings.SiteUrl.Port != url.Port)
return base.IsHtmlUrlValidForRewrite(contextUrl, url);
Note: This will never happen in a production enviornment, since you would never render links with port numbers, but since this was part of some functionalty that needed to be tested on the testenviornment, it is not so easy to tell the customer: "But I am sure it will work in production..".
Download the provider here