In a current EPiServer project, I was using the Web3 component from Networked Planet for topic map relations.
Web3 was installed as an IIS website on the server using Windows Authentication.
The website interacting with Web3 (customer's main site) was set up alongside it in IIS, normally using Forms+Anonymous authentication.
Both sites were set up with custom hostheaders instead of just listening to a specific port.
After installation on the server, I browsed to the Web3 site to begin creating topic map schemas and relations. Browsing the site caused a login dialog to appear (as it should when using Windows auth.) Problem was, the login dialog did not accept the credentials even though they were valid for a Windows user account on the server. After several attempts, the server returned "HTTP 401.1 - Unauthorized: Access is denied due to invalid credentials".
It turns out that there is a security feature introduced in .NET 3.5 SP1 which fails if a request is made to a website which has a custom hostheader different from the server hostname. In effect, the request fails to loopback to the server it came from, where the website is located.
NOTE: This only applies when the request is made from the same server that the website is located on, i.e. when you are logged onto the webserver.
If browsing the site from an external machine, the Windows Authentication should work fine.
Fixing this involves a registry hack and a server reboot, which is described in this Microsoft article: http://support.microsoft.com/kb/896861/en-us
Big thanks go out to the NetworkedPlanet support team which helped discover the solution after hours of error tracing!