Cannot get past login screen with Mac OSX and Firefox

[kderoos]

We are having difficulty with Mac OSX stations with Firefox 3.6.10 getting past the login screen.

Server is Cognos 8 BI version 8.4 fix pack 2 and is configured to use Windows domain authentication.  Windows users using IE are not prompted for a password.  Windows users using Firefox 3.6.X must modify the network.automatic-ntlm.trusted-uris preference to include the server's domain name (i.e. dw.contoso.com).

We're able to get those Macs who have been bound to the contoso.com Active Directory Domain to successfully log in with Firefox by modifying the same Firefox preference as noted above, but those stations that have not been bound to the domain are continually prompted for a user id and password.  Regardless of what user id/password combination we use (joeuser, joeuser@contoso.com, contoso/joeuser) it is rejected. 

We are able to log in with the Macs using Safari without modifying any preferences BUT Safari is not a supported browser and sometimes crashes on larger reports.

Any thoughts or insight would be greatly appreciated.

[kderoos]

I should mention that binding the problematic Macs isn't feasible in the near term due to organizational politics. 

I ended up installing Wireshark on my Mac and did some packet sniffing using both Firefox and Safari.

Reviewing the resulting packets side by side I found that the Firefox browser was returning a "401 - Unauthorized URL" message. Reviewing IBM support documentation, the document http://www-01.ibm.com/support/docview.wss?uid=swg21375539 indicates that it's an IIS issue and that by enabling Basic Authentication in IIS individuals can get in.

Basic Authentication sends clear text passwords though, so not something I'm interested in doing. 

Anyone have any ideas?

[kderoos]

Figured this out so am posting an internal support document I wrote that describes the problem and resolution.

Symptoms
Mac users attempting to access reports in Cognos are prompted for a net id and password. No combination of net id and password works.

More Information
Web browser support for this product is limited to Internet Explorer (preferred) or Firefox per vendor http://www-01.ibm.com/support/docview.wss?uid=swg27017522#browsers. The Safari browser included with Mac OSX is not supported by the vendor.

Cause
Firefox on the Mac requires a Kerberos ticket from CONTOSO.COM to access Cognos. Macs that are member stations of the CONTOSO.COM Active Directory obtain valid tickets upon login. All other Macs must first obtain a ticket before attempting to access Cognos.

Resolution
This is a list of potential resolutions for this issue. IT staff should review and test solutions to determine which of these are workable and would fit within support efforts of departmental IT Staff. Solutions are grouped together on whether a Mac is a member station of the DEPARTMENT.CONTOSO.COM windows domain or not. Each numbered solution should be considered a standalone solution.

Mac not bound to a domain
1. Bind Mac to CONTOSO.COM windows domain
2. Individual must retrieve Kerberos ticket to CONTOSO.COM prior to logging into e-Data
     a. Mac OSX 10.6.X //System/Library/CoreServices/Ticket Viewer
     b. Mac OSX 10.5.X //System/Library/CoreServices/Kerberos

Mac bound to DEPARTMENT.CONTOSO.COM windows domain
     1. Unbind Mac from existing domain.
     2. Bind Mac to CONTOSO.COM
     3. Allow user to delete DEPARTMENT.CONTOSO.COM Kerberos ticket and obtain a CONTOSO.COM Kerberos ticket when needed (either manually or programatically)
     4. Supply user with a Windows Virtual Machine to use with e-Data
     5. Supply user with Windows Machine

Hope this helps others.