crminthefield: Kerberos in Load Balanced Environments

[Blog bot]

Источник: http://blogs.msdn.com/b/crminthefiel...ironments.aspx
==============

A very common topic we tend to get many questions on is delegation and troubleshooting delegation usually via one of the many topics that impact this: SPN’s, Kerberos, Load Balancers, 401 errors, etc. Many times detailed environment knowledge is required to make any conclusions, so rather than documenting a complex flowchart that covers every possible scenario, I am going to draft up a fairly common environment and document the various steps required to make sure Kerberos is working for that environment. Kerberos only works properly when everything is setup correctly, so troubleshooting issues can be very frustrating and time consuming. If there is any one issue in the chain, authentication will fail and give you errors like: 
  – “Login failed for user: NT AUTHORITY\ANONYMOUS LOGON”
  – “Login failed for: NT AUTHORITY\SYSTEM”
  – “HTTP Error 401.1 – Unauthorized: Access is denied”
If you’re troubleshooting one of these issues I suggest reading through all the steps in this article and verifying your configuration, then move on to the troubleshooting section.

A few notes regarding this post:

• These concepts apply to any web applications/services that are referenced by a name other than their computer name in Active Directory (regardless of a physical load balancers presence), in this example I’m using Dynamics CRM as the web application
• Kerberos authentication is required if you are persisting a user’s credentials or delegating the user’s credentials through more than 1 hop (this scenario is commonly referred to as “double hop”)
• For the purpose of demonstration I am illustrating unconstrained delegation. Constrained delegation is an optional way to define which SPN’s allow delegation of credentials (essentially a more specific mapping for delegation).
• For information on CRM 2011 setup and questions on splitting roles and distributing services for better scaling please refer to my other blog post titled: “Server Setup Frequently Asked Questions” (http://bit.ly/CRMSetupFAQ).
  • In addition the FAQ article includes information about the MSCRMSandboxService SPN which may be required depending on your configuration.

This post has the following assumptions:

• CRM 2011 is installed on your CRM servers: NDCRMSrv1, NDCRMSrv2, NDCRMSrv3
• CRM 2011 is installed to databases located on the SQL Server: NDSQLSrv1 (this is for the purpose of demonstration only, CRM does support using multiple SQL servers to host the various CRM databases)
• SSRS 2008 (or R2) is installed on your reporting servers: NDRPTSrv1 and NDRPTSrv2 and these servers are configured in a Scale-Out deployment (using the same SSRS configuration databases) instructions are here: http://bit.ly/SXtvAZ
• You have access (or have a process) to request changes or make changes to Active Directory
• You have not set any SPN’s for these services in your environment and you have no duplicate SPN’s for these services or host machines.
• Your domain functional level is 2003 or greater
• You are using Active Directory integrated authentication for Dynamics CRM.

1) Service Principal Names (SPN’s):

First thing we want to focus on is setting our SPN’s correctly. A service principal Name (SPN) is a way to map identities in Active Directory to various services (SQL, HTTP, etc) hosted in your domain. This mapping is an essential part of issuing and decrypting Kerberos tickets within your environment. Below, in the troubleshooting section, you can see how a workstation will ask your DC’s for a Kerberos ticket via the service name which, for HTTP services, is determined by the URL of the service.

If you run into any scenarios where you believe there must be duplicate SPN’s you’re most likely trying to use more than one identity to host a single service. For example: you’ll notice in the above graphic that we’re hosting the CRM services under the hostname crm.contoso.com and with a single identity: contoso\CRMWeb. If you wanted to add a second website which would use a different port and a different identity, but would use the same URL, you have two options: 1) use a different hostname/URL to differentiate the services 2) register the SPN’s using a port number to differentiate them.

To configure the SPN’s I recommend mapping out all the various services you’re hosting. For HTTP services you’ll also want to map out the URL’s used to access the service*, for this example we have the following potential URL’s:

Potential CRM Service URL’s: Potential Report Service URL’s: - http://crm
- http://crm.contoso.com
- http://ndcrmsrv1
- http://ndcrmsrv1.contoso.com
- http://ndcrmsrv2
- http://ndcrmsrv2.contoso.com
- http://ndcrmsrv3
- http://ndcrmsrv3.contoso.com - http://reports
- http://reports.contoso.com
- http://ndrptsrv1
- http://ndrptsrv1.contoso.com
- http://ndrptsrv2
- http://ndrptsrv2.contoso.com

*NOTE: there has been some debate on the need for registering both the Fully Qualified Domain Name (FQDN) SPN as well as the netbios name SPN, generally speaking, I’ll register both just to make sure all bases are covered. In addition, if users will only be using the “friendly” url or alias (in this instance: http://CRM) then registering the SPN’s for each server is not required – but if you plan on using the server names to access the web services (even if it’s for testing) it’s a good idea to register those as well.

Next, you’ll want to construct a table of the various services that will require an SPN. Please note: if you had more than one SQL server or any custom web applications or web services that use different URL’s those should also be added to this table.

Identity/User Hosting the Service Service Type Service Principal Name(s)

CRMWeb@contoso.com HTTP http/crm
http/crm.contoso.com
http/ndcrmsrv1
http/ndcrmsrv1.contoso.com
http/ndcrmsrv2
http/ndcrmsrv2.contoso.com
http/ndcrmsrv3
http/ndcrmsrv3.contoso.com

reportService@contoso.com HTTP http/reports
http/reports.contoso.com
http/ndrptsrv1
http/ndrptsrv1.contoso.com
http/ndrptsrv2
http/ndrptsrv2.contoso.com

SQLService@contoso.com MSSQLSvc MSSQLSvc/NDSQLSrv1.contoso.com

2) Registering the SPN’s:

Now that we have a full list of all the SPN’s we’re going to set them up in Active Directory. The easiest way to set an SPN is to use the SetSpn utility in Windows. You will have to have elevated permissions to add SPN’s in Active Directory. The syntax to use for adding the above SPN’s is as follows: SetSpn –S {spnToAdd} {AD Object}. An example of adding the crm.contoso.com SPN and the SQL SPN would be:
SetSpn -S HTTP/crm.contoso.com contoso\crmweb
SetSpn -S MSSQLSvc/NDSQLSrv1.contoso.com contoso\sqlservice

Repeat this process for each SPN, I’ve found it helpful to have an Excel spreadsheet of my SPN’s and account names and check them off as I add them. You’ll notice that I’ve documented using –S instead of –A, –S is a new option that is used to check for duplicates before adding the SPN – if there are duplicates you’ll receive an error and the SPN will not be added. 

3) Configuring CRM’s IIS Server for Kernel Mode Authentication:

IIS 7 and 7.5 can take advantage of Kernel Mode authentication and CRM configures it’s website to use it. Kernel Mode auth is desired as it improves authentication performance and prevents authentication problems with application pools using custom identities. However, when using a host name that does not match the server name(s) we need to tell IIS to use the Application Pool’s identity (as it’s common on all your servers) and not the system identity this allows the Kerberos tickets to be decrypted by any server behind the load balancer. To do this we’re going to set the UseAppPoolCredentials property in the IIS configuration.

Check the value of the webservers WindowsAuthentication configuration
%systemroot%\System32\inetsrv\appcmd.exe list config -section:system.webServer/security/authentication/windowsAuthentication


Set “UseAppPoolCredentials” in the webservers configuration file:
%systemroot%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -useAppPoolCredentials:true /commit:apphost

3.1)Kerberos Auth Persistence (optional)

Configuring persistence of Kerberos credentials will reduce the challenge/responses on the same TCP session resulting in fewer 401’s and better performance. If you analyze a Fiddler trace of communication between the user and the server you’ll notice that most successful responses from a webserver are proceeded by at least one 401, even when the requests are for the same TCP session between the client and server. By setting the AuthPersistNonNTLM property to “true” the webserver will persist the credentials of the user over the existing TCP sessions – dramatically reducing the number of 401’s for the user. If you’re looking in Fiddler you will also notice a Persist-Auth header in the response indicating auth was persisted for that request. If you’re interested in another tweak for better performance through compression of WCF traffic be sure to check out this blog posting: “Enable WCF Compression to Improve CRM 2011 Network Performance”

Check the value of the webservers WindowsAuthentication configuration
%systemroot%\System32\inetsrv\appcmd.exe list config -section:system.webServer/security/authentication/windowsAuthentication




Set “authPersistNonNTLM” in the webservers configuration file:
%systemroot%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/authentication/windowsAuthentication -authPersistNonNTLM:true /commit:apphost

4)Configure your accounts to allow for Kerberos Delegation:

To make sure delegation will succeed in any given environment there are a couple of settings that must be in place before delegation is allowed.

1. Any services that persist credentials must be allowed to delegate in Active Directory. For this example, we must enable the CRMWeb account to delegate. To do this
   • login to a server with credentials allowing you to edit the given service account
   • Open Active Directory Users & Computers (from the run box type: DSA.msc and press Ok)
   • Click on the root of your domain (in this case it would be Contoso.com)
   • Click on the Find button in the toolbar of Active Directory Users & Computers:
   • Find your service account (CRMWeb) and open that objects properties
   • Click on the Delegation tab (this will only show up once the account has an SPN registered – otherwise it is hidden)
   • Make sure to “Trust this user for delegation to any service (Kerberos only)
     
     NOTE: if you wish to implement constrained delegation you would select Specified services only and select which SPN’s to use. I recommend setting up unconstrained delegation first, then constraining it after you’ve verified that it works

5) Troubleshooting:

• Client Browser IE Configuration: Double check Internet Explorer for the “Enable windows authentication” option:
  • 
• How you access CRM: Kerberos authentication cannot function without connectivity to a domain controller, thus if you are accessing over the internet Kerberos auth will likely fail unless you’re on a VPN or on your corporate network. If you’re accessing CRM over the internet, CRM has a configuration specifically designed to work over the internet called “Claims Based Authentication with Internet Facing Deployment (IFD)”.
• Check your SPN’s – Windows Server 2008 now includes SetSPN which can be used to list the SPN’s. No special permission should be required to read these SPN’s. For example: to check the CRMWeb account for SPN’s open a command prompt and type the following: setspn –L contoso\crmweb. This will list all the SPN’s found under the CRMWeb account.
• Event Logs: Check your servers and client System & Security event logs. The System log can contain errors from KDC or Kerberos which might contain useful information and sometimes you’ll find useful failed audit entries in the Security log (be careful with these as they can be benign as well). If this doesn’t yield any results also check the Global Catalog domain controller System Event logs for KDC errors.
• Check DNS: For each server (CRM, SQL, reports, etc) run nslookup and make sure you get the correct IP address, in addition make sure the reverse lookup is working as well (ie: nslookup {ip} should return you the correct host name).
• Active Directory Replication: As mentioned above, the SPN’s are stored in Active Directory. If Active Directory replication is delayed or broken the SPN’s will not replicate and you can get inconsistent results. I suggest first checking with your Active Directory admin and having them check replication status and the NTDS logs. If necessary you can use a tool like LDP.exe to bind to each global catalog and check for the SPN’s, but this can be time consuming.
• Netmon: Capture a netmon trace from your client
  • Close all of your Internet Explorer browsers
  • Start netmon 3.4 and press the New Capture button. Once the new capture tab appears, press the Start Capture button
  • From the run box on your computer type in your server URL: http://crm.contoso.com (for this example)
  • Once the prompt appears stop the netmon capture
  • Enter the following into the “Display Filter” box: KrbError (see in the graphic below)
  • Then press the “Apply” button in Display Filter box to filter your Frame Summary
  • Within the Frame Summary window click on the Kerberos Errors displayed. The frame summary should show you the DC name or IP that responded to the request as well as the error:

• • In the Frame Details expand the KRB_ERROR to view the details of the error, this shows exactly which service name failed along with the realm info and other details. In this scenario you would see a failure for the service HTTP/crm.contoso.com or HTTP/crm depending on how you entered the URL into the browser. This Kerberos error is telling you that it cannot find a suitable Service Principal Name matching the service.
• Enable Kerberos Event Loggingon the server and on your client, you will likely see an error of: “KDC_ERR_PREAUTH_REQUIRED” or “KDC_ERR_S_PRINCIPAL_UNKNOWN” when you attempt to login. Please keep in mind that enabling Kerberos event logging will show you benign errors as well as valid errors to investigate, so I recommend only doing this to validate other errors or once you’ve exhausted other techniques.
• User delegation blocked. If you have a user that is constantly prompted but other users are not prompted
  1. • login to a server with credentials allowing you to edit the given user account in Active Directory
     • Open Active Directory Users & Computers (from the run box type: DSA.msc and press Ok)
     • Click on the root of your domain (in this case it would be Contoso.com)
     • Click on the Find button in the toolbar of Active Directory Users & Computers:
     • Find the account and open the properties for that user
     • Click on the “Account” tab and make sure the following checkbox is unchecked
       
• Check for duplicate SPN’s:
  • Please remember, all SPN’s must be unique, if you register a duplicate SPN then neither will function properly (SPN’s are case insensitive). There are a couple of techniques you can use to check for duplicate SPN’s, I’m going to focus on the use of Windows Server 2008’s version of SetSPN. There are two approaches to checking for duplicates using SetSPN:
    1. Check for any duplicate SPN’s for the entire forest – this is a very handy option but in true enterprise environments this can take some time to run and can consume a lot of memory. To do this you run the following command at a command prompt:
       SetSpn.exe –X
    2. Search for a specific SPN across the domain for duplicates. For example, if I wanted to search my domain for any duplicates of the SPN HTTP/crm.contoso.com I would use the following syntax:
       SetSpn.exe –Q HTTP/crm.contoso.com
• Use DelegConfig to test Kerberos Authentication and setup: http://bit.ly/PX2b1X 

As always I’ll do my best to keep up with comments. If you want to keep in touch with our team you can follow us here (http://blogs.msdn.com/CRMInTheField) as well as on Twitter, if you have a Microsoft Premier support contract and wish to work with a member of our team ask your TAM about the PFE offerings we have for Dynamics CRM, and if you want to connect with us at conferences we can be found speaking and attending Dynamics Convergence. We’ll keep any other events or opportunities to connect up to date here and on Twitter.

Thanks!

Sean McNellis





Источник: http://blogs.msdn.com/b/crminthefiel...ironments.aspx