These notes are rough, but might prove useful in the future (or to someone else who had the same problem). I wanted to join my newly created OpenSolaris 2008.11 installation to my experimental Windows Server 2008 Active Directory. I followed the tutorial in the Solaris CIFS Administration Guide, but when I attempted the actual join command, it failed with LOGIN_FAILURE.
Here's the answer:
Firstly, make sure you have the IP address setup correctly, name added in /etc/hosts, DNS setup correctly in /etc/resolv.conf and /etc/nsswitch.conf. You make sure you are on the same domain as the Active Directory ("windows.zone" in my case).
Secondly, make sure your system clock is synchronized with the domain controller:
# ntpdate dc01.windows.zone
Then setup Kerberos by editing /etc/krb5/krb5.conf. This is documented in the manual.
Install the SMB Server using the package manager and start it up using:
# svcadm enable -r smb/server
Then try to join the domain:
# smbadm join -u administrator windows.zone
For me, this is where it failed. The mailing lists suggest that the problem might be related to smb signing. On the DC, I opened up the Group Policy Management tool and changed the following:
Computer Configuration\Policies\Administrative Templates\System\Net
Logon\Allow Cryptography Algorithms Compatible with Windows NT 4.0 -> Enabled
I then ran a gpupdate /force.
Finally, I read that the sharectl command on the OpenSolaris server should be run to use NTLMv2 authentication:
# sharectl set -p lmauth_level=2 smb
I re-ran the join command, and it worked properly. OpenSolaris CIFS server now part of the Active Directory domain.