Federation Services Single Sign-on with SharePoint not Working if I log into CRM First

I have both Dynamics CRM 2011 and SharePoint 2010 using ADFS 2.0 for claims authentication integrated together using the CRM List Component. If I visit SharePoint first, then visit CRM, I am not prompted to log in again. However, if I log into CRM first then I visit SharePoint or have an iframe into SharePoint on a CRM form, I am redirected to the ADFS log in form by SharePoint. Are there additional requirements for ADFS to provide a single sign-on instead of prompting for credentials twice or ways that I could diagnose the issue?