SSL CA causes server to stall for a minute or two and then goes back to normal...

I installed an SSL cert on a website on my server. The only issues is that when I have the CA installed and access the site through https it will stall and not receive any pings or anything for about 2 mins. After that it will start working normal again until I try to go to another page on the https site. Once I removed the CA it worked just fine. Does anyone know what is causing this?
Thanks