I was getting this error on all my callouts, initially they all were running fine. Recently we had tried upgrading CRM 3.0 to CRM 4.0 and had failed in that, so we had CRM 3.0 reinstalled on that machine.
This time the port was different i.e. 100 instead of 5555. And we were using our own config files for callout where we were setting crm’s Url. I modified the url there, followed by an IISRESET but still same error.
Finally i updated my web references( in visual studio) for my callout to point to the new url with the new port, rebuild it and deployed this callout. This resolved the issue.