Castiel wasn't baking down from this human, he also didn't understand the reason for this man's hesitation.

"Suit yourself," he said, "But I believe we should stick together. My powers may be weakened, but I can tell that there are some very strong creatures in these halls. I do not think staying here is an option either."