Spain vs Latin America: how different is their Spanish?

Spanish is spoken across three continents, and learners worry constantly about picking the wrong version. But compared to English or Portuguese, the variation is surprisingly small. The vowels stay consistent whether you are in Mexico, Colombia or Spain, and most of what really changes is slang, rhythm and a couple of consonants. I have found that after a day or two in a new country you stop hearing the accent at all and just hear the meaning. Which Spanish accent do you find hardest to follow?