The Language is All Backwards
I'm told that Korean is one of the hardest languages in the world for English-speakers to learn. When I was first studying the language, I didn't really sense this though. Spanish had seemed pretty hard to me too and improving my Korean appeared to be just a matter of memorizing words and grammatical structures, and then practicing to assemble them together into sentences.
But having advanced beyond the beginner stages, I relate a little more with the assessment of how hard Korean is for English-speakers; this is reinforced by comments from Japanese saying how easy Korean is for them (and how difficult English is!) because the underlying grammars of Japanese and Korean are so similar.
The radically different grammatical structures and phrasing concepts between English and Korean make it much harder to get used to the Korean ways of saying things.
It is also much more difficult to intuitively guess new ways of putting words together; if I haven't studied a specific usage before, it's pretty risky to suppose it will work in Korean just because that's the way we say it in English.
It's Downright Impossible Sometimes to Speak Korean with Koreans Who Want to Speak English
Another obstacle for many of us in learning Korean is the fact that we first came to Korea as English teachers. Becoming an English teacher in Korea can be a deal with the devil: easy short-term access to a job in exchange for permanently handicapping oneself in the learning of Korean. Once we reach Korea, start teaching English and establish friendships with Koreans wanting to learn English, it takes a tremendous amount of initiative to break out of that and find a Korean-speaking social group.
In fact, even those not coming to Korea to teach English still find that those in their circle either a) already speak excellent English and don't want to chat in baby Korean with a foreigner or b) don't speak excellent English and want to practice English.
This is not a small issue; it is one I still struggle with even after all these years. But I should point out that in many cases, I choose the easy way of seeking out an English-speaking option (sometimes by default; sometimes deliberately) and then feeling victimized for not getting to improve my Korean. In these cases, I have nobody to blame but myself.