A matter of labels

Sometimes I wonder if calling myself a fantasy writer or even speculative fiction writer is the best approach when talking with some people. Let me be clear, I'm not embarrassed to identify myself as a fantasy writer, and I think when talking with other speculative fiction readers and writers, that's the most accurate label.

When I meet new people, though, or when people I know want to know more about what I write...I'm not sure it gets the right idea across. "Speculative fiction," while in many cases the term I prefer, doesn't work for those situations, because they just don't know what that means. I can tell by their follow-up questions or later conversation that they imagine something very different. But swords, spaceships, elves, dragons, fire-summoning wizards...these rarely if ever show up in my fiction (swords more than the rest, but even that not as a prominent feature), and those are the things they seem to assume.

But the dilemma is...what would get a more accurate image across to someone who isn't steeped in the broad fields of speculative fiction? I've considered magical realism and surrealism. I have work that I think would fit each of those, and they'd probably conjure up a more accurate picture. But...the pedant in me wants to argue that most of what I write doesn't really fit either of those terms, or at least not the image I have when I hear them used. I recently saw a reference to "imaginative literature," as a tradition that includes writers like Borges and Calvino. I like that. Certainly both writers have influenced my writing significantly. So for someone familiar with that usage, it might get across a better idea. But for the most part it runs into the same problem as speculative fiction--most people are just going to say, "Umm, what's that?"

Maybe "literary fantasy"? Within the field, there's some definite resistance to the term--it smacks of arrogant pretension or something, I suppose. I used to use that in my head to describe the stories I liked, but I've shied away from saying or writing it. But I think to someone without those in-genre associations (and honestly, we're a relatively small and insular group who tend to over-estimate our numbers and influence) the image is going to be closer.

I don't know, though. Any other suggestions? To be clear, I'm not talking about rigid definitions of where the lines or genre are or anything of the sort. Just the best way to explain myself to others.