Jeff VanderMeer is curious about what "experimental" means with regards to science fiction and fantasy. It's a great discussion and I urge people to check it out.
Jeff specifically mentions Samuel Delany's great novel Dhalgren, which to many people is the living archetype of experimental science fiction. The high praise one hears for Delany's novel – or condemnation, depending on if a person actually enjoys reading experimental fiction – is justified.
However, I've always found the term "experimental fiction" to also be a little puzzling. The term is usually used by people to describe fiction with an explicitly experimental narrative – think James Joyce's Ulysses and all the novels which have mimicked it's stream of consciousness style over the years. However, to apply the term experimental to science fiction should mean so much more than simply how a novel is written. Since science fiction pushes the bounds of human experience, experimental SF should not only push the bounds of narrative style but do the same with the novel's subject matter, characters, themes, vision, and so on.
To my mind, the SF novel which does all of this is Gene Wolfe's Book of the New Sun. The four parts of the novel, and its related coda, experiment with all elements of what we think a novel can be. In addition, it creates a vision of the future which hasn't aged a bit in the three decades since it was written. The Book of the New Sun is experimental science fiction at its best.