Barnett offers some reasons, but also makes an assertion that I’m not 100% on board with:
Adams ultimately succeeded in mining a very rare yet rich seam of comedy that meant he was loved by both the science fiction community and the mainstream book audience who might not consider themselves science fiction readers. It’s hard to fathom what his secret was – if we could, then more people would be doing the same.
I’ll go at this backwards. First off, it’s not difficult at all to fathom what Douglas Adams’ secret was. What Douglas Adams was doing was farce — a very specific, British version of it. Hitchhiker was right down the street from Monty Python, and both desperately beloved by nerds. My personal belief as to why this is has to do with the Python/Hitchhiker being very obviously comedy (i.e., even the most socially clueless nerds could see that it was broadly funny) and for being literate in a highly quotable way (i.e., easy to share and get a laugh from your friends with). It’s why so many of us dig Eddie Izzard today — he’s all that farce-y stuff and a bag of chips.
The reason more people aren’t doing the same is not because they don’t know what it is but because is because it is so amazingly hard to do. Any sort of comedy or humor is difficult to write, mind you; it just looks easy (or at the very least is supposed to look easy). But to do a very specific type of humor — in this case British farce — is even harder to do, especially if one is not already a practitioner of the form. Douglas Adams was: the man got credited for a Monty Python TV bit, one of only two non-Pythons to claim that honor (the other being Neil Innes), and outside of Python he wrote with Graham Chapman.
Unfortunately for most science fiction/fantasy authors who would later attempt Adam’s brand of comedy, they weren’t practiced British farcisists, and their attempts at the form would fall flat. In my opinion, only Gaiman and Pratchett’s Good Omens comes even close to the form, and the two of them are British, so that certainly helps (but, obviously, is not sufficient in itself; it helps both gentlemen are otherwise very funny).
Add to this the problem that for years, it at least seemed that if you wanted to do humor in science fiction/fantasy, the type of humor you had to do was Hitchhikers-like, because it had sold millions and was the most obvious example of what humor in the genre was. If you have as the most prominent example of humor in the genre a variant of humor that is particularly hard to do, it’s not entirely surprising to discover that even many years on, there are very few valid pretenders to the throne. This one reason why I’m fond of saying that the success of Hitchhiker’s was an extinction-level event for any other sorts of humor in written science fiction, just like Star Wars was an extinction-level event for the type of science fiction that was being made in Hollywood before it: It was so clearly, obviously, blindingly popular that it just obliterated everything else in the field.
To be clear, this is not saying this is the fault of Douglas Adams, who did nothing wrong; quite the opposite, of course — he was delightfully and blazingly funny in a way few have been before or since. He earned his spot on that throne. He just happened to get there via a path that’s hard for anyone else to follow.
I am certainly not attempting British-style farce. I love the stuff, but writing-wise it’s not my natural humorous tradition. Nor am I particularly concerned about taking science fiction’s humor crown from Adams; that’s like trying to take the melodic pop crown from Lennon/McCartney. You can try, but they’re just going to call you “Beatlesque.”
What I would like to do instead is continue to expand the sorts of humor that work in science fiction. Because as much as I love Adams’ British farce — and I do — we’re long past the time where other sorts of humor should be commercially successful in science fiction, and for humor to come in through the genre’s front door, rather than being snuck through the side as it often is now. Instead of guessing who will be the next Douglas Adams, let’s get to work on the first whomevers, doing humor in science fiction and fantasy in their own way.
And in fact that’s the thing I am proudest of regarding Redshirts: It’s funny and it’s even absurd, and it’s unapologetically so — we didn’t have to pretend it was an action-y novel that just happened to be funny. We sold it as humor, and people bought it as humor, and generally speaking, they seem to like it as humor. It’s not the next Hitchhiker. It’s just Redshirts, and that’s enough. In a post-Hitchhiker world that was not insignificant. And I like to think King Douglas, long may he reign, would be just fine with that.