Its simply that the Clojure filetype is built on the Lisp filetype, and uses the Lisp highlighting lexer. But `#"` is undefined in Common Lisp so it skips the opening " and starts a string at the closing ".
Highlighting is done by the Scintilla editing widget we use, which is a separate [project](www.scintilla.org) so you should report it there and maybe it can be added since `#"` is undefined in Lisp.