Is a scoring system based on the binary representation of the code ok?
Sure. Any objective scoring system you can come up with is valid. Just people may dislike it if it's inelegant or unfair.
How to define widely?
Without a definitive list of allowed encodings, you can't. It would be better to say "only preexisting encodings are allowed". This is our default here anyway so you don't really even need to say it. If someone designed an encoding to specifically win your challenge they would be majorly breaking our rules.
Has someone already compiled a list of allowed encodings I could use, or should I just provide some as the only ones allowed?
Maybe take a look at http://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings , but I advise against a list.
Could there be any problem with languages that use unprintable characters?
I don't think so. Unprintable chars have a normal binary representation the same as printable ones, so a scoring system based on the binary representation applies regardless.
Should the string-version of the hex-Dump be graded?
If you mean convert the hex values to binary and score that, then that's the same as the binary representation.
If you mean convert the hex string characters (0-9A-F) to binary in the chosen encoding and score that, then no don't do that, it would be weird.
By the way, you don't need to worry about "giving away the challenge". People who cheat by creating challenge-specific languages/encodings are quickly downvoted. In fact making your challenge public in the Sandbox is more likely to make your challenge better than worse.
