For example, could I invent a language that contained only two characters (for example 1, and 0) and use it to answer a question (assuming it fulfilled all the necessary site criteria), and then score it as 1 byte per 8 characters? Or would this still require 1 byte per character?
If there's an interpreter that supports this, yes
On PPCG a language is defined by its interpreter. If there's already an interpreter which can read the condensed format (8 characters stored in one byte) and properly execute the code, by all means use it. If there's not, feel free to create one and use that.
Please note that if the interpreter is newer than the challenge, it is not allowed to compete unless the challenge specifically allows it. But you can still answer challenges with that interpreter, as long as you mark the answer as "non-competing".