So with that out of the way, scoring by file size really is the only viable method to determine a byte count. Metadata gets annoying in these cases, but often it's possible to remove or golf some of that metadata by hand while still keeping the file readable for the language's implementation. If that's possible, you're not required to submit the file as exported by the language, but you can shorten those parts as much as possible. The only important requirement is that the scored file can be correctly processed by the implementation.
Now of course counting bytes this way is often a bit annoying for visual languages, especially because it can be non-obvious what modifications to the visual program will actually reduce the file size. Often, you'd want to score these by components or similarly. However, it doesn't make sense to do so in a regular code-golf competition. "Code golf" means scoring by bytes and nothing else. However, if you can come up with a good and interesting scoring that is closer to the building blocks of your language, you can always write a language-specific atomic-code-golf challenge, where a problem needs to be solved with this language and the score is tailored specifically to the language at hand. Note that language-specific challenges in general are frowned upon, but if you have a good reason for limiting the challenge to a single language then go for it (and using a language-specific scoring for a language where counting bytes doesn't make sense seems like a good reason to me). I'd definitely recommend posting such a challenge idea in the sandbox first though to make sure the scoring is fair and the chosen problem is a good fit for the language.