I want my system filter to freeze all mails greater than 500K in size, but to exclude those to a specific domain. However, I don't seem to be able to use $domain in a system filter.

You cannot do this in a system filter, because a single message may have multiple recipients, some in the special domain, and some not. That is also the reason why $domain is not set in a system filter. If you want to take actions on a per-recipient basis, you have to do it in a router. However, freezing is not appropriate, because freezing stops all deliveries. You could, however, delay delivery to all but the special domains by using something like this:

delay_if_too_big:
  driver = redirect
  domains = !the.special.domain
  condition = ${if >{$message_size}{500K}{yes}{no}}
  allow_defer
  data = :defer: message too big.

However, there isn't an easy way of releasing > such messages at present.