I would like add some custom headers to selected outgoing mail based on a specific domain and the subject line.

To the remote_smtp transport, add something like

headers_add = ${if and{\
              {eq{$domain}{spec.dom}}\
              {matches{$h_subject:}{whatever}} }\
              {Content-Type: text/html; charset="us-ascii"} fail }

This example shows a Content-Type: header, but you can have anything you like, and multiple headers can be inserted by using \n to separate them.