Is there a way to make airplay-ng-3 so it doesn't send a broadcast?

I think that you are misusing airplay-ng-3 suite : this framework is intended for a "lower level" manipulation. You don't need to be on the same network as target, or even know its IP address. When using aireplay, you should already have a clear understanding of "lower level" network topology : you should already know (e.g. through airodump) which client is associated with which AP.

  Note: When I mean "client" and "AP", I'm talking about their BSSID's (which is often the MAC address).

Once there, you can tell aireplay to deauth client (identified by BSSID) by impersonating the AP (still identified by its BSSID), e.g. :

  aireplay-ng -0 100 -a 00:14:6C:7E:40:80 -c 00:0F:B5:34:30:30 ath0

Note: Previous command is from https://www.aircrack-ng.org/doku.php?id=deauthentication; I just change -0 1 by -0 100 because in real cases, you need more than 1 deauth packet to effectively deauth target. If you do so, aireplay will inject fake wifi packets in order to break the link between client and AP. If you intend to break the passphrase, you should capture the (re)assotiating challenges (e.g. with airodump) and then break the key (e.g. with aircrack).