There's a way to restrict access to a user account or set of user accounts via PAM (and by extension, SSH)—the obviously named pam_access module. It's available on Gentoo Linux in sys-libs/pam, and on Debian Linux (and I assume the derivatives) in libpam-modules.
In order to enable this module for SSH, you have to edit the SSH's PAM file (Gentoo: /etc/pam.d/sshd; Debian: /etc/pam.d/ssh) to enable the access module: account required pam_access.so
There's some pretty good documentation in /etc/security/access.conf (at least, in the default distribution of it) on how to configure the file, but one thing that it doesn't say explicitly is that you can use IP address blocks in CIDR notation to denote access privileges. For instance, if I wanted to limit bob to the local network (192.168.0.*) and the VPN (172.16.*). The configuration line for that would be:
-:ALL EXCEPT bob:192.168.0.0/24 172.16.0.0/16
