The loudest voices may be remote-work advocates, but plenty of other people can’t get back to the office fast enough.