Would it be frowned upon the upper class in a Victorian society to be a liberal unionist?
In Victorian England, Whigs were the conservatives and anyone else was a Tory. There were only the two parties. Unionist didn’t mean then, what it does now, because the unions were not a political party yet. A unionist was someone who belonged to something, so a liberal unionist would be a Tory. When she says they dine with us, or ‘come in the evening, at any rate’ that means they are accepted into Victorian homes, but not quite fully. That’s what the ‘come in the evening’ routine is about. They are invited to drop by after dinner, or perhaps to dine if there are no other guests.
