See the Arukh haShulchan OC 637:5 (from yesterday's Arukh haShulchan Yomi
http://aishdas.org/ahs-yomi ) about using someone else's sukkah without
asking.
The Taz says we assume that ("ustama kein hu") people are okay with your
using their sukkah to fulfil a mitzvah when it is unoccupied.
But not if he does care ("im maqpid") or he is using his sukkah already,
as "mistama" he would care about others joining him when he's eating.
(Maybe we are worried they're too polite to say they're bothered?)
Either way, notice when it comes to sukkah, we the rule only applies if
we don't have reason to believe the owner would object.
Tir'u baTov!
-Micha