You can have as many connections and datasources as you want--provided you don't run overcomplicated filters and nested actions.
Typically the reason for performance issues happens in inefficient or non-delegable filters. If this is related to your other post on LookUps in galleries, then that would be the likely culprit.
Having fewer controls are better. But while you could consolidate many things into galleries, you need to be sure you aren't overcomplicating things inside the gallery. Remember that every record in the gallery needs to be repeated, so if you're doing something complicated, it is multiplied by the number of things in the gallery.