It can depend on several things. For example if you're loading your graphics from a dedicated directory on your server or if you're loading iFrames and other sponsor-hosted advertising materials. I've also had better success choosing to serve banners from a dedicated directory versus saving them in the OpenX database.
Regarding how many zones on a page, I use a lot of zones on some sites/pages. One thing you definitely want to do is use the
Single Page Call method, which will cut down on the number of HTTP requests your page has to send to serve the banners.
Aside from those things, you can also consider ad positioning in your page design, so that some or most of the other important things will load quickly even if the overall page takes longer to load. As a simple example, making sure the sponsor-hosted graphics aren't at the top of the page can help ensure that someone else's slow server response won't mean a surfer clicks away from your page because he's not seeing anything interesting soon enough.
HTH