I can give a little perspective from how Ultima Online did this sort of thing, as I helped code a server emulator for it (
UOX. Basically, the map is divided into "blocks" of 8 tiles by 8 tiles. Then, depending on an option in the configuration file, the server would either load the ENTIRE map into memory, or load (I think) the offset in bytes into the map file of each block (*much* less data than loading the whole map, but still pretty fast). Then, when the data was needed for collision detection or what have you, it would seek to that point in the file and load the appropriate block(s), and cache it in memory so that the next time that block was requested it would be faster. Obviously if you chose the pre-load option it is all cached at startup. I think there must have been an option of how many blocks to cache at once...say you set it to 100. After the 100th block is cached, it will start overwriting either the oldest one or the least-frequently accessed one, I forget.
But anyway, I think that''s a decent option. You only have useful data in memory (not that one part of the map no one ever goes to, heh), and it is nearly as fast to seek to a spot in a file, load maybe 200 bytes, as to just read it from memory (and after the cache, it will be in memory next time).
Another benefit of this is you could divide the "blocks" across several servers, either statically or dynamically.
All in all, it was a pretty good system I think!
Anthracks