After a summer packed full of trips and fun, I dove right into the problem of multiplayer latency. I can now say that I truly understand why seasoned developers often try to steer new developers away from online multiplayer. At this point, online multiplayer has probably taken more than 30% of my development budget. It is not a feature to be taken lightly. My advice for any developer planning to tackle online multiplayer for the first time would be to keep your game simple - at least as simple as a first person shooter.
At this point I have solved the latency problem for movement and targeting, and it works extremely well. I have also implemented an acceptable solution for collision detection and syncing balls after collisions. I have decided not to take the time to provide a technical explanation of how I accomplished this. Instead let me simply say that wrestling with latency involves making a series of tradeoffs.
I absolutely hate playing a game where its impossible to hit an opponent because their position keeps teleporting around every time there is an update. This will never happen in my game. The players all move about smoothly from one position to another - regardless how bad the latency may be. I also hate playing a game where my character dies from an attack that clearly missed me. Again this will never happen in my game. However, the tradeoff is that in heavy latency a ball that seemed to hit an opponent may miss. To mitigate this possibility, I slow down balls on the attacking player's machine so we have time to display the opponents dodge anim. This solution works well, but not in every instance. The tradeoff of allowing thrown balls to fall out of sync is that I eventually have to bring them back into sync. This means that while characters and thrown balls all move smoothly, the ball may pop into sync after a collision if the player is experiencing heavy latency. This seemed an acceptable compromise, since this would happen only after a player has dodged the incoming ball.
At the moment I'm wrapping up a few remaining multiplayer bugs, and am getting computer AI to follow the same latency management proceedures as do human controlled players. Ever onward!