If you like to carry around a water bottle, Royal Caribbean asks not to fill it directly from the dispensers for hygiene reasons. If that’s the case, you can ask your stateroom attendant for some ice. Drinking tap water in your stateroom is also perfectly safe, although it might not always be cold.
The quality of the tap water on board is generally high as it undergoes thorough filtration.