It isn't so much that you have to be realistic, but you have to be consistent. Having written a boat load of games in all genres, I can tell you from experience that a player will expect a consistent set of rules. The rules don't need to be based on reality, but they need to be self-consistent. If you introduce inconsistent behaviors, then you need to make the inconsistencies make sense to the player.
You can do the slow-to-start and turn-on-a-dime-at-speed, as long as you are consistent in this behavior, and you have come up with a good reason for this type of behavior. The goal is to get the player to buy into the game and you have to give them reasons why they should buy into the game. Without giving the player a good reason for the ship's behavior, they will think it is either bad programming, or inconsistent rules and both will cause the player to hit the escape key.
In this situation you could have an inertia meter displayed on the screen. When the inertia meter hits zero, you can spin on a dime. By showing the player that inertia only effects the ship at low speed you have explained the reason for the behavior and also enhanced the game play by giving the player a goal to achieve: keep the ship going at high speed.
You might be interested in a book I am writing. I have a rough-draft of the first chapter
here where I talk about about these types of issues.
No one is trying to talk you out of your game design, we are just trying to help.