If you test software that allows users to navigate using hyperlinks you need to think about new windows. It sounds simple enough, links send the user off to a different place, but context should determine whether the link opens in a new window, or tab, or re-uses the existing one.
During an E-commerce checkout you expect the “Buy now” button, or link, to open in the window or tab you’re using. The same thing with “Sign Up” flows, they are part of the main journey and so should be re-using the main window. It would be pretty strange to end up with multiple tabs each displaying part of a “Sign Up” journey.
What about “FAQ” or “Terms of Service” pages? Well it depends. If you click one of these links in the main site footer then it makes sense for the window to be re-used. If however I click a “Terms of Service” link from a checkout or payment page then I really hope it opens in a new window or tab to avoid interrupting my purchase.
Resuming journeys
Have you ever tried to purchase something from a website only to be sent off into a “Sign Up” flow? If you were lucky you completed the “Sign Up” and were gracefully returned to the page you were originally on. Sadly this isn’t always the case. As a user it is hugely frustrating when websites make you repeat actions just because they wanted you to do something else first. Make this easy. Return users to the page they were originally on.
Indicating behaviour
Links to external websites or services should open new windows as the default behaviour. In addition, external links, and mailtos, should be labeled with the standard icon to indicate that they will take you away from the current site. Have a look around a service such as Spotify to see these icons in action.
Interaction between tabs
More complex tab thinking leads to considering the interaction between tabs. I often search for items on Amazon and open several tabs to compare different items. After I’ve compared the items I might add one or two to my basket. I expect my session to exist across all the tabs. That means that I am logged in on all the tabs, and my basket is gathering items from all the tabs. Checking out on any one of the tabs should show me a basket containing all of my selected items.
Application
Following these guidelines will help make your website intuitive to use. However the question of context must still be answered. What happens on your website when you open a new tab and log in? Are both tabs now showing the correct state? Is that correct for your service? How about if you log out? Or purchase something? Think about the standard behaviour but always do what is right in your context.
Testing the Mind 
I’ve never been entirely convinced about the right approach, My general rule is to open links in the same window, unless it’s downloading a PDF when I use a new tab (and tell people I’m doing it). I’m not sure that’s right and it doesn’t quite match the way I instincively use websites, which is pretty much as you describe. I’ve been persuaded that most people expect websites to behave the way I do links, rather than the way I prefer. For years I’ve wanted to do some sort of study to see what people actually do, as opposed to what they say they prefer. Are you aware of one?
In the meantime I think it’s important to be consistent. I might not be right, but at least I’m wrong in a predictable way.
Thanks for commenting, James.
I think link behaviour will become increasingly important as more actions move online. Opening links from an article is a fairly low risk action. Maybe some people won’t have the experience they expect but little will be lost. Once you start thinking about submitting a Tax Return or paying a bill then the importance increases. The problem with opening links in the same window is loss of user entered data.
A study into user behavious would be very interesting. I’m not aware of one. Maybe someone else is?
Since I never really found a satisfactory answer back when I tested browser applications: how do you deal with user settings in the browser? Are the solutions you present (and I agree 100% with those) only valid if they agree with user settings? Someone might for example have settings that open every link in a new window. How would you deal with that? Can or rather should those settings be overridden? Maybe you have some ideas about that 🙂
Great question! My opinion would be to respect user settings. I think everybody has the right to set their mobile, or web browser, to behave as they want it to. Sometimes that might lead to less than ideal user behavior but I generally find people accept that their choices resulted in the degraded experience. For example I have known browser extensions to interfere with website Javascript. Usually once the user know the reason behind the weirdness they’re fine, and either accept the situation or turn off that extension.