Excluding any member of a group from conversation, whether by language or the selection of the topic, is rude. If it is necessary to explain something in Japanese very briefly, you can say, “Please excuse us for a moment, but I would like to use Japanese to explain something to her.” Then, once you have communicated in Japanese, revert to English and say, “Sorry. I couldn’t explain in English, so it was easier to use Japanese,” and perhaps mention what you were discussing. If the Japanese person does not understand English at all, then you have no choice in the matter.
Japanese tend not to expect that a person with blond hair and blue eyes can speak Japanese, but do not assume that. Whether you are in Tokyo, Atlanta or New York, you will encounter people who have lived in Japan or at least studied the basics of Japanese. It is not polite to talk about strangers in front of them, and besides, they may understand every word you are saying! In one memorable experience in an elevator with two Japanese men, an American friend overheard their conversation about how one of the men spent his evening with a woman of the streets. The men assumed that they were speaking in the ‘privacy’ of Japanese language. The American friend knew Japanese rather well and was greatly entertained as a result.