1915 Letting Go

I was having a conversation with someone on the phone the other night and it came time to end it. No particular tension on either side of the conversation, just one of those feelings I got that it was time to stop blathering. So I said, "Well, I'll let you go."
The person on the other end accepted that graciously, hung up, and we were on our merry respective ways.
But naturally, I thought about that phrase I used. How neither of us had expressed any desire to be freed in any way. Certainly not the other person, who would have continued the conversation indefinitely. It was me who wanted to go. Or actually be let go.
Funny, when you "let someone go" the implication is they are struggling on some level to be free. And it's their choice. And you are just nicely acceding to their unspoken request. But the truth is, when we use the phrase, we are usually the ones anxious to take a powder.
Another phrase I don't get at all. Take a powder. As in go to the powder room? That place your mom and aunt and grandmother always went to "powder their noses." Which never looked any more or less shiny when they returned?
Or take a powder as in some sleeping potion? Ready to knock yourself out and retire for the evening.
And when you tell someone to take a powder it's worse. Like "let you go" is the passive/aggressive euphemism for firing someone. "Sorry George, we're going to have to let you go."
"But gee Boss, I don't want to go anywhere. So you don't have to let me. I'll just stay if it's all the same to you."
"Sorry. When you first came here you were fired up. Now you're just fired. Take a powder."
America, ya gotta love it.