Maybe your were doing something like ths:
1. Make the selection with the rectangular marquee tool as you described
2. Switch to the Magic Wand Tool
3. Hold down the Alt (Option) key and click on the background color within the selection
The Magic Wand Tolerance value can affect the selection