Knowledge Issue: Color Blindness

1. Over the summer, while at the beach with my friends, one of my friends and I brought two very similar towels. While both of our towels had no designs and only one color on each, his was purple and mine was blue. Due to my color blindness, I had a difficult time distinguishing the towels from one another. I can tell the difference between blue and purple, but when the shades look very similar from my point of view I cannot distinguish between the two. When we were leaving at the end of the day, I grabbed the wrong towel on accident, sparking an argument between my friend and I as to which towel was mine and which one was his. After asking the other people that were there, they agreed that I have taken the purple towel and not my blue one. Instead of continuing the argument, I decided to just agree with them and accept their judgment.
2. The aforementioned situation involves sense perception and secondhand knowledge (to an extent).
3. Differences in perception play the primary role in this situation. My friends, to my knowledge, do not have any problems with their vision in terms of perceiving color. On the other hand, I have difficulty distinguishing between close shades of blue and purple, especially dark shades of both when placed together. The only reason for me to believe that the towel is blue is based on secondhand knowledge, so I do not know for myself whether this is true unless someone else tells me. However, I am still fairly capable of deciphering between most shades of blue and purple, so I might have been right. I cannot see through other people's eyes, so I cannot verify whether what they see is different from what I see. The only way I can verify which is blue and which is purple is through a majority consensus.
4. Knowing I cannot tell the difference between these two colors/shades, I cannot trust my own eyes when determining which color is what. However, if I cannot trust my own eyes, how can I trust anyone else's eyesight? Considering that a majority of men are afflicted with some type of color blindness, my friend could as well be color blind but unaware of it. Furthermore, I cannot see through my friend's eyes, so I cannot tell whether his perception is truly different than mine.
5. If our own senses are inaccurate, how can we trust anyone else's sense perception to be more accurate, especially if we cannot perceive anything through their point of view?