Celebrated on the first Monday of September (in the U.S., Canada, and Bermuda), the Labor Day holiday is used to honor the achievements of workers. In observation, many workers do not work on this day.
It's a well-known fashion "rule" to refrain from wearing white after labor day. There isn't a fixed reason for this, at least no recognized one. But there are speculations. The most obvious is the weather conditions associated with fall as opposed to summer are more likely to dirty clean, white clothing. White is considered a summer color, therefore "shouldn't" be worn in the fall just as fall colors "shouldn't" be worn in the spring.
My take? Fashion risks exist to be taken. And, used in moderation, any color can be used at any time.
Happy Labor Day! Hopefully you had fun barbecuing, shopping, and perhaps even working, ironically enough!