Obviously we have to make our own value judgements for ourselves and this includes whom we wish to associate with. In the grand collective "all of us" of economics this means a lot. For example, when someone (not Calum) holds "We could all change the world for the better overnight if we all worked together (as a species)" as a view as well as this other view: "I must make X and Y work with Z and nobody else for the good of society," X and Y's human brains have been taken out of the economic equation. Let's forget about these two peoples rights for a second and just think about the simple economics of it. X and Y can no longer make a judgement or calculation that decides who they associate with. These two people are slaves - they might be the only two people that can invent a lightbulb. Suddenly they cannot invent the lightbulb.
An easier way to picture it is if World War One came earlier and Thomas Eddison didn't get to sit around making 1,000 filaments that didn't work, contributing nothing in the eyes of society, instead he was forced to 'save the world' and it wasn't until a decade later that this mans invention saved everyone from the industry of making candles and its many stages (the first is killing animals for the wax, getting the materials from the wick, making the candle mould, pouring the candle. Millions of people would be tied up in these industries when there is a more economical way. The lightbulb by putting these jobs out of use also added massive labor pool resources as a side effect.
Now, freedom isn't always pretty, but for as long as we can act against laws and orders given with no consideration of the conseqences, for as long as men can make mistakes, or for as long as men commit crimes of passion and such, basically forever, the world is never going to be perfect in this sense. There is always going to be someone doing something stupid that doesn't harm people that seems stupid anyway. Justin Bieber is an idiot and Hitler would shoot him but Justin Bieber's existence to me is a sign of artistic freedom. It is also the sign of a bankrupt culture.
Ayn Rand doesn't just want everyone to live like hermits, she wants everyone to act in their self-interest more and think about their own long term existence. In the simplest words, Objectivists are working hard to create a second enlightenment. When you look at the upsurge in readership, attendance at Objectivist lectures, and sales of lecture videos at the Ayn Rand institute a lot of people are becoming enlightened by this amazing philosophy.