Is it a good idea to leave a full time PM job and be your own Boss?

I am Co- Founder of one company and Project Manager of another. Being Co-Founder was that leap for me. I decided to be my own boss. But it meant doing everything. I think working as full time Project Manager is easier. There are certain skills I am not good at. What we do in that case? Hire someone for that task. But still that is not easy if you are starting at minimum or zero level.
If you are confident that having no work for months is not going to harm you financially than I guess nothing should stop you from being your own boss.

In our realm, there are quite a few that have left the public sector to work for themselves as consultants. The key points have been mentioned, and contacts/ network is key. Pay is generally very good (CAN$150K+) for experienced persons, but work can be sporadic, and competition is ever increasing as more make the leap. A former colleague I spoke to a few months back made the leap, but is struggling, but he chose a niche market - startups. I've also recently attended a few local PEng seminars, and those that have started small businesses seem to be struggling. To be an entrepreneur takes a special breed of person to be successful.