No you're not. You just need to keep in mind that their ships outnumber yours no matter what, so you have to make sure that the ships you do have are much mroe powerful than theirs. Then you can hold them off. Once you invade and colonize thier planet, their numbers should decrease.
You have to build lots of mines and invest in the economy heavily early so you don't have this problem. Mines are best for income, then factories, trade alliences, trade routes, morale, keeping your ships in green space and having your fleet fully repaired and fueled..Carriers can help alot with that last one. Expansive colonization to get revenue from more planets can be good too, but don't colonize faster than your economy can handle.