One system would be to have some multi-purpose/backbone/generalist ships that can fill in anywhere but also have no outstanding strengths or weaknesses with a wider array of specialist ships. Anti-fighter, anti-capital, maybe something particularly effective against workers, etc. My current project uses a variant of this type with corvettes, frigates, destroyers, cruisers, and carriers each having different roles in the fleet. Frigates and cruisers are generalists with some extra anti-station stuff on the cruisers. Corvettes are kind of like a cross between scouts and stock destroyers. Destroyers intercept missiles and have anti-fighter weapons. Carriers launch fighters to attack from a safe location. Pros are that it's easy for players to learn (counter this type of ship with this other type of ship, regardless of race) and it's easy to balance. Cons are that individual ships of a particular type may not have a whole lot of character compared to other systems and the coding can be tedious.
Or you could have damage and armor types with a rock-paper-scissors dynamic. Warcraft 1-3, Starcraft 1 (and 2, I expect), the Command & Conquer series, and even Mass Effect 2 and 3 all used this kind of approach. Pros are that it's pretty easy to visualize what the game will look like and it doesn't usually take players too long to figure the system out. Cons are that it takes a lot of damage exceptions (the default damage on an ordnance may be just a reference for you to decide how much more or less it should do to a particular ship) and, though this one is subjective, it's really common.
I've also considered an advantage/disadvantage system for a mod. Each ship would have a number of advantages (ex: long range, fast speed, bonus damage to stations, etc.) and an equal number of disadvantages (ex: reduced accuracy, unshielded, short sensor range, etc.). In this case, costs would be scaled to health and damage and having every advantage countered by a disadvantage would hopefully provide the rest of the balance. It also allows individual ship classes to have more character than a rigid type system and would be relatively easy to do the ODFs for, particularly with having few, if any, exceptions for damage, hit chance, etc. Each advantage and disadvantage would be listed in the ship's tooltip. Pros are that this system allows for completely unique ship classes and has more likely variations than the second option while being easier to code. Cons are that the potentially massive number of variations may take players longer to learn. I'm not really familiar with any games that use a system like this so I'm not sure.
Or you could go for a mix. A battleship could describe a unit with a specific role defined by it's use of specific damage and armor types along with a specific set of advantages and disadvantages. Individual battleship classes would be differentiated by an additional advantage and an additional disadvantage to make them distinct.
Whether or not caps should be used is also something to consider. The Super Star Destroyer could be a massive anti-everything ship that takes a fleet to kill, but then you should probably either make it prohibitively expensive (which can keep them from ever being built) or put a cap on them. You could also use a general population cap like the stock officer system. This can keep people from simply spamming ships and force them to build their fleet more carefully, but it's harder to limit the entire fleet in a balanced fashion than to limit specific units.