X= initial number of creatures.
Cost -> Damage
In rainbow
4
-> 2X|3X
8
-> 5X|7X
12
-> 9X|12X
In mono
4
-> X+1|X+2
8
-> 2X+3|2X+5
12
-> 3X+6|3X+9
Mono gets the option of Mitosis so it can pay the casting cost for either Coalescence or Common Ancestor.
Mono usage is weaker than Rainbow usage.
The +1|+1 upgrade helps Rainbow much more than Mono.
The Mono usage is fairly close to balanced if not already.
Conclusion: It would be better if the upgrade was a cost decrease (Which helps mono and rainbow) and if Coalescence cost more than Common Ancestor (So the mono usage could use the cheaper cost for the weaker effect). I might even suggest base stats of 0|0 since they get a minimum of +1|+1 when played via Coalescence.
With the stat based changes but without the cost based changes
X= initial number of creatures.
Cost -> Damage
In rainbow
4
-> X
8
-> 3X
12
-> 6X
In mono
4
-> X
8
-> 2X+1
12
-> 3X+3