See the diagram for the forces between the charges. I have shown forces on a charge at B in the square ABCD. The system consists of all charges. So we have to balance forces on all charges -Q and +q.
Forces on -q:
For any positive or negative value of q
or Q, the charge q will be in equilibrium. This is because if symmetry and cancellation of the forces of the four -Q on q.
Let the size of square be L. The attractive force F_BO exerted by one -Q on -q at the center is:
F_BO = 1/4πε * (-Q)*q / (L/√2)² = - 1/4πε
* 2 Qq / L²
This attraction force is along the diagonal and towards -Q. Now, The other -Q charge at the opposite vertex of the square also
exerts the same attractive force F in the opposite direction. So they are cancelled. Similarly for
the other two charges -Q at other corners of the square.
Hence the charge +q at the center O is in equilibrium.
Forces on -Q at any corner
Now the for the charge - Q at the vertices. Each of the charges is repelled by the other three charges at other vertices
On the charge at B, the forces exerted on -Q are , F_AB, F_CB, F_DB and F_OB respectively. F_AB = repelling force of charge -Q at A on charge at B. Repelling force F_CB is due to the charge at C on charge at B. F_DB is the repelling force on charge at B by charge at D. Now the attractive force F_OB is due to the charge +q on charge at B.
Magnitude of F_AB = F_CB = (1/4π∈ ) * (-Q)²/L²
Net Resulting force of these two forces is along the direction of diagonal DB and it is equal to F_AB * √2 because the angle between F_AB and diagonal is 45°, AB and CB are at 90°. Vector addition of F_AB and F_CB gives us that.
Force F_DB = (1/4π∈) * (-Q)² / (L√2)² is along DB direction.
Adding vector wise, we get net resultant force of F_AB, F_CB and F_DB as
F_net = Q²/(4π∈) [ √2 + 1/2) /L² = [Q²/4π∈L² ] * (1+2√2)/2
The attractive force of -q is along BO and is = F_OB = - (1/4π∈) * qQ/ (L/√2)²
= (1/4π∈)* 2 qQ /L²
equating these two forces F_net and F_OB, we get
q = Q(1+2√2)/4