Let masses at the vertices of a square be M.
Distance from center to a vertex = d = a/√2
Potential energy of the mass m when it is at the center of the square = work done by an gravitational force in bringing mass m from infinity to the center =
4 * (- G M m / d) = -4√2 GMm/a
If the kinetic energy = +4√2 GMm/a is given to the mass m, then it will go to infinity with zero speed. It will have zero total energy at infinity.
Let the minimum velocity to escape the gravitational attraction of the system of the masses at the vertices = v
Hence KE = - PE as KE + PE = 0
1/2 m v² = 4√2 GMm/a
v = √(8√2GM/a )