Your formula is wrong.
Gravity above Earth's surface is g' = G Me / (r+h)²
Force of gravity = m g'
See the diagram enclosed.
When the mass m is below earth's surface at a depth of h , then there are two forces acting on it. One is the gravitational force of Earth inside radius (R-h) and another, the gravity due to Shell of thickness h.
The net force of attraction of the outer shell of thickness h is zero. This is Because of symmetry and direction of force being spread in 360 degrees from all sides.
Density of Earth, = Me / (4πR³/3)
Mass of Earth with in radius (R-h) = Me * [ 4π(R-h)³/3 ] / (4πR³/3)
Me' = Me (R-h)³ / R³
Hence the gravity (acceleration) g' below Earth's surface =
g' = G Me' / (R-h)² = G Me (R-h) / R³
= ( GMe / R² ) [ 1 - h / R ]
g' = g [ 1 - h/R ]
m g' = m g ( 1 - h/R )