In the Arctic and Antarctic, they do. The ice cap at the North Pole is entirely over ocean; on the other hand, the ice is only a few feet deep. Oceans don't freeze solid for several reasons: for one thing, they contain a lot of water, which circulates around the world. Water from warmer oceans (and from areas near underground volcanoes) flows into the Arctic, taking some of the chill off. Another important fact is that oceans contain salt water, which has a higher freezing point than fresh water. Furthermore, most of the salt leaves the water as some of it freezes, so the water that remains becomes even saltier. This remaining, briny water has an even higher freezing point, and is also more dense, so it tends to sink to the bottom of the ocean, letting warmer water come to the top, below the ice. This warmer water is actually insulated by the ice, which reflects the sun's rays, and it helps warm the surface and prevent the ice from thickening further.