The solution is simple you find the nearest point on the square to the center of the circle
nearX = min(max(squareX1, circleX), squareX2); // where squareX1 < squareX2
nearY = min(max(squareY1, circleY), squareY2); // where squareY1 < squareY2
nearZ = min(max(squareZ1, circleZ), squareZ2); // where squareZ1 < squareZ2
now you have a point which is clipped to the dimensions of the box but which is also the closest point within the box to the circle. From here is just a simple hypotenuse to see if the distance is less than the circle radius
distance = nearX*nearX + nearY*nearY + nearZ*nearZ;
if (distance > circleR*circleR) { return 0.0; }
So now you know that a collision has occured and you have reduced the collision response to something equivelent to a simple circle circle. At this point I usually do the following
distance = 1.0/sqrt(distance); // now it's time to apply the square root
averageX = (nearX+circleX) / 2.0; // take the average between the two points
averageY = (nearY+circleY) / 2.0; // take the average between the two points
averageZ = (nearZ+circleZ) / 2.0; // take the average between the two points
circleX = averageX - circleRadius*distance*nearX;
circleY = averageY - circleRadius*distance*nearY;
circleZ = averageZ - circleRadius*distance*nearZ;
I use this simple response because I find that the closer you try to mimic real physics the more unstable the simulation becomes. It is still possible for a circle to smash through a box using this algorithm but it is fairly stable.
Edit: Opps I meant something like this for response the above is for two dynamic circles but here one circle should be static
distance = 1.0/sqrt(distance); // now it's time to apply the square root
circleX = nearX - circleR*distance*(nearX-circleX);
circleY = nearY - circleR*distance*(nearY-circleY);
circleZ = nearZ - circleR*distance*(nearZ-circleZ);