I never dug too deep into physics algorithms. You might find something useful in the book Real Time Collision Detection. I know some 3rd party physics engines I've used will “put objects to sleep if they are considered not moving”. This concept might help a little bit, but it sounds like you have an actual bug. Post a video of how bad the jitter is.

real-time collision detection is great for the narrow-phase collision primitives, but it doesn't talk much about the actual rigid body simulation.

In general, you have to make sure that your system doesn't gain energy. This can happen through both small numerical errors, and through the specific penetration response/penalty functions.

The big stick to hit this problem with, is damping. Linear damping is somewhat helpful (each frame, set velocity to velocity * 0.99,) but angular damping is crucial! The reason for this is that even a small amount of penetration along the corners of a cube (which are the most likely to penetrate,) can generate significant change in angular momentum. You may want to damp angular movement quite hard (angvel = angvel * 0.97 each step, say!) or use a magnitude-based damping – the larger the angular velocity, the more it gets damped.

Another nice trick is to move penalty impulses towards center of gravity. This will change “rotational bounces” more towards “linear bounces,” so you don't get those crazy-high angular values. When adding a penalty impulse calculate its position as a blend of (penetration-position, center-of-gravity.) Even something drastic like a 50:50 blend can look convincing while being much nicer, numerically.