Why does butter sometimes design unstable filters?
Filters whose poles are either close together or close to the unit circle are highly sensitive to roundoff and quantization error. Both high order Butterworth filters and Butterworth filters with very low or very high cutoff frequencies have poles close to each other and the unit circle. Therefore, attempts to design these types of Butterworth filters will often generate filters with strange-looking, almost random transfer functions. Unfortunately, since roundoff error is a hardware-limited effect, the only solution for this problem is to use a different type of filter designer, such as ellip or cheby2.
