Experimental and theoretical studies devoted to investigation of chaotic systems with a mixed phase space such as H atoms interacting with strong microwave fields and quantum billiards are reported. We describe combined experimental-theoretical study to realize and control the behavior of nonlinear common (joint) resonances that occur when excited H atoms are driven by linearly polarized microwave field consisting of two commensurate frequencies. We show how the relative phase of the field can be used to affect the resonances in the mixed phase space and, by this means, control the ionization probability. We also study the regime of quantum cantori in quarter-stadium billiards. Experimentally, the stadium billiard is simulated by a thin quarter-stadium microwave cavity. The eigenfunctions of the quarter-stadium billiard with the parameter are reconstructed in the cantori regime using a field perturbation technique and a circular wave expansion method. We show that in the quantum cantori regime the rescaled localization length of the eigenfunctions does not depend on average on . Support by KBN grant No. 2 P03B 023 17 is acknowledged.