This paper complements the instability theory of frontal waves investigated by Orlanski (1968), and reinterprets the unstable modes obtained. First, the stability of a frontal model is reconsidered by using a matrix method. The major part of Orlanski's (1968) result is verified but some flaws are found in some parameter regions: unstable modes do not exist over the entire Ri–Ro region. Also, the features of the neutral waves in the one-layer subsystems are studied, in order to determine the instability of the full two-layer system. As a result, the unstable mode called the (B)-mode by Orlanski (1968) and suggested by Sakai (1989) to be Rossby-Kelvin instability caused by a resonance between a Rossby wave and a gravity wave, proves to be a geostrophic unstable mode caused by resonance between a Rossby wave and the Rossby-gravity mixed mode. In addition, some of the analytical conclusions about the stability of this frontal model are explained by the features of the neutral waves in the one-layer subsystem.

Flows in a cylindrical tank over a rotating bottom are investigated by laboratory experiments. Despite the axisymmetry of the experimental setup, various anisotropic phenomena are observed. The slow rotation of the bottom disk induces a circular flow according to the axisymmetric environment, but polygonal vortices form under faster rotation. Between these two vortex flow states, the flow undergoes a transition with clear hysteresis during which the elliptical shape assumed under faster rotation is retained when the rotation is subsequently slowed to rates that previously supported axisymmetric flow. Sloshing is also observed; here, a calm circular flow state alternates with an oscillation of the water surface along the sidewall of the container. A phase diagram showing the phenomena observed under different combinations of the initial water depth at rest and the rotation rate of the bottom disk is developed following thorough experimental testing over a wide range of parameter values. The features of the dependences of the range for each phenomenon to occur on these parameters are also elucidated.

The linear stability of a front whose lower layer has a uniform potential vorticity is investigated. The results are compared with the unstable modes of a frontal model which consists of two homogeneous flows with an interacting interface. Unstable modes with a phase speed close to the velocity of the basic flow in the lower layer, which exist in the frontal model of two homogeneous flows, are stabilized in the present model. This feature is explained by the absence of Rossby waves in the reduced one-layer problem of the lower layer resulting from the uniformity of its potential vorticity. The results show that the stability of a front is strongly affected by the potential vorticity distribution. Careful attention is needed for the application of the frontal model with uniform potential vorticity, which is a critical situation.

Normal modes of shallow water waves in a channel wherein the Coriolis parameter and the depth vary in the spanwise direction are investigated based on the conservation of the number of zeros in an eigenfunction. As a result, it is generally shown that the condition for transition modes (Kelvin modes and mixed Rossby-gravity modes) to exist, besides Rossby and Poincaré modes, is determined only by boundary conditions. A Kelvin mode is interpreted as a modification of a Kelvin wave or a boundary wave along a closed boundary, and a mixed Rossby-gravity mode as a modification of an inertial oscillation or a boundary wave along an open boundary. Transition modes appearing in edge and continental-shelf waves, equatorial waves and free oscillations over a sphere are systematically understood by applying the theory in this paper.

The interpretation of an unstable mode in terms of resonance between neutral waves is extended to a critical layer instability. It is found that the concept of resonance can be applied to the critical layer instability, if continuous modes are taken into account: The instability occurs due to a resonance between a non-singular mode and a superposition of several continuous modes whose phase speeds are close to the velocity at the critical level. These continuous modes have pseudomomentum with its sign opposite to the gradient of the potential vorticity at the critical level, which is consistent with the interpretation of the instability in terms of resonance between neutral waves.