One of the challenges in the next generation wireless communication systems is to provide high quality multimedia services. To address this issue, the approach we take in this dissertation is cross-layer design of communication systems.

In the first part of the dissertation, we address transmission of progressive images or scalable video using hierarchical modulation. Thanks to the progressive features, progressive bitstreams enable each user in a multi-user network to decode the source at the rate that their channel allows. These progressive sources have the feature that they have gradual differences of importance in the bitstreams. One would like to have gradual differences in unequal error protection (UEP) to correspond to the gradual differences in importance. However, hierarchical modulation, which is often used for UEP and is currently employed in the Digital Video Broadcasting (DVB) standard, provides only a limited number of UEP levels. By multiplexing hierarchical modulation, we propose a high performance multilevel UEP system for the transmission of progressive sources.

In the second part, we consider the transmission of a layered source in a multiple-input multiple-output (MIMO) system for broadcast scenarios. We first analyze the tradeoff between two different MIMO approaches, Alamouti coding and spatial multi-plexing, having the same transmission rate. For analytical tractability, we consider high SNR approximate (minimum distance) bit error rates (BER) for both MIMO approaches. Based on this, we propose superposition MIMO coding, where two different MIMO approaches are hierarchically combined such that low-rate high priority components of the source are Alamouti coded, high-rate low priority components are spatially multiplexed, and the two different components are superposed. It is demonstrated that in broadcast scenarios, the proposed MIMO coding maximizes the performance of a layered source which has different data rates for its components.

In the third part of the dissertation, we analyze the performance of n-channel symmetric FEC-based multiple description coding for a progressive mode of transmission. Multiple description source coding has recently emerged as an attractive framework for robust multimedia transmission over packet erasure channels. In the analysis, we consider transmission over orthogonal frequency division multiplexing (OFDM) networks in a frequency-selective, slowly-varying, Rayleigh faded environment.