BSc CSIT (TU) Science Multimedia Computing (BSc CSIT, CSC467) Question Paper 2081 Nepal

Multimedia System

A multimedia system is a computer-controlled, integrated system that can capture, store, process, transmit and present information in more than one medium — combining text, graphics, images, audio, video and animation — in a synchronized and interactive manner. At least one continuous (time-dependent) medium such as audio or video must be involved.

Characteristics of Multimedia Data

• Voluminous (large data size): Uncompressed audio, image and especially video produce huge data volumes (e.g. raw video ≈ tens of MB per second).
• Time dependence (continuous media): Audio and video are isochronous — samples must be delivered at fixed rates; late data is useless.
• Need for synchronization: Different media streams (e.g. lip-sync of audio and video) must be temporally coordinated.
• High bandwidth and real-time requirements: Streaming demands guaranteed throughput and bounded delay/jitter.
• Compressibility: Multimedia data contains spatial/temporal redundancy and so is amenable to compression.
• Interactivity: Users can navigate, seek, pause and control the presentation.

Storage and Coding Requirements

Because raw data is so large, multimedia systems must compress data using coding standards:

Example of raw size: a $640\times480$ true-color (24-bit) image needs $640\times480\times3 = 921{,}600$ bytes ($\approx 0.9$ MB). One second of such video at 25 fps needs $\approx 22$ MB, so compression and high-capacity storage (CD/DVD/SSD) are essential.

Components of a Multimedia Computing System

1. Capture devices – cameras, microphones, scanners, video/audio digitizers.
2. Storage devices – hard disks/SSD, optical media (CD/DVD/Blu-ray), large-capacity servers.
3. Processing hardware – CPU/GPU, sound and graphics cards, codecs (compression/decompression).
4. Communication/network – high-bandwidth networks for streaming and conferencing.
5. Presentation/output devices – display screens, speakers, projectors.
6. Software – authoring tools, media players, operating-system multimedia support (file formats, drivers, real-time scheduling).

Together these support the acquisition → compression/storage → processing → transmission → synchronized presentation pipeline that defines a multimedia computing system.

Multimedia Synchronization

Multimedia synchronization is the task of maintaining the correct temporal and spatial relationships among the media objects that make up a multimedia presentation, so that they are rendered in the intended order and at the intended times (e.g. audio matching the moving lips in a video).

Intra-media (Intra-stream) Synchronization

This maintains the internal timing within a single continuous medium. Each media unit (e.g. a video frame or audio sample) must be presented at the correct instant relative to the previous units, preserving the playback rate.

• Example: displaying a 25 fps video so that one frame appears every 40 ms; or playing audio samples at exactly 44.1 kHz.
• Goal: avoid jitter and rate variation inside one stream.

Inter-media (Inter-stream) Synchronization

This maintains the temporal relationship between two or more different media streams.

• Classic example: lip-sync between the audio stream and the video stream (skew must stay within about ±80 ms to be unnoticed).
• Other examples: a slide changing exactly when the narration reaches a point; subtitles appearing with the spoken line.
• Goal: keep the relative skew between streams within acceptable limits.

Reference Model for Multimedia Synchronization (Four-Layer Model)

A widely used model structures synchronization into four layers, each offering a service to the layer above:

1. Media Layer – deals with a single continuous media stream as a sequence of Logical Data Units (LDUs); provides device-level read/write of samples/frames (intra-stream timing).
2. Stream Layer – operates on continuous streams and groups of streams; provides guarantees on throughput, delay and jitter, and handles intra-stream synchronization for grouped streams.
3. Object Layer – integrates all media streams and time-independent objects (text, images) into a complete presentation; hides the difference between continuous and discrete media and handles inter-stream coordination.
4. Specification Layer – an open layer where authors specify the synchronization requirements (using interval-based, axes-based, hierarchical or event-based specification methods); it maps the author's intent down to the object layer.

This layered reference model separates what must be synchronized (specification) from how it is enforced at runtime (object/stream/media layers).

Digital Image Representation

A digital image is a 2-D array of pixels (picture elements). Each pixel stores an intensity/color value. For a color image with bit depth $b$ bits per channel and 3 channels, each pixel needs $3b$ bits; a $W\times H$ true-color (24-bit) image needs $W\times H\times 3$ bytes. Spatial resolution = pixel count; color depth = bits per pixel.

Color Models

RGB (Red, Green, Blue) — additive

Colors are produced by adding light in the three primaries. Used by emissive devices (monitors, cameras, scanners). $(0,0,0)$ = black, $(255,255,255)$ = white. Suited to display, not to printing.

CMYK (Cyan, Magenta, Yellow, Black) — subtractive

Colors are produced by subtracting (absorbing) light using inks on paper. Used in printing. Black (K) is added because mixing C, M, Y inks gives a muddy dark, not true black, and saves ink. Approximate conversion: $C=1-R',\; M=1-G',\; Y=1-B'$ (on normalized values), then K is extracted.

YUV / YCbCr — luminance + chrominance

Separates brightness from color:

• Y = luma (brightness),
• U/Cb = blue-difference chroma, V/Cr = red-difference chroma.

Conversion (BT.601):

This model is used in TV, JPEG and MPEG because it decorrelates color and lets the chroma be compressed more heavily than luma.

Color Sampling and Chroma Subsampling

The human eye is more sensitive to brightness (luma) than to color (chroma). Therefore the chroma channels can be sampled at a lower spatial resolution than luma without visible quality loss — this is chroma subsampling.

Notation $J{:}a{:}b$ over a $4\times2$ block:

• 4:4:4 – no subsampling; full chroma for every pixel.
• 4:2:2 – chroma sampled at half horizontal resolution (shared by 2 pixels).
• 4:2:0 – chroma sampled at half horizontal and half vertical resolution (one chroma sample per $2\times2$ block). Used in JPEG, MPEG, H.264.

Benefit: 4:2:0 stores chroma at one quarter the samples, cutting data size by about 50% versus 4:4:4 with little perceptual loss — a key reason YCbCr is preferred over RGB for compression.

Huffman Coding

Huffman coding is a lossless, variable-length, entropy-coding technique that assigns shorter codewords to more frequent symbols and longer codewords to rarer ones, producing an optimal prefix-free code (no codeword is a prefix of another).

Construction (example)

Symbols and frequencies: A=45, B=13, C=12, D=16, E=9, F=5 (total 100).

1. Treat each symbol as a leaf node with its frequency.
2. Repeatedly take the two lowest-frequency nodes and merge them into a parent whose frequency is their sum.
3. Repeat until one tree (root) remains.
4. Label left edges 0 and right edges 1; the path from root to a leaf is that symbol's code.

Merging order: (F5+E9)=14, (C12+B13)=25, (14+D16)=30, (25+30)=55, (45+55)=100.

Resulting codes:

Average length $= (45\cdot1+13\cdot3+12\cdot3+16\cdot3+9\cdot4+5\cdot4)/100 = 224/100 = 2.24$ bits/symbol, versus 3 bits for fixed-length coding — a clear saving.

Working

Encoding replaces each symbol with its codeword. Decoding walks the tree bit-by-bit from the root until a leaf is reached; the prefix property guarantees unambiguous decoding. Huffman coding is used inside JPEG, MP3 and ZIP.

Run-Length Encoding (RLE)

Run-length encoding is a simple lossless compression technique that replaces consecutive repetitions (runs) of the same symbol with a single value and a count, instead of storing each repeated symbol separately.

Principle

A run of $n$ identical symbols s is stored as the pair (n, s) (or (s, n)). It is most effective when data contains long runs of identical values, e.g. simple images, fax (bi-level) images, and icons.

Example

Input string:

AAAAABBBCCDAA

Encoded as count–symbol pairs:

5A 3B 2C 1D 2A

The 13-character input is represented by 5 pairs.

For a binary/fax line such as 0000000111110000 it becomes 7(0) 5(1) 4(0).

Characteristics

• Lossless – original data is fully recoverable.
• Very fast and simple to implement.
• Best for low-complexity / repetitive data; for data with few runs (e.g. natural photos), it can even expand the data, so it is often combined with other schemes (e.g. RLE of zero runs in JPEG after DCT/quantization).

Role of DCT and Quantization in JPEG Compression

JPEG compresses an image in $8\times8$ pixel blocks (after RGB→YCbCr conversion and optional chroma subsampling). Two key stages are the DCT and quantization.

Discrete Cosine Transform (DCT)

The 2-D DCT transforms each $8\times8$ block of pixel values from the spatial domain to the frequency domain:

• It produces 64 DCT coefficients: one DC coefficient (average/low frequency, top-left) and 63 AC coefficients (higher frequencies).
• Image energy is compacted into a few low-frequency coefficients; most high-frequency coefficients are near zero.
• The DCT itself is lossless and reversible; it only reorganizes the information so redundancy can be removed.

Quantization

Each DCT coefficient is divided by a value from an $8\times8$ quantization table and rounded:

• This is the only lossy step in JPEG.
• High-frequency coefficients (to which the eye is less sensitive) get larger quantization step sizes, so many become zero and are discarded.
• The quality/compression trade-off is controlled here: larger $Q$ values → more zeros → smaller file but lower quality.

Together

DCT concentrates the visually important information into a few coefficients; quantization then throws away the perceptually unimportant (mostly high-frequency) detail. The resulting block — full of zeros — is then zig-zag scanned and run-length + Huffman coded to achieve the final compression.

I-, P- and B-frames in MPEG

MPEG video exploits temporal redundancy between successive frames using three frame (picture) types organized in a Group of Pictures (GOP).

Details

• I-frame (Intra-coded): self-contained; uses only spatial (intra-frame) compression. Acts as a reference and a random-access/refresh point, and starts each GOP. Largest in size.
• P-frame (Predictive-coded): stores only the difference from the most recent preceding I- or P-frame, using forward motion compensation. Smaller than I but depends on the earlier frame.
• B-frame (Bidirectionally predicted): predicted from both a past and a future reference frame, giving the best compression. B-frames are never used as references, so they cause no error propagation. (Because they need a future frame, encoding/decoding order differs from display order.)

A typical GOP looks like I B B P B B P B B P ….

RGB vs CMYK Color Models

Explanation

• RGB is additive: colored light is emitted and combined; adding all three at full intensity yields white. Ideal for screens that produce their own light.
• CMYK is subtractive: colored inks on white paper absorb part of the incident white light and reflect the rest; combining inks removes more light, tending toward black. The extra K (black) is used because mixed CMY inks give a muddy brown rather than pure black, and using K saves ink and gives sharper text.
• Conversion (approx., normalized 0–1): $C=1-R,\;M=1-G,\;Y=1-B$, then $K=\min(C,M,Y)$ is factored out. Because gamuts differ, some bright RGB screen colors cannot be printed exactly in CMYK.

Sampling and Quantization of Digital Audio

Converting a continuous (analog) sound wave into a digital signal involves two steps: sampling (discretizing time) and quantization (discretizing amplitude) — together forming PCM (Pulse Code Modulation).

Sampling

The continuous waveform is measured at regular time intervals; each measurement is a sample. The sampling rate (frequency) $f_s$ is the number of samples per second (Hz).

• Nyquist theorem: to capture a signal whose highest frequency is $f_{max}$ without aliasing, $f_s \ge 2 f_{max}$.
• Example: human hearing reaches ~20 kHz, so CD audio uses $f_s = 44.1$ kHz ($\ge 2\times20\text{ kHz}$).
• Too low a rate causes aliasing (high frequencies fold into wrong, lower frequencies).

Quantization

Each sampled amplitude (a real value) is rounded to the nearest level out of a finite set determined by the bit depth $n$. With $n$ bits there are $2^n$ levels.

• Example: 16-bit audio → $2^{16}=65{,}536$ levels.
• The rounding error is quantization noise; more bits → finer steps → lower noise and higher dynamic range (≈ $6.02n$ dB).

Resulting Data Size

For CD-quality stereo: $44100 \times 16 \times 2 = 1{,}411{,}200$ bits/s ≈ 1.41 Mbit/s (≈ 10 MB/minute uncompressed), which is why audio compression (MP3/AAC) is used.

Lossy vs Lossless Compression

Explanation

• Lossless compression encodes data more efficiently (e.g. Huffman or RLE) so the exact original bits are recoverable. It is mandatory where every bit matters (programs, text, legal/medical images).
• Lossy compression exploits limits of human perception (e.g. the eye's insensitivity to high spatial frequencies, the ear's masking effects) to throw away data that the user is unlikely to notice. This yields far smaller files, which is why it dominates images, audio and video.

Examples: A .png logo (lossless) can be restored bit-for-bit; a .jpg photo (lossy) at high compression shows blocking artifacts and cannot be restored to the original.

Characteristics of Multimedia Data and Storage Requirements

Characteristics of Multimedia Data

• Voluminous (large size): Uncompressed image, audio and especially video generate very large data volumes.
• Continuous / time-dependent media: Audio and video are isochronous — units must be delivered and played at a fixed rate; late data is worthless.
• Real-time and high-bandwidth needs: Capture and playback require guaranteed throughput and bounded delay and jitter.
• Need for synchronization: Multiple streams (e.g. audio + video lip-sync) must keep their temporal relationship.
• Redundancy / compressibility: Contains spatial and temporal redundancy, so it compresses well (JPEG, MPEG, MP3).
• Interactivity: Supports user control such as seek, pause and navigation.

Storage Requirements

Because raw data is so large, multimedia needs high-capacity, high-throughput storage and compression.

Example (raw sizes):

• $640\times480$ true-color (24-bit) image: $640\times480\times3 = 921{,}600 \approx 0.9$ MB.
• CD-quality stereo audio: $44100\times16\times2 \approx 1.41$ Mbit/s ≈ 10 MB/minute.
• Raw video at 25 fps of the above frame ≈ $0.9\text{ MB}\times25 \approx 22$ MB/s — over a GB per minute.

Consequences for storage:

1. Large capacity media (HDD/SSD, optical CD/DVD/Blu-ray, servers).
2. High transfer rate / low access time to sustain real-time playback.
3. Compression (lossy and lossless) to make storage and transmission feasible.
4. Specialized file systems/buffering to deliver continuous media with bounded jitter.

Multimedia Synchronization

Multimedia synchronization is the process of maintaining the correct temporal (and spatial) relationships between the media objects in a presentation, so they are played in the intended order and timing (e.g. audio matching video, subtitles matching speech).

Intra-media vs Inter-media Synchronization

Explanation

• Intra-media synchronization ensures the internal continuity of one stream: each logical data unit (frame/sample) is presented at the right instant relative to the previous one, so playback is smooth and at the correct rate.
• Inter-media synchronization coordinates several streams together. The classic case is lip-sync, where audio–video skew should stay within roughly ±80 ms to remain unnoticed. It also covers discrete media, e.g. an image or text appearing at a defined moment in the audio.