BSc CSIT (TU) Science Multimedia Computing (BSc CSIT, CSC467) Question Paper 2082 Nepal

Why Speech Synthesis is Difficult

Speech synthesis (text-to-speech, TTS) is the artificial production of human speech from text. It is a difficult task because natural human speech is highly variable and context-dependent, and the machine must reproduce not just words but the naturalness, intonation and emotion of a human speaker.

Cases and Issues that Make it Hard

• Pronunciation ambiguity (homographs): The same spelling can be read differently depending on meaning, e.g. read (present vs past), lead (metal vs guide), bass (fish vs music). The system must disambiguate using context.
• Letter-to-sound (grapheme-to-phoneme) mismatch: Many languages, especially English, are not phonetic — though, through, tough, cough are spelled similarly but pronounced very differently.
• Prosody (intonation, stress, rhythm): Correct pitch, loudness and duration are needed so the sentence does not sound flat/robotic. A question and a statement of the same words need different intonation.
• Co-articulation: Adjacent sounds influence each other; phonemes are not produced in isolation, so simply joining recorded sounds creates unnatural transitions.
• Numbers, abbreviations and symbols: Text like Dr., St., 1990, $5, Ph.D. must be expanded correctly (Doctor/Drive, Saint/Street) before synthesis — this is text normalization.
• Emotion and naturalness: Conveying emotion, emphasis and a natural voice quality is very hard for a machine.
• Language/dialect/speaker variability: Accents, speaking rate and voice differences make a single model insufficient.

Techniques for Speech Generation

1. Articulatory Synthesis

Models the physical human vocal tract (tongue, lips, glottis, vocal cords) using mathematical/physical models and produces speech by simulating air flow. Most accurate in principle but very complex and computationally expensive, so rarely used in practice.

2. Formant (Rule-based) Synthesis

Generates speech from a set of rules and parameters (formant frequencies, pitch, noise) using an additive/source-filter model rather than any recorded human speech.

• Advantages: very small footprint, intelligible at high speeds, fully controllable.
• Disadvantages: sounds robotic/unnatural. Used in older systems and assistive devices.

3. Concatenative Synthesis

Builds speech by joining together small recorded units of real human speech taken from a database.

• Unit types: phonemes, diphones (phone-to-phone transitions), syllables, words.
• Unit-selection synthesis picks the best-matching recorded units from a large database to minimize joins.
• Advantages: sounds natural because it uses real speech.
• Disadvantages: large database needed; audible glitches at join points and limited flexibility in changing voice/emotion.

4. Statistical Parametric / HMM-based Synthesis

Uses statistical models (e.g. Hidden Markov Models) trained on speech to generate parameters (spectrum, pitch, duration), which a vocoder turns into sound. Smaller and more flexible than concatenative, but slightly muffled quality.

5. Deep-Learning (Neural) Synthesis

Modern systems (e.g. WaveNet, Tacotron-style models) use deep neural networks to generate highly natural, near-human speech directly from text, learning prosody and voice characteristics from data. They give the best naturalness at the cost of heavy computation/training.

Summary

Speech synthesis is hard chiefly because of text normalization, grapheme-to-phoneme conversion, prosody, co-articulation and naturalness. Techniques range from physically-modeled (articulatory) and rule-based (formant) approaches to data-driven (concatenative, HMM and neural) approaches, with neural methods currently giving the most natural results.

Methods for Controlling Animation

Animation control determines how the motion of objects over time is specified and played back. The three common methods are frame-based, key-frame and timeline control.

Frame-based Animation

Every individual frame is drawn/stored separately and played back in sequence at a fixed rate, like a traditional flip-book or film.

• The animator (or system) must define each complete frame.
• Simple to understand and to play, but storage-heavy and labor-intensive since nothing is generated automatically.

Key-frame Animation

The animator defines only the important key frames (extreme positions, such as start and end of a motion); the in-between frames are generated automatically by interpolation (a process called tweening / in-betweening).

• Far less work and storage than frame-based since intermediate frames are computed.
• Interpolation can be linear or smooth (spline) for natural motion.
• Widely used in 2-D/3-D animation tools (Flash, Maya, etc.).

Timeline (Procedural / Script) Control

Animation is controlled along a timeline using scripts, parameters or procedures rather than drawing frames. Objects, their properties (position, scale, color) and events are placed on a time axis, and the system computes the display at each instant.

• Supports interactivity and event-driven changes.
• Allows precise control and reuse; behavior can be programmed (e.g. with a scripting language).

Comparison

Transmission of Animation

Animation can be delivered to a remote viewer in two main ways:

1. Symbolic Representation (Transmit the description)

Instead of sending pixels, the objects, their attributes and the operations/commands that produce the animation are transmitted; the receiver regenerates (renders) the frames locally.

• Advantages: very low bandwidth (only commands/parameters sent), resolution-independent, scalable.
• Requirement: the receiver must have the rendering capability and the same objects/fonts; output may differ slightly between machines.
• Example: vector/scripted animation (e.g. SVG/Flash-style command streams).

2. Pixmap (Pixel-by-pixel) Representation (Transmit the rendered frames)

The animation is rendered at the sender and the resulting bitmap frames (or compressed video) are transmitted.

• Advantages: receiver needs only a player; the result looks identical everywhere.
• Disadvantages: high bandwidth because full frames/video are sent; this is usually combined with video compression (MPEG/H.264) to make it feasible.

Note

The choice involves a bandwidth-vs-compatibility trade-off: symbolic transmission saves bandwidth but needs a capable, consistent renderer, while pixmap transmission is universally playable but bandwidth-intensive (hence compression and streaming are used).

Entropy Coding

Entropy coding is a class of lossless compression in which symbols are encoded based purely on their statistical probability of occurrence, independent of the meaning of the media. According to information theory, the average information content (entropy) of a source is

where $p_i$ is the probability of symbol $i$. Entropy is the theoretical lower bound on the average number of bits per symbol. Entropy coders assign shorter codes to more probable symbols to approach this bound.

• Examples: Huffman coding, Arithmetic coding, Run-Length Encoding (RLE).
• Because it only removes statistical redundancy, it is lossless and fully reversible.

Hybrid Coding

Hybrid coding combines two or more compression techniques — typically a source/transform (often lossy) stage followed by an entropy (lossless) stage — to achieve a much higher overall compression ratio than any single method.

• The lossy part removes perceptually unimportant information (e.g. DCT + quantization).
• The lossless entropy part then removes remaining statistical redundancy (e.g. RLE + Huffman).
• Examples: JPEG (DCT + quantization + RLE + Huffman) and MPEG/H.26x (DCT + motion compensation + quantization + entropy coding) are hybrid coders.

Lossy Sequential DCT-based Model (Baseline JPEG)

The baseline JPEG mode is the lossy, sequential, DCT-based model — the most widely used JPEG mode. "Sequential" means the image is encoded block by block in a single left-to-right, top-to-bottom pass. Its stages are:

Image --> Color transform & subsampling --> 8x8 blocks --> FDCT
      --> Quantization --> Zig-zag scan --> DC (DPCM) + AC (RLE)
      --> Entropy coding (Huffman) --> Compressed JPEG

1. Color transform & subsampling: RGB is converted to YCbCr; chroma is subsampled (e.g. 4:2:0) since the eye is less sensitive to color.
2. Block splitting: the image is divided into $8\times8$ blocks.
3. Forward DCT (FDCT): each block is transformed from the spatial to the frequency domain, producing one DC and 63 AC coefficients; energy concentrates in low frequencies. (Lossless step.)
4. Quantization: each coefficient is divided by a value from an $8\times8$ quantization table and rounded. This is the only lossy step; high frequencies get coarser steps, so many coefficients become zero.
5. Zig-zag scanning: coefficients are read in a zig-zag order so that low-frequency (non-zero) values come first and the many trailing zeros are grouped together.
6. DC and AC coding: the DC coefficient is coded differentially (DPCM) against the previous block's DC; the AC coefficients are run-length encoded (run of zeros, value).
7. Entropy coding: the result is finally Huffman coded (entropy coding) to produce the compressed stream.

Decoding simply reverses these steps (entropy decode → de-zig-zag → dequantize → IDCT → color convert). Because quantization discards data, baseline JPEG is lossy, and because it pipelines a transform+quantization stage with an entropy stage, it is a classic hybrid coder.

Multimedia System

A multimedia system is a computer-controlled, integrated system capable of capturing, storing, processing, transmitting and presenting information in more than one medium — combining text, graphics, images, audio, video and animation — in a synchronized and interactive way. It must involve at least one continuous (time-dependent) medium such as audio or video.

Properties of a Multimedia System

• Computer-controlled: the system is controlled by a computer that coordinates capture, processing and presentation of the media.
• Integrated: different media (text, image, audio, video) are handled by a single integrated system rather than separate devices.
• Digital representation: the information the system handles is digitally represented, enabling processing, storage and transmission.
• Use of continuous and discrete media: it combines time-independent (discrete) media — text, images — with time-dependent (continuous) media — audio, video.
• Interactivity: the user can interact with and control the presentation (navigate, seek, pause, choose paths).
• Synchronization: the media must be temporally synchronized (e.g. lip-sync of audio and video).
• Large data volume and real-time needs: continuous media are voluminous and require high bandwidth, real-time delivery and compression.

In short, a multimedia system is defined by being computer-controlled, integrated, digital, and able to present multiple, synchronized, interactive media including at least one continuous stream.

Phases of Image Synthesis

Image synthesis is the process of generating a realistic 2-D image from a description of a 3-D scene (its objects, lights, camera and surface properties) — i.e. computer-generated imagery / rendering. It proceeds through the following phases:

1. Modeling (Scene Description)

The objects and the scene are described geometrically — their shapes (polygons, surfaces), positions, sizes and structure are defined in a 3-D coordinate space, along with surface/material properties.

2. Setting Lighting and Viewing (Scene Setup)

The light sources (type, position, intensity, color) and the camera/viewpoint (position, direction, field of view, projection) are specified. This determines how the scene is illuminated and from where it is observed.

3. Projection and Geometric Transformation

The 3-D scene is transformed and projected onto the 2-D view (image) plane using transformations (translation, rotation, scaling) and a projection (perspective or parallel), mapping world coordinates to screen coordinates.

4. Visibility / Hidden-Surface Removal

Surfaces and parts of objects that are hidden behind others from the chosen viewpoint are removed (e.g. using the z-buffer / depth-buffer or painter's algorithm), so only the visible surfaces are drawn.

5. Shading, Illumination and Texturing

The color and brightness of each visible surface are computed using an illumination model (ambient, diffuse, specular), shading techniques (flat, Gouraud, Phong), shadows, and texture mapping to add surface detail and realism.

6. Rasterization and Display (Image Generation)

The shaded surfaces are converted to pixels (rasterized) and stored in the frame buffer, then displayed as the final synthesized image. Anti-aliasing may be applied to smooth jagged edges.

Together these phases take a scene from an abstract 3-D description to a finished, displayable 2-D image.

Audio vs Video as a User Interface

Whether audio or video is the more effective user interface depends on the context and task — neither is universally superior. A justified comparison is needed.

Where Video is More Effective

• Video carries far more information (it combines images, motion, text and usually audio), so it is better for demonstrations, spatial/visual tasks and showing how something looks or works.
• It engages the visual channel, which humans process quickly for spatial relationships.
• Examples: a tutorial showing on-screen steps, a navigation map, a product demo, a video conference where seeing facial expressions matters.

Where Audio is More Effective

• Audio is eyes-free and hands-free, so it is better when the user's eyes/hands are busy (e.g. driving, cooking, walking) or for visually impaired users (screen readers).
• It needs much less bandwidth and storage than video and works on low-end/connectivity-poor devices.
• It is good for alerts, notifications, voice assistants and feedback that should not require looking at a screen.

Comparison

Justification / Conclusion

For conveying rich, visual or spatial information, video is more effective because it uses the high-bandwidth visual channel and combines multiple media. However, for hands-free, eyes-free, low-bandwidth or accessibility situations, audio is more effective. The best user interfaces often combine both (e.g. video with narration), so effectiveness is determined by the task, user context and available resources rather than the medium alone.

System-Software Abstractions for Multimedia Programming

Writing multimedia applications directly against raw hardware is impractical, so system software provides layers of abstraction that hide hardware details and expose convenient interfaces for handling continuous and discrete media. The main abstractions are:

1. Device Abstraction (Device Drivers)

The operating system provides device drivers that hide the specifics of cameras, sound cards, video capture cards and displays. The programmer accesses devices through a uniform interface (open/read/write/control) instead of hardware registers.

2. Media / Stream Abstraction

Continuous media are abstracted as streams of Logical Data Units (LDUs) — e.g. video as a stream of frames, audio as a stream of samples. The programmer works with read/write of media units at a given rate, while the system handles buffering and timing.

3. File and File-System Abstraction

Media are stored and accessed as files in standard formats (e.g. WAV, JPEG, MPEG). The file-system abstraction provides naming, structure and continuous-media-aware storage/retrieval (buffering, prefetching) so that real-time playback is sustained.

4. Process / Thread and Real-Time Scheduling Abstraction

Media handling is organized into processes/threads, and the OS provides real-time scheduling so that time-critical media tasks meet their deadlines (bounded delay and jitter), guaranteeing smooth playback.

5. Communication / Networking Abstraction

For distributed multimedia, system software offers communication abstractions (sockets, streaming protocols, transport with QoS) that hide network details and provide throughput/delay guarantees for streaming and conferencing.

6. Programming Interface / Toolkit Abstraction (APIs)

High-level APIs and toolkits (multimedia libraries) provide ready operations to capture, encode/decode, synchronize and present media, so applications are written in terms of media objects rather than low-level details.

Together these abstractions let a programmer treat media as streams, files, processes and devices through clean interfaces, while the system enforces the real-time and synchronization guarantees underneath.

Video Conferencing

Video conferencing is a real-time, interactive multimedia communication technology that lets two or more participants at different locations see and hear each other simultaneously through synchronized live audio and video over a network. It combines captured video and audio, compresses them (e.g. H.264/H.323/WebRTC), transmits them over the network, and presents them in real time, often with extra features such as screen/document sharing and chat.

• Requirements: camera, microphone, display, speakers, codecs, sufficient bandwidth and low delay/jitter for natural interaction, and audio-video synchronization (lip-sync).
• Uses: remote meetings, online education, telemedicine, and collaboration.

Editing Digital Video

Digital video editing is the process of manipulating and rearranging video clips to produce a final program, using non-linear editing (NLE) software that allows random access to any frame. Typical operations are:

1. Capture / Import: acquire footage from a camera/file and store it as digital clips.
2. Trimming and cutting: remove unwanted parts by cutting clips and setting in/out points.
3. Arranging on a timeline: place clips in the desired order on a timeline (the basis of non-linear editing).
4. Transitions: add transitions between clips (cut, fade, dissolve, wipe) for smooth scene changes.
5. Adding effects and titles: apply visual effects, color correction, filters, titles and captions.
6. Audio editing: add/adjust soundtrack, narration and sound effects, and ensure audio-video synchronization.
7. Rendering and export: render the edited timeline and export/encode it to a target format (e.g. MP4/H.264) at the required resolution and bitrate.

Because editing is non-linear, any clip or frame can be accessed and modified directly, making digital video editing flexible and non-destructive to the original footage.

Characteristics of a Multimedia System

• Computer-controlled: the system is coordinated by a computer.
• Integrated: multiple media are handled by a single integrated system.
• Digital representation: information is represented and processed digitally.
• Combines continuous and discrete media: time-dependent media (audio, video) together with time-independent media (text, image).
• Interactivity: the user can control and navigate the presentation.
• Synchronization: media streams are kept temporally synchronized (e.g. lip-sync).
• Large data volume / high bandwidth / real-time: continuous media are voluminous and need compression and real-time delivery.

Representation of a Digital Image

A digital image is represented as a two-dimensional array (matrix) of pixels (picture elements). Each element corresponds to a small point of the image and stores a numeric value representing its intensity/color.

• Spatial resolution: the image has a width $W$ and height $H$, i.e. $W\times H$ pixels; more pixels means higher resolution and detail.
• Color depth (bit depth): the number of bits per pixel determines how many colors/levels each pixel can take. With $n$ bits there are $2^n$ levels.
• 1 bit → black & white (bi-level).
• 8 bits → 256 gray levels (grayscale) or an indexed palette.
• 24 bits → true color (8 bits each for R, G, B → about 16.7 million colors).
• Pixel value: a grayscale pixel stores one value; a color pixel stores a tuple, e.g. $(R, G, B)$.

Storage Size

The raw size of an image is:

Example: a $640\times480$, 24-bit image needs $640\times480\times3 = 921{,}600$ bytes ($\approx 0.9$ MB), which is why images are usually compressed (e.g. JPEG, PNG).

Thus a digital image is fundamentally a matrix of pixels characterized by its resolution and color/bit depth.

Why We Need to Compress Multimedia Data

Multimedia data (image, audio and especially video) is extremely voluminous in raw form, so compression is essential.

• Reduce storage: raw video can need over 1 GB per minute; compression makes it feasible to store on disks, optical media and servers. (E.g. a $640\times480$ 24-bit image ≈ 0.9 MB; raw CD audio ≈ 10 MB/minute.)
• Reduce transmission bandwidth: networks have limited bandwidth, so compression lowers the bitrate needed for streaming and conferencing.
• Enable real-time delivery: smaller data meets real-time throughput and latency limits for playback over networks.
• Reduce cost: less storage and bandwidth means lower cost.
• Feasibility of applications: without compression, applications like video streaming, video calls, IPTV and DVDs would be impractical.

Compression is possible because multimedia data contains spatial, temporal and perceptual redundancy that can be removed (losslessly or lossily).

Roles of the User Interface in a Multimedia System

The user interface (UI) is the layer through which the user interacts with the multimedia system; it plays several important roles:

• Presentation / output: organizes and presents the multiple media (text, image, audio, video) to the user in a coherent, synchronized way.
• Interaction and control: lets the user control playback and navigation — play, pause, seek, stop, volume, choose content paths (interactivity).
• Navigation: provides menus, links, buttons and hyperlinks so the user can move through the (often hypermedia) content easily.
• Input handling: accepts user input through devices (keyboard, mouse, touch, voice, gestures) and translates it into system actions.
• Feedback: gives the user feedback (visual/audio cues) about the current state and the result of actions.
• Usability and accessibility: makes the system easy, intuitive and accessible to different users, improving the overall user experience.

In short, the UI mediates presentation, interaction, navigation, input and feedback, and its quality largely determines how usable and effective the multimedia system is.

Media Integration

Media integration is the process of combining different media types — text, graphics, images, audio, video and animation — into a single, coherent and synchronized multimedia presentation or application, so they work together as a unified whole rather than as separate elements.

• It involves temporal integration (synchronizing time-dependent media, e.g. audio with video) and spatial integration (arranging media on the screen/layout).
• Goal: present multiple media in a consistent, synchronized and meaningful way so the user perceives one integrated experience.
• Example: a video with synchronized narration, on-screen subtitles and background music, plus interactive buttons.

Expressing Media as Types, Files and Processes

In a multimedia system, each medium can be viewed/expressed at three levels of abstraction:

1. Media as a Type

A medium is treated as a data type with defined properties and operations. The type captures the nature of the medium — e.g. it is continuous (audio, video) or discrete (text, image) — along with attributes (resolution, sampling rate, color depth) and the operations valid on it (play, display, encode). This lets programs handle media in a type-safe, abstract way.

2. Media as a File

A medium is stored and exchanged as a file in a defined format/container that encodes the data plus metadata (e.g. WAV, MP3 for audio; JPEG, PNG for images; MPEG/MP4 for video). The file abstraction provides persistent storage, naming and standard structure so media can be saved, transferred and read back consistently across systems.

3. Media as a Process

When media is being captured, processed, transmitted or presented, it is handled by active processes/threads that operate on the media in real time. For continuous media, these processes must meet timing/real-time constraints (correct rate, bounded jitter) and may be connected as a pipeline (capture → encode → transmit → decode → present). The process abstraction captures the dynamic, runtime behavior of the medium.

Summary

The type view describes what the medium is, the file view describes how it is stored/exchanged, and the process view describes how it is actively handled at runtime — together giving a complete way to express media in a multimedia system.

Issues in Designing a User Interface

Designing a good multimedia user interface (UI) involves several issues that must be carefully addressed:

• Usability / ease of use: the interface must be simple, intuitive and easy to learn, minimizing user effort and errors.
• Consistency: layout, controls, colors and behavior should be consistent throughout so users can predict how things work.
• Navigation structure: clear navigation (menus, links, structure) is needed so users do not get lost, especially in large hypermedia content.
• Media selection and integration: choosing the right media for the content and integrating/synchronizing them effectively (not overloading the user).
• Response time and performance: the UI must give timely feedback and response, since delays frustrate users (important for continuous media).
• Feedback: the system should clearly indicate state and the result of actions (visual/audio cues).
• Accessibility: support different users, including those with disabilities (alternatives for audio/visual content).
• Aesthetics and screen layout: pleasant, uncluttered layout with good use of color, space and contrast.
• Different devices / responsiveness: the UI must work across different screen sizes and platforms.

Roles of Multimedia in Telemedicine

Telemedicine is the delivery of healthcare services remotely using ICT; multimedia plays central roles in it:

• Remote consultation (video conferencing): live audio-video lets doctors and patients interact at a distance, enabling diagnosis and follow-up without travel.
• Transmission of medical images: high-quality images (X-ray, CT, MRI, ultrasound) are captured, stored and transmitted for remote diagnosis (tele-radiology), often needing lossless compression to preserve detail.
• Storing and sharing patient records: multimedia electronic health records combine text, images, audio notes and video for sharing among specialists.
• Real-time monitoring: audio/video and signal data (e.g. ECG) can be monitored remotely in real time.
• Medical education and training: multimedia videos, simulations and animations are used to train doctors and educate patients.
• Tele-surgery / specialist support: synchronized real-time video allows expert guidance during procedures.

Thus multimedia in telemedicine improves access to healthcare, remote diagnosis, collaboration and education, especially valuable for rural and underserved areas; it requires reliable bandwidth, synchronization and security/privacy of medical data.