Nomadik Application Processor
Andrea Gallo Giancarlo Asnaghi
ST is #1 world-wide leader in Digital TV and Consumer Audio
MP3 Portable Digital Satellite Radio Player Digital Car Radio
Set Top Box DVD Player
MMDSP+ inside more than 200 million produced chips
January 14, 2009
ST leader in mobile phone chips
January 14, 2009
Nomadik
Nomadik is based on this heritage providing:
Unrivalled multimedia performances Very low power consumption Scalable performances
January 14, 2009
Best Application Processor 2004
Lowest power consumption Scalable performance Video/Audio quality Cost-effective Nominees: Intel XScale PXA260, NeoMagic MiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611
January 14, 2009
Nomadik
Nomadik is a family of Application Processors
Distributed processing architecture ARM9 + multiple Smart Accelerators Support of a wide range of OS and applications Seamless integration in the OS through standard API drivers and MM framework
January 14, 2009
roadmap
...
January 14, 2009
Some Nomadik products on the market...
January 14, 2009
STn8815 block diagram
January 14, 2009
Nomadik : a true real time multiprocessor platform
SDRAM
Unlimited Space Limited Bandwidth
SRAM (Level 2 Cache for Video)
General Purpose System DMA
ARM926 (L1 + L2)
Master OS Peripherals
Memory Controller
multi-layer AHB bus
RTOS Multi-thread
RTOS (Scheduler FSM) MMDSP+ Video
NAND Flash
Unlimited Space No Bandwidth Mass storage
MMDSP+ Audio
133 MHz, 24-bit
66 MHz, 16-bit
HW Acceleration
Audio Peripherals
January 14, 2009
Video Peripherals
Context & task relation: Logical view
ARM boundary (WinCE & Linux OS type)
GUI, User Applications Multiple Address Space Application engines & servers
Virtual world
User side System side (kernel) Single Address Space
System calls
IO process IO
Drivers & Kernel components
HW/IO process Flat Address Space IO
Physical world SAA boundary
SVA boundary
January 14, 2009
Picture Post Processing
January 14, 2009
Examples of Video Resize filters targeting Mobile TV
Resized: zoom out (preview) Regular: display 1:1 scale Wide: zoom to full height Panoramic crop and zoom to full width Stretch zoom to full width and height
January 14, 2009
Rotate
January 14, 2009
Hardware/Software partitioning
MPEG4 software stack
Sequence ARM SW Group of picture Picture Group of Macro-Block HW/FW Macro-Block Block
Firmware
Handled by Software on the host CPU. High flexibility, low computation OS friendly
Handled by Hardware. No flexibility, high computation
January 14, 2009
MM integration in an open OS
Application (Player,) MM Framework
Audio Source Filter Codec Filter/Codec Video Filter/Codec Audio Sink Video Sink Virtual Media pipeline User side Kernel side PCM if SAA HW Driver (control path) SVA HW Driver (control path)
(plug-ins based)
Integration/Adaptation layer
SAA Audio Codec Video Codec Post-
SVA
Display FrameBuffer
Processing
January 14, 2009
Concurrent execution
t1 t2 t3 t4 t5
ARM
Read & Parse
Read & Parse Audio Decode
Read & Parse Audio Decode Audio Playback
Read & Parse Audio Decode Audio Playback Video Decode Video Display
Read & Parse Audio Decode Audio Playback Video Decode Video Display
SAA
SVA
Video Decode
Video Decode Video Display
January 14, 2009
Power Management support
RUN mode :
~200mA
Full clock speed 64 393MHz
Full clock speed
Doze mode :
ARM in WFI mode PLL1 & 2 OFF, clocks = 32 kHz 32 kHz All peripheral clocks are gated
Standby mode :
ARM in WFI mode (Wait For Interrupt) PLL1 & 2 are running 64-393MHz
All peripheral clocks are gated
Wake up: 3ms
Sleep mode :
ARM in WFI mode PLL1 & 2 OFF, clocks are stopped except the non switchable part @ 32 kHz All peripheral clocks are gated
~mA
Power Save mode: Power Save mode
ARM in WFI mode PLL1 OFF, PLL2 Running 19MHz
Deep -Sleep mode:
- Main part of the logic is no more powered - The non switchable part is clocked @32 kHz
~ A
CPU & Bus clocked at XTAL frequency All peripheral clocks unused are gated
January 14, 2009
SW architecture Linux
Screenshots and pictures
WVGA 800x480 @ 24bpp Philips TPO display
20% CPU
3% CPU
January 14, 2009
Average CPU load
VGA 30fps 2Mbps Xine FFMPEG 10-20% GSTLaunch MPEG4 27% GSTLaunch H264 37% Playbin OpenGL-ES panel carousel demo ARM load 3%
Lauterbach ETM trace H264 VGA 30fps 2Mbps AAC
Lauterbach ETM trace MPEG4 VGA 30fps 2Mbps MP3
+20pp
Khronos compliance ongoing
January 14, 2009
System overview for STn8815
cmp K+ architecture Application Framew ork + Application Framework + Azingo + Google Android + Access ALP + Trolltech Qtopia + OpenMoko + Poky
Full open source Linux 2.6.20 kernel OpenedHand PokyLinux for system test In line with GMAE, Limo and Maemo
Multimedia HCL + SAA HCL + SGA HCL + SVA HCL
Middleware Communication + Connection Manager + Bluetooth + Wi-Fi + Ethernet + Telephony Connectiv ity + SD/MMC + USB Hi-Speed MEMS + Gesture Library + MEMS Driver Security + Cryptoengine driver + IPSec + OpenSSL + Security Graphic User Interface + Matchbox + GTK+ Camera + Camera ISP + ISP Driver OpenGL-ES 1.1 + Clutter + EGL + OpenGL Driver + OpenGL-ES 1.1 + GLES Library GStreamer + GStreamer + OpenMAX-IL + FFMPEG + SAA Block Library Pow er Management + PM framework + Governor + Battery Manager + CPUFreq + Linux DPM + APM + LiMo PM
Kernel and driver level Multimedia Interfaces + ALSA + V4L2 + X.11 + xvideo + EXA + DirectFB File systems + JFFS2 + YAFFS + CRAMFS + RAMFS + MTD Kernel and Driv ers + DMA + I2C + Linux kernel 2.6.20 + MSP + SPI + UART Multimedia Driv ers + SAA Driver + SGA Lib and Driver + SVA Driver
Power Management
January 14, 2009
A/V features at a glance
Video
MPEG4 / H.264 decoder up to VGA 30fps 2Mbps MPEG4 encoder up to VGA 30fps H.263 codec JPEG codec MP3, AAC / eAAC+, WMA G.711, G.723, G.726, G.729 AMRnb / AMRwb Full duplex noise and acoustic echo cancelling Effects, e.g. equaliser, reverberation, spatial sound
Audio
January 14, 2009
Multimedia interfaces
cmp Multimedia Interfaces
Standard Linux interfaces
ALSA
X.11
DirectFB
V4L2
xvideo flow
EXA use use use
use
use
Nomadik drivers
Kernel and Drivers: :MSP
Multimedia Drivers: :SVA Driver
Multimedia Drivers: :SGA Lib and Driver
January 14, 2009
Graphic and video subsystem
2D Graphics
X.11 EXA for copy, solid fill, compositing Good acceleration up to 3x on big pixmaps Penalties for smaller primitives ARM + L2C is much faster ARM for EGL and parsing SVA+FPU for TnL SGA for rendering Good fit for smart GUI, not for 3D games Xvideo extension with hw acceleration in SVA Resize, rotate, colour conversion and blitting
3D Graphics
Video rendering
January 14, 2009
Multimedia with FFMPEG
cmp Karakum PMP application Xine Linphone Web Browser Minimo Matchbox use flow flow flow flow flow FFMPEG audio codecs flow flow BlockLib V4L2 API flow flow ALSA SAA Driver flow SVA driver use SGA Driver use SGA user lib use FFMPEG video codecs EXA flow flow flow use Xvideo use Xserver use use use GTK+
flow
flow
flow
flow
AudioCodec
frame buffer
January 14, 2009
GStreamer and OpenMAX layers
cmp Static View APPLICATION
Gstreamer Based Player App
instantiate PCM_Data_Buffer flow
instantiate
FRAMEWORK
instantiate
instantiate
GSTREAMER:: GstAlsaSink
GSTREAMER_OMX: :GstOmxMp3Dec
GSTREAMER_OMX: :GstOmxMpeg4Dec
flow
GSTREAMER:: GstXVImageSink
OMX::OMX_Core
INTEGRATION
OMX:: OMX_AudioDecComponent
OMX:: OMX_VideoDecComponent
XVideo
ST_FFmpeg::ffmpeg libavcodec SAA User Lib::Blocklib
KERNEL
Linux DDrivers::ALSA driver
Linux DDrivers::SAA driver
Linux DDrivers::SVA driver
NOMADIK HW
Nomadik HW::Audio Codec (ex. 5095)
Nomadik HW:: SAA
Nomadik HW:: SVA
Nomadik HW:: Frame Buffer
January 14, 2009
Google Android on Nomadik
APPLICATIONS
Home Contacts Phone Browser ...
APPLICATION FRAMEWORK
Activity Manager Package Manager Window Manager Telephony Manager Content Providers Resource Manager View System Location Manager Notification Manager XMPP Service
LIBRARIES
Surface Manager OpenGL|ES SGL Media Framework FreeType SSL SQLite WebKit libc
ANDROID RUNTIME
Core libraries
Dalvik Virtual Machine
LINUX KERNEL
Display Driver USB Driver Camera Driver Keypad Driver Bluetooth Driver WiFi Driver Modem Driver Crypto Engine Driver Flash Memory Driver A/V Driver Binder (IPC) Driver Power Management
January 14, 2009
Multiple clipping video rendering
Minimise bandwidth usage when not needed!!!
R1
R2
R3
January 14, 2009
Thread Model
Play bin thread
gst player
Input file gst media parser
gst XVImageSink
gst-omx mpeg4 dec plugin
gst-omx mp3 dec plugin
gst ALSA sink
XVLib
omx mpeg4 dec component
omx mp3 dec component
ALSA lib
omx mpeg4 msg thread
OMX mpeg4 dec thread
OMX mp3 dec thread
omx mp3 msg thread
omx mpeg4 dec msg handler
omx mpeg4 dec buf manager
omx mp3 dec buf manager
omx mp3 dec msg handler
ffmpeg mpeg4 dec
ffmpeg mp3 dec
SAA block lib
Linux SVA driver
Linux SAA driver
January 14, 2009
Power Management
CPUfreq to scale voltage and frequency CPUIdle to switch amongst running, idle, sleep and deep sleep Custom API to control clock gating to all internal IPs Feasibility on-going with DPM
PowerTop
flow /sysfs
/sysfs
/sysfs
Kernel Space
CpuIdle use
cpufreq
Constraint Manager
Drivers and Bus Drivers
Driver Interface
Dynamic Tick (Wakeup event)
Governor
January 14, 2009
DVFS Dynamic Voltage Frequency Scaling
stm Operating Points PH1
100.8/100.8 MHz 201.6/100.8 MHz
1.0V
1.2V
264.0/132.0 MHz
1.2V
326.4/163.2 MHz Boot
1.2V
Initial
393.6/131.2 MHz 489.6/163.2 MHz
1.26V
1.45V
(lab only)
January 14, 2009
PM Dump Tool
PM monitor to optimise operating modes and ULP, estimate consumption
(Under development)
UART SDMMC MCDE TVOUT AUDIO SAA SVA BT WIFI LAN CPU 0 CPU 1
35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms
I (mA) V Clock
100 1.2 393
95 1.2 393
20 0.9 64
50 1 166
10 0.8 19
100 1.2 393
95 1.2 393
20 0.9 64
50 1 166
10 0.8 19
100 1.2 393
95 1.2 393
20 0.9 64
January 14, 2009
Thank you!
