RealTime
Content
What is a real-time system?
♦ A system that maintains a continuous and timely interaction with its environment
Cruise-Control System
Brake Accelerator Wheel Rotation Desired Speed Commands
Real-Time Systems
Throttle Value
Manas Saksena University of Pittsburgh
University of Pittsburgh
Manas Saksena
1
University of Pittsburgh
Manas Saksena
2
Real-Time Systems
♦ Real-time computer system
corrects depends on − when the results are produced (timeliness) in addition to − what results are produced
Timing Constraints
♦ Speed Control
sample sensor value (speed) every ‘100’ ms, − periodic activity − inter-arrival time (period) output actuator command within ‘60’ ms − deadline for each arrival
♦ Embedded System
The computer system is part of a larger system Examples: − automobile control system(s) − air-traffic control
Manas Saksena 3
♦ Brake Press
stop cruise control within ‘50’ ms − arrival rate: irregular − deadline
Manas Saksena 4
University of Pittsburgh
University of Pittsburgh
Hard vs Soft Real-Time
♦ Hard Real-Time
Missing deadlines means system failure Missing deadlines can cause serious loss Deadlines may be missed occasionally
Value Functions
♦ Safety Critical ♦ Soft Real-Time
value Soft real-time Hard real-time
♦ Most real systems have a mix of hard and soft real -time components ♦ The distinction between hard and soft real -time is somewhat subjective ♦ Soft real-time is not non real-time
deadline
University of Pittsburgh
Manas Saksena
5
University of Pittsburgh
Manas Saksena
6
1
Traditional Real-Time Applications
♦ Avionics ♦ Data Acquisition ♦ Digital Controllers ♦ Defense Systems ♦ Industrial Process Control
Emerging Embedded Real-Time Applications
♦ Cell-phones, VoIP phone, PDA’s ♦ MP3 players ♦ Set-top boxes, Game Consoles ♦ Automotive Systems ♦ Network Boxes ♦ On Demand Servers ♦ Embedded Devices
University of Pittsburgh
Manas Saksena
7
University of Pittsburgh
Manas Saksena
8
Real-time Control Systems Real-Time Software Design Basics
Controller
Reference input
A/D A/D
Control-raw computation
D/A
sensor
Plant
actuator
University of Pittsburgh
Manas Saksena
9
University of Pittsburgh
Manas Saksena
10
Software Structure: State Transition Diagrams
State Machines
initial state
Initialize Object
state machine state
initialization
external trigger? Wait for Event Handle Event
manual trigger
cruise/startControl()
Take actions ISR: to set/clear events Change system state
automatic
shutdown/
action expression final state
Terminate Object
timeout/doControl()
transition
University of Pittsburgh Manas Saksena 11 University of Pittsburgh Manas Saksena 12
2
State Transition Diagrams: Design
Schedulability Analysis
♦ Are my timing requirements being met? ♦ Real-Time System Model
manual
automatic
A set of events − Maximum arrival rate for each event A handler for each event − Maximum execution time for each handler A deadline for each event − Finish time (e) – Arrival time (e) <= Deadline
Response Time
University of Pittsburgh
Manas Saksena
13
University of Pittsburgh
Manas Saksena
14
Software Structure: Cyclic Executives
Real-Time System Software
A
B
C
D
A
B
C
D
♦ Concurrency
real-world events occur concurrently − multi-tasking simplifies software structure
¡ ♦ Replace Interrupts by Polling
More predictable behavior
♦ Static Cyclic Schedule ¡
Repeated at run-time
♦ Timed Events
need timers to trigger timed events − build timer support
♦ Implemented by a single loop ¢¤£¦¥¦§©¨
©¨¦¦ ©©
©¤
© !
©#"$
%
University of Pittsburgh Manas Saksena 15
♦ Timeliness of Response
Different events have different requirements How to structure software to satisfy all the requirements
Manas Saksena 16
University of Pittsburgh
Real-time System Development
Cross-Development for Embedded Systems
♦ Use the host to ¡ ¡ edit, compile, and build application programs
Target system Application tasks
Development host Compiler, debugger, loader, simulator, shell monitor NT OS (Solaris) Pentium PC station SUN workstation
configure the target
♦ At ¡ the target embedded system, use tools to
load, execute, debug, and monitor (performance and timing)
Real-time OS Input output
♦ Iterative process
PC Host + development tools RS-232 (console) application tasks RTOS (vxWorks) Target (MBX860)
Hardware
RS-232 Ethernet
Ethernet (load, debug, monitor)
University of Pittsburgh
Manas Saksena
17
University of Pittsburgh
Manas Saksena
18
3
Traditional Real-Time OS Requirements
Real-Time Operating Systems and Real-Time Software Design
♦ Multi-Tasking kernel
Fixed priority, preemptive scheduling Semaphores, Message Queues, etc. − Priority Inheritance
♦ Inter-task communication ♦ Timers ♦ Device Drivers ♦ Interrupt Management
Do this with low and predictable overheads.
Manas Saksena
University of Pittsburgh
Manas Saksena
19
University of Pittsburgh
Manas Saksena
20
Real-time OS
G Functions: &
task management, − scheduling, dispatcher − communication (pipe, queue) − synchronization (semaphore, event)
Schedulable Entities
Task’s Thread of Control Event Queue
' ' ' '
memory management time management device driver interrupt service
Application Kernel
External interrupt
Interrupt dispatch
Interrupt service Scheduling & dispatcher Task execution
Timer interrupt
Scheduler
Task Queue
Time service & events
Signal Queue
System calls (trap)
Services (create thread, sleep, notify, send,…)
kernel University of Pittsburgh Manas Saksena 21 University of Pittsburgh Manas Saksena 22
Real-Time OS: Examples
♦ Traditional Real-Time OSes
VxWorks, pSOS, QNX, LynxOS ThreadX, Ercos, Nucleus Solaris, NT, Linux
Response Time Factors
♦ Execution Time
Amount of time taken in the absence of resource contention
♦ Minimal Real-Time OSes ♦ General Purpose OS Made Real -Time
♦ Higher Priority Non Schedulable Entities
Interrupt Handlers Time taken by higher priority work Time taken by lower priority work − Often, due to contention on some shared resource
Manas Saksena 24
♦ Scheduling ♦ Shared Resources
♦ There are many other real-time operating systems. ♦ Many developers roll out their own real-time operating system
University of Pittsburgh
Manas Saksena
23
University of Pittsburgh
4
Scheduling
♦ Tasks
Primary scheduling entities Scheduled by the kernel Done within the task Usually makes sense with “event-driven” tasks
Real-Time Software Design
♦ Task Centric Design ¡
¡ Tasks are primary units for scheduling ¡ Application = Set of Tasks
Task Activation by signals
♦ Events or Messages
♦ Signals
Interrupts to tasks Useful for activating tasks (e.g., timer expiry)
♦ Event Centric Design ¡
¡ Events are primary units for scheduling ¡ Application = Set of Events and their Responses
Single Task or Multiple Tasks
University of Pittsburgh
Manas Saksena
25
University of Pittsburgh
Manas Saksena
26
Task Centric Design
Initialize Task Wait Control Read Inputs Perform Work Write Outputs
Event Centric Design - Single Task
♦ Event Scheduling Initialize
♦ Task Scheduling ¡ ¡ Preemptive or Non-Preemptive
Often, priority based mechanism
¡
¡ Non-Preemptive
Wait for Event Process Event Priority Based ♦ Event Priorities
♦ Task Priorities ¡
Application Controlled
¡
Application Controlled
Terminate Terminate Task
University of Pittsburgh Manas Saksena 27 University of Pittsburgh Manas Saksena 28
Event Centric Design - Multiple Tasks
Initialize Wait for Event Process Event Initialize Wait for Event Process Event
Task Centric Design
Terminate
Terminate
♦ Task Priorities and Scheduling ¡ ¡ Fixed, or Dynamic
Preemptive Scheduling
University of Pittsburgh Manas Saksena 29 University of Pittsburgh
Manas Saksena
Manas Saksena
30
5
Abstract Design Model
♦ Identify Events
Interrupts Timers
Implementing Tasks
♦ Two Common Ways
Interrupt Handlers Threads − Need an RTOS that supports threads (also called tasks in many RTOSes)
♦ Event Handlers
Code Segments − call these actions
task
♦ Interrupt Handlers
fast response times difficult to program correctly often IH will do some minimal work and then wake up a task to the real work
Manas Saksena 32
♦ Typically, mix of both
event action
University of Pittsburgh
Manas Saksena
31
University of Pittsburgh
Multi-Tasking
♦ Communications
Shared Memory − Semaphores/Mutexes to Control Access Messages − Message Queues
Key Concepts
♦ Tasks
Encapsulate thread of control Schedulable Unit Each task implements “one” function
♦ Scheduling
Priorities − IH may be viewed as higher priority tasks Preemptive Priority Scheduling − kernel is usually fully preemptive
University of Pittsburgh Manas Saksena 33
♦ Protected Objects
Encapsulate Shared Data (Resource) Mutually exclusive operation of Object Methods
University of Pittsburgh
Manas Saksena
34
Response Time Components
¡ ♦ Execution
Execution time of the task itself, running alone
Execution and Preemption
¡ ♦ Preemption
Execution delayed by higher priority tasks Execution delayed by lower priority tasks
HIGHER PRIORITY TASK P MY TASK PREEMPTION TIME BLOCKING TIME B
LOWER PRIORITY TASK
priority
¡ ♦ Blocking (Priority inversion)
time
University of Pittsburgh Manas Saksena 35 University of Pittsburgh Manas Saksena 36
6
Preemption Analysis
♦ Each higher priority task has a preemption effect ♦ Depends on Number of Arrivals and Execution Time of the Higher Priority Task
I j ( R ) = NumArrival s j ( R ) × C j
Number of Arrivals of Task “j” in a window of interval R Preemption Effect from Task “j”
University of Pittsburgh
Unbounded Priority Inversion
τ τ
high
S1
low
τ τ τ τ
High and low priority tasks share a resource – This can lead to UNBOUNDED Blocking
s1
highest
Attempts to lock S1
high medium s1 s1 s1
low
Response Time
Manas Saksena 37
Normal execution
s1
Execution in Critical Section
Unbounded Priority Inversion
University of Pittsburgh Manas Saksena 38
Highest Locker Protocol
τ τ
high
Priority Inheritance Protocol
τ τ
high
S1
low
τ τ τ τ
♦ High and low priority tasks share a critical section ♦ Ceiling Priority of CS = Highest Priority Task that can use the CS ♦ Execute CS at Ceiling priority
Ready s1 Ready
S1
low
♦ High and low priority tasks share a critical section ♦ Inherit priority of blocked task in critical section
highest
high medium s1 s1 s1
τ τ τ τ
highest
Attempts to lock S1
high Ready medium s1 s1 s1
s1
low
low
Normal execution
University of Pittsburgh
Execution in Critical Section
39
Normal execution
University of Pittsburgh
s1
Execution in Critical Section
40
Manas Saksena
Manas Saksena
Example of Chained Blocking
τ τ τ
1
Blocking Analysis
♦ Common Property
A lower priority task can cause blocking only while it is in a critical section
S1 S2
2
3
• • •
Same task set as sample problem Assumes priority inheritance used May need to wait for all possible critical sections
Attempts to lock S2 S1 S2
τ τ τ
Attempts to lock S1
1
S1 S1
♦ Highest Locker
The critical section must have a ceiling priority that is higher (or equal) than task priority Only one such CS can cause blocking
2
S2
3
S2
♦ Priority Inheritance Protocol
Multiple CS can cause blocking in a “chained manner” The effect is sum of the largest such chain
University of Pittsburgh Manas Saksena 42
Normal execution
S1 S2
Execution in Critical Section using S 1 Execution in Critical Section using S 2
41
University of Pittsburgh
Manas Saksena
7
Response Time Analysis
Ri = Ci + Bi +
Response Time Execution time of task
Design/Implementation Choices
♦ Non-Preemptive Scheduling ¡
Similar Analysis, but consider that − No preemption effect once task is given CPU − Blocking effect due to non-preemptive execution
(
j∈hp ( i )
I j ( Ri )
¡ ♦ Resource Contention Protocols
Different protocols have different blocking effects
Preemption from higher priority tasks Assumptions ♦ No Self Interference ♦ Bounded Blocking
♦ Priority Assignment ¡
Preemption and Blocking Effects Depend on Task Priorities
Blocking time from lower priority tasks (priority inversion)
Ri ≤ Ti
)
) Task Response times depend on all these three factors. ) These represent “design/implementation” choices.
43 University of Pittsburgh Manas Saksena
Functional Correctness does (should) not depend on the choice.
44
University of Pittsburgh
Manas Saksena
Why Linux?
Linux as a Real-Time Operating System
♦ Reliable, Full-featured Operating System ¡ ¡ Rich multi-tasking support ¡ Security, Protection
¡
Networking Support − TCP/IP, RSVP, SIP, MPLS, H.323 Multimedia Support − JPEG, MPEG, GSM Unix Lineage − Familiar environment for many users/developers POSIX Compliance
¡ ♦ Standard, Known Environment and API’s
Manas Saksena
¡
University of Pittsburgh
Manas Saksena
45
University of Pittsburgh
Manas Saksena
46
Why Linux?
♦ The Cost Factor
Free run-time royalties A global team of programmers enhancing the environment literally all the time Availability of libraries, tools, and device drivers Source Code Access allowing “peeking inside the hood” (and customizing as necessary)
Why Linux?
♦ Small Embedded Systems
Modular Kernel, possible to configure the kernel to suitable size Customizable Root File System BusyBox Utilities
♦ The Open Source Factor
♦ High-End Embedded Systems
High-Availability Clustering SMP Support
♦ The Popularity Factor
Excellent textbooks and documentation
Manas Saksena 47 University of Pittsburgh
University of Pittsburgh
Manas Saksena
48
8
Linux API: Tasking
♦ Process
Encapsulates a thread of control and an address space − Address space may be shared giving threads in effect Schedulable Entity
Linux API: POSIX, SVR4, BSD
¡ ♦ POSIX 1003.1.b (Real-Time Extensions) ¡ Priority Scheduling ¡ Memory Locking ¡ Clocks and Timers
Real-Time Signals
♦ Threads
Are processes to the Linux kernel − Scheduled by the Linux kernel Can be created such that they share the address space with the parent process, effectively giving threads
University of Pittsburgh Manas Saksena 49
¡ ♦ POSIX 1003.1.c (Thread Extensions) ¡ Using pthreads library ¡ Thread creation, destruction, etc.
Mutexes, Condition Variables
¡ ♦ SVR4 IPC ¡ Shared Memory
Semaphores
¡ ♦ Networking:
BSD Sockets
University of Pittsburgh Manas Saksena 50
Linux Internals Architecture
Linux Internals: Scheduling
♦ Schedulable Entities ¡
Processes − Real-Time Class: SCHED_FIFO or SCHED_RR − TimeSharing Class: SCHED_OTHER Real-Time processes have − Application defined priority − Higher priority than time-sharing processes
Modules
ipc
¡
mm Process Scheduler vfs
net
¡ ♦ Non Schedulable Entities ¡
Interrupt Handlers − Have priorities, and can be nested Bottom Halves & Task Queues − Run on schedule, ret from system call, ret from interrupt
Manas Saksena 52
Device Drivers
Core Mechanisms
University of Pittsburgh
Manas Saksena
51
University of Pittsburgh
Linux and Real-Time: Problems
♦ Timer Granularity
Many real-time tasks are driven by timer interrupts In Standard Linux, the timer is set to expire at 10 ms intervals
Linux Internals: Priority Inversion
♦ Non Preemptible Kernel
A process running in the kernel cannot be preempted by another (higher) priority process Introduces, unwanted priority inversion
♦ Scheduler Predictability
Linux scheduler keeps tasks in an unsorted list Requires a scan of all tasks to make a scheduling decision Scales poorly as number of tasks increases, and is especially poor for real-time performance
♦ Non Schedulable Entities
Interrupt Handlers Bottom Half Handlers No mechanism to bound priority inversion in the presence of resource sharing
Manas Saksena 54
♦ Resource Sharing
University of Pittsburgh
Manas Saksena
53
University of Pittsburgh
9
The Real-Time Linux Challenge
Approaches to Real-Time Linux
♦ Approaches limiting Real-time and Non Real-time Task Interactions ¡
Compliant Kernel Approach
¡
How to leverage the advantages of Linux, while making it suitable for real-time systems?
− LynxOS/Blue Cat Linux Thin Kernel Approach − RTLinux/RTAI
♦ Approaches that integrate Real-time and Non Realtime tasks ¡
Core Kernel Approach
¡
− TimeSys Linux/RT, Hard Hat Linux Resource Kernel Approach − TimeSys Linux/RT
University of Pittsburgh
Manas Saksena
55
University of Pittsburgh
Manas Saksena
56
Approaches to Real-Time Linux
Compliant Kernel Approach
Compliant Kernel Approach
Linux Development Tools And Environment Linux Development Tools And Environment
Dual Kernel Approach
Linux System Call API
Linux System Call API
Core Kernel Approach Linux Kernel
(Embedded Applications)
Real-Time Kernel (Real-Time Applications)
Resource Kernel Approach
University of Pittsburgh
Manas Saksena
57
University of Pittsburgh
Manas Saksena
58
Compliant Kernel Approach
♦ Basic Claim
Linux is defined by its API and not by its internal implementation The real-time kernel is a non Linux kernel
Compliance
♦ 100% Linux API
Support all of Linux kernel API Any Linux application can run on real -time kernel − Development can be done on Linux Host, with rich set of host tools for development All Linux libraries are trivially available to run on real-time kernel − Third party software Achieving 100% Linux API is non-trivial − Consider the amount of effort put on Linux kernel development
59 University of Pittsburgh Manas Saksena 60
♦ Implications
♦ Implications
No benefits from the Linux kernel Not possible to benefit from the Linux kernel evolution Not possible to use Linux hardware support Not possible to use Linux device drivers
University of Pittsburgh Manas Saksena
10
Approaches to Real-Time Linux
Compliant Kernel Approach
User-Level
The Thin Kernel Approach
Linux Process Linux Process
Dual Kernel Approach
Kernel-Level
Core Kernel Approach
Real-Time Task
Real-Time Task
Real-Time Task
Linux Kernel
Real-Time Kernel (RT-Linux or RTAI)
Resource Kernel Approach
Hardware
University of Pittsburgh
Manas Saksena
61
University of Pittsburgh
Manas Saksena
62
Linux Preemptability
♦ Linux kernel is run as the lowest priority task in the real-time kernel
Requires small changes to the Linux kernel
Programming Model
Real-Time Tasks Non Real-Time Tasks
♦ Linux kernel can be preempted at any time by the real-time kernel ♦ Real-Time kernel is a small multi -tasking kernel providing multi-tasking, inter-task communication and synchronization, and a fixed priority scheduler
Very tight response times for rt-tasks
University of Pittsburgh Manas Saksena 63
♦ Real-Time Tasks
Run in kernel Limited API, restricted to what is provided by the real-time kernel
♦ Non Real-Time Tasks
Run in user-space under Linux
Manas Saksena 64 University of Pittsburgh
Programming Model
♦ Memory Protection
Not available for real-time tasks A single errant real-time task can bring the system down
Soft and Dynamic Real-Time
♦ Static Decomposition into ¡
Real-time parts, and non real-time parts
♦ Duplication of Functionality
Currently: − Multi-tasking, task scheduling, inter-task communication and synchronization Future Extensibility − E.g., IP protocol stack, file system, …
♦ Consider ¡ ¡ Real-time often involves many shades of gray (soft real -time)
The soft/hardness of real-time may be dynamically decided
¡ ♦ Implications ¡
Support soft real-time tasks in real-time kernel − Limited API Support soft real-time tasks in Linux kernel − No performance guarantees
♦ Starvation of non real-time tasks
University of Pittsburgh Manas Saksena 65 University of Pittsburgh Manas Saksena 66
11
Approaches to Real-Time Linux
Compliant Kernel Approach
Core Kernel Approach
♦ Basic Ideas
Make the kernel more suitable for real-time Ensure that the impact of changes is localized so that − Kernel upgrades can be easily incorporated − Kernel reliability and scalability is not compromised
Dual Kernel Approach
Core Kernel Approach
♦ Mechanisms
Static Configuration − Can be configured at compile time Dynamic Configuration − Using loadable kernel modules
Manas Saksena 68
Resource Kernel Approach
University of Pittsburgh
Manas Saksena
67
University of Pittsburgh
Preemptive Kernel
♦ Linux kernel is non-preemptive
Simplifies synchronization in the kernel since kernel data structures that are not touched by interrupts need not be locked. Introduces potentially large priority inversions
Minimize Non-Schedulable Work
♦ Schedulable Entities
Processes or Threads Interrupt Handlers Bottom Halves
♦ Non Schedulable Entities
♦ But, Linux is already SMP safe
That is, the hooks needed to make it preemptive are already there. ♦ Solution Use SMP lock hooks to make the kernel preemptible
University of Pittsburgh Manas Saksena 69
♦ Solution Use schedulable entities (interrupt threads) to do the bulk of the work These can be prioritized according to application requirements
University of Pittsburgh Manas Saksena 70
Priority Inheritance
♦ A lower priority process holds a resource needed by a higher priority process
Can lead to unbounded priority inversion, unless some form of priority inheritance is used. ♦ Solution Provide a kernel mutex implementation that supports priority inheritance Use the mutex for locks in the kernel Export the mutex to user level processes using system calls
High Accuracy Timers
♦ Linux uses a periodic heartbeat timer (by default 10 ms) to implement timers.
Provides poor accuracy Increasing the frequency improves accuracy, but increases overheads
♦ Solution Use a one-shot “hardware” timer to support all software timers • Use a 10 ms periodic timer to do regular Linux timer work Can provide high accuracy at a low overhead
71 University of Pittsburgh Manas Saksena 72
University of Pittsburgh
Manas Saksena
12
Predictable Scheduler
♦ Current Linux Scheduler
Does not scale well as the number of processes increases
Core Kernel Approach
♦ Allows the use of most if not all existing Linux primitives, applications, and tools.
Need to avoid primitives that can take extended time in the kernel
♦ Real-Time Scheduler
Predictable, Low Overhead Separate real-time and non real-time − Non real-time processes do not effect real-time scheduling behavior Implement a better scheduler
University of Pittsburgh Manas Saksena 73
♦ Solution(s)
♦ Allows the use of most existing device drivers written to support Linux.
Need to avoid poorly written drivers that unfairly hog system resources
♦ Robustness and Reliability
Core kernel modifications can effect robustness, but source is available
Manas Saksena 74
University of Pittsburgh
URLs
♦ www.embedded.com
Embedded Systems Conference − papers available online
♦ http://cs-www.bu.edu/pub/ieee-rts/Home.html
IEEE Real-Time Technical Committee Page − many useful links
University of Pittsburgh
Manas Saksena
75
13
♦ A system that maintains a continuous and timely interaction with its environment
Cruise-Control System
Brake Accelerator Wheel Rotation Desired Speed Commands
Real-Time Systems
Throttle Value
Manas Saksena University of Pittsburgh
University of Pittsburgh
Manas Saksena
1
University of Pittsburgh
Manas Saksena
2
Real-Time Systems
♦ Real-time computer system
corrects depends on − when the results are produced (timeliness) in addition to − what results are produced
Timing Constraints
♦ Speed Control
sample sensor value (speed) every ‘100’ ms, − periodic activity − inter-arrival time (period) output actuator command within ‘60’ ms − deadline for each arrival
♦ Embedded System
The computer system is part of a larger system Examples: − automobile control system(s) − air-traffic control
Manas Saksena 3
♦ Brake Press
stop cruise control within ‘50’ ms − arrival rate: irregular − deadline
Manas Saksena 4
University of Pittsburgh
University of Pittsburgh
Hard vs Soft Real-Time
♦ Hard Real-Time
Missing deadlines means system failure Missing deadlines can cause serious loss Deadlines may be missed occasionally
Value Functions
♦ Safety Critical ♦ Soft Real-Time
value Soft real-time Hard real-time
♦ Most real systems have a mix of hard and soft real -time components ♦ The distinction between hard and soft real -time is somewhat subjective ♦ Soft real-time is not non real-time
deadline
University of Pittsburgh
Manas Saksena
5
University of Pittsburgh
Manas Saksena
6
1
Traditional Real-Time Applications
♦ Avionics ♦ Data Acquisition ♦ Digital Controllers ♦ Defense Systems ♦ Industrial Process Control
Emerging Embedded Real-Time Applications
♦ Cell-phones, VoIP phone, PDA’s ♦ MP3 players ♦ Set-top boxes, Game Consoles ♦ Automotive Systems ♦ Network Boxes ♦ On Demand Servers ♦ Embedded Devices
University of Pittsburgh
Manas Saksena
7
University of Pittsburgh
Manas Saksena
8
Real-time Control Systems Real-Time Software Design Basics
Controller
Reference input
A/D A/D
Control-raw computation
D/A
sensor
Plant
actuator
University of Pittsburgh
Manas Saksena
9
University of Pittsburgh
Manas Saksena
10
Software Structure: State Transition Diagrams
State Machines
initial state
Initialize Object
state machine state
initialization
external trigger? Wait for Event Handle Event
manual trigger
cruise/startControl()
Take actions ISR: to set/clear events Change system state
automatic
shutdown/
action expression final state
Terminate Object
timeout/doControl()
transition
University of Pittsburgh Manas Saksena 11 University of Pittsburgh Manas Saksena 12
2
State Transition Diagrams: Design
Schedulability Analysis
♦ Are my timing requirements being met? ♦ Real-Time System Model
manual
automatic
A set of events − Maximum arrival rate for each event A handler for each event − Maximum execution time for each handler A deadline for each event − Finish time (e) – Arrival time (e) <= Deadline
Response Time
University of Pittsburgh
Manas Saksena
13
University of Pittsburgh
Manas Saksena
14
Software Structure: Cyclic Executives
Real-Time System Software
A
B
C
D
A
B
C
D
♦ Concurrency
real-world events occur concurrently − multi-tasking simplifies software structure
¡ ♦ Replace Interrupts by Polling
More predictable behavior
♦ Static Cyclic Schedule ¡
Repeated at run-time
♦ Timed Events
need timers to trigger timed events − build timer support
♦ Implemented by a single loop ¢¤£¦¥¦§©¨
©¨¦¦ ©©
©¤
© !
©#"$
%
University of Pittsburgh Manas Saksena 15
♦ Timeliness of Response
Different events have different requirements How to structure software to satisfy all the requirements
Manas Saksena 16
University of Pittsburgh
Real-time System Development
Cross-Development for Embedded Systems
♦ Use the host to ¡ ¡ edit, compile, and build application programs
Target system Application tasks
Development host Compiler, debugger, loader, simulator, shell monitor NT OS (Solaris) Pentium PC station SUN workstation
configure the target
♦ At ¡ the target embedded system, use tools to
load, execute, debug, and monitor (performance and timing)
Real-time OS Input output
♦ Iterative process
PC Host + development tools RS-232 (console) application tasks RTOS (vxWorks) Target (MBX860)
Hardware
RS-232 Ethernet
Ethernet (load, debug, monitor)
University of Pittsburgh
Manas Saksena
17
University of Pittsburgh
Manas Saksena
18
3
Traditional Real-Time OS Requirements
Real-Time Operating Systems and Real-Time Software Design
♦ Multi-Tasking kernel
Fixed priority, preemptive scheduling Semaphores, Message Queues, etc. − Priority Inheritance
♦ Inter-task communication ♦ Timers ♦ Device Drivers ♦ Interrupt Management
Do this with low and predictable overheads.
Manas Saksena
University of Pittsburgh
Manas Saksena
19
University of Pittsburgh
Manas Saksena
20
Real-time OS
G Functions: &
task management, − scheduling, dispatcher − communication (pipe, queue) − synchronization (semaphore, event)
Schedulable Entities
Task’s Thread of Control Event Queue
' ' ' '
memory management time management device driver interrupt service
Application Kernel
External interrupt
Interrupt dispatch
Interrupt service Scheduling & dispatcher Task execution
Timer interrupt
Scheduler
Task Queue
Time service & events
Signal Queue
System calls (trap)
Services (create thread, sleep, notify, send,…)
kernel University of Pittsburgh Manas Saksena 21 University of Pittsburgh Manas Saksena 22
Real-Time OS: Examples
♦ Traditional Real-Time OSes
VxWorks, pSOS, QNX, LynxOS ThreadX, Ercos, Nucleus Solaris, NT, Linux
Response Time Factors
♦ Execution Time
Amount of time taken in the absence of resource contention
♦ Minimal Real-Time OSes ♦ General Purpose OS Made Real -Time
♦ Higher Priority Non Schedulable Entities
Interrupt Handlers Time taken by higher priority work Time taken by lower priority work − Often, due to contention on some shared resource
Manas Saksena 24
♦ Scheduling ♦ Shared Resources
♦ There are many other real-time operating systems. ♦ Many developers roll out their own real-time operating system
University of Pittsburgh
Manas Saksena
23
University of Pittsburgh
4
Scheduling
♦ Tasks
Primary scheduling entities Scheduled by the kernel Done within the task Usually makes sense with “event-driven” tasks
Real-Time Software Design
♦ Task Centric Design ¡
¡ Tasks are primary units for scheduling ¡ Application = Set of Tasks
Task Activation by signals
♦ Events or Messages
♦ Signals
Interrupts to tasks Useful for activating tasks (e.g., timer expiry)
♦ Event Centric Design ¡
¡ Events are primary units for scheduling ¡ Application = Set of Events and their Responses
Single Task or Multiple Tasks
University of Pittsburgh
Manas Saksena
25
University of Pittsburgh
Manas Saksena
26
Task Centric Design
Initialize Task Wait Control Read Inputs Perform Work Write Outputs
Event Centric Design - Single Task
♦ Event Scheduling Initialize
♦ Task Scheduling ¡ ¡ Preemptive or Non-Preemptive
Often, priority based mechanism
¡
¡ Non-Preemptive
Wait for Event Process Event Priority Based ♦ Event Priorities
♦ Task Priorities ¡
Application Controlled
¡
Application Controlled
Terminate Terminate Task
University of Pittsburgh Manas Saksena 27 University of Pittsburgh Manas Saksena 28
Event Centric Design - Multiple Tasks
Initialize Wait for Event Process Event Initialize Wait for Event Process Event
Task Centric Design
Terminate
Terminate
♦ Task Priorities and Scheduling ¡ ¡ Fixed, or Dynamic
Preemptive Scheduling
University of Pittsburgh Manas Saksena 29 University of Pittsburgh
Manas Saksena
Manas Saksena
30
5
Abstract Design Model
♦ Identify Events
Interrupts Timers
Implementing Tasks
♦ Two Common Ways
Interrupt Handlers Threads − Need an RTOS that supports threads (also called tasks in many RTOSes)
♦ Event Handlers
Code Segments − call these actions
task
♦ Interrupt Handlers
fast response times difficult to program correctly often IH will do some minimal work and then wake up a task to the real work
Manas Saksena 32
♦ Typically, mix of both
event action
University of Pittsburgh
Manas Saksena
31
University of Pittsburgh
Multi-Tasking
♦ Communications
Shared Memory − Semaphores/Mutexes to Control Access Messages − Message Queues
Key Concepts
♦ Tasks
Encapsulate thread of control Schedulable Unit Each task implements “one” function
♦ Scheduling
Priorities − IH may be viewed as higher priority tasks Preemptive Priority Scheduling − kernel is usually fully preemptive
University of Pittsburgh Manas Saksena 33
♦ Protected Objects
Encapsulate Shared Data (Resource) Mutually exclusive operation of Object Methods
University of Pittsburgh
Manas Saksena
34
Response Time Components
¡ ♦ Execution
Execution time of the task itself, running alone
Execution and Preemption
¡ ♦ Preemption
Execution delayed by higher priority tasks Execution delayed by lower priority tasks
HIGHER PRIORITY TASK P MY TASK PREEMPTION TIME BLOCKING TIME B
LOWER PRIORITY TASK
priority
¡ ♦ Blocking (Priority inversion)
time
University of Pittsburgh Manas Saksena 35 University of Pittsburgh Manas Saksena 36
6
Preemption Analysis
♦ Each higher priority task has a preemption effect ♦ Depends on Number of Arrivals and Execution Time of the Higher Priority Task
I j ( R ) = NumArrival s j ( R ) × C j
Number of Arrivals of Task “j” in a window of interval R Preemption Effect from Task “j”
University of Pittsburgh
Unbounded Priority Inversion
τ τ
high
S1
low
τ τ τ τ
High and low priority tasks share a resource – This can lead to UNBOUNDED Blocking
s1
highest
Attempts to lock S1
high medium s1 s1 s1
low
Response Time
Manas Saksena 37
Normal execution
s1
Execution in Critical Section
Unbounded Priority Inversion
University of Pittsburgh Manas Saksena 38
Highest Locker Protocol
τ τ
high
Priority Inheritance Protocol
τ τ
high
S1
low
τ τ τ τ
♦ High and low priority tasks share a critical section ♦ Ceiling Priority of CS = Highest Priority Task that can use the CS ♦ Execute CS at Ceiling priority
Ready s1 Ready
S1
low
♦ High and low priority tasks share a critical section ♦ Inherit priority of blocked task in critical section
highest
high medium s1 s1 s1
τ τ τ τ
highest
Attempts to lock S1
high Ready medium s1 s1 s1
s1
low
low
Normal execution
University of Pittsburgh
Execution in Critical Section
39
Normal execution
University of Pittsburgh
s1
Execution in Critical Section
40
Manas Saksena
Manas Saksena
Example of Chained Blocking
τ τ τ
1
Blocking Analysis
♦ Common Property
A lower priority task can cause blocking only while it is in a critical section
S1 S2
2
3
• • •
Same task set as sample problem Assumes priority inheritance used May need to wait for all possible critical sections
Attempts to lock S2 S1 S2
τ τ τ
Attempts to lock S1
1
S1 S1
♦ Highest Locker
The critical section must have a ceiling priority that is higher (or equal) than task priority Only one such CS can cause blocking
2
S2
3
S2
♦ Priority Inheritance Protocol
Multiple CS can cause blocking in a “chained manner” The effect is sum of the largest such chain
University of Pittsburgh Manas Saksena 42
Normal execution
S1 S2
Execution in Critical Section using S 1 Execution in Critical Section using S 2
41
University of Pittsburgh
Manas Saksena
7
Response Time Analysis
Ri = Ci + Bi +
Response Time Execution time of task
Design/Implementation Choices
♦ Non-Preemptive Scheduling ¡
Similar Analysis, but consider that − No preemption effect once task is given CPU − Blocking effect due to non-preemptive execution
(
j∈hp ( i )
I j ( Ri )
¡ ♦ Resource Contention Protocols
Different protocols have different blocking effects
Preemption from higher priority tasks Assumptions ♦ No Self Interference ♦ Bounded Blocking
♦ Priority Assignment ¡
Preemption and Blocking Effects Depend on Task Priorities
Blocking time from lower priority tasks (priority inversion)
Ri ≤ Ti
)
) Task Response times depend on all these three factors. ) These represent “design/implementation” choices.
43 University of Pittsburgh Manas Saksena
Functional Correctness does (should) not depend on the choice.
44
University of Pittsburgh
Manas Saksena
Why Linux?
Linux as a Real-Time Operating System
♦ Reliable, Full-featured Operating System ¡ ¡ Rich multi-tasking support ¡ Security, Protection
¡
Networking Support − TCP/IP, RSVP, SIP, MPLS, H.323 Multimedia Support − JPEG, MPEG, GSM Unix Lineage − Familiar environment for many users/developers POSIX Compliance
¡ ♦ Standard, Known Environment and API’s
Manas Saksena
¡
University of Pittsburgh
Manas Saksena
45
University of Pittsburgh
Manas Saksena
46
Why Linux?
♦ The Cost Factor
Free run-time royalties A global team of programmers enhancing the environment literally all the time Availability of libraries, tools, and device drivers Source Code Access allowing “peeking inside the hood” (and customizing as necessary)
Why Linux?
♦ Small Embedded Systems
Modular Kernel, possible to configure the kernel to suitable size Customizable Root File System BusyBox Utilities
♦ The Open Source Factor
♦ High-End Embedded Systems
High-Availability Clustering SMP Support
♦ The Popularity Factor
Excellent textbooks and documentation
Manas Saksena 47 University of Pittsburgh
University of Pittsburgh
Manas Saksena
48
8
Linux API: Tasking
♦ Process
Encapsulates a thread of control and an address space − Address space may be shared giving threads in effect Schedulable Entity
Linux API: POSIX, SVR4, BSD
¡ ♦ POSIX 1003.1.b (Real-Time Extensions) ¡ Priority Scheduling ¡ Memory Locking ¡ Clocks and Timers
Real-Time Signals
♦ Threads
Are processes to the Linux kernel − Scheduled by the Linux kernel Can be created such that they share the address space with the parent process, effectively giving threads
University of Pittsburgh Manas Saksena 49
¡ ♦ POSIX 1003.1.c (Thread Extensions) ¡ Using pthreads library ¡ Thread creation, destruction, etc.
Mutexes, Condition Variables
¡ ♦ SVR4 IPC ¡ Shared Memory
Semaphores
¡ ♦ Networking:
BSD Sockets
University of Pittsburgh Manas Saksena 50
Linux Internals Architecture
Linux Internals: Scheduling
♦ Schedulable Entities ¡
Processes − Real-Time Class: SCHED_FIFO or SCHED_RR − TimeSharing Class: SCHED_OTHER Real-Time processes have − Application defined priority − Higher priority than time-sharing processes
Modules
ipc
¡
mm Process Scheduler vfs
net
¡ ♦ Non Schedulable Entities ¡
Interrupt Handlers − Have priorities, and can be nested Bottom Halves & Task Queues − Run on schedule, ret from system call, ret from interrupt
Manas Saksena 52
Device Drivers
Core Mechanisms
University of Pittsburgh
Manas Saksena
51
University of Pittsburgh
Linux and Real-Time: Problems
♦ Timer Granularity
Many real-time tasks are driven by timer interrupts In Standard Linux, the timer is set to expire at 10 ms intervals
Linux Internals: Priority Inversion
♦ Non Preemptible Kernel
A process running in the kernel cannot be preempted by another (higher) priority process Introduces, unwanted priority inversion
♦ Scheduler Predictability
Linux scheduler keeps tasks in an unsorted list Requires a scan of all tasks to make a scheduling decision Scales poorly as number of tasks increases, and is especially poor for real-time performance
♦ Non Schedulable Entities
Interrupt Handlers Bottom Half Handlers No mechanism to bound priority inversion in the presence of resource sharing
Manas Saksena 54
♦ Resource Sharing
University of Pittsburgh
Manas Saksena
53
University of Pittsburgh
9
The Real-Time Linux Challenge
Approaches to Real-Time Linux
♦ Approaches limiting Real-time and Non Real-time Task Interactions ¡
Compliant Kernel Approach
¡
How to leverage the advantages of Linux, while making it suitable for real-time systems?
− LynxOS/Blue Cat Linux Thin Kernel Approach − RTLinux/RTAI
♦ Approaches that integrate Real-time and Non Realtime tasks ¡
Core Kernel Approach
¡
− TimeSys Linux/RT, Hard Hat Linux Resource Kernel Approach − TimeSys Linux/RT
University of Pittsburgh
Manas Saksena
55
University of Pittsburgh
Manas Saksena
56
Approaches to Real-Time Linux
Compliant Kernel Approach
Compliant Kernel Approach
Linux Development Tools And Environment Linux Development Tools And Environment
Dual Kernel Approach
Linux System Call API
Linux System Call API
Core Kernel Approach Linux Kernel
(Embedded Applications)
Real-Time Kernel (Real-Time Applications)
Resource Kernel Approach
University of Pittsburgh
Manas Saksena
57
University of Pittsburgh
Manas Saksena
58
Compliant Kernel Approach
♦ Basic Claim
Linux is defined by its API and not by its internal implementation The real-time kernel is a non Linux kernel
Compliance
♦ 100% Linux API
Support all of Linux kernel API Any Linux application can run on real -time kernel − Development can be done on Linux Host, with rich set of host tools for development All Linux libraries are trivially available to run on real-time kernel − Third party software Achieving 100% Linux API is non-trivial − Consider the amount of effort put on Linux kernel development
59 University of Pittsburgh Manas Saksena 60
♦ Implications
♦ Implications
No benefits from the Linux kernel Not possible to benefit from the Linux kernel evolution Not possible to use Linux hardware support Not possible to use Linux device drivers
University of Pittsburgh Manas Saksena
10
Approaches to Real-Time Linux
Compliant Kernel Approach
User-Level
The Thin Kernel Approach
Linux Process Linux Process
Dual Kernel Approach
Kernel-Level
Core Kernel Approach
Real-Time Task
Real-Time Task
Real-Time Task
Linux Kernel
Real-Time Kernel (RT-Linux or RTAI)
Resource Kernel Approach
Hardware
University of Pittsburgh
Manas Saksena
61
University of Pittsburgh
Manas Saksena
62
Linux Preemptability
♦ Linux kernel is run as the lowest priority task in the real-time kernel
Requires small changes to the Linux kernel
Programming Model
Real-Time Tasks Non Real-Time Tasks
♦ Linux kernel can be preempted at any time by the real-time kernel ♦ Real-Time kernel is a small multi -tasking kernel providing multi-tasking, inter-task communication and synchronization, and a fixed priority scheduler
Very tight response times for rt-tasks
University of Pittsburgh Manas Saksena 63
♦ Real-Time Tasks
Run in kernel Limited API, restricted to what is provided by the real-time kernel
♦ Non Real-Time Tasks
Run in user-space under Linux
Manas Saksena 64 University of Pittsburgh
Programming Model
♦ Memory Protection
Not available for real-time tasks A single errant real-time task can bring the system down
Soft and Dynamic Real-Time
♦ Static Decomposition into ¡
Real-time parts, and non real-time parts
♦ Duplication of Functionality
Currently: − Multi-tasking, task scheduling, inter-task communication and synchronization Future Extensibility − E.g., IP protocol stack, file system, …
♦ Consider ¡ ¡ Real-time often involves many shades of gray (soft real -time)
The soft/hardness of real-time may be dynamically decided
¡ ♦ Implications ¡
Support soft real-time tasks in real-time kernel − Limited API Support soft real-time tasks in Linux kernel − No performance guarantees
♦ Starvation of non real-time tasks
University of Pittsburgh Manas Saksena 65 University of Pittsburgh Manas Saksena 66
11
Approaches to Real-Time Linux
Compliant Kernel Approach
Core Kernel Approach
♦ Basic Ideas
Make the kernel more suitable for real-time Ensure that the impact of changes is localized so that − Kernel upgrades can be easily incorporated − Kernel reliability and scalability is not compromised
Dual Kernel Approach
Core Kernel Approach
♦ Mechanisms
Static Configuration − Can be configured at compile time Dynamic Configuration − Using loadable kernel modules
Manas Saksena 68
Resource Kernel Approach
University of Pittsburgh
Manas Saksena
67
University of Pittsburgh
Preemptive Kernel
♦ Linux kernel is non-preemptive
Simplifies synchronization in the kernel since kernel data structures that are not touched by interrupts need not be locked. Introduces potentially large priority inversions
Minimize Non-Schedulable Work
♦ Schedulable Entities
Processes or Threads Interrupt Handlers Bottom Halves
♦ Non Schedulable Entities
♦ But, Linux is already SMP safe
That is, the hooks needed to make it preemptive are already there. ♦ Solution Use SMP lock hooks to make the kernel preemptible
University of Pittsburgh Manas Saksena 69
♦ Solution Use schedulable entities (interrupt threads) to do the bulk of the work These can be prioritized according to application requirements
University of Pittsburgh Manas Saksena 70
Priority Inheritance
♦ A lower priority process holds a resource needed by a higher priority process
Can lead to unbounded priority inversion, unless some form of priority inheritance is used. ♦ Solution Provide a kernel mutex implementation that supports priority inheritance Use the mutex for locks in the kernel Export the mutex to user level processes using system calls
High Accuracy Timers
♦ Linux uses a periodic heartbeat timer (by default 10 ms) to implement timers.
Provides poor accuracy Increasing the frequency improves accuracy, but increases overheads
♦ Solution Use a one-shot “hardware” timer to support all software timers • Use a 10 ms periodic timer to do regular Linux timer work Can provide high accuracy at a low overhead
71 University of Pittsburgh Manas Saksena 72
University of Pittsburgh
Manas Saksena
12
Predictable Scheduler
♦ Current Linux Scheduler
Does not scale well as the number of processes increases
Core Kernel Approach
♦ Allows the use of most if not all existing Linux primitives, applications, and tools.
Need to avoid primitives that can take extended time in the kernel
♦ Real-Time Scheduler
Predictable, Low Overhead Separate real-time and non real-time − Non real-time processes do not effect real-time scheduling behavior Implement a better scheduler
University of Pittsburgh Manas Saksena 73
♦ Solution(s)
♦ Allows the use of most existing device drivers written to support Linux.
Need to avoid poorly written drivers that unfairly hog system resources
♦ Robustness and Reliability
Core kernel modifications can effect robustness, but source is available
Manas Saksena 74
University of Pittsburgh
URLs
♦ www.embedded.com
Embedded Systems Conference − papers available online
♦ http://cs-www.bu.edu/pub/ieee-rts/Home.html
IEEE Real-Time Technical Committee Page − many useful links
University of Pittsburgh
Manas Saksena
75
13
