1

CHAPTER 1 INTRODUCTION

In video surveillance, video signals from multiple remote locations are displayed on several TV screens which are typically placed together in a control room. In the so-called third generation surveillance systems (3GSS), all the parts of the surveillance systems will be digital and consequently, digital video will be transmitted and processed. Additionally, in 3GSS some 'intelligence' has to be introduced to detect relevant events in the video signals in an automatic way. This allows filtering of the irrelevant time segments of the video sequences and the displaying on the TV screen only those segments that require the attention of the surveillance operator. Motion detection is a basic operation in the selection of significant segments of the video signals. Once motion has been detected, other features can be considered to decide whether a video signal has to be presented to the surveillance operator. If the motion detection is performed after the transmission of the video signals from the cameras to the control room, then all the bit streams have to be previously decompressed; this can be a very demanding operation, especially if there are many cameras in the surveillance system. For this reason, it is interesting to consider the use of motion detection algorithms operating in the compressed (transform) domain. In this thesis we present a motion detection algorithm in the compressed domain with a low computational cost. In the following Section, we assume that video is compressed by using motion JPEG (MJPEG), i.e. each frame is individually JPEG compressed. Motion detection from a moving observer has been a very important technique for computer vision applications. Especially in recent years, for autonomous driving systems and driver supporting systems, vision-based navigation method has received more and more attention worldwide.

2

One of its most important tasks is to detect the moving obstacles like cars, bicycles or even pedestrians while the vehicle itself is running in a high speed. Methods of image differencing with the clear background or between adjacent frames are well used for the motion detection. But when the observer is also moving, which leads to the result of continuously changing background scene in the perspective projection image, it becomes more difficult to detect the real moving objects by differencing methods. To deal with this problem, many approaches have been proposed in recent years. Previous work in this area has been mainly in two categories: 1) Using the difference of optical flow vectors between background and the moving objects, 2) calibrating the background displacement by using camera’s 3D motion analysis result. Calculate the optical flow and estimate the flow vector’s reliability between adjacent frames. The major flow vector, which represents the motion of background, can be used to classify and extract the flow vectors of the real moving objects. However, by reason of its huge calculation cost and its difficulty for determining the accurate flow vectors, it is still unavailable for real applications. To analysis the camera’s 3D motion and calibrate the background is another main method for moving objects detection. For on-board camera’s motion analysis, many motion-detecting algorithms have been proposed which always depend on the previous recognition results like road lane-marks and horizon disappointing. These methods show some good performance in accuracy and efficiency because of their detailed analysis of road structure and measured vehicle locomotion, which is, however, computationally expensive and over-depended upon road features like lane-marks, and therefore lead to unsatisfied result when lane mark is covered by other vehicles or not exist at all. Compare with these previous works, a new method of moving objects detection from an on-board camera is presented in this paper. To deal with the background-change problem, our method uses camera’s 3D motion analysis results to calibrate the background scene. With pure points matching and the introduction of camera’s Focus of Expansion (FOE), our method is able to determine camera’s rotation and translation parameters theoretically by using only three pairs of matching points between adjacent frames, which make it faster and more efficient for real-time applications.

3

A neural network, also known as a parallel distributed processing network, is a computing paradigm that is loosely modeled after cortical structures of the brain. It consists of interconnected processing elements called nodes or neurons that work together to produce an output function. The output of a neural network relies on the cooperation of the individual neurons within the network to operate. Processing of information by neural networks is characteristically done in parallel rather than in series (or sequentially) as in earlier binary computers or Von Neumann machines. Since it relies on its member neurons collectively to perform its function, a unique property of a neural network is that it can still perform its overall function even if some of the neurons are not functioning. In other words it is robust to tolerate error or failure. All neural networks take numeric input and produce numeric output. The transfer function of a unit is typically chosen so that it can accept input in any range, and produces output in a strictly limited range (it has a squashing effect). An artificial neural network (ANN), also called a simulated neural network (SNN) or commonly just neural network (NN) is an interconnected group of artificial neurons that uses a mathematical or computational model for information processing based on a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. There are different topologies of neural networks that may be employed for time series modeling. In our investigation we used radial basis function networks which have shown considerably better scaling properties, when increasing the number of hidden units, than networks with sigmoid activation function. RBF networks were introduced into the neural network literature by Broom head/Lowe and Poggio/Girosi in the late 1980s. The RBF network model is motivated by the locally tuned response observed in biologic neurons, e.g. in the visual or in the auditory system. RBFs have been studied in multivariate approximation theory, particularly in the field of function interpolation. The RBF neural network model is an alternative to multilayer perceptron which is perhaps the

4

most often used neural network architecture. A radial basis function network (RBF), therefore, has a hidden layer of radial units, each actually modeling a Gaussian response surface. Since these functions are nonlinear, it is not actually necessary to have more than one hidden layer to model any shape of function: sufficient radial units will always be enough to model any function. In surveillance system estimation of motion is of great importance, which enables the various types of operations to be performed on the detected object. When using motion estimation, an assumption is made that the objects in the scene have only translational motion. This assumption holds as long as there is no camera pan, zoom, changes in luminance, or rotational motion (quite an assumption!). After the process of estimation, the detected motion has to be extracted. With the obtained boundary, two objects (with background) can then be extracted from two image frames (both current image frame and previous image frame). Extracting the moving object from its background can be done by the edge enhancement network and the background remover. In algorithm level, complexity, regularity and precision are main factors that directly affect the power consumed in extracting an algorithm for motion estimation. Concurrency and modularity are the requirements on algorithms that are intended to execute on low power architecture. This project aims to reduce the power consumption of motion estimation at algorithm level and architectural level by using neural network concept.

5

1.1

PROBLEM STATEMENT

The goals for this thesis have been the following. One goal has been to compile an introduction to the motion detection

algorithms. There exist a number of studies but complete reference on real time motion detection is not as common .we have collected materials from journals, papers and conferences and proposed approach that can be best to implement a real time motion detection. Another goal has been to search for algorithms that can be used to implement the RBF neural network. A third goal is to evaluate their performance with regard to motion detected. These properties were chosen because they have the greatest impact on the implementation effort. A final goal has been to design and implement an algorithm including object extraction. This should be done in high level language or matlab. The source code should be easy to understand so that it can serve as a reference on the standard for designers that need to implement real time motion detection.

6

CHAPTER 2 OVERVIEW OF NEURAL NETWORKS

Neural network theory is sometimes used to refer to a branch of computational science that uses neural networks as models to simulate or analyze complex phenomena and/or study the principles of operation of neural networks analytically. It addresses problems similar to artificial intelligence (AI) except that AI uses traditional computational algorithms to solve problems whereas neural networks use 'networks of agents' (software or hardware entities linked together) as the computational architecture to solve problems. Neural networks are trainable systems that can "learn" to solve complex problems from a set of exemplars and generalize the "acquired knowledge" to solve unforeseen problems as in stock market and environmental prediction. i.e., they are self-adaptive systems. Traditionally, the term neural network has been used to refer to a network of biological neurons. In modern usage, the term is often used to refer to artificial neural networks, which are composed of artificial neurons or nodes. Thus the term 'Neural Network' has two distinct connotations: 1. Biological neural networks are made up of real biological neurons that are connected or functionally-related in the peripheral nervous system or the central nervous system. In the field of neuroscience, they are often identified as groups of neurons that perform a specific physiological function in laboratory analysis.

2. Artificial neural networks are made up of interconnecting artificial neurons (usually simplified neurons) designed to model (or mimic) some properties of biological neural networks. Artificial neural networks can be used to model the modes of operation of biological neural networks, whereas cognitive models are theoretical models that mimic cognitive brain functions without

7

necessarily using neural networks while artificial intelligence are well-crafted algorithms that solve specific intelligent problems without using neural network as the computational architecture.

2.1 The brain, neural networks and computers

While it is accepted by most scientists that the brain is a type of computer, it is a computer with a vastly different architecture to the computers that most of us are familiar with. The brain is massively parallel, even more so than advanced multiprocessor computers. This means that simulating the behavior of a brain on traditional computer hardware is necessarily slow and inefficient. Neural networks, as used in artificial intelligence, have traditionally been viewed as simplified models of neural processing in the brain, even though the relation between this model and brain biological architecture is very much debated. To answer this question, David Marr has proposed various levels of analysis which provide us with a plausible answer for the role of neural networks in the understanding of human cognitive functioning. The question of what is the degree of complexity and the properties that individual neural elements should have in order to reproduce something resembling animal intelligence is a subject of current research in theoretical neuroscience. Historically computers evolved from von Neumann architecture, based on sequential processing and execution of explicit instructions. On the other hand origins of neural networks are based on efforts to model information processing in biological systems, which may rely largely on parallel processing as well as implicit instructions based on recognition of patterns of 'sensory' input from external sources. In other words, rather than sequential processing and execution, at their very heart, neural networks are complex statistic processors.

8

2.2 Artificial Neural networks

An artificial neural network (ANN), also called a simulated neural network (SNN) or commonly just neural network (NN) is an interconnected group of artificial neurons that uses a mathematical or computational model for information processing based on a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. In more practical terms neural networks are non-linear statistical data modeling tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data.

2.3 Background

An artificial neural network involves a network of simple processing elements (neurons) which can exhibit complex global behavior, determined by the connections between the processing elements and element parameters. One classical type of artificial neural network is the Hopfield net. In a neural network model, simple nodes (called variously "neurons", "neurodes", "PEs" ("processing elements") or "units") are connected together to form a network of nodes — hence the term "neural network". While a neural network does not have to be adaptive per se, its practical use comes with algorithms designed to alter the strength (weights) of the connections in the network to produce a desired signal flow. In modern software implementations of artificial neural networks the approach inspired by biology has more or less been abandoned for a more practical approach based on statistics and signal processing. In some of these systems neural networks or parts of neural networks (such as artificial neurons) are used as components in larger systems that combine both adaptive and non-adaptive elements.

9

2.4 Models

Neural network models in artificial intelligence are usually referred to as artificial neural networks (ANN); these are essentially simple mathematical models defining a function such functions. . Each type of ANN model corresponds to a class of

Fig 1 Artificial Neural Network

10

Fig 2 A complex neural network

2.5 Employing artificial neural networks

Perhaps the greatest advantage of ANN is their ability to be used as an arbitrary function approximation mechanism which 'learns' from observed data. However, using them is not so straightforward and a relatively good understanding of the underlying theory is essential.

•

Choice of model: This will depend on the data representation and the application. Overly complex models tend to lead to problems with learning. Learning algorithm: There are numerous tradeoffs between learning algorithms. Almost any algorithm will work well with the correct hyper parameters for training on a particular fixed dataset. However selecting and tuning an algorithm for training on unseen data requires a significant amount of experimentation.

•

•

Robustness: If the model, cost function and learning algorithm are selected appropriately the resulting ANN can be extremely robust.

With the correct implementation ANN can be used naturally in online learning and large dataset applications. Their simple implementation and the existence of mostly

11

local dependencies exhibited in the structure allows for fast, parallel implementations in hardware.

2.6 Types of neural networks 2.6.1 Feed forward neural network

The feed forward neural networks are the first and arguably simplest type of artificial neural networks devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

2.6.2 Single-layer perceptron

The earliest kind of neural network is a single-layer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. In this way it can be considered the simplest kind of feed-forward network. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called McCulloch-Pitts neurons or threshold neurons. A perceptron can be created using any values for the activated and deactivated states as long as the threshold value lies between the two. Most perceptrons have outputs of 1 or -1 with a threshold of 0 and there is some evidence that such networks can be trained more quickly than networks created from nodes with different activation and deactivation values. Perceptrons can be trained by a simple learning algorithm that is usually called the delta rule. It calculates the errors between calculated output and sample output data, and uses this to create an adjustment to the weights, thus implementing a form of gradient descent.

12

Single-unit perceptrons are only capable of learning linearly separable patterns; in 1969 in a famous monograph entitled Perceptrons Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

2.6.3 Multilayer layer perceptron

This class of networks consists of multiple layers of computational units, usually interconnected in a feed-forward way. Each neuron in one layer has directed connections to the neurons of the subsequent layer. In many applications the units of these networks apply a sigmoid function as an activation function. The universal approximation theorem for neural networks states that every continuous function that maps intervals of real numbers to some output interval of real numbers can be approximated arbitrarily closely by a multi-layer perceptron with just one hidden layer. This result holds only for restricted classes of activation functions, e.g. for the sigmoid functions. Multi-layer networks use a variety of learning techniques, the most popular being back-propagation. Here the output values are compared with the correct answer to compute the value of some predefined error-function. By various techniques the error is then fed back through the network. Using this information, the algorithm adjusts the weights of each connection in order to reduce the value of the error function by some small amount. After repeating this process for a sufficiently large number of training cycles the network will usually converge to some state where the error of the calculations is small. In this case one says that the network has learned a certain target function. To adjust weights properly one applies a general method for non-linear optimization that is called gradient descent. For this, the derivative of the error function with respect to the network weights is calculated and the weights are then changed such that the error decreases (thus going downhill on the surface of the error function). For this reason back-propagation can only be applied on networks with differentiable.

13

Fig 3 XOR perceptron A three layer Perceptron net capable of calculating XOR. The numbers within the perceptrons represent each perceptrons' explicit threshold. The numbers that annotate arrows represent the weight of the inputs. This net assumes that if the threshold is not reached, zero (not -1) is output. Note that the bottom layer of inputs is not always considered a real perceptron layer.

2.6.4 Radial basis function (RBF) network

Radial Basis Functions are powerful techniques for interpolation in multidimensional space. A RBF is a function which has built into a distance criterion with respect to a centre. Radial basis functions have been applied in the area of neural networks where they may be used as a replacement for the sigmoidal hidden layer transfer characteristic in multi-layer perceptrons.

2.6.5 Echo State Network

The Echo State Network (ESN) is a recurrent neural network with a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change and be learned. ESN are good to (re)produce temporal patterns.

14

2.6.6 Stochastic neural networks

A stochastic neural network differs from a regular neural network in the fact that it introduces random variations into the network. In a probabilistic view of neural networks, such random variations can be viewed as a form of statistical sampling, such as Monte Carlo sampling.

2.6.7 Neuro-fuzzy networks

A neuro-fuzzy network is a fuzzy inference system in the body of an artificial neural network. Depending on the FIS type, there are several layers that simulate the processes involved in a fuzzy inference like fuzzification, inference, aggregation and defuzzification. Embedding an FIS in a general structure of an ANN has the benefit of using available ANN training methods to find the parameters of a fuzzy system.

15

CHAPTER 3 RBF NETWORK

3.1 Radial Functions

Radial functions are a special class of function. Their characteristic feature is that their response decreases (or increases) monotonically with distance from a central point. The centre, the distance scale, and the precise shape of the radial function are parameters of the model, all fixed if it is linear. A typical radial function is the Gaussian which, in the case of a scalar input, is

Its parameters are its centre c and its radius r. The figure illustrates a Gaussian RBF with centre c = 0 and radius r = 1. A Gaussian RBF monotonically decreases with distance from the centre. In contrast, a multiquadric RBF which, in the case of scalar input, is

monotonically increases with distance from the centre. Gaussian-like RBFs are local (give a significant response only in a neighbourhood near the centre) and are more commonly used than multiquadric-type RBFs which have a global response.

3.2 Radial Networks

A RBF is a function which has built into a distance criterion with respect to a centre. Radial basis functions have been applied in the area of neural networks where they may be used as a replacement for the sigmoidal hidden layer transfer characteristic in multi-layer perceptrons. RBF networks have 2 layers of processing: In the first, input is mapped onto each RBF in the 'hidden' layer. The RBF chosen is usually a Gaussian. In regression problems the output layer is then a linear combination of hidden layer values representing mean predicted output. The

16

interpretation of this output layer value is the same as a regression model in statistics. In classification problems the output layer is typically a sigmoid function of a linear combination of hidden layer values, representing a posterior probability. Performance in both cases is often improved by shrinkage techniques, known as ridge regression in classical statistics and known to correspond to a prior belief in small parameter values (and therefore smooth output functions) in a Bayesian framework. RBF networks have the advantage of not suffering from local minima in the same way as multi-layer perceptrons. This is because the only parameters that are adjusted in the learning process are the linear mapping from hidden layer to output layer. Linearity ensures that the error surface is quadratic and therefore has a single easily found minimum. In regression problems this can be found in one matrix operation. In classification problems the fixed non-linearity introduced by the sigmoid output function is most efficiently dealt with using iterated reweighed least squares. RBF networks have the disadvantage of requiring good coverage of the input space by radial basis functions. RBF centers are determined with reference to the distribution of the input data, but without reference to the prediction task. As a result, representational resources may be wasted on areas of the input space that are irrelevant to the learning task. A common solution is to associate each data point with its own centre, although this can make the linear system to be solved in the final layer rather large, and requires shrinkage techniques to avoid over fitting. Associating each input datum with an RBF leads naturally to kernel methods such as Support Vector Machines and Gaussian Processes (the RBF is the kernel function). All three approaches use a non-linear kernel function to project the input data into a space where the learning problem can be solved using a linear model. Like Gaussian Processes, and unlike SVMs, RBF networks are typically trained in a Maximum Likelihood framework by maximizing the probability (minimizing the error) of the data under the model. SVMs take a different approach to avoiding over fitting by maximizing instead a margin. RBF networks are outperformed in most

17

classification applications by SVMs. In regression applications they can be competitive when the dimensionality of the input space is relatively small.

3.3 RBF Architecture

Artificial networks typically have three layers: an input layer, a hidden layer with a non-linear RBF activation function and a linear output layer. The output, , of the network is thus

Where N is the number of neurons in the hidden layer, ci is the center vector for neuron i, and ai are the weights of the linear output neuron. In the basic form all input are connected to each hidden neuron. The norm is typically taken to be the Euclidean distance and the basis function is taken to be Gaussian.

.

The Gaussian basis functions are local in the sense that far away from the center of that neuron. RBF networks are universal approximates on a compact subset of function with arbitrary precision. The weights ai, and the data.

.

Changing parameters of one neuron has only a small effect for input values that are

. This means

that a RBF network with enough hidden neurons can approximate any continuous

, and β are determined in a manner that optimizes the fit between

18

Fig 4 Architecture of a radial basis function network.

3.4 Training

In a RBF network there are three types of parameters that need to be chosen to adapt the network for a particular task: the center vectors ci, the output weights wi, and the RBF width parameters βi. In the sequential training of the weights are updated at each time step as data streams in. For some tasks it makes sense to define an objective function and select the parameter values that minimize its value. The most common objective function is the least squares function

Where,

.

19

We have explicitly included the dependence on the weights. Minimization of the least squares objective function by optimal choice of weights optimizes accuracy of fit.

3.5 Interpolation

RBF networks can be used to interpolate a function values of that function are known on finite number of when the points:

. Taking the known points xi to be the centers of the radial basis functions and evaluating the values of the basis functions at the same points gij = ρ( | | xj − xi | | ) the weights can be solved from the equation

It can be shown that the interpolation matrix in the above equation is non-singular, if the point’s x_i are distinct, and thus the weights w can be solved by simple linear algebra:

3.6 Function approximation

If the purpose is not to perform strict interpolation but instead more general function approximation or classification the optimization is somewhat more complex because there is no obvious choice for the centers. The training is typically done in two phases first fixing the width and centers and then the weights. This can be justified by considering the different nature of the non-linear hidden neurons versus the linear output neuron.

20

3.7 Training the basis function centers

Basis function centers can be either randomly sampled among the input instances or found by clustering the samples and choosing the cluster means as the centers. The RBF widths are usually all fixed to same value which is proportional to the maximum distance between the chosen centers.

3.8 Pseudoinverse solution for the linear weights

After the centers ci have been fixed, the weights that minimize the error at the output are computed with a linear pseudo inverse solution: , Where the entries of G are the values of the radial basis functions evaluated at the points xi: gji = ρ (| | xj − ci | |). The existence of this linear solution means that unlike Multi-layer perceptron (MLP) networks the RBF networks have a unique local minimum (when the centers are fixed).

3.9 Advantages/Disadvantages

• • • RBF trains faster than a MLP. Another advantage that is claimed is that the hidden layer is easier to interpret Although the RBF is quick to train, when training is finished and it is being

than the hidden layer in an MLP. used it is slower than a MLP, so where speed is a factor a MLP may be more appropriate.

21

CHAPTER 4 OVERVIEW OF MOTION DETECTION ALGORITHMS

Given a number of sequential video frames from the same source the goal is to detect the motion in the area observed by the source. When there is no motion all the sequential frames have to be similar up to noise influence. In the case when motion is present there is some difference between the frames. For sure, each lowcost system has some aspect of noise influence. And in case of no motion every two sequential frames will not be the identical. This is why the system must be smart enough to distinguish between noise and real motion. When the systems are calibrated and stable enough the character of noise is that every pixel value may be slightly different from that in other frame. And in first approximation it is possible to define some noise per pixel threshold parameter (adaptable for any given state) the meaning of which is how the pixel value (of the same oriented pixel in two sequential frames) might differ but actually the indicating value is the same one. More precisely, if the pixel with coordinates (Xa,Ya) in frame A differs from the pixel with coordinates (Xb,Yb) in frame B less than on TPP (threshold per pixel) value so we will see them as pixels with equal values. And we can write it by formulae: Pixel (Xa, Ya) equal to Pixel (Xb, Yb) I if {abs (Pixel (Xa,Ya)-Pixel(Xb,Yb)) < TPP } By adapting the TPP value to current system state we can make the system to be noise-stable. By applying this threshold operation to every pixel pair we may assume that all the preprocessed pixel values are noise-free. The element of noise that is not cancelled will be significantly small relative to other part. Ok, if so we have to post-process these values to detect the motion if any. As it was memorized above we have manipulate with different pixels inside two sequential frames to make conclusion about the motion.

22

Firstly, to make the system sensitive enough we have not to fix the TPP value too big. It mean that keeping the sensitivity of the system high in any two frames there will be some little number (TPP related) of different pixels. And in this case we have not to see them as noise. It is the first of the reasons to define a TPF (threshold per frame) value (adaptable for any given state) the meaning of which is how many pixels at least, inside two sequential frames must differ in order to see them as motion. The second reason to deal with TPF is to filter (to drop) small motion. For instance, by playing with TPF values we can neutralize motion of the small object (bugs etc.) by still detect the motion of people. And we can write the exact meaning of TPF by formulae: Let’s define NDPPP to be the Number of Different Pre-Processed by TPP Pixels. So, There is a motion i.e. NDPPP > TPF. Both of TPP and TPF values are variable through the UI to get the optimal system sensitivity. Also the TPF value has its visual equivalent and it is used as following. After the pixels pre-processing (by TPP) lets color all static (which do not include motion) pixels by lets say black color and all the dynamic (which indicate the motion) pixels will be left with their original color. This will bring the effect of motion extraction. In the other words, all the static parts of the frames will be black, and only the moving parts will be seen normally. The enabling/disabling of this effect is possible to control through the GUI. The Camera Manager provides routines for acquiring video frames from CCD cameras. Any process can request a video frame from any video source. The system manages a request queue for each source and executes them cyclically.

23

CHAPTER 5 PROPOSED DETECTION SYSTEM

This chapter presents the main software design and implementation issues. It starts by describing the general flow chart of the main program that was implemented in MATLAB. It then explains each component of the flow chart with some details. Finally it shows how the graphical user interface GUI was designed.

5.1 Basic Architecture

Fig 5 A basic architecture of surveillance system The above block diagram shows the surveillance system which consists of a camera system which monitors the particular area, a video daughter card which transmits the video signal to electrical signal, a network card which helps in connecting to a network and motion detection algorithm (SAD and Correlation) along with RBF network.

24

5.2 Main Program Flow Chart

The main task of the software was to read the still images recorded from the camera and then process these images to detect motions and take necessary actions accordingly. Figure 6 below shows the general flow chart of the main program.

Start

Setup & Initializations

Flag=1

What is Flag value?

Flag=0

Image Acquisition

Motion Detection Algorithm

Break & clear

No

Is image > threshold

Yes

Actions on Motion Detection

Data Record

Stop

Figure 6 Main Program Flow Diagram

25

It starts with general initialization of software parameters and objects setup. Then, once the program started, the flag value which indicates whether the stop button was pressed or not is checked. If the stop button was not pressed it start reading the images then process them using one of the two algorithms as the operator was selected. If a motion is detected it starts a series of actions and then it go back to read the next images, otherwise it goes directly to read the next images. Whenever the stop button is pressed the flag value will be set to zero and the program is stopped, memory is cleared and necessary results are recorded. This terminates the program and returns the control for the operator to collect the results. The next sections explain each process of the flow chart in figure 6 with some details.

5.2.1 Setup and Initializations

Start

Launch GUI

No

Start button pressed

Yes

Read Threshold Value

Read Algorithm Type

Setup Serial Port

Setup Video Object

Stop

Figure 7 Setup and Initializations Process

26

Figure 7 show the flow chart for the setup and initialization process. This process includes the launch of the graphical user interface (GUI) where the type of motion detection algorithm is selected and threshold value (the amount of sensitivity of the detection) is being initialized. Also, during this stage a setup process for both the serial port and the video object is done. This process takes approximately 15 seconds to be completed,(depending on the specifications of the PC used) for the serial port it starts by selecting a communication port and reserving the memory addresses for that port, then the PC connect to the device using the communication setting that was mentioned in the previous chapter. The video object is part of the image acquisition process but it should be setup at the start of the program.

5.2.2 Image acquisition

Start Read First Frame Convert to Grayscale

Read Second Frame

Convert to Grayscale Stop

Figure 8 Image acquisitions Process

27

After setup stage the image acquisition starts as shown in figure 8 above. This process reads images from the PC camera and save them in a format suitable for the motion detection algorithm. There were three possible options from which one is implemented. The first option was by using auto snapshots software that takes images automatically and save them on a hard disk as JPEG format, and then another program reads these images in the same sequence as they were saved. It was found that the maximum speed that can be attained by this software is one frame per second and this limits the speed of detection. Also, synchronization was required between both image processing and the auto snapshot software’s where next images need to be available on the hard disk before processing them. The second option was to display live video on the screen and then start capturing the images from the screen. This is a faster option from the previous approach but again it faced the problem of synchronization, when the computer monitor goes into a power saving mode where black images are produced all the time during the period of the black screen. The third option was by using the image acquisition toolbox provided in MATLAB 6.5.1 or higher versions. The image acquisition toolbox is a collection of functions that extend the capability of MATLAB. The toolbox supports a wide range of image acquisition operations, including acquiring images through many types of image acquisition devices, such as frame grabbers and USB PC cameras, also viewing a preview of the live video displayed on monitor and reading the image data into the MATLAB workspace directly. For this project video input function was used to initialize a video object that connects to the PC camera directly. Then preview function was used to display live video on the monitor. Get snapshot function was used to read images from the camera and place them in MATLAB workspace.

28

The later approach was implemented because it has many advantages over the others. It achieved the fastest capturing speed at a rate of five frames per seconds depending on algorithm complexity and PC processor speed. Furthermore, the problem of synchronization was solved because both capturing and processing of images were done using the same software. All read images were converted it into a two dimensional monochrome images. This is because equations in other algorithms in the system were designed with such image format.

5.2.3 Motion Detection Algorithm

A motion detection algorithm was applied on the previously read images. There were two approaches to implement motion detection algorithm. The first one was by using the two dimensional cross correlation while the second one was by using the sum of absolute difference algorithm. These are explained in details in the next two sub sections.

5.3 Motion Detection Using Sum of Absolute Difference (SAD)

This algorithm is based on image differencing techniques. mathematically represented using the following equation:

D (t ) = 1 N

It is

∑ I (t

i

) − I (t j )

Where N is the number of pixels in the image used as scaling factor,

I (t i ) is the image I at time i ,

I (t j )

is the image I at time j and

D (t ) is the normalized sum of absolute difference for that time.

In an ideal case when there is no motion

I (t i ) = I (t j )

29

and D(t ) = 0 . However noise is always presented in images and a better model of the images in the absence of motion will be

I (t i ) = I (t j ) + n( p )

Where n( p ) is a noise signal. The value D (t ) that represents the normalized sum of absolute difference can be used as a reference to be compared with a threshold value as shown in figure 9 below. The figure also shows a test case that contains a large change in the scene being monitored by the camera this was done by moving the camera. During the time before the camera was moved the SAD value was around 1.87 and when the camera was moved the SAD value was around 2.2. If the threshold for detection was fixed around the value less than 2.2 it will continuously detect motion after the camera stop moving.

Figure 9 Direct Thresholds for SAD Values

30

This approach solve the need for continuously re-estimate the threshold value. Choosing a threshold of 1*10-3 will detect the times when only the camera is moved. This results into a robust motion detection algorithm that can not be affected by illumination change and camera movements.

5.3.1 Actions on Motion Detection

Before explaining series of actions happen when motion is detected it is worth to mention that the values of variance that was calculated whether it was above or below the threshold will be stored in an array, where it will be used later to produce a plot of frame number Vs. the variance value. This plot helps in comparing the variance values against the threshold to be able to choose the optimum threshold value. Whenever the variance value is less than threshold the image will be dropped and only the variance value will be recorded. However when the variance value is greater than threshold sequence of actions is being started as shown in figure 10 below. Start

Time Date Frame#

Update Log File Display Image

Convert Image to Frame

Trigger Serial Port

Stop

Figure 10 Actions on Motion Detection

31

As the above flow chart show a number of activities happen when motion is detected. First the serial port is being triggered by a pulse from the PC; this pulse is used to activate external circuits connected to the PC. Also a log file is being created and then appended with information about the time and date of motion also the frame number in which motion occur is being recorded in the log file. Another process is to display the image that was detected on the monitor. Finally the image that was detected in motion will be converted to a movie frame and will be added to the film structure.

5.3.2 Break and clear Process

After motion detection algorithm applied on the images the program checks if the stop button on GUI was pressed. If it was pressed the flag value will be changed from one to zero and the program will break and terminate the loop then it will return the control to the GUI. Next both serial port object and video object will be cleared. This process is considered as a cleaning stage where the devises connected to the PC through those objects will be released and the memory space will be freed.

5.3.3 Data Record

Finally when the program is terminated a data collection process starts where variable and arrays that contain result of data on the memory will be stored on the hard disk. This approach was used to separate the real time image processing from results processing. This has the advantage of calling back these data whenever it is required. The variables that are being stored from memory to the hard disk are variance values and the movie structure that contain the entire frames with motion. At this point the control will be returned to the GUI where the operator can callback the results that where archived while the system was turned on. Next section will explain the design of the GUI highlighting each button results and callbacks.

32

START

IMAGE ACQUISTION FRAME SEPARATION

DIVIDE QUADRANTS SUM OF ABSOLUT DIFFERENCE

>T

DATA RECORD

Fig 11 Flow chart for SAD algoritham

Fig 12 Frame separation

Fig 13 Divide Quadrants

33

5.3.4 Graphical User Interface Design

The GUI was designed to facilitate interactive system operation. GUI can be used to setup the program, launch it, stop it and display results.

Start

Clear all Previous Work

Variable Initialization & Setup

Launch program

Call Selected main Program

Terminate Program

View Results

Yes

Start Again

NO

Exit

End

Figure 14 GUI flow Chart

34

During setup stage the operator is promoted to choose a motion detection algorithm and select degree of the detection sensitivity Whenever the start/stop toggle button is pressed the system will be launched and the selected program will be called to perform the calculations until the start/stop button is pressed again which will terminate the calculation and return control to GUI. Results can be viewed as a log file, movie and plot of frame number vs. variance value. Figure 14 illustrate a flow chart of the steps performed using the GUI.

5.4 Motion detection using Correlation Network

A correlation neural network (CNN) which accounts for velocity sensitive responses of neurons is suitable for analog circuit implementation of motiondetection systems and has been successfully implemented on CMOS. The CNN utilizes local motion detectors to correlate signals sampled at one location in the image with those sampled after a delay at adjacent locations; however, an edgedetection process is required in practical motion detection systems with the CNNs. The term correlation can also mean the cross-correlation of two functions or electron correlation in molecular systems. In probability theory and statistics, correlation, also called correlation coefficient, indicates the strength and direction of a linear relationship between two random variables. In general statistical usage, correlation or co-relation refers to the departure of two variables from independence, although correlation does not imply causation. In this broad sense there are several coefficients, measuring the degree of correlation, adapted to the nature of data. A number of different coefficients are used for different situations. The best known is the Pearson product-moment correlation coefficient, which is obtained by dividing the covariance of the two variables by the product of their standard deviations.

35

5.4.1 Mathematical properties

The correlation ρX, Y between two random variables X and Y with expected values μX and μY and standard deviations σX and σY is defined as:

Where E is the expected value of the variable and cov means covariance. Since μX = E(X), σX2 = E(X2) − E2(X) and likewise for Y, we may also write

The correlation is defined only if both of the standard deviations are finite and both of them are nonzero. It is a corollary of the Cauchy-Schwarz inequality that the correlation cannot exceed 1 in absolute value. The correlation is 1 in the case of an increasing linear relationship, −1 in the case of a decreasing linear relationship, and some value in between in all other cases, indicating the degree of linear dependence between the variables. The closer the coefficient is to either −1 or 1, the stronger the correlation between the variables. If the variables are independent then the correlation is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. Here is an example: Suppose the random variable X is uniformly distributed on the interval from −1 to 1, and Y = X2. Then Y is completely determined by X, so that X and Y are dependent, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, independence is equivalent to uncorrelated ness. A correlation between two variables is diluted in the presence of measurement error around estimates of one or both variables, in which case disattenuation provides a more accurate coefficient.

36

5.4.2 Geometric Interpretation of correlation

The correlation coefficient can also be viewed as the cosine of the angle between the two vectors of samples drawn from the two random variables. This method only works with centered data, i.e., data which have been shifted by the sample mean so as to have an average of zero. Some practitioners prefer an uncentered (non-Pearson-compliant) correlation coefficient. See the example below for a comparison. As an example, suppose five countries are found to have gross national products of 1, 2, 3, 5, and 8 billion dollars, respectively. Suppose these same five countries (in the same order) are found to have 11%, 12%, 13%, 15%, and 18% poverty. Then let x and y be ordered 5-element vectors containing the above data: x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18). By the usual procedure for finding the angle between two vectors, the uncentered correlation coefficient is:

Note that the above data were deliberately chosen to be perfectly correlated: y = 0.10 + 0.01 x. The Pearson correlation coefficient must therefore be exactly one. Centering the data (shifting x by E(x) = 3.8 and y by E(y) = 0.138) yields x = (-2.8, -1.8, -0.8, 1.2, 4.2) and y = (-0.028, -0.018, -0.008, 0.012, 0.042), from which

as expected.

37

5.4.3 Interpretation of the size of a correlation

Several authors have offered guidelines for the interpretation of a correlation coefficient. Cohen (1988), for example, has suggested the following interpretations for correlations in psychological research, in the table in the bottom. As Cohen himself has observed, however, all such criteria are in some ways arbitrary and should not be observed too strictly. This is because the interpretation of a correlation coefficient depends on the context and purposes. A correlation of 0.9 may be very low if one is verifying a physical law using high-quality instruments, but may be regarded as very high in the social sciences where there may be a greater contribution from complicating factors

Correlation Small Medium Large

Negative −0.29 to −0.10 −0.49 to −0.30 −1.00 to −0.50 Table 1

Positive 0.10 to 0.29 0.30 to 0.49 0.50 to 1.00

Fig 15 An unit network of two-dimensional CCN.

38

START

IMAGE ACQUISTION FRAME SEPARATION DIVIDE QUADRANTS

CORRELATION NETWORK

Decisi on

DATA RECORD

Fig 16 Flow chart for correlation

39

CHAPTER 6 PROPOSED OBJECT EXTRACTION

Many attempts have been made to extract data from video and film in a form suitable for use by animators and modelers. Such an approach is attractive, since motions and movements for people and animals may be obtained in this way that would be difficult using mechanical or magnetic motion capture systems. Visual extraction is also appealing since it is non-intrusive and has the potential to capture, from film, the motion and characteristics of people or animals long dead or extinct. Almost all attempts to perform visual extraction have been based around bespoke computer vision applications which are difficult for non-experts to use or adapt to their own needs. This paper presents a generic approach to extracting data from video. Whilst our approach allows low-level information to be extracted we show that higher-level functionality is available also. This functionality can be utilized in a manner that requires little knowledge of the underlying techniques and principles. Our approach is to approximate an image using principal component analysis, and then to train a multi-layer perceptron to predict the feature required by the user. This requires the user to hand-label the features of interest in some of the frames of the image sequence. One of the aims of this work is to keep to a minimum the number of frames that need to need labeled by the user. The trained multi-layer perceptron is then used to predict features for images that have never been labeled by the user. Other attempts to extract useful information from video sequences include the use of edge-detection and contour or edge tracking, template matching and template tracking. All such systems work well in some circumstances, but fail or require adaptation to meet the requirements of new users. For instance, in the case of template tracking, the user needs to be aware of the kinds of features that can be tracked well in an image and also choose a suitable template size. This is not a trivial task for non-specialists.

40

6.1 Method

The main steps in extraction using our system are detailed below: The user selects the sequence (or set) of images for which they wish data to be extracted from. This may well comprise of several shorter clips taken from different parts of a film. These images have some pre-processing performed on them (principal components analysis) to reduce each image to a small set of numbers. The user decides what feature(s) they wish to extract and labels this feature by hand in a fraction of the images chosen at random. The labeling process may involve clicking on a point to be tracked, labeling a distance or ratio of distances, measuring an angle, making a binary decision (yes/no, near/far etc.) or classifying the feature of interest into one of several classes. Once this ground-truth data is available, a neural network is trained to predict the feature values in images that have not been labeled by the user.

6.2 Feature Extraction

Principal components analysis (also known as eigenvector analysis) has been used extensively in computer vision for image reconstruction, pattern matching and classification. Given the i th image in a sequence of images, each of which consists of M pixels, we form the vector x i by concatenating the pixels of the image in raster scan order and removing the mean image of the sequence. The matrix X is created using the x i's as column vectors. Traditionally, the principal modes, qi, are extracted by computing X T i=iqi (1) Xq Where i's are the Eigen values.

41

a measure of the amount of variance each of the eigen vectors accounts for. Unfortunately, the matrix XXT is typically too large to manipulate since it is of size M by M. Such computation is wasteful anyway since only N princi pal modes are meaningful, where N is the number of example images. In all our work N M. Therefore we compute: X X =i ui(2) T u and we can obtain the qi's that we actually require using: q =X i i u (3)

In practice only the first P modes are used, P30 N. The principal mode extracted from a short film clip is shown in Figure 1 and is used later to help an animator to construct a cartoon version of the clip. It is tempting to think that such modes could be used directly to predict, say, the rotation of the man's shoulders. However, the second mode also encodes information about shoulder movement and it is only by combining information from many modes that rotation can be reliably predicted.

42

REFERENCE;

[1] ' Special issue on third generation surveillance systems', froc. IEEE, 2001, 89, JAIN, R., KASTURI, R., and SCHUNCK, B.G. This paper gives the detailks about the surveillance systems [2] 'Machine vision' (McGraw-Hill Inc., 1995) PONS, J., PRADES-NEBOT, J., ALBIOL, A,, and MOLINA, J.his paper provides the details about intelligence. [3] 'Motion video sensor in the compressed domain'. SCS Euromedia Conf., Valencia, Spain, 2001, This paper provides the details about compressed domain. [4] Y. Song, A perceptual approach to human motion detection and labeling. PhD thesis, California Institute of Technology, 2003. This paper provides the details about human motion detection [5] N. Howe, M. Leventon, and W. Freeman, “Bayesian reconstruction of 3D human motion from single-camera video,” Tech. Rep. TR-99-37, Mitsubishi Electric Research Lab, 1999 This paper provides the details about 3d human detection. [6] L. Goncalves, E. D. Bernardo, E. Ursella, and P. Perona, “Monocular tracking of the human arm in 3D,” in Proc. 5th Int. Conf. Computer Vision, (Cambridge, Mass), pp. 764– 770, 1995.This paper provides the details about 3d human detection. [7] S. Wachter and H.-H. Nagel, “Tracking persons in monocular image sequences,” Computer Vision and Image Understanding, vol. 74, pp. 174–192, 1999. This paper provides details about motion detection in image sequences. algorithms in artificient

43

44

CHAPTER 1 INTRODUCTION

In video surveillance, video signals from multiple remote locations are displayed on several TV screens which are typically placed together in a control room. In the so-called third generation surveillance systems (3GSS), all the parts of the surveillance systems will be digital and consequently, digital video will be transmitted and processed. Additionally, in 3GSS some 'intelligence' has to be introduced to detect relevant events in the video signals in an automatic way. This allows filtering of the irrelevant time segments of the video sequences and the displaying on the TV screen only those segments that require the attention of the surveillance operator. Motion detection is a basic operation in the selection of significant segments of the video signals. Once motion has been detected, other features can be considered to decide whether a video signal has to be presented to the surveillance operator. If the motion detection is performed after the transmission of the video signals from the cameras to the control room, then all the bit streams have to be previously decompressed; this can be a very demanding operation, especially if there are many cameras in the surveillance system. For this reason, it is interesting to consider the use of motion detection algorithms operating in the compressed (transform) domain. In this thesis we present a motion detection algorithm in the compressed domain with a low computational cost. In the following Section, we assume that video is compressed by using motion JPEG (MJPEG), i.e. each frame is individually JPEG compressed. Motion detection from a moving observer has been a very important technique for computer vision applications. Especially in recent years, for autonomous driving systems and driver supporting systems, vision-based navigation method has received more and more attention worldwide.

2

One of its most important tasks is to detect the moving obstacles like cars, bicycles or even pedestrians while the vehicle itself is running in a high speed. Methods of image differencing with the clear background or between adjacent frames are well used for the motion detection. But when the observer is also moving, which leads to the result of continuously changing background scene in the perspective projection image, it becomes more difficult to detect the real moving objects by differencing methods. To deal with this problem, many approaches have been proposed in recent years. Previous work in this area has been mainly in two categories: 1) Using the difference of optical flow vectors between background and the moving objects, 2) calibrating the background displacement by using camera’s 3D motion analysis result. Calculate the optical flow and estimate the flow vector’s reliability between adjacent frames. The major flow vector, which represents the motion of background, can be used to classify and extract the flow vectors of the real moving objects. However, by reason of its huge calculation cost and its difficulty for determining the accurate flow vectors, it is still unavailable for real applications. To analysis the camera’s 3D motion and calibrate the background is another main method for moving objects detection. For on-board camera’s motion analysis, many motion-detecting algorithms have been proposed which always depend on the previous recognition results like road lane-marks and horizon disappointing. These methods show some good performance in accuracy and efficiency because of their detailed analysis of road structure and measured vehicle locomotion, which is, however, computationally expensive and over-depended upon road features like lane-marks, and therefore lead to unsatisfied result when lane mark is covered by other vehicles or not exist at all. Compare with these previous works, a new method of moving objects detection from an on-board camera is presented in this paper. To deal with the background-change problem, our method uses camera’s 3D motion analysis results to calibrate the background scene. With pure points matching and the introduction of camera’s Focus of Expansion (FOE), our method is able to determine camera’s rotation and translation parameters theoretically by using only three pairs of matching points between adjacent frames, which make it faster and more efficient for real-time applications.

3

A neural network, also known as a parallel distributed processing network, is a computing paradigm that is loosely modeled after cortical structures of the brain. It consists of interconnected processing elements called nodes or neurons that work together to produce an output function. The output of a neural network relies on the cooperation of the individual neurons within the network to operate. Processing of information by neural networks is characteristically done in parallel rather than in series (or sequentially) as in earlier binary computers or Von Neumann machines. Since it relies on its member neurons collectively to perform its function, a unique property of a neural network is that it can still perform its overall function even if some of the neurons are not functioning. In other words it is robust to tolerate error or failure. All neural networks take numeric input and produce numeric output. The transfer function of a unit is typically chosen so that it can accept input in any range, and produces output in a strictly limited range (it has a squashing effect). An artificial neural network (ANN), also called a simulated neural network (SNN) or commonly just neural network (NN) is an interconnected group of artificial neurons that uses a mathematical or computational model for information processing based on a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. There are different topologies of neural networks that may be employed for time series modeling. In our investigation we used radial basis function networks which have shown considerably better scaling properties, when increasing the number of hidden units, than networks with sigmoid activation function. RBF networks were introduced into the neural network literature by Broom head/Lowe and Poggio/Girosi in the late 1980s. The RBF network model is motivated by the locally tuned response observed in biologic neurons, e.g. in the visual or in the auditory system. RBFs have been studied in multivariate approximation theory, particularly in the field of function interpolation. The RBF neural network model is an alternative to multilayer perceptron which is perhaps the

4

most often used neural network architecture. A radial basis function network (RBF), therefore, has a hidden layer of radial units, each actually modeling a Gaussian response surface. Since these functions are nonlinear, it is not actually necessary to have more than one hidden layer to model any shape of function: sufficient radial units will always be enough to model any function. In surveillance system estimation of motion is of great importance, which enables the various types of operations to be performed on the detected object. When using motion estimation, an assumption is made that the objects in the scene have only translational motion. This assumption holds as long as there is no camera pan, zoom, changes in luminance, or rotational motion (quite an assumption!). After the process of estimation, the detected motion has to be extracted. With the obtained boundary, two objects (with background) can then be extracted from two image frames (both current image frame and previous image frame). Extracting the moving object from its background can be done by the edge enhancement network and the background remover. In algorithm level, complexity, regularity and precision are main factors that directly affect the power consumed in extracting an algorithm for motion estimation. Concurrency and modularity are the requirements on algorithms that are intended to execute on low power architecture. This project aims to reduce the power consumption of motion estimation at algorithm level and architectural level by using neural network concept.

5

1.1

PROBLEM STATEMENT

The goals for this thesis have been the following. One goal has been to compile an introduction to the motion detection

algorithms. There exist a number of studies but complete reference on real time motion detection is not as common .we have collected materials from journals, papers and conferences and proposed approach that can be best to implement a real time motion detection. Another goal has been to search for algorithms that can be used to implement the RBF neural network. A third goal is to evaluate their performance with regard to motion detected. These properties were chosen because they have the greatest impact on the implementation effort. A final goal has been to design and implement an algorithm including object extraction. This should be done in high level language or matlab. The source code should be easy to understand so that it can serve as a reference on the standard for designers that need to implement real time motion detection.

6

CHAPTER 2 OVERVIEW OF NEURAL NETWORKS

Neural network theory is sometimes used to refer to a branch of computational science that uses neural networks as models to simulate or analyze complex phenomena and/or study the principles of operation of neural networks analytically. It addresses problems similar to artificial intelligence (AI) except that AI uses traditional computational algorithms to solve problems whereas neural networks use 'networks of agents' (software or hardware entities linked together) as the computational architecture to solve problems. Neural networks are trainable systems that can "learn" to solve complex problems from a set of exemplars and generalize the "acquired knowledge" to solve unforeseen problems as in stock market and environmental prediction. i.e., they are self-adaptive systems. Traditionally, the term neural network has been used to refer to a network of biological neurons. In modern usage, the term is often used to refer to artificial neural networks, which are composed of artificial neurons or nodes. Thus the term 'Neural Network' has two distinct connotations: 1. Biological neural networks are made up of real biological neurons that are connected or functionally-related in the peripheral nervous system or the central nervous system. In the field of neuroscience, they are often identified as groups of neurons that perform a specific physiological function in laboratory analysis.

2. Artificial neural networks are made up of interconnecting artificial neurons (usually simplified neurons) designed to model (or mimic) some properties of biological neural networks. Artificial neural networks can be used to model the modes of operation of biological neural networks, whereas cognitive models are theoretical models that mimic cognitive brain functions without

7

necessarily using neural networks while artificial intelligence are well-crafted algorithms that solve specific intelligent problems without using neural network as the computational architecture.

2.1 The brain, neural networks and computers

While it is accepted by most scientists that the brain is a type of computer, it is a computer with a vastly different architecture to the computers that most of us are familiar with. The brain is massively parallel, even more so than advanced multiprocessor computers. This means that simulating the behavior of a brain on traditional computer hardware is necessarily slow and inefficient. Neural networks, as used in artificial intelligence, have traditionally been viewed as simplified models of neural processing in the brain, even though the relation between this model and brain biological architecture is very much debated. To answer this question, David Marr has proposed various levels of analysis which provide us with a plausible answer for the role of neural networks in the understanding of human cognitive functioning. The question of what is the degree of complexity and the properties that individual neural elements should have in order to reproduce something resembling animal intelligence is a subject of current research in theoretical neuroscience. Historically computers evolved from von Neumann architecture, based on sequential processing and execution of explicit instructions. On the other hand origins of neural networks are based on efforts to model information processing in biological systems, which may rely largely on parallel processing as well as implicit instructions based on recognition of patterns of 'sensory' input from external sources. In other words, rather than sequential processing and execution, at their very heart, neural networks are complex statistic processors.

8

2.2 Artificial Neural networks

An artificial neural network (ANN), also called a simulated neural network (SNN) or commonly just neural network (NN) is an interconnected group of artificial neurons that uses a mathematical or computational model for information processing based on a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. In more practical terms neural networks are non-linear statistical data modeling tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data.

2.3 Background

An artificial neural network involves a network of simple processing elements (neurons) which can exhibit complex global behavior, determined by the connections between the processing elements and element parameters. One classical type of artificial neural network is the Hopfield net. In a neural network model, simple nodes (called variously "neurons", "neurodes", "PEs" ("processing elements") or "units") are connected together to form a network of nodes — hence the term "neural network". While a neural network does not have to be adaptive per se, its practical use comes with algorithms designed to alter the strength (weights) of the connections in the network to produce a desired signal flow. In modern software implementations of artificial neural networks the approach inspired by biology has more or less been abandoned for a more practical approach based on statistics and signal processing. In some of these systems neural networks or parts of neural networks (such as artificial neurons) are used as components in larger systems that combine both adaptive and non-adaptive elements.

9

2.4 Models

Neural network models in artificial intelligence are usually referred to as artificial neural networks (ANN); these are essentially simple mathematical models defining a function such functions. . Each type of ANN model corresponds to a class of

Fig 1 Artificial Neural Network

10

Fig 2 A complex neural network

2.5 Employing artificial neural networks

Perhaps the greatest advantage of ANN is their ability to be used as an arbitrary function approximation mechanism which 'learns' from observed data. However, using them is not so straightforward and a relatively good understanding of the underlying theory is essential.

•

Choice of model: This will depend on the data representation and the application. Overly complex models tend to lead to problems with learning. Learning algorithm: There are numerous tradeoffs between learning algorithms. Almost any algorithm will work well with the correct hyper parameters for training on a particular fixed dataset. However selecting and tuning an algorithm for training on unseen data requires a significant amount of experimentation.

•

•

Robustness: If the model, cost function and learning algorithm are selected appropriately the resulting ANN can be extremely robust.

With the correct implementation ANN can be used naturally in online learning and large dataset applications. Their simple implementation and the existence of mostly

11

local dependencies exhibited in the structure allows for fast, parallel implementations in hardware.

2.6 Types of neural networks 2.6.1 Feed forward neural network

The feed forward neural networks are the first and arguably simplest type of artificial neural networks devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

2.6.2 Single-layer perceptron

The earliest kind of neural network is a single-layer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. In this way it can be considered the simplest kind of feed-forward network. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called McCulloch-Pitts neurons or threshold neurons. A perceptron can be created using any values for the activated and deactivated states as long as the threshold value lies between the two. Most perceptrons have outputs of 1 or -1 with a threshold of 0 and there is some evidence that such networks can be trained more quickly than networks created from nodes with different activation and deactivation values. Perceptrons can be trained by a simple learning algorithm that is usually called the delta rule. It calculates the errors between calculated output and sample output data, and uses this to create an adjustment to the weights, thus implementing a form of gradient descent.

12

Single-unit perceptrons are only capable of learning linearly separable patterns; in 1969 in a famous monograph entitled Perceptrons Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

2.6.3 Multilayer layer perceptron

This class of networks consists of multiple layers of computational units, usually interconnected in a feed-forward way. Each neuron in one layer has directed connections to the neurons of the subsequent layer. In many applications the units of these networks apply a sigmoid function as an activation function. The universal approximation theorem for neural networks states that every continuous function that maps intervals of real numbers to some output interval of real numbers can be approximated arbitrarily closely by a multi-layer perceptron with just one hidden layer. This result holds only for restricted classes of activation functions, e.g. for the sigmoid functions. Multi-layer networks use a variety of learning techniques, the most popular being back-propagation. Here the output values are compared with the correct answer to compute the value of some predefined error-function. By various techniques the error is then fed back through the network. Using this information, the algorithm adjusts the weights of each connection in order to reduce the value of the error function by some small amount. After repeating this process for a sufficiently large number of training cycles the network will usually converge to some state where the error of the calculations is small. In this case one says that the network has learned a certain target function. To adjust weights properly one applies a general method for non-linear optimization that is called gradient descent. For this, the derivative of the error function with respect to the network weights is calculated and the weights are then changed such that the error decreases (thus going downhill on the surface of the error function). For this reason back-propagation can only be applied on networks with differentiable.

13

Fig 3 XOR perceptron A three layer Perceptron net capable of calculating XOR. The numbers within the perceptrons represent each perceptrons' explicit threshold. The numbers that annotate arrows represent the weight of the inputs. This net assumes that if the threshold is not reached, zero (not -1) is output. Note that the bottom layer of inputs is not always considered a real perceptron layer.

2.6.4 Radial basis function (RBF) network

Radial Basis Functions are powerful techniques for interpolation in multidimensional space. A RBF is a function which has built into a distance criterion with respect to a centre. Radial basis functions have been applied in the area of neural networks where they may be used as a replacement for the sigmoidal hidden layer transfer characteristic in multi-layer perceptrons.

2.6.5 Echo State Network

The Echo State Network (ESN) is a recurrent neural network with a sparsely connected random hidden layer. The weights of output neurons are the only part of the network that can change and be learned. ESN are good to (re)produce temporal patterns.

14

2.6.6 Stochastic neural networks

A stochastic neural network differs from a regular neural network in the fact that it introduces random variations into the network. In a probabilistic view of neural networks, such random variations can be viewed as a form of statistical sampling, such as Monte Carlo sampling.

2.6.7 Neuro-fuzzy networks

A neuro-fuzzy network is a fuzzy inference system in the body of an artificial neural network. Depending on the FIS type, there are several layers that simulate the processes involved in a fuzzy inference like fuzzification, inference, aggregation and defuzzification. Embedding an FIS in a general structure of an ANN has the benefit of using available ANN training methods to find the parameters of a fuzzy system.

15

CHAPTER 3 RBF NETWORK

3.1 Radial Functions

Radial functions are a special class of function. Their characteristic feature is that their response decreases (or increases) monotonically with distance from a central point. The centre, the distance scale, and the precise shape of the radial function are parameters of the model, all fixed if it is linear. A typical radial function is the Gaussian which, in the case of a scalar input, is

Its parameters are its centre c and its radius r. The figure illustrates a Gaussian RBF with centre c = 0 and radius r = 1. A Gaussian RBF monotonically decreases with distance from the centre. In contrast, a multiquadric RBF which, in the case of scalar input, is

monotonically increases with distance from the centre. Gaussian-like RBFs are local (give a significant response only in a neighbourhood near the centre) and are more commonly used than multiquadric-type RBFs which have a global response.

3.2 Radial Networks

A RBF is a function which has built into a distance criterion with respect to a centre. Radial basis functions have been applied in the area of neural networks where they may be used as a replacement for the sigmoidal hidden layer transfer characteristic in multi-layer perceptrons. RBF networks have 2 layers of processing: In the first, input is mapped onto each RBF in the 'hidden' layer. The RBF chosen is usually a Gaussian. In regression problems the output layer is then a linear combination of hidden layer values representing mean predicted output. The

16

interpretation of this output layer value is the same as a regression model in statistics. In classification problems the output layer is typically a sigmoid function of a linear combination of hidden layer values, representing a posterior probability. Performance in both cases is often improved by shrinkage techniques, known as ridge regression in classical statistics and known to correspond to a prior belief in small parameter values (and therefore smooth output functions) in a Bayesian framework. RBF networks have the advantage of not suffering from local minima in the same way as multi-layer perceptrons. This is because the only parameters that are adjusted in the learning process are the linear mapping from hidden layer to output layer. Linearity ensures that the error surface is quadratic and therefore has a single easily found minimum. In regression problems this can be found in one matrix operation. In classification problems the fixed non-linearity introduced by the sigmoid output function is most efficiently dealt with using iterated reweighed least squares. RBF networks have the disadvantage of requiring good coverage of the input space by radial basis functions. RBF centers are determined with reference to the distribution of the input data, but without reference to the prediction task. As a result, representational resources may be wasted on areas of the input space that are irrelevant to the learning task. A common solution is to associate each data point with its own centre, although this can make the linear system to be solved in the final layer rather large, and requires shrinkage techniques to avoid over fitting. Associating each input datum with an RBF leads naturally to kernel methods such as Support Vector Machines and Gaussian Processes (the RBF is the kernel function). All three approaches use a non-linear kernel function to project the input data into a space where the learning problem can be solved using a linear model. Like Gaussian Processes, and unlike SVMs, RBF networks are typically trained in a Maximum Likelihood framework by maximizing the probability (minimizing the error) of the data under the model. SVMs take a different approach to avoiding over fitting by maximizing instead a margin. RBF networks are outperformed in most

17

classification applications by SVMs. In regression applications they can be competitive when the dimensionality of the input space is relatively small.

3.3 RBF Architecture

Artificial networks typically have three layers: an input layer, a hidden layer with a non-linear RBF activation function and a linear output layer. The output, , of the network is thus

Where N is the number of neurons in the hidden layer, ci is the center vector for neuron i, and ai are the weights of the linear output neuron. In the basic form all input are connected to each hidden neuron. The norm is typically taken to be the Euclidean distance and the basis function is taken to be Gaussian.

.

The Gaussian basis functions are local in the sense that far away from the center of that neuron. RBF networks are universal approximates on a compact subset of function with arbitrary precision. The weights ai, and the data.

.

Changing parameters of one neuron has only a small effect for input values that are

. This means

that a RBF network with enough hidden neurons can approximate any continuous

, and β are determined in a manner that optimizes the fit between

18

Fig 4 Architecture of a radial basis function network.

3.4 Training

In a RBF network there are three types of parameters that need to be chosen to adapt the network for a particular task: the center vectors ci, the output weights wi, and the RBF width parameters βi. In the sequential training of the weights are updated at each time step as data streams in. For some tasks it makes sense to define an objective function and select the parameter values that minimize its value. The most common objective function is the least squares function

Where,

.

19

We have explicitly included the dependence on the weights. Minimization of the least squares objective function by optimal choice of weights optimizes accuracy of fit.

3.5 Interpolation

RBF networks can be used to interpolate a function values of that function are known on finite number of when the points:

. Taking the known points xi to be the centers of the radial basis functions and evaluating the values of the basis functions at the same points gij = ρ( | | xj − xi | | ) the weights can be solved from the equation

It can be shown that the interpolation matrix in the above equation is non-singular, if the point’s x_i are distinct, and thus the weights w can be solved by simple linear algebra:

3.6 Function approximation

If the purpose is not to perform strict interpolation but instead more general function approximation or classification the optimization is somewhat more complex because there is no obvious choice for the centers. The training is typically done in two phases first fixing the width and centers and then the weights. This can be justified by considering the different nature of the non-linear hidden neurons versus the linear output neuron.

20

3.7 Training the basis function centers

Basis function centers can be either randomly sampled among the input instances or found by clustering the samples and choosing the cluster means as the centers. The RBF widths are usually all fixed to same value which is proportional to the maximum distance between the chosen centers.

3.8 Pseudoinverse solution for the linear weights

After the centers ci have been fixed, the weights that minimize the error at the output are computed with a linear pseudo inverse solution: , Where the entries of G are the values of the radial basis functions evaluated at the points xi: gji = ρ (| | xj − ci | |). The existence of this linear solution means that unlike Multi-layer perceptron (MLP) networks the RBF networks have a unique local minimum (when the centers are fixed).

3.9 Advantages/Disadvantages

• • • RBF trains faster than a MLP. Another advantage that is claimed is that the hidden layer is easier to interpret Although the RBF is quick to train, when training is finished and it is being

than the hidden layer in an MLP. used it is slower than a MLP, so where speed is a factor a MLP may be more appropriate.

21

CHAPTER 4 OVERVIEW OF MOTION DETECTION ALGORITHMS

Given a number of sequential video frames from the same source the goal is to detect the motion in the area observed by the source. When there is no motion all the sequential frames have to be similar up to noise influence. In the case when motion is present there is some difference between the frames. For sure, each lowcost system has some aspect of noise influence. And in case of no motion every two sequential frames will not be the identical. This is why the system must be smart enough to distinguish between noise and real motion. When the systems are calibrated and stable enough the character of noise is that every pixel value may be slightly different from that in other frame. And in first approximation it is possible to define some noise per pixel threshold parameter (adaptable for any given state) the meaning of which is how the pixel value (of the same oriented pixel in two sequential frames) might differ but actually the indicating value is the same one. More precisely, if the pixel with coordinates (Xa,Ya) in frame A differs from the pixel with coordinates (Xb,Yb) in frame B less than on TPP (threshold per pixel) value so we will see them as pixels with equal values. And we can write it by formulae: Pixel (Xa, Ya) equal to Pixel (Xb, Yb) I if {abs (Pixel (Xa,Ya)-Pixel(Xb,Yb)) < TPP } By adapting the TPP value to current system state we can make the system to be noise-stable. By applying this threshold operation to every pixel pair we may assume that all the preprocessed pixel values are noise-free. The element of noise that is not cancelled will be significantly small relative to other part. Ok, if so we have to post-process these values to detect the motion if any. As it was memorized above we have manipulate with different pixels inside two sequential frames to make conclusion about the motion.

22

Firstly, to make the system sensitive enough we have not to fix the TPP value too big. It mean that keeping the sensitivity of the system high in any two frames there will be some little number (TPP related) of different pixels. And in this case we have not to see them as noise. It is the first of the reasons to define a TPF (threshold per frame) value (adaptable for any given state) the meaning of which is how many pixels at least, inside two sequential frames must differ in order to see them as motion. The second reason to deal with TPF is to filter (to drop) small motion. For instance, by playing with TPF values we can neutralize motion of the small object (bugs etc.) by still detect the motion of people. And we can write the exact meaning of TPF by formulae: Let’s define NDPPP to be the Number of Different Pre-Processed by TPP Pixels. So, There is a motion i.e. NDPPP > TPF. Both of TPP and TPF values are variable through the UI to get the optimal system sensitivity. Also the TPF value has its visual equivalent and it is used as following. After the pixels pre-processing (by TPP) lets color all static (which do not include motion) pixels by lets say black color and all the dynamic (which indicate the motion) pixels will be left with their original color. This will bring the effect of motion extraction. In the other words, all the static parts of the frames will be black, and only the moving parts will be seen normally. The enabling/disabling of this effect is possible to control through the GUI. The Camera Manager provides routines for acquiring video frames from CCD cameras. Any process can request a video frame from any video source. The system manages a request queue for each source and executes them cyclically.

23

CHAPTER 5 PROPOSED DETECTION SYSTEM

This chapter presents the main software design and implementation issues. It starts by describing the general flow chart of the main program that was implemented in MATLAB. It then explains each component of the flow chart with some details. Finally it shows how the graphical user interface GUI was designed.

5.1 Basic Architecture

Fig 5 A basic architecture of surveillance system The above block diagram shows the surveillance system which consists of a camera system which monitors the particular area, a video daughter card which transmits the video signal to electrical signal, a network card which helps in connecting to a network and motion detection algorithm (SAD and Correlation) along with RBF network.

24

5.2 Main Program Flow Chart

The main task of the software was to read the still images recorded from the camera and then process these images to detect motions and take necessary actions accordingly. Figure 6 below shows the general flow chart of the main program.

Start

Setup & Initializations

Flag=1

What is Flag value?

Flag=0

Image Acquisition

Motion Detection Algorithm

Break & clear

No

Is image > threshold

Yes

Actions on Motion Detection

Data Record

Stop

Figure 6 Main Program Flow Diagram

25

It starts with general initialization of software parameters and objects setup. Then, once the program started, the flag value which indicates whether the stop button was pressed or not is checked. If the stop button was not pressed it start reading the images then process them using one of the two algorithms as the operator was selected. If a motion is detected it starts a series of actions and then it go back to read the next images, otherwise it goes directly to read the next images. Whenever the stop button is pressed the flag value will be set to zero and the program is stopped, memory is cleared and necessary results are recorded. This terminates the program and returns the control for the operator to collect the results. The next sections explain each process of the flow chart in figure 6 with some details.

5.2.1 Setup and Initializations

Start

Launch GUI

No

Start button pressed

Yes

Read Threshold Value

Read Algorithm Type

Setup Serial Port

Setup Video Object

Stop

Figure 7 Setup and Initializations Process

26

Figure 7 show the flow chart for the setup and initialization process. This process includes the launch of the graphical user interface (GUI) where the type of motion detection algorithm is selected and threshold value (the amount of sensitivity of the detection) is being initialized. Also, during this stage a setup process for both the serial port and the video object is done. This process takes approximately 15 seconds to be completed,(depending on the specifications of the PC used) for the serial port it starts by selecting a communication port and reserving the memory addresses for that port, then the PC connect to the device using the communication setting that was mentioned in the previous chapter. The video object is part of the image acquisition process but it should be setup at the start of the program.

5.2.2 Image acquisition

Start Read First Frame Convert to Grayscale

Read Second Frame

Convert to Grayscale Stop

Figure 8 Image acquisitions Process

27

After setup stage the image acquisition starts as shown in figure 8 above. This process reads images from the PC camera and save them in a format suitable for the motion detection algorithm. There were three possible options from which one is implemented. The first option was by using auto snapshots software that takes images automatically and save them on a hard disk as JPEG format, and then another program reads these images in the same sequence as they were saved. It was found that the maximum speed that can be attained by this software is one frame per second and this limits the speed of detection. Also, synchronization was required between both image processing and the auto snapshot software’s where next images need to be available on the hard disk before processing them. The second option was to display live video on the screen and then start capturing the images from the screen. This is a faster option from the previous approach but again it faced the problem of synchronization, when the computer monitor goes into a power saving mode where black images are produced all the time during the period of the black screen. The third option was by using the image acquisition toolbox provided in MATLAB 6.5.1 or higher versions. The image acquisition toolbox is a collection of functions that extend the capability of MATLAB. The toolbox supports a wide range of image acquisition operations, including acquiring images through many types of image acquisition devices, such as frame grabbers and USB PC cameras, also viewing a preview of the live video displayed on monitor and reading the image data into the MATLAB workspace directly. For this project video input function was used to initialize a video object that connects to the PC camera directly. Then preview function was used to display live video on the monitor. Get snapshot function was used to read images from the camera and place them in MATLAB workspace.

28

The later approach was implemented because it has many advantages over the others. It achieved the fastest capturing speed at a rate of five frames per seconds depending on algorithm complexity and PC processor speed. Furthermore, the problem of synchronization was solved because both capturing and processing of images were done using the same software. All read images were converted it into a two dimensional monochrome images. This is because equations in other algorithms in the system were designed with such image format.

5.2.3 Motion Detection Algorithm

A motion detection algorithm was applied on the previously read images. There were two approaches to implement motion detection algorithm. The first one was by using the two dimensional cross correlation while the second one was by using the sum of absolute difference algorithm. These are explained in details in the next two sub sections.

5.3 Motion Detection Using Sum of Absolute Difference (SAD)

This algorithm is based on image differencing techniques. mathematically represented using the following equation:

D (t ) = 1 N

It is

∑ I (t

i

) − I (t j )

Where N is the number of pixels in the image used as scaling factor,

I (t i ) is the image I at time i ,

I (t j )

is the image I at time j and

D (t ) is the normalized sum of absolute difference for that time.

In an ideal case when there is no motion

I (t i ) = I (t j )

29

and D(t ) = 0 . However noise is always presented in images and a better model of the images in the absence of motion will be

I (t i ) = I (t j ) + n( p )

Where n( p ) is a noise signal. The value D (t ) that represents the normalized sum of absolute difference can be used as a reference to be compared with a threshold value as shown in figure 9 below. The figure also shows a test case that contains a large change in the scene being monitored by the camera this was done by moving the camera. During the time before the camera was moved the SAD value was around 1.87 and when the camera was moved the SAD value was around 2.2. If the threshold for detection was fixed around the value less than 2.2 it will continuously detect motion after the camera stop moving.

Figure 9 Direct Thresholds for SAD Values

30

This approach solve the need for continuously re-estimate the threshold value. Choosing a threshold of 1*10-3 will detect the times when only the camera is moved. This results into a robust motion detection algorithm that can not be affected by illumination change and camera movements.

5.3.1 Actions on Motion Detection

Before explaining series of actions happen when motion is detected it is worth to mention that the values of variance that was calculated whether it was above or below the threshold will be stored in an array, where it will be used later to produce a plot of frame number Vs. the variance value. This plot helps in comparing the variance values against the threshold to be able to choose the optimum threshold value. Whenever the variance value is less than threshold the image will be dropped and only the variance value will be recorded. However when the variance value is greater than threshold sequence of actions is being started as shown in figure 10 below. Start

Time Date Frame#

Update Log File Display Image

Convert Image to Frame

Trigger Serial Port

Stop

Figure 10 Actions on Motion Detection

31

As the above flow chart show a number of activities happen when motion is detected. First the serial port is being triggered by a pulse from the PC; this pulse is used to activate external circuits connected to the PC. Also a log file is being created and then appended with information about the time and date of motion also the frame number in which motion occur is being recorded in the log file. Another process is to display the image that was detected on the monitor. Finally the image that was detected in motion will be converted to a movie frame and will be added to the film structure.

5.3.2 Break and clear Process

After motion detection algorithm applied on the images the program checks if the stop button on GUI was pressed. If it was pressed the flag value will be changed from one to zero and the program will break and terminate the loop then it will return the control to the GUI. Next both serial port object and video object will be cleared. This process is considered as a cleaning stage where the devises connected to the PC through those objects will be released and the memory space will be freed.

5.3.3 Data Record

Finally when the program is terminated a data collection process starts where variable and arrays that contain result of data on the memory will be stored on the hard disk. This approach was used to separate the real time image processing from results processing. This has the advantage of calling back these data whenever it is required. The variables that are being stored from memory to the hard disk are variance values and the movie structure that contain the entire frames with motion. At this point the control will be returned to the GUI where the operator can callback the results that where archived while the system was turned on. Next section will explain the design of the GUI highlighting each button results and callbacks.

32

START

IMAGE ACQUISTION FRAME SEPARATION

DIVIDE QUADRANTS SUM OF ABSOLUT DIFFERENCE

>T

DATA RECORD

Fig 11 Flow chart for SAD algoritham

Fig 12 Frame separation

Fig 13 Divide Quadrants

33

5.3.4 Graphical User Interface Design

The GUI was designed to facilitate interactive system operation. GUI can be used to setup the program, launch it, stop it and display results.

Start

Clear all Previous Work

Variable Initialization & Setup

Launch program

Call Selected main Program

Terminate Program

View Results

Yes

Start Again

NO

Exit

End

Figure 14 GUI flow Chart

34

During setup stage the operator is promoted to choose a motion detection algorithm and select degree of the detection sensitivity Whenever the start/stop toggle button is pressed the system will be launched and the selected program will be called to perform the calculations until the start/stop button is pressed again which will terminate the calculation and return control to GUI. Results can be viewed as a log file, movie and plot of frame number vs. variance value. Figure 14 illustrate a flow chart of the steps performed using the GUI.

5.4 Motion detection using Correlation Network

A correlation neural network (CNN) which accounts for velocity sensitive responses of neurons is suitable for analog circuit implementation of motiondetection systems and has been successfully implemented on CMOS. The CNN utilizes local motion detectors to correlate signals sampled at one location in the image with those sampled after a delay at adjacent locations; however, an edgedetection process is required in practical motion detection systems with the CNNs. The term correlation can also mean the cross-correlation of two functions or electron correlation in molecular systems. In probability theory and statistics, correlation, also called correlation coefficient, indicates the strength and direction of a linear relationship between two random variables. In general statistical usage, correlation or co-relation refers to the departure of two variables from independence, although correlation does not imply causation. In this broad sense there are several coefficients, measuring the degree of correlation, adapted to the nature of data. A number of different coefficients are used for different situations. The best known is the Pearson product-moment correlation coefficient, which is obtained by dividing the covariance of the two variables by the product of their standard deviations.

35

5.4.1 Mathematical properties

The correlation ρX, Y between two random variables X and Y with expected values μX and μY and standard deviations σX and σY is defined as:

Where E is the expected value of the variable and cov means covariance. Since μX = E(X), σX2 = E(X2) − E2(X) and likewise for Y, we may also write

The correlation is defined only if both of the standard deviations are finite and both of them are nonzero. It is a corollary of the Cauchy-Schwarz inequality that the correlation cannot exceed 1 in absolute value. The correlation is 1 in the case of an increasing linear relationship, −1 in the case of a decreasing linear relationship, and some value in between in all other cases, indicating the degree of linear dependence between the variables. The closer the coefficient is to either −1 or 1, the stronger the correlation between the variables. If the variables are independent then the correlation is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. Here is an example: Suppose the random variable X is uniformly distributed on the interval from −1 to 1, and Y = X2. Then Y is completely determined by X, so that X and Y are dependent, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, independence is equivalent to uncorrelated ness. A correlation between two variables is diluted in the presence of measurement error around estimates of one or both variables, in which case disattenuation provides a more accurate coefficient.

36

5.4.2 Geometric Interpretation of correlation

The correlation coefficient can also be viewed as the cosine of the angle between the two vectors of samples drawn from the two random variables. This method only works with centered data, i.e., data which have been shifted by the sample mean so as to have an average of zero. Some practitioners prefer an uncentered (non-Pearson-compliant) correlation coefficient. See the example below for a comparison. As an example, suppose five countries are found to have gross national products of 1, 2, 3, 5, and 8 billion dollars, respectively. Suppose these same five countries (in the same order) are found to have 11%, 12%, 13%, 15%, and 18% poverty. Then let x and y be ordered 5-element vectors containing the above data: x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18). By the usual procedure for finding the angle between two vectors, the uncentered correlation coefficient is:

Note that the above data were deliberately chosen to be perfectly correlated: y = 0.10 + 0.01 x. The Pearson correlation coefficient must therefore be exactly one. Centering the data (shifting x by E(x) = 3.8 and y by E(y) = 0.138) yields x = (-2.8, -1.8, -0.8, 1.2, 4.2) and y = (-0.028, -0.018, -0.008, 0.012, 0.042), from which

as expected.

37

5.4.3 Interpretation of the size of a correlation

Several authors have offered guidelines for the interpretation of a correlation coefficient. Cohen (1988), for example, has suggested the following interpretations for correlations in psychological research, in the table in the bottom. As Cohen himself has observed, however, all such criteria are in some ways arbitrary and should not be observed too strictly. This is because the interpretation of a correlation coefficient depends on the context and purposes. A correlation of 0.9 may be very low if one is verifying a physical law using high-quality instruments, but may be regarded as very high in the social sciences where there may be a greater contribution from complicating factors

Correlation Small Medium Large

Negative −0.29 to −0.10 −0.49 to −0.30 −1.00 to −0.50 Table 1

Positive 0.10 to 0.29 0.30 to 0.49 0.50 to 1.00

Fig 15 An unit network of two-dimensional CCN.

38

START

IMAGE ACQUISTION FRAME SEPARATION DIVIDE QUADRANTS

CORRELATION NETWORK

Decisi on

DATA RECORD

Fig 16 Flow chart for correlation

39

CHAPTER 6 PROPOSED OBJECT EXTRACTION

Many attempts have been made to extract data from video and film in a form suitable for use by animators and modelers. Such an approach is attractive, since motions and movements for people and animals may be obtained in this way that would be difficult using mechanical or magnetic motion capture systems. Visual extraction is also appealing since it is non-intrusive and has the potential to capture, from film, the motion and characteristics of people or animals long dead or extinct. Almost all attempts to perform visual extraction have been based around bespoke computer vision applications which are difficult for non-experts to use or adapt to their own needs. This paper presents a generic approach to extracting data from video. Whilst our approach allows low-level information to be extracted we show that higher-level functionality is available also. This functionality can be utilized in a manner that requires little knowledge of the underlying techniques and principles. Our approach is to approximate an image using principal component analysis, and then to train a multi-layer perceptron to predict the feature required by the user. This requires the user to hand-label the features of interest in some of the frames of the image sequence. One of the aims of this work is to keep to a minimum the number of frames that need to need labeled by the user. The trained multi-layer perceptron is then used to predict features for images that have never been labeled by the user. Other attempts to extract useful information from video sequences include the use of edge-detection and contour or edge tracking, template matching and template tracking. All such systems work well in some circumstances, but fail or require adaptation to meet the requirements of new users. For instance, in the case of template tracking, the user needs to be aware of the kinds of features that can be tracked well in an image and also choose a suitable template size. This is not a trivial task for non-specialists.

40

6.1 Method

The main steps in extraction using our system are detailed below: The user selects the sequence (or set) of images for which they wish data to be extracted from. This may well comprise of several shorter clips taken from different parts of a film. These images have some pre-processing performed on them (principal components analysis) to reduce each image to a small set of numbers. The user decides what feature(s) they wish to extract and labels this feature by hand in a fraction of the images chosen at random. The labeling process may involve clicking on a point to be tracked, labeling a distance or ratio of distances, measuring an angle, making a binary decision (yes/no, near/far etc.) or classifying the feature of interest into one of several classes. Once this ground-truth data is available, a neural network is trained to predict the feature values in images that have not been labeled by the user.

6.2 Feature Extraction

Principal components analysis (also known as eigenvector analysis) has been used extensively in computer vision for image reconstruction, pattern matching and classification. Given the i th image in a sequence of images, each of which consists of M pixels, we form the vector x i by concatenating the pixels of the image in raster scan order and removing the mean image of the sequence. The matrix X is created using the x i's as column vectors. Traditionally, the principal modes, qi, are extracted by computing X T i=iqi (1) Xq Where i's are the Eigen values.

41

a measure of the amount of variance each of the eigen vectors accounts for. Unfortunately, the matrix XXT is typically too large to manipulate since it is of size M by M. Such computation is wasteful anyway since only N princi pal modes are meaningful, where N is the number of example images. In all our work N M. Therefore we compute: X X =i ui(2) T u and we can obtain the qi's that we actually require using: q =X i i u (3)

In practice only the first P modes are used, P30 N. The principal mode extracted from a short film clip is shown in Figure 1 and is used later to help an animator to construct a cartoon version of the clip. It is tempting to think that such modes could be used directly to predict, say, the rotation of the man's shoulders. However, the second mode also encodes information about shoulder movement and it is only by combining information from many modes that rotation can be reliably predicted.

42

REFERENCE;

[1] ' Special issue on third generation surveillance systems', froc. IEEE, 2001, 89, JAIN, R., KASTURI, R., and SCHUNCK, B.G. This paper gives the detailks about the surveillance systems [2] 'Machine vision' (McGraw-Hill Inc., 1995) PONS, J., PRADES-NEBOT, J., ALBIOL, A,, and MOLINA, J.his paper provides the details about intelligence. [3] 'Motion video sensor in the compressed domain'. SCS Euromedia Conf., Valencia, Spain, 2001, This paper provides the details about compressed domain. [4] Y. Song, A perceptual approach to human motion detection and labeling. PhD thesis, California Institute of Technology, 2003. This paper provides the details about human motion detection [5] N. Howe, M. Leventon, and W. Freeman, “Bayesian reconstruction of 3D human motion from single-camera video,” Tech. Rep. TR-99-37, Mitsubishi Electric Research Lab, 1999 This paper provides the details about 3d human detection. [6] L. Goncalves, E. D. Bernardo, E. Ursella, and P. Perona, “Monocular tracking of the human arm in 3D,” in Proc. 5th Int. Conf. Computer Vision, (Cambridge, Mass), pp. 764– 770, 1995.This paper provides the details about 3d human detection. [7] S. Wachter and H.-H. Nagel, “Tracking persons in monocular image sequences,” Computer Vision and Image Understanding, vol. 74, pp. 174–192, 1999. This paper provides details about motion detection in image sequences. algorithms in artificient

43

44