Cloud Tp

A TERM PAPER REPORT ON

MANAGING APPLICATIONS IN CLOUD







Submitted by

Saurabh Tomar (11108332)
Roll No:A26








Submitted to

Mr. Dhawleshwar Rao Ch.
School of Computer Science and Engineering
(Novemeber 2014)
Contents

1. Introduction 3

2. Scope and Scenario 4

3. Overview of Solution 5

4. Solutions Diagram 5

5. Managing Rapid Deployments 10

6. Managing User Data Analysis 11

7. Managing Complexity 12

8. References 14
Introduction
A growing majority of business applications today require a level of complexity not easily
addressed by a single hosting service. One application may include multiple logical services,
different programming languages, differing types of data storage needs, and a range of runtime
requirements. In addition, teams are generally composed of developers with vastly different areas
of expertise and experience. Requirements change at a pace that often leads to a level of
complexity that is challenging to manage by even the largest teams.
The Cloud Platform has evolved to provide a complete set of tools for managing even the most
complex application requirements. From a fully managed application to custom virtual machines, it
gives developers the flexibility to balance administrative overhead with technical demands. On
Cloud Platform, applications can be easily developed as separate components with vastly different
technical and business requirements. These components can have their own performance
configuration, budget, authorization and versioning while sharing a common set of resources. The
shared resources enable a tighter integration than individual applications communicating over
traditional APIs. In addition, components benefit from an integrated security model and centralized
management of a Cloud project
[1]
. Developing on the Cloud Platform provides a range of tools for
managing growing application complexity.
Whether you’re building an application from the ground up or moving an existing custom
application
[2]
to the cloud, the Cloud Platform can provide the level of performance and
functionality appropriate for your business.
In this solution, we’ll show the right tools for:
• Different developer expertise
• Different data storage requirements
• Interactive and batch processing requirements
• Custom analysis engines
• Logically separate business components







"
Scope and Scenario
This paper is designed for Solutions Architects and CTOs interested in managing the deployment
and scaling of a large scale application on the Cloud Platform. The example presented in the
paper showcases how to handle many of the complexities encountered in deploying large
applications. We provide an introduction to the tools of the platform and provide links to deeper
dives where relevant. The sample discussed in this paper was kept intentionally simple to focus on
solving typical complexities rather than going deep into app-specific functionality.
To illustrate how the Cloud Platform enables you to manage complex applications in the cloud, we
will walk through a real-world scenario of building a video sharing community. In this scenario,
users will upload videos from a mobile phone. The content will be transcoded for playback in
multiple formats and finally presented on a web community site for watching and commenting. The
demo application for the scenario includes the following components:
• Android application for video upload and mobile backend
• Video transcoding pipeline
• Community website for video watching and commenting
• Comment analysis via Hadoop
The following technical challenges will be addressed:
• Multiple backend languages
• Multiple independent developers or teams
• Mix of managed applications and custom binaries
• Multiple data storage requirements
• Dynamic scaling for batch processing.









#
Overview of the Solution
Our proposed solution makes use of the following products from the Cloud Platform:
• App Engine
• App Engine Task Queues
• Cloud Storage
• Cloud SQL
• Compute Engine
• Cloud Datastore
• Mobile Cloud Endpoints
• App Engine Modules
The solution also uses the following third-party and open source tools] (not endorsed by ):
• Apache Hadoop
• FFmpeg
Solutions Diagram
The following architecture diagram illustrates the components in the solution:





$
The following components are shown in the diagram:
Reference Component Description
1 Browser client Typical web browser
2 PHP video sharing site Web property for browsing, commenting, and sharing videos
3 Cloud SQL Relational database service used for user and video metadata
4 Mobile video uploader
app
Android application for producing and uploading videos
5 Java mobile video API
backend
Backend for publishing videos from the Android application
6 Cloud Storage Distributed file storage for video content
7 Video transcode
pipeline
System to transcode incoming videos to various playback formats
8 Task Queue Queueing service for batch task execution
9 Hadoop cluster Hadoop cluster used to analyze user data
10 Cloud Datastore Highly scalable and distributed NoSQL database used in this sample to store
comments for the videos and video transcoding statistics









%
Managing the Deployment and Scaling of Web
Services
For the deployment and maintenance of a typical web property, there is a significant amount of
overhead associated with the administration of the underlying network, systems, and services.
App Engine alleviates those concerns by providing a completely managed application platform. It
automatically handles scaling out and back resources in response to demand. Versioning is
another powerful feature that allows developers to easily handle rollbacks, divide customer traffic
across releases, and conduct slow rollout of new releases. For web workloads, App Engine helps
a developer to completely focus on building the best applications.
The web part of the video sharing site was built on App Engine to leverage these unique benefits.
This component was written in one of the web’s most popular languages, PHP. PHP support on
App Engine brings all the benefits of automatic scaling, versioning, and performance paired with
the rapid development enabled by a language built for the web. Most existing PHP applications
will run with minimal changes in the App Engine environment.
Since many web development teams are familiar with MySQL, we opted to use a database
technology that should be very familiar to them: Google Cloud SQL. Cloud SQL provides
compatibility with the MySQL functionality and SQL protocol, allowing for a simple transition of
existing workloads. In addition, it handles the complexity of high availability and backups behind
the scenes. In the sample, Cloud SQL was used for the primary storage of user data and video
metadata.
For storage of video files and thumbnails, we opted for Cloud Storage. Cloud Storage supports
large file sizes and direct delivery to our application clients. It also provides an optional, built-in
edge cache for global distribution of our video content. For uploads, Cloud Storage provides time-
limited writeable URLs to allow file uploads directly from the client. This reduces the need for
spending valuable application resources relaying files to storage.
For user activity on the site, namely comments, we anticipated a growth at a substantially higher
scale and opted to use App Engine’s NoSQL database, High Replication Datastore. HRD provides
a level of scalability unmatched by pure SQL solutions and is a good match for high volume data
access. The choice also opened us up to the use of the newly released Cloud Datastore, providing
access to our user activity across the Cloud Platform and externally.




&


Managing Batch Processing Pipelines
Most large web sites today require some level of batch processing. The workloads may include
processing of uploaded content, updating user data, managing external notifications, or data
analysis. Regardless of the business demand, batch processing pipelines are inherently difficult to
build, scale, and maintain. The Cloud Platform provides the tools and services to simplify the task
of building a scaleable and reliable processing pipeline. From a programmable compute
environment to a range of storage options, the platform is tuned for distributed batch processing.
Our scenario uses a batch processing pipeline to manage the uploaded video content for the
application. Most video sharing sites use batch processing for their uploaded content to do
everything from transcoding, watermarking, and/or digital rights management. The demand for
these services tends to be extremely variable based on the sporadic upload of content versus a
more steady consumption behavior. Additionally, the various components of a video pipeline tend
to require custom applications or code that are not suitable for a strictly PaaS (Platform as a
Service) environment. Both factors make Compute Engine an excellent fit. Scaling for a typical




'
data center or managed hosting typically requires purchasing capacity ahead of time or paying for
hour-long blocks. Compute Engine provides extremely fast spin-up for instances (in most cases,
in seconds) and fast turn-down making it a great fit for this type of workload. Additionally, Compute
Engine provides extremely consistent performance as you scale out.
For our environment, we utilize a combination of App Engine and Compute Engine to spin up a
scalable video transcoding service on demand. Utilizing an orchestration engine written in Python
on App Engine, we control Compute Engine instance behavior using the available JSON API and
coordinated transcoding activity using Task Queues. The setup allows us to dynamically allocate
capacity in response to demand and existing load. In addition, a periodic script checks the load in
the environment and reduces capacity when it was no longer needed. This provides a strong
economic benefit given the recent change to sub-hour billing on Compute Engine. Task Queues
provide a way to accept, buffer and distribute requests across the video transcoding instances.

Figure 3: Video Transcoding Pipeline





()
Managing Rapid Mobile Deployments
The Cloud Platform provides a suite of tools to enable rapid mobile application development. A
combination of Cloud Endpoints and App Engine provides a platform for exposing backend server
technology as an API for external consumption. Without Cloud Endpoints, adding a mobile
application to an existing web site usually requires a level of planning for a flexible API and strong
authentication. Endpoints handles authentication and API generation automatically for the
developer, freeing them up to build out the application functionality. In addition, the use of App
Engine simplifies scaling and ongoing maintenance for the API server.
To add a mobile application to our video sharing site, the API server was built on App Engine. We
also made use of Google Cloud Endpoints to facilitate authentication and rapid API development.
Additionally, we chose to implement the backend in Java to illustrate the flexibility of language
choices between different business components. Our mobile backend facilitates the upload of
video content from the mobile application to Cloud Storage, creation of metadata in Cloud SQL
and integration into the transcoding service via Task Queues.
The mobile video application was written on Android utilizing the generated Cloud Endpoints
libraries to quickly integrate the authentication and upload of videos. Additionally, videos are
uploaded directly to Cloud Storage to avoid any additional latency or processing on the App
Engine API Server. Authentication is handled seamlessly through Cloud Endpoints and permission
is delegated to the Cloud Storage bucket for storing video content. Permission delegation is an
option on the bucket allowing users of the application to directly upload files. The combination of
simplified authentication and API generation allowed for the rapid development of a complete
mobile media solution.

Figure 4: Mobile Deployment Architecture




((
Managing User Data Analysis
The analysis of user activity in applications can lead to an explosion of data along with the systems
and software to retrieve meaningful results. Given the sporadic nature of most analysis, provisioned
environments can sit idle for periods of time. However, when performing analysis, these systems can
consume large amounts of computing resources and storage in order to complete in reasonable time
frames. These characteristics make it a good fit for Compute Engine. The ability to rapidly spin up
instances in response to a growing workload and to scale back when idle allows both small and large
analysis jobs to be launched on demand. Additionally, the availability of capacity for expansion
prevents bottlenecks associated with typical data processing environments.
To process the expected volume of comments on our video application, we opted to use a Hadoop
installation running on Compute Engine. Again, the Compute Engine API makes it easy to launch a
Hadoop cluster of varying sizes for processing data. In our case, we are pulling comment data from the
Cloud Datastore associated with our application and making a simple sentiment calculation for
comments associated with a video. The Cloud Datastore data is automatically shared between the App
Engine and the Compute Engine instances belonging to the same project. The end result of computing
negative or positive total sentiment is written back to a Datastore entity associated with the video.

Figure 5: Hadoop Deployment Architecture




(*
Managing Complexity
Given the range of logical services in our application, we made heavy use of several key features
of the Google Cloud Platform. Together, they provide a strong base for supporting complex
business applications in the cloud.
Cloud Endpoints
Cloud Endpoints is a collection of tools and services for generating REST APIs and client libraries
from an App Engine application to simplify both web client and mobile client access. The
combination allows for rapid rollout of an API server for Android applications. Cloud Endpoints
simplifies communication with an App Engine application opening up the benefits of the available
services, including: Cloud Datastore, Cloud Storage, Mail, Task Queues, etc. Behind the scenes, it
converts the REST API calls to method invocation on the backend application and secures each
call with OAuth 2.0 authentication. Existing development environments, including Eclipse, Android
Studio and Apache Maven, currently offer functionality to generate a client library from within a
project. Cloud Endpoints technology removes the overhead of API development and allows
developers to focus on what really matters, their product.
To get started with Google Cloud Endpoints, check out the From Nothing to Nirvana in
Minutes session given at Google I/O 2013 and the developer documentation for Cloud Endpoints.
In addition, Google Cloud Endpoints are showcased in several solution samples including the Web
App, the Mobile Shopping Assistant, and the Mobile Backend Starter.
Compute Engine API
The Compute Engine API provides a programmatic method to control a set of virtual machines in
the Google Cloud Environment. Bare virtual machines can be launched and built up to needed
specifications. Alternatively, machine images can be created beforehand to reduce the time
required to launch them. Auto scaling can be achieved through periodic checks of machine load.
Through the use of App Engine Task Queues, you can partition your work and horizontally scale
your processing environment. The benefits of Google Compute Engine can be harnessed to
provide batch or custom processing for practically any workload.
For more information on coordinating with the Compute Engine API, check out our Load Balanced
Gaming Server Sample. Note that even though the sample uses gaming as a scenario, many
design patterns can be generalized to other scenarios as well.
Shared Data Storage




(+
Data storage is deeply integrated into the Google Cloud Platform. From basic file storage to
transaction object storage, there is a service to meet the needs of your application:
• Cloud Storage provides a storage service for objects and files up to terabytes in size.
• Cloud SQL provides a relational SQL database based on MySQL.
• Cloud Datastore (App Engine Datastore) provides a NoSQL schemaless object datastore,
with a query engine and atomic transactions.
Each of these services share a common theme of accessibility. They are built to be accessed from
anywhere in the Cloud Platform and externally as well. All services represent a range of storage
options for application developers. Whether you need the simplicity of a pure SQL implementation
or opt for the scalability of NoSQL, there is an available database option for your application.
Authentication and authorization are automatically handled by the Google Cloud Platform. By
default, applications are authorized to access other resources belonging to the same project.
Service Account is a type of role account specific to your project that enables application access
to resources. Outside of the Cloud Platform, certificate-based service accounts are available to
enable secure access from remote locations. Service Accounts can be easily created and revoked
from within the Cloud Project.
App Engine Modules
App Engine Modules allow teams to partition applications in components with separate
requirements for:
• Scaling
• Deployments
• Versioning
• Performance Settings
Typical large scale applications are divided into logical services supporting the various aspects of
the business. These components can support a number of areas including a corporate site, web
applications, mobile access, customer API access, and administration portals. These different
sites usually carry different service levels, expected traffic, and processing requirements.
Supporting these requirements in a typical managed environment usually forces the developer to
choose the highest level of performance for all applications. App Engine Modules opens up the
ability to customize an application into logical units but still maintain the benefits of a fully
managed environment including optional auto scaling, automatic deployment versioning, and
traffic splitting.
The end result is an environment built for multiple teams and/or development languages. One App
Engine application can support the diverse requirements of larger application.




("

References
1.https://cloud.google.com/developers/articles/managing-complex-
applications-in-the-cloud/

2.http://www-935.ibm.com/services/us/gbs/application-management/cloud-
application-management/

3.http://www.networkcomputing.com/cloud-infrastructure/cloud-application-
management-complexity-a-growing-challenge-/d/d-id/1233380?