Dynamics in Video-Streaming via P2P

Video-streaming is clearly a popular application for both live and on-demand services. Peer-to-peer model emerged as a solution providing scalability and ease of deployment. French researchers I. Ullah et.al. published the results from their experiments on “A survey and synthesis of user behavior measurements in P2P streaming systems“.

The paper seems a little repetitive and with expected but controversial results.

1. They made a good point in considering user behavior and analyzing metric relationships in order to establish better peering

2. Most of the results are quite obvious: popularity of a video will affect session duration, or the day of the week affecting network delays

3. Their conclusions are based on their survey that presents very diverse and non-related experiments, meaning that the context is not taken into account to really understand the environmental factors and network performance

Still, there are a few video streaming strategies and user-metrics that give a good picture of this popular application…

First a view to the classic content delivery model: Client/Server. This approach has limited bandwidth and it’s unscalable. A step forward is the inclusion of CDNs (Content Delivery Networks). CDNs distribute content geographically, for better access. This reduces delays but incurs on high deployment costs.

Another strategy is IPTV, a telco-managed Internet Television. It works via IP multicast (multicasting delivers a message to a selected subgroup of the network, each of these hosts is in charge of continuing the distribution to subgroups deeper down in the “tree”). There are two main drawbacks: it’s unscalable because it requires changes at infrastructure level and ISPs are mostly disinterested in its implementation.

P2P! Each peer takes the role of host and relays video stream to each other. It doesn’t require any infrastructure change. It wouldn’t need router support, the end-system support suffices. The streaming flow depends and is directly affected by certain external factors and by user behavior.

User metrics to be considered:

• channel/ video popularity [long term vs instantaneous]
• session duration
• online duration: has been mostly ignored! (useful for exchange P2P)
• arrival/ departure
• download/ upload

A key input was the online duration versus session duration. Even if a user is not watching the specified video, it may serve as a host as long as it remains online.

External factors can be environmental or network-specific:

• Environmental: day-of-week, day-scale, channel/ content type
• Network Performance: delays, partner discovery, quality, failure rate

The combination of these 3 groups will give the required understanding to have a better service and benefit users, service operators and network operators.

Measurement approaches: “Reinventing the wheel”

Is it good or not to “Reinvent the wheel“?!

We can surely list many cases on which we are better off looking for an existing approach and implement it in our current problem. These cases vary from very simple with straight-forward solutions to very tough and challenging scenarios.

Internet Measurement requires solutions that move widely from one extreme to the other, depending on your goal. And even if the Internet has been connected for several decades now, this topic is still under research; and not only in academics, but the industry is still learning and incorporating new discoveries (read more about this from a previous post).

Presenting a simple example for evaluation

A hands-on example is capturing and measuring the access points signal in a building. An access point can be a home router we use to connect. Let’s also consider that whenever a WiFi connection is shared across a wide extension, access points are located at key points to repeat the signal with specified parameters: power, channel, authentication, etc.

What applications can get from measuring this?

1) If you are the one who needs to identify the key points of each access point, you want to make sure that you cover the whole area with the least number of devices to manage, right?

2) The second application is more interesting, and it is a current research open-item: indoor geo-location. The concept is clear, find where the user is standing at a certain point in time, taking into consideration halls, rooms, floor level, etc.

3) A third one is an open idea for the future: shared “private/semi-public” WiFi access”. Right now, certain open spaces offer Internet connection if you login with a specific user/password on the offered area. Wouldn’t it be nice that if you are walking in a public space, your phone connects automatically when a network from your provider is accessible? And not only the open connections, but closed ones can be offered under a special contract and easily identified by common prefix usually left as the access point name: ESSID. The mechanism would be similar to WiFi connections on moving devices like buses.

Anyway, there are many applications that can be derived from a single measurement experiment. Now, back to our original question: should we reinvent the wheel or use an existing  mechanism / tool? The measurement looks (and is) simple enough, the Internet is a network… and there certainly are multiple analysis schemes for networks.

Well, as inviting as the no-rework seems, I would recommend a customized approach, meaning… reinvent the wheel! It is for a very specific purpose and the error range should be very small. It will require a little bit more work, but it will be worth it. I will definitely deepen on this in a future post! After some testing with app #2, existing approaches were okay but did not give as satisfactory results as the custom one. I was surprised by two things:

1. That customization and rework won over existing approaches
2. The little extra effort it required to achieve great results

Relevance of Internet Measurement

Internet Measurement is a tricky title because the term is very broad and it may not be easily understood, unless you’ve been for a while in this field.

Besides all the Internet offerings, such as content, instant communication, networking, applications with multiple functionalities… There’s a fascinating quantitative world within it that has not been fully explored. Many approaches have aroused, mainly because network topologies must be understood to scale, plan and organize what is known as traffic engineering.

So, yes! Internet Measurement is the general term to cover the analysis of Internet infrastructure, traffic on the Internet and applications performance. Beware, these applications refer to network applications!

What kind of applications exactly?

They can range from early text-based utilities such as file transfer and remote login to the more recent advent of the Web, electronic commerce, and multimedia streaming. Some applications include: DNS, peer-to-peer and networked games.

The information retrieved from the Internet can provide:

• Applications structure & user behavior
• Workload performance
• Availability
• Internet Security
• Traffic architecture

With this knowledge, ISPs (Internet Service Providers) design packets distribution and new physical connections in order to reduce costs. Analysts can benefit from these insights in several ways. Besides a deep understanding of Internet architecture, some interesting connections can be discovered.

For example, some countries in Africa (i.e. Benin, Togo, Ghana) are not connected between them. If they wish to communicate, they redirect their traffic from their local ISP to an ISP in France, which is the original service provider, and then back to Africa, causing delays and definitely unnecessary traffic and energy consumption. By knowing this, analysts can find out who is behind the service providing in each country. Perhaps, they could even understand  further strategies, either from the country government (in cases when the Internet is government regulated as in China, Iran, Saudi Arabia, etc.) or from the corresponding ISPs. Main strategies are often classified as either transit or peering.

E-learning platforms

E-learning technologies have revolutionized quite unexpectedly in the last 2 years! The dynamics sparked by Khan Academy’s success led to initiatives involving massive amounts of people… therefore its name: MOOC (massive open online course).

There are already a few major players with different strategies and innovations. Their growth and driving strategies can be contrasted in the following image. It shows the launch date and up-to-today users by important MOOC providers, such as Coursera, edX and Udacity.

This sudden arousal didn’t appear out of nowhere. All MOOC providers have been influenced and based on expertise acquired by previous platforms and online education solutions. For example, Open University‘s initiatives are being adapted into the coming launch of Future Learn, which before its release date has partnered with more than 15 universities in the UK. The previously-revolutionary OpenCourseWare presents a direct influence in the roots of edX.

MOOC platforms

A very popular platform is Moodle; it launched in late 2012 with a strong focus on content interaction.

KhanAcademy released its own platform. Initially openly-available, but now it is restricted and with very little information online.

Another platform player who released its open-source framework in 2012 is Google. The involvement of a tech-giant promises a lot! Even though Google Course Builder has had constant updates, it is still a very technical solution. GCB has limited market penetration and limited interoperability.

The latter applies to most, if not all, existing platforms. Internet enhances inter-platform approaches… Still, platforms lack this little push to allow simple and smooth interoperable integration.

Luckily, platforms like Google Course Builder and Moodle have open code for creative developers to participate in this new technological wave. A GCB extension presenting a first step to achieve this mentioned usability can be found under Research papers.

Netflix uncovered!

Netflix is the leading provider of online movies and TV shows, video streaming, in the US and Canada. Traffic management measures taken by Netflix impact the network infrastructure and users’ bandwidth availability. Netflix experienced fast growing demands and responded to the technical challenges to enhance availability and scalability; Netflix serves its massive amounts of content in an infrastructure-less fashion.

Researchers from the University of Minnesota and Bell-Labs produced a thorough and extensive research in 2011 to uncover Netflix infrastructure and video streaming strategies (see paper Unreeling Netflix: Understanding and Improving Multi-CDN Movie Delivery, Adhikari et.al.). Netflix popularity between users is supported by the use of multiple Content Distribution Networks (CDNs) that react to different bandwidth conditions offering discussable performance.

Infrastructure-less system
Netflix manages Internet video delivery with little infrastructure of its own. Netflix combines multiple third-party services and resorts to the use of cloud services. CDNs serve out video streaming. Amazon AWS (SimpleDB, S3) and Apache Cassandra are used for file storage. Microsoft Silverlight is the video playback platform.
After observing, monitoring, testing and measuring the network while playing videos in different locations and with different users, the authors obtained a picture of Netflix main structure, process and algorithms.

Architecture

1. Netflix handles itself user account registration and payment information
2. In the Amazon cloud Netflix runs: user sign-in, mobile device support, CDN routing, content ingestion, log recording and analysis, and DRM
3. Videos are served by 3 CDNs: Akamai, LimeLight and Level-3
4. Streaming protocol: DASH (Dynamic Streaming over HTTP)
   1. videos encoded at different quality levels
   2. videos are divided in small “chunks”
   3. the client requests one chunk at a time via HTTP
   4. next-chunk quality is calculated based on bandwidth rate measures from previous chunk

CDNs features

• Akamai: designed to simplify delivery of standards-based, on-demand video by “in the network” packaging
• Level 3: offers Managed Video Network Services (MVNS), a fully managed video delivery, IP and networking solution
• Limelight: maximizes dynamic content delivery (commerce accelerator) and increase application availability, response times and security (portal accelerator)

Netflix Process

1. Download playback application (Microsoft)
2. User authentication (Amazon)
3. Manifest file retrieval (Amazon) with streaming controls, parameters and client-specific instructions.
4. Download trickplay chunks from specified CDN, ranked in manifest file (Akamai, LimeLight or Level-3)
5. Periodic “heartbeat or log” communication from user to server (Amazon)

Performance

• CDN selection is “static”; when communication degrades, Netflix lowers quality to the lowest level before switching to another CDN; most users use only one
• “the last mile is still the bottleneck for streaming video” (universities with better access links receive higher bandwidth than residential sites)
• Each CDN performance changes over time, throughout the day
• CDNs vary significantly over geographic locations

Suggested optimization

• Optimal CDN selection strategy: better CDN selection (time/location best-performing) and combining CDN providers throughout delivery achieves a 54-70% improvement

Future work

If you thought this was interesting and cool (because discovering it from scratch IS)… you may find more interesting to know that in 2012 Netflix changed it all! Netflix OpenConnect is their new design approach.

So a new unreeling, as well as the following topics will help understand the real pros of this and future initiatives…

• Analysis of other video streaming delivery systems, i.e. Hulu
• Develop practical HTTP adaptive streaming: utilize multiple CDNs simultaneously

IT trends

We had a conference given by Deloitte regarding future trends in Information and Communication Technologies. I highlight the conference subject in red because I really don’t believe the mentioned technologies represent the future trends. I know the presenter said that technology evolves quickly and this presentation may not be valid in the near future, but in my opinion, it is long-overdue… keep reading and I’ll conclude with my reasons to state this.

So, the technologies they enforced us to focus on and pay attention to are linked one with the other and one enhancing and driving the following one:

1. Cloud :: cloud computing is not really a technology but a medium to deliver it… it is clear that the industry has shifted in this direction with the inclusion of SaaS, PaaS, NaaS, IaaS and everything-as-a-service… This has introduced many applications and business expansion possibilities without big investments while enhancing scalability. There’s a good video about the cloud that I showed in a previous post.
2. Mobility :: with the big smartphones’ adoption rate, every business should better be  online and providing accessible mobile services. There’s a lot more to this that was not deepened; like usability, privacy and security challenges and why not, some real applications showing the future of mobile apps: interaction (gamification aspects), platforms, competition, etc.
3. Social :: needless to explain much about this part… we were told about the importance of social networks and social network analysis, and some success and failure cases. They quickly “showed”/explained the famous Social Media Revolution video, yes, the one I actually used for my Social Media MOOC. My only complaint is that it was VERY consumer-oriented. And there are many social media strategies and initiatives within companies or for B2B, it would have been interested to hear something about it…
4. Big Data :: we were shown information about Google File System, Hadoop implementations and some success stories (even its politics targeted use, some about it in this post). Data analysis is important with BIG challenges in handling enormous amounts of data, semantic and Natural Language Processing (NLP).

And finally, coming back to my fist statement about these old technologies… We’ve heard about these four concepts for a long time, and maybe we have been involved in many of them for several years. If you google just a little, you can find the history and first steps for each of these, and find out they have been in talks or in development even before I started high-school (I’m referring to the ’90s). Even the last one, big data! The first paper about page ranks, and Hadoop-type infrastructures (originally from Google) appeared in late 1990s and was further studied and implemented in several industries in the early 2000s. Nowadays there are some out-of-the-box applications comprising these four concepts (I’ll link you to a paper I wrote about this in the near future).

But there is a reason Deloitte is talking about them and why they are, or may be perceived as IT trends. Consulting firms’ clients are general businesses, not necessarily innovators, like Google or Facebook. This means, their clients are not the first ones to integrate technologies, but they wait until they are proved and necessary for their businesses (actually this was a statement from another consulting firm to other topics that were discussed regarding consulting activities). Nowadays the cloud, mobility, social and big data are crucial! These are THE TODAY! And, if today you want to work on IT, you should be familiar and be able to work with them!!

Creating your MOOC

Disruptive education brings new paradigms, platforms and tools. This disruptiveness brings new ways of teaching and of learning; and it happens nowadays aided by technology!

So, what tools to use and why? There are course tools and lecture tools. To have your course live you need your content to be hosted by your own or with the use of a course platform (e.g. Khan platform or Google Course Builder) or by a site, online course oriented or not (i.e. Bb’s courseSites or Google sites).

When it comes to create your content, the tough time begins! Not only because of the planning and innovation integration, but… there are SO many resources available nowadays to support teaching, you’d better get a clear idea of what you want and how you want it beforehand.

I’ll try to summarize some tools I found useful on the way…

1. Online Presentation
   Your video lecture can be only digital, with an embedded person, with a hand shown or both. During the presentation, the presenter can write, display images, and/or use a powerpoint, an experiment, webpage or app.
   • Prezi

   You can create and host your “dynamic” presentations. The way your slides flow is cool; it allows you to zoom in and out easily so you can navigate through your presentation and deliver contents in a more natural way.

2. Animation
   • Explain Everything

   Narrate, annotate and animate your videos or images.

   • GoAnimate or PowToon or Articulate Storyline

   This is just one of the many available. They have templates with different characters and scenarios. You write the dialogues and they produce your video!

   • ShowMe

   Create tutorials with a tablet: draw and write at will.

3. Computer recording
   • Camtasia

   This is the most popular and complete program. If you need it once, you can just download the trial or pay the license for the program.

   • QuickTime Player (I believe it’s only for Mac users)

   It plays the same trick, just go to File>New>ScreenRecording. You can change options like including or not sound or your mouse movements, and selecting a section of the screen to record.

   • Screencast or Screenr or Record My Desktop

   It’s a digital recording of computer screen.

4. Assignments creation
   • Google Forms

   If you know or have access to your students, you can create your questionnaire in Google Docs and share it. Every time someone answers it you will get a new record a your Doc spreadsheet for evaluation.

   • Genghis

   It is a research project that creates a simple tool for professors to create Khan Academy-style exercises with parameters, random variables, etc.

   • ProProfs

   If you want your students to have automatic feedback. The free version will only let you keep scores and statistics for 10 attempts.

   • Google Course Platform

   As part of Google Course Builder, you can create quizzes and assignments (not very user-friendly, but you can follow the tutorial and examples).

5. Surveys/ Polls
   • Doodle

   It’s a very intuitive and popular online service to ask a specific question and have people answering with their preferences.

   • Google Forms

   Same mechanism explained before.

It depends on you and your objectives to decide what is better… Online we can find many things that are ready but it will always require additional work to put everything together. Depending on your needs, you may even choose to create most of your content and applications from scratch. Be creative! And start sharing your knowledge online (it seems very fashionable today)…

More about Real-Time!!

Paulo Verissimo is a professor at Lisbon University with a broad knowledge and research experience in Computer Science focusing on distributed architectures and its adaptability and security. He was our invited speaker last Wednesday January 9th, 2013 for Real-Time Distributed Systems class. Paulo is a good speaker and provides great insights to its audience. In this talk, he approached security issues and challenges for embedded and real-time systems.

The first part of the talk was spent talking about embedded systems. Embedded systems are moving from being simple and integrated into networked-embedded-systems. This new trend points towards a more complex way to manage multiple systems and inter-connected controls. But that’s not it, the future is including a broader view: systems of networked embedded systems.

If additionally, we include a real-time functionality, these systems must master complexity, modularity, autonomy, and dynamics of configurations and heterogeneity of compositions. All these challenges must be faced at the time of design and development. But also, depending on the nature and specifics of the application the system will also face pervasiveness of devices, ubiquity of computations, lack of perceived global state, unreliability of communication, uncertainty of timeliness (delays), insecurity, etc.

Some current and/or in-development applications using this “future” scheme of real-time embedded systems are:

• Autonomous teams of robots
• Assisted terrestrial/ aerial transportation (e.g. transit traffic): it involves cooperative functionality (i.e. HIDENETS: HIghly DEpendable ip-based NETworks and Services)
• Avionic applications and free-flight (e.g. Karyon: EU project using autonomous mobile systems as unmanned aerial vehicles – UAVs, or cooperative smart cars)

 

Uncertainty, or not predictability, is one of the main challenges. Considering it, there must be a balance between cost and performance to not sacrifice any of the two. He introduced a safe kernel, involving 1-10%, for partially-sync || hybrid systems.

To analyze these systems, it’s important to consider three components: Object, Body and Environment. The “additional” is the body. Body refers to the near environment, which is being integrated in systems with the use of sensors (for example, in planes it’s important to capture vibration sensing). The objects are the ones that consume and produce events; its communication involves the activity messages and flow/sensing.  The environment is important because it’s critical for systems to adapt to changes with a certain amount of predictability. Proofs of correctness are based on assumptions. These assumptions must be defined to offer coverage.  This coverage is analyzed and modified by monitoring the environment and detecting deviations. In mature technologies, coverage for accidental failures tends to be high.

Main threats are present at the weakest links: the assumptions (time, clock, phase, etc.). And these are vulnerabilities that are already attacked and there even exist toolkits to assist hackers.

Adaptivity and hybridization are especially present in wireless-embedded systems where security, control and performance must be carefully evaluated. Securing individual components (such as chips, PLCs, industrial PCs) is important but it’s not the end of it. Safety needs to be both internal and across boundaries. Secure systems include secured components, modular interconnection techniques/ architectures, trustworthy and resilient glue algorithms, etc.

One of the key things about cyber and physical security is to not feel confident and not attackable. Beware of the little and simple things. And remind constantly that:

• control systems are attackable
• private lines are insecure
• must not trust/ beware of intranets, firewalls, intrusion  detectors, software, network packets

 

To conclude, Paulo recommended to always be alert and to follow the “perpetual rejuvenate” technique, meaning, refresh and go back to zero on a constant basis. A quote that I personally liked a lot and summarizes perfectly systems’ security is

“What can be used can be abused”.

Social Media Tools MOOC

Social Media (SM) Tools is a subject about current SM strategies, applications and implementations presented in a MiniMooc thanks to two social media strategists in the US.

There are three main branches concerning SMT: listening, sharing and collaborating! This 30-minute course (MOOC) pretends to explain and exemplify the different categories and steps required to get started with these tools.

What will you find?

• General course overview and link to presentation
• Lecture videos
• Final assessment and survey

How was it created?

We created two versions: quick demo site and one course using Google Course Builder platform.

Google Course Builder is based on Python integrated into HTML + JS. You can host it within google using Google App Engine, which offers you the infrastructure to build and host web applications. All of it is very straight-forward, even though some technical knowledge may be needed.

The whole preparation and set of tools needed at each step can be seen in this presentation.

(Interactive) Individualized Integrated BOOK

It is said that this will be the very FIRST IIB: individualized–integrated book! And I am in the experiment group to test it, well test a chapter…

Here are the details:

• It is an iOS app (iPad only)
• The chapter talks about Page Ranking (chapter 3 from this book)
• App launched by Princeton University research team
• 100 people is in the experiment
  • I am from group B: I read first the book, and then play with the IIB
• It integrates supplementary material like videos, wiki info, relevant forum search threads, and links to a coursera course that teaches from this book
• At the end of each section, some questions are answered so the book can adapt to each individual‘s understanding
• There’s a final survey including demographics, interactivity usefulness/efficiency and overall experience

I read the book chapter and it’s quite short and very easy to read; besides I am familiar with the topic (from previous research, courses and a talk presentation I already talked about). I am still playing around with the app before submitting my survey…

The experience was good, I’ll highlight some of my comments:

• The app is still somewhat slow, you have to be patient!
• The app has no logo yet! So you only have an awful gray icon
• The additional features (besides iPad’s copy/define, they added Wikipedia, book wiki and course forums) are somewhat useful, maybe Wikipedia the most; but each opens a new screen, it would be nice to have something embedded and the option to expand “out” if you need it
• They included the course explanatory videos in the corresponding section… It was good, except that they are SLOW (meant to be part of a class explanation) I would had liked to have the speed-changing feature as in the course platform
• I would recommend removing the gray bookmarks per paragraph, they didn’t seem to do anything
• Before moving on new sections, you must answer some questions so the IIB can learn what would be relevant to you… once you click on the lock, there’s no way back; I’m wondering how usable would it be to have a warning before moving to the questions
• With all this interactivity, I was surprised not to find optional examples and exercises to be solved and measure your progress
• The app allows you to add comments (notes) and links, these are share with everybody else… it is cool, except I didn’t know this was possible; I only saw one comment and was confused about what it meant or where did it come from… if you add one, you cannot modify/delete it afterwards
• I still don’t figure out, if possible, how would the book be reusable and shareable

You can see how it works with the tutorial video they sent us…  I’ll be posting more about it when we also get some feedback about how all the experiment went. And let’s wait and see what improvements they incorporate!