Mesh with Rev Caching Configuration

If you are using Rev with your DME(s), you will automatically utilize a shared cache that takes advantage of the DME mesh architecture. The goal of this architecture is for all your DMEs to communicate and reduce out-of-network bandwidth costs as a result. This, in turn, will reduce bandwidth for both DME to Rev-cloud communications and within your network topology from DME to DME. The DME mesh architecture also focuses on delivering content as close to the edge as possible. Once the content is within the DME mesh architecture, DMEs communicate between themselves to distribute the content effectively.

In terms of reducing network bandwidth, VBrick’s first approach is to support reducing the number of DMEs to which Rev will pre-positioned content. This reduces out-of-band network bandwidth, as well as storage requirements for the DME. Once content is ready, Rev instructs all pre-positioned DMEs to retrieve it. When the DMEs receive the download command, they first inspect their local storage to see if they already have the content and, if so, they are done.

For DMEs that do not have the content, a specific, randomly assigned, rotating DME (from the set of pre-positioned DMEs) will immediately begin downloading the content while the other DMEs wait a random amount of time (no more than 5 minutes). This allows the first DME to be seeded with the content, and subsequent download requests will utilize the DME mesh architecture if possible. (Larger content may require more time than is allotted and thus be requested directly from Rev until it is seeded in the mesh.) After the wait, each of the other DMEs will begin by first inspecting the mesh for the content, or download it from Rev as necessary.

Note: Pre-positioning is determined by customers. DMEs are defined as pre-positioned by system administrators within the Rev interface. Please see Rev Online help for details.

In terms of improving playback, the DME mesh also focuses on state-of-the-art edge caching and multi-protocol first-time. Once the content is in the DME mesh, VBrick leverages several technologies so it can be distributed and retrieved seamlessly by VBrick players.

To illustrate the mesh architecture, several common use cases are presented below. Consider the example of 3 DMEs, DME-1 (prepositioned), DME-2, and DME-3 all reachable. DME-1 is the only prepositioned DME, so any new content uploaded to Rev will automatically be downloaded to DME-1. DME-1 currently has the following stored videos: video1.mp4, video2.m3u8 (HLS).

●Case 1: HTTP. A player requests to play video2.m3u8 from DME-2. DME-2 first checks locally for the HLS, but cannot find it. DME-2 then checks with the mesh. The mesh reports DME-1 has the HLS file and delivers it to DME-2. DME-2 caches the HLS file and then delivers it to the player. Playback begins for the user from DME-2. DME-2 will (first-time) cache all the .ts files that make up the HLS stream. Any additional (new) player requests for the HLS stream to DME-2 will pull from its cache.

This is the general solution when dealing with HTTP based content. If it is not local, then the Mesh is inspected. If it is found in the mesh, it is cached on the second DME. If it is not in the Mesh, then the player will fall back to Rev delivery.

●Case 2: RTMP. A player requests to play video1.mp4 using RTMP from DME-2. DME-2 first checks locally for the video1.mp4 file, but cannot find it. DME-2 then queries the mesh. The mesh reports DME-1 has the file. We cannot use the shared cache for delivery across RTMP to the player, so we redirect the player to DME-1. Playback begins for the user from DME-1.

DME-2, in parallel, has identified missing requested content that is present in the mesh, but not locally present. DME-2 will then initiate a mesh-copy of the content to get a local copy of video1.mp4. Once local, this copy can be provided directly for any subsequent RMTP requests. This is the general solution when dealing with requests across RTMP.

●Case 3. Ultimate Fallback. A player requests a video from DME-2 that is not present or in any reachable DMEs. At this point, if it is not on the DME-2 and it is not within the mesh architecture, the request will fail and the player will fall back and play from Rev. Playback begins for the user from Rev.

DME-2, in parallel, has identified missing requested content that is not present in the mesh, and not locally present. DME-2will then notify Rev of the cache-miss content. Rev will receive the notification and then immediately re-preposition the content to all pre-positioned DMEs and the DME that reported the miss.

These case studies illustrate the basic use cases of retrieving data and how the mesh identifies, locates, and distributes content. This provides for a much more efficient delivery and distribution of content, but it also fills up our DMEs.

Starting with DME version 3.7, VBrick introduced an automated process for removing older content. This is done to keep enough storage space ready for additional downloads. The system monitors disk storage and when it reaches a predefined threshold, content is then evaluated on the DME and deleted based on a modified LRU (least recently used) algorithm. This algorithm identifies and orders content (in the UploadedVideos directory) by when it was last watched. The rationale is the standard caching theory that important content is watched more often than less important content. It then inspects the Mesh to determine if the content is locatable on another DME. The DME then removes sufficient content (to meet the threshold) ONLY if it exists elsewhere within the Mesh.

All DMEs will be automatically included within the mesh and utilized for content location and distribution. As such, reachability (ability to connect) between the DMEs is a critical issue for the mesh architecture. The mesh architecture has limited usefulness if DMEs cannot reach each other.

When moving into the DME mesh, the following guidelines should be considered:

●Do not add DMEs to Rev during peak use times. Adding a DME will cause a mesh update across all DMEs. This will cause playback disruptions as the mesh resets.

●Do not add DMEs to Rev during any live event. This will cause playback disruptions as the mesh resets. Adding a DME will cause a mesh update across all DMEs. This will cause playback disruptions as the mesh resets.

●Every meshed DME must have reachability to at least one other DME. DMEs rely on other DMEs to get and share content. If they cannot communicate, then they cannot share content. Visit the System Configuration > Caching page to see peered DMEs and their reachability status.

●Every meshed DME must have reachability to at least one other prepositioned DME. As mentioned above, DMEs need DMEs to share content. However, there should be reachability to at least 1 prepositioned DME for all DMEs. By doing this, any prepositioned content will be available to the non-prepositioned DME.

●Every stream and video file must have a unique name across the DMEs and within the mesh. This is a critical component of the mesh. Within the mesh, streams and pieces of streams (e.g., HLS .ts files) are flowing between DMEs that are serving them to players. When a duplicate name is entered into the system from separate DMEs this causes the mesh to deliver incorrect packets of streams to players. If you are transmuxing, transrating or creating new streams on DMEs please verify that the names are unique.

○Plan for the appropriate use of your DMEs. Are they simply reflectors? VOD storage? Stream manipulation and transmuxing? Configuration depends on your topology and use.

○Most importantly, evaluate your network topology to best utilize the mesh. VBrick Customer Service and Professional Services are a good resource for questions and solutions.

As noted, see the “Preposition DME Content” topic in Rev Online help for details on how to preposition content in Rev.

○Navigate to System Configuration > Rev Interface and make sure the Rev Interface has been enabled and is running. See: Rev Interface for details if you need instructions.

○As a short cut, in the bottom of each page to the far right should be two True or False indicators, one on top of the other. The top indicator is True if the Rev Interface is Enabled, and the bottom indicator is True if the Rev Interface Service is running. Both should be True for correct communication to Rev. If either is false, reset the Rev Interface by going to the System Configuration > Rev Interface page verify your settings and uncheck the Rev Enabled checkbox, click Apply, re-enable it, and click Apply again.

○Navigate to the System Configuration > Streaming page and verify the Cache System Resources (Memory/Disk) setting. This is an overall setting that controls the amount of caching resources the system will use. Please see the Caching topic for more details.

○Navigate to System Configuration > Caching. On this page, you can see how the DME is configured, HLS Remote Hosts, and all the peer DMEs within the Mesh. Inspect each peer DME to identify connectivity (via the Reachable column) and when the last connection was made.

○The HTTP Port and ICP Port should be listed at the top of the page. Best practice is to use using default port numbers (80 & 31030) for proper mesh usage.

○Within the HLS Remote Hosts section you may see 1 or more hostnames – these are set by Rev if your account is enabled and using the Video Conference Recording and Streaming functionality. Otherwise, these will be blank. Do not edit these fields.

○Review how this DME is configured, it will be identified as setup/or not for VOD playback and prepositioning. If the setting is (or should be) different, visit Rev to centrally reset it.

○Within the Alternative Sources section you should see the list of your peer DMEs (to this current DME). This DME should be able to reach at least one prepositioned DME, OR this DME should be prepositioned. If you cannot see all of the peers, please use the Display dropdown list at the top of the page to increase the display size.

○There is also a Lock icon that can be hovered over to display status on certificate installation. This represents if a DME is locked down to https serving or not.

○Click the Auto-Configure from Rev button to perform a Mesh Update from Rev. This will request from Rev all mesh information (list of peer DMEs) and process each as necessary. If the DME detects any change between the current settings and the new list downloaded from Rev , then the DME will locally resets its caching system. This may have an impact on existing streams and viewers.

Because each DME can be configured differently within your deployment, this should be repeated on each DME you wish to inspect.

Tip[1]: If you do not want to utilize the distributed caching mechanism, DME Mesh, then set all DMEs to preposition content on Rev.

Tip[2]: DMEs can be set as prepositioned on Rev. You cannot set it within the DME.

Tip[3]: From your DME, if there are no other reachable prepositioned DMEs (identified within the table) then this DME cannot utilize the Mesh and is stranded. Consider making this DME prepositioned.

If you are an existing customer, when you upgrade to Rev v7.6 and DME v3.6 (or beyond), your DMEs will be automatically defaulted as prepositioned.