How-To: Super Scalable Streaming with HLS
We’ve just launched AudioEngine, a free Docker-ecosystem based system for completely managing your stream deployment. This includes HLS and DASH, as we outlined in this tutorial. You can read up on it, and get the code, here.
This tutorial will run you through how to build a crazy scalable streaming platform for your radio station, using HLS. This is a long one, so grab a cup of coffee or a Coke.
First, you’ll need to sign up to Google’s Project Shield. This is totally free for independent (e.g. community) radio stations. You can do that here. If you want to use another provider, that’s fine also.
You also require a Linux (or other UNIX-y) server. Although nginx builds fine on Windows, we haven’t tested it.
Although this works on most platforms, there are some exceptions. Notably, in Internet Explorer on Windows 7 and below. For that, you can probably use Flash as a polyfill. We’re not using HE-AAC here, as it’s not supported in all major browsers. If you want to (you’ll break support for Flash/IE, Firefox, and probably Opera – far too many users) add “-profile aac_he_v2”.
Setting Up the Server
On your Linux server, make sure you have GCC installed.
On CentOS/Fedora/etc., run yum -y install gcc gcc-c++ make zlib-devel pcre-devel openssl-devel git autoconf automake cmake pkgconfig.
On a Debian machine, you can run apt-get -y install build-essential zlib1g-dev libpcre3 libpcre3-dev libbz2-dev libssl-dev tar unzip git autoconf automake cmake pkgconfig. If you’re using another flavour of Linux (or UNIX), find the packages above and install ’em.
Next, we need to get and compile nginx with the “nginx-ts-module”. You can also use nginx-rtmp-module, but there’s no point unless you want a full RTMP server.
Move to the tmp directory with cd /tmp
To do this, go to the nginx website and download the latest stable version. For instance, run wget http://nginx.org/download/nginx-1.12.1.tar.gz
Extract it. tar -xf nginx-1.12.1.tar.gz
Next, run git clone https://github.com/arut/nginx-ts-module.git to download nginx-ts-module
Now, cd nginx-1.12.1
Build nginx by running:
./configure \
--user=nginx \
--group=nginx \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--with-http_ssl_module \
--with-pcre \
--with-file-aio \
--with-http_realip_module \
--without-http_scgi_module \
--without-http_uwsgi_module \
--without-http_fastcgi_module \
--add-module=/tmp/nginx-ts-moduleThen, make && make install
Now, to install it on boot, run these commands:
useradd -r nginxwget -O /etc/init.d/nginx https://gist.github.com/sairam/5892520/raw/b8195a71e944d46271c8a49f2717f70bcd04bf1a/etc-init.d-nginx
The on-boot commands may differ based on your operating system. Google for “build nginx ” and see what it advises there.
Excellent. We’ve built it!
Configuring nginx
In /etc/nginx/nginx.conf, add the following server block:
server {
listen 80 default;
root /srv/stream;
client_max_body_size 0;
location /stream {
allow 127.0.0.1;
deny all;
ts;
ts_hls path=/srv/stream/hls segment=10s segments=30;
ts_dash path=/srv/stream/dash segment=10s segments=30;
}
location ~ \.(h|mpd|m3u8)$ {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Cache-Control' 'max-age=9';
}
location ~ \.(ts|mp4)$ {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Cache-Control' 'max-age=86400';
}
}Next, run mkdir /srv/stream /srv/stream/hls /srv/stream/dash /srv/stream/hls/stream /srv/stream/dash/stream to make our web folders for nginx to serve. Then, chown nginx: /srv/stream/dash
Now, we can start nginx with service nginx start and chkconfig nginx on.
Our HLS stream becomes available at http://host-name/hls/stream/index.m3u8, and DASH at http://host-name/dash/stream/index.mpd.
Adding the Plumbing
We have our origin server configured now. Next, we need to pipe some audio into it.
First, install ffmpeg. We need it with “libfdk-aac” so we can get a good quality stream. This will use the most modern HE-AACv2 codec, as it ensures the highest quality at the lowest bitrates. You can follow a guide here for CentOS on how to do so. For Ubuntu/Debian, make sure you have installed autoconf, automake, cmake, and pgkconfig (we did this above). You can then follow the CentOS guide, which should be mostly the same.
In /usr/local/run.sh, add the following (replacing the stream.cor bit with your regular stream URL):
#!/bin/bash
while [ true ]; do
ffmpeg -i https://stream.cor.insanityradio.com/path_to_icecast_mp3 -c:a libfdk_aac -b:a 128k -f mpegts http://127.0.0.1/stream
sleep 0.1
doneThese commands tell ffmpeg to grab a copy of your Icecast stream, encode it with High Efficiency AAC, and send it to the server so that it can repackage it for browsers. If it crashes, it will automatically restart after 1/10th of a second (this is intentional, as if the system is ill-configured, the batch job would choke available system resources).
If you want to get even better quality, you can use an RTP stream or something else in the ffmpeg command. The reason I’m not using this in the example above is that, to achieve that, it requires an overhaul of your streaming architecture to use PCM on ingest. If you have a Barix based STL, you can configure it to RTP send to your streaming server, thus saving the need for an extra audio interface.
Append to /etc/rc.local the following line to instruct the system to automatically restart our script on boot:
/usr/local/run.sh&(You could do this using systemd or initv for production, but this works well enough)
Excellent. When you run /usr/local/run.sh& or reboot, you should be able to access your streams.
Setting up the CDN
CDNs are designed to copy your content so that it can scale. For instance, instead of having one server (our origin) serving 100,000 clients, we can send this to lots of “edge” servers. Providers make this easy and are more cost efficient.
We’re using Project Shield, because it’s totally free for indie media, and harnesses the same power of Google. Other freebies exist like CloudFlare, but CloudFlare’s terms of service don’t let you use it for multimedia.
The nginx config we used above should allow your CDN to work properly from the onset. It will cache the manifest/index files (that instruct the client which audio segments to get) for 10 seconds. After 10 seconds, the files will have changed so we want the edges to update. The media files will be cached for a day so once the CDN grabs it once, it will never need to again.
Usage & Testing
I’ll post some sample player code on GitHub, but you probably want to use the HLS stream with hls.js. You can test it on this page. Why not DASH? nginx-ts-module, at the time of writing, has tiny gaps between audio segments. These are pretty audible to an average listeners, so until that is fixed we might as well continue using HLS.
Once again, the streaming URLs will be http://yourcdnhost.com/hls/stream/index.m3u8 (HLS), and http://yourcdnhost.com/dash/stream/index.mpd (DASH).
Do consider setting up SSL. It requires a couple of tweaks in your nginx configuration, and LetsEncrypt makes it much easier. Google are working on support for LetsEncrypt in Shield – hopefully when you read this it will be much easier to use, instead of having to manually replace SSL/TLS certificates every 60 days.
Going Further
The nginx-rtmp-module, similar to install, provides adaptive bitrate HLS (but not DASH). However, there is a bug in Mobile Safari that adds silent gaps between the segments. The TS module will soon support now supports adaptive HLS, this guide will be updated when that happens. You can also hack it yourself.