We have an Underlay IPFS cluster! It lives at
underlay.store and exposes two subdomains:
gateway.underlay.store is for fetching files from IPFS. This includes files that are pinned to the cluster (which will be fast to serve) but also works as a gateway to any file that anyone has on IPFS, anywhere (which will be a little slower to serve, but at least you can get them all in the same place). This subdomain is public! We want anyone to be able to use our gateway.
cluster.underlay.store is the API endpoint for adding and removing files from the cluster. This subdomain is private and not open to the Internet.
underlay.store domain was registered on name.com in December 2018. It uses Cloudflare’s nameservers.
The cluster is a group of EC2 instances all running 1) an IPFS daemon and 2) an IPFS Cluster daemon. IPFS Cluster is project maintained by Protocol Labs that lets groups of IPFS nodes share responsibility for storing a collective set of files (“distributed pinset orchestration”) by running a distributed consensus algorithm (Raft).
Each cluster peer is a
t3.medium instance attached to a 512GB external EBS volume mounted at
/dev/sdb. The user data script for each instance that handles installing and configuring IPFS and IPFS Cluster, and bootstrapping into consensus with the rest of the cluster lives at this GitHub Gist.
The gateway has a CNAME record pointing to an AWS application load balancer that forwards HTTP and HTTPS traffic to port
8080 of its target group, which includes all of the cluster peers. This means that requests to
gateway.underlay.store are forwarded one of the peers, and AWS decides which.
/ipfs/ at the start of the path; it’s necessary.
Cloudflare is configured to cache all pages under the
gateway subdomain indefinitely.
A cluster subdomain is a separate application load balancer that exposes APIs for adding and removing files (and configuring the cluster itself). There are two sets of APIs on two respective ports:
The load balancer forwards port
5001 to port
9095 of its target group, which is the IPFS API Proxy endpoint of the cluster. This API endpoint perfectly mimics the regular IPFS HTTP API documented here, except it applies pin operations to the entire cluster instead of just the single node. You can read more about the proxy API here.
The load balancer also forwards port
9095 to port
9095 of its target group, which is the HTTP API for IPFS Cluster itself. This includes equivalents for adding and removing files, but also other cluster-specific endpoints like adding and removing peers and generating health report graphs. These routes are documented here.
The upshot is that there’s a single endpoint
cluster.underlay.store that can behave exactly as if it’s a single IPFS node (right down to the API!) but is actually a collection of peers in consensus, all handling a share of the traffic.
curl -F firstname.lastname@example.org http://cluster.underlay.store:5001/api/v0/add
The two load balancers on AWS are configured with security groups
gateway accepts incoming traffic from anywhere on ports
443 (although Cloudflare is configured to upgrade every http connection to https on
cluster restricts incoming traffic to the
priorart-file-parser server’s security group. It’s important to protect it from the broader internet because then anyone can make us download and store stuff.
But if you just want to test it out, add your IP to the
cluster security group and refresh
https://cluster.underlay.store:5001/api/v0/id a bunch of times! It should roughly alternate between the peer ids of the cluster peers.
Here are some useful things that are pinned to the cluster: