Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Hosting recipes

Each recipe sets up one host for a chronozarr store: the bucket, public read access, CORS and caching. chronozarr publish then uploads the store to S3, R2, Google Cloud Storage or Azure (Publish a permanent link). For another host, each recipe also gives the put function for the upload script.

The recipes have not all been run against live accounts. Hosting observations records which ones were.

Amazon S3 with CloudFront

Keep the bucket private. CloudFront reads it through an origin access control (OAC). The distribution in this recipe is public: anyone with its URL reads the store (Viewing a private store). CloudFront gives HTTPS on your own domain and caching. It is also the only way to add Timing-Allow-Origin in front of S3.

  1. Create the bucket and block public access.

    BUCKET=my-chronozarr-stores
REGION=us-east-1
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION"   # outside us-east-1 add --create-bucket-configuration LocationConstraint="$REGION"
aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

  2. Create a CloudFront distribution in the console or with aws cloudfront create-distribution. Use these settings:

    • Origin: the REST endpoint of the bucket, BUCKET.s3.REGION.amazonaws.com.
    • Origin access: origin access control, with signing on.
    • Viewer protocol policy: redirect HTTP to HTTPS.
    • Allowed methods: GET, HEAD, OPTIONS.
    • Cache policy: CachingOptimized. The cache key has no headers and no query strings.
    • Response headers policy: the custom policy from step 4.

  3. Attach this bucket policy. It allows only that distribution. Fill in the account id and the distribution id. The second statement lets S3 answer 404 for a file that does not exist. Without it, S3 answers 403, and chronozarr doctor fails the store. It also lets anyone with the distribution's address list the names of the objects in the bucket, at the address's root. Keep only public stores in this bucket, or set a default root object on the distribution.

    {
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCloudFrontRead",
    "Effect": "Allow",
    "Principal": { "Service": "cloudfront.amazonaws.com" },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-chronozarr-stores/*",
    "Condition": { "StringEquals": { "AWS:SourceArn": "arn:aws:cloudfront::<account-id>:distribution/<distribution-id>" } }
  }, {
    "Sid": "AllowCloudFrontNotFound",
    "Effect": "Allow",
    "Principal": { "Service": "cloudfront.amazonaws.com" },
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::my-chronozarr-stores",
    "Condition": { "StringEquals": { "AWS:SourceArn": "arn:aws:cloudfront::<account-id>:distribution/<distribution-id>" } }
  }]
}

  4. Create the response headers policy with aws cloudfront create-response-headers-policy --response-headers-policy-config file://chronozarr-headers.json. The policy adds the CORS headers with OriginOverride, so the bucket needs no CORS configuration. It also adds Timing-Allow-Origin.

    {
  "Name": "chronozarr-store",
  "CorsConfig": {
    "AccessControlAllowOrigins": { "Quantity": 1, "Items": ["*"] },
    "AccessControlAllowHeaders": { "Quantity": 1, "Items": ["Range"] },
    "AccessControlAllowMethods": { "Quantity": 2, "Items": ["GET", "HEAD"] },
    "AccessControlAllowCredentials": false,
    "AccessControlExposeHeaders": { "Quantity": 4, "Items": ["Content-Range", "Content-Length", "ETag", "Accept-Ranges"] },
    "AccessControlMaxAgeSec": 86400,
    "OriginOverride": true
  },
  "CustomHeadersConfig": {
    "Quantity": 1,
    "Items": [{ "Header": "Timing-Allow-Origin", "Value": "*", "Override": true }]
  }
}

  5. Publish with chronozarr publish my_store --destination s3://$BUCKET/PREFIX --public-url https://<distribution-domain>/PREFIX. Or upload with the upload script, and define put as:

    put() { aws s3 cp "$1" "s3://$BUCKET/$PREFIX/${1#./}" --cache-control "$CC" --only-show-errors; }

The store URL is https://<distribution-domain>/$PREFIX.

Notes:

  • The upload stores Cache-Control on each object. CloudFront passes it through. You do not invalidate anything, because prefixes are immutable.
  • aws s3 cp labels extensionless objects binary/octet-stream. CloudFront compresses only a fixed list of content types, for objects between 1,000 and 10,000,000 bytes. binary/octet-stream is not on the list. Shards are served as stored. Confirm this with Verifying by hand.
  • A public bucket without CloudFront also serves Range and Cache-Control. It needs an S3 CORS configuration instead of the response headers policy. It cannot send Timing-Allow-Origin.
  • S3 sends CORS headers only on requests that carry an Origin header.

S3 CORS configuration for a public bucket:

{
  "CORSRules": [{
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "HEAD"],
    "AllowedHeaders": ["Range"],
    "ExposeHeaders": ["Content-Range", "Content-Length", "ETag", "Accept-Ranges"],
    "MaxAgeSeconds": 86400
  }]
}

Apply it with aws s3api put-bucket-cors --bucket "$BUCKET" --cors-configuration file://s3-cors.json.

Amazon S3, public bucket

A public bucket is the shortest route on AWS. Anyone can read every object in it, so keep it for public stores. S3 gives no custom domain and no Timing-Allow-Origin; use CloudFront for those.

  1. Create the bucket, and allow a public bucket policy on it.

    BUCKET=my-chronozarr-stores
REGION=us-east-1
aws s3api create-bucket --bucket "$BUCKET" --region "$REGION"   # outside us-east-1 add --create-bucket-configuration LocationConstraint="$REGION"
aws s3api put-public-access-block --bucket "$BUCKET" --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=false,RestrictPublicBuckets=false

  2. Attach this bucket policy with aws s3api put-bucket-policy --bucket "$BUCKET" --policy file://public-read.json. The first statement lets anyone read the objects. The second lets S3 answer 404 for a file that does not exist; without it, S3 answers 403 and chronozarr doctor fails the store. It also lets anyone list the names of the objects.

    {
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "PublicRead",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-chronozarr-stores/*"
  }, {
    "Sid": "PublicNotFound",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::my-chronozarr-stores"
  }]
}

  3. Publish. --apply-cors adds the CORS rule that the viewer needs.

    chronozarr publish my_store --destination "s3://$BUCKET/my_store-v1" --apply-cors

The store URL is https://BUCKET.s3.REGION.amazonaws.com/my_store-v1. publish derives it, checks it and prints the viewer link.

Cloudflare R2

data.chronozarr.org is served this way. See deploy/README.md.

  1. Create the bucket, set its CORS rules and add a custom domain.

    BUCKET=my-chronozarr-stores
npx wrangler r2 bucket create "$BUCKET" --location enam
npx wrangler r2 bucket cors set "$BUCKET" --file r2-cors.json --force
npx wrangler r2 bucket domain add "$BUCKET" --domain data.example.com --zone-id <zone id>

    Save the rule below as r2-cors.json (the repository keeps it as deploy/r2-cors.json). It uses wrangler's rule format. The S3 CORS array format does not work here:

    {
  "rules": [{
    "allowed": {
      "origins": ["*"],
      "methods": ["GET", "HEAD"],
      "headers": ["Range", "If-Match", "If-None-Match", "Content-Type"]
    },
    "exposeHeaders": ["Content-Range", "Content-Length", "ETag", "Accept-Ranges"],
    "maxAgeSeconds": 86400
  }]
}

  2. Upload with the upload script. Define put as:

    put() { npx --no-install wrangler r2 object put "$BUCKET/$PREFIX/${1#./}" --file "$1" --cache-control "$CC" --remote > /dev/null; }

  3. Create a Cache Rule and a Transform Rule on the hostname. Both are in the dashboard under Rules. Both need zone write access. The wrangler login token used for deploys does not have it.

Wrangler needs about 2 seconds per object. An unsharded store has thousands of objects. For the first upload of an unsharded store, use an S3 client against the S3 endpoint of R2. Each later append uploads only the objects it wrote (appending). Upload times are in Measurement log.

Cache Rule

Cloudflare does not cache objects without a file extension by default. Chunk keys have none, so their responses stay cf-cache-status: DYNAMIC until you add a Cache Rule.

Create the rule in the dashboard under Caching, then Cache Rules:

  1. Set the field to Hostname, the operator to equals, and the value to your data hostname. The expression is (http.host eq "data.example.com").
  2. Set Cache eligibility to Eligible for cache.
  3. Set Edge TTL to Use cache-control header if present, bypass cache if not.
  4. Under Caching, then Configuration, set Browser Cache TTL to Respect Existing Headers.

The upload script sets Cache-Control on every object (the cache classes). The rule follows those headers. Chunks stay cached for a year, and zarr.json expires after 300 seconds, so an append shows up within five minutes.

Step 3 also keeps 404 responses out of the cache. A missing chunk means fill, and an append later creates it.

Step 4 matters. The zone default raises any shorter browser lifetime to four hours, so zarr.json would reach browsers with max-age=14400.

Do not match on a URI Full wildcard. That expression matches nothing, and every response stays DYNAMIC. The dashboard warns that a hostname rule "may not apply to your traffic" for an R2 hostname. The warning is wrong for this case. The verification is in Measurement log.

Timing-Allow-Origin

Add a Transform Rule of type Modify Response Header. Use the same hostname expression. Set the action to Set static, with Timing-Allow-Origin = *.

Object size and miss cost

A cache miss on a range read pulls the whole object from R2, so the cost grows with object size. In the live unsharded store the largest chunk object is 1.8 MB, and a miss is cheap. In a sharded store, a smaller shard_time makes a miss cheaper. Cloudflare limits the size of a cacheable object by plan. Measurements are in Measurement log.

One cached copy per requesting site

With a CORS policy, R2 sends Vary: Origin, so Cloudflare caches a separate copy of each object for each site that requests it. Visitors of one site share a cache. A site that serves the reader or the viewer from its own origin starts with a cache miss on every object. So does a client that sends no Origin header (Python, GDAL, curl). An iframe of the hosted viewer sends Origin: https://chronozarr.org. A Transform Rule that removes Vary does not merge the copies. Measurements are in Measurement log.

r2.dev

The public development URL (wrangler r2 bucket dev-url enable) is rate limited. Use it for a first check, and use a custom domain for anything public.

Google Cloud Storage

  1. Create the bucket, make it public and set its CORS rules.

    BUCKET=my-chronozarr-stores
gcloud storage buckets create "gs://$BUCKET" --location=US --uniform-bucket-level-access
gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" --member=allUsers --role=roles/storage.objectViewer   # anyone can read and list the objects
gcloud storage buckets update "gs://$BUCKET" --cors-file=gcs-cors.json

    GCS takes one responseHeader list. It uses the list for both Access-Control-Allow-Headers and Access-Control-Expose-Headers. Put Content-Range in the list next to Range. This is gcs-cors.json:

    [{
  "origin": ["*"],
  "method": ["GET", "HEAD"],
  "responseHeader": ["Content-Type", "Range", "Content-Range", "Content-Length", "ETag", "Accept-Ranges"],
  "maxAgeSeconds": 86400
}]

  2. Upload with the upload script. Define put as:

    put() { gcloud storage cp "$1" "gs://$BUCKET/$PREFIX/${1#./}" --cache-control="$CC" --quiet; }

The store URL is https://storage.googleapis.com/$BUCKET/$PREFIX.

Notes:

  • GCS honors CORS on storage.googleapis.com. It does not honor CORS on storage.cloud.google.com, which authenticates with cookies.
  • A public object with no Cache-Control is served with public, max-age=3600. The upload command above overrides it.
  • GCS rejects allUsers when the organization enforces public access prevention. Change that policy for this bucket, or put a backend bucket behind Cloud CDN.
  • You cannot set Timing-Allow-Origin on a bucket or an object. A backend bucket behind an external HTTPS load balancer can send it as a custom response header: gcloud compute backend-buckets create ... --custom-response-header='Timing-Allow-Origin: *'. The header is a MAY. Skip it unless you benchmark the viewer.
  • Do not upload objects with Content-Encoding: gzip. GCS applies transcoding, which changes the bytes it serves.

Azure Blob Storage

  1. Allow anonymous reads on the storage account, and create a container whose public access level is Blob. An organization policy can forbid both. Then put Azure Front Door in front of the container, and pass its address as --public-url.

    ACCOUNT=mystorageaccount
az storage account update --name "$ACCOUNT" --resource-group my-group --allow-blob-public-access true
az storage container create --name stores --account-name "$ACCOUNT" --public-access blob --auth-mode login

  2. Sign in with az login, or set AZURE_STORAGE_CONNECTION_STRING.

  3. Publish. CORS for Blob storage applies to the whole storage account. --apply-cors adds the viewer rule after the existing rules. Azure allows five rules.

    chronozarr publish my_store --destination "az://$ACCOUNT/stores/my_store-v1" --apply-cors

The store URL is https://ACCOUNT.blob.core.windows.net/stores/my_store-v1.

Source Cooperative

Source Cooperative (Radiant Earth) hosts open datasets. It serves them through a data proxy at https://data.source.coop. An object has the address https://data.source.coop/<account>/<product>/<path>.

You do not configure CORS or cache headers on this host. The proxy is in beta. The storage behind it (S3, GCS, Azure or R2) is not your choice.

Expect chronozarr doctor to pass checks 1 to 8 and 12. It warns on cache-control for a versioned prefix. It reports edge cache and timing-allow-origin as info. Observations: Measurement log.

You need an account and upload access for the product. If the product page shows no upload option, write to hello@source.coop.

Choose one of two upload routes.

Route 1: the web UI

  1. Open the product page.
  2. Click the lock icon, then Edit Mode.
  3. Drag in the level directories (0/, 1/, ...), and volatility/ if the store has one.
  4. Drag in the root zarr.json by itself, last.

The UI does not order uploads, so step 4 is a separate action. A sharded store (--shard) has about a hundred objects, which is workable here. An unsharded store has thousands of objects, which is not.

Route 2: an S3 client through the proxy

  1. Run source-coop login to get temporary credentials.

  2. Create an AWS profile named source-coop, as the Source docs describe.

  3. Upload with the upload script. Define put as:

    ACCOUNT=my-account
PRODUCT=my-product
put() {
  aws s3 cp "$1" "s3://$ACCOUNT/$PRODUCT/$PREFIX/${1#./}" \
    --endpoint-url https://data.source.coop --profile source-coop \
    --cache-control "$CC" --only-show-errors
}

The store URL is https://data.source.coop/$ACCOUNT/$PRODUCT/$PREFIX.

Check the cache-control line of doctor. The proxy may not store the --cache-control value.

Upload without chronozarr publish

chronozarr publish runs the upload order of Immutable prefixes and upload order for S3, R2, Google Cloud Storage and Azure. For another host, or to upload by hand, use the script below with the put function of your host's recipe.

The upload script

The three phases work for every host. The script below runs them.

  1. Save the script as upload.sh.
  2. Define put from the recipe for your host above.
  3. Run bash upload.sh.

put FILE uploads ./FILE, relative to the store directory, to $PREFIX/FILE with Cache-Control: $CC.

#!/usr/bin/env bash
set -euo pipefail
STORE=data/stores/my_aoi/chronozarr-2   # local store directory
PREFIX=my_aoi/chronozarr-2              # new for every encode
CC="public, max-age=31536000, immutable"
 
# put() goes here, from the recipe for your host
 
export -f put
export STORE PREFIX CC BUCKET   # plus anything else put() reads
cd "$STORE"
# 1. chunk and shard objects
find . -type f ! -name zarr.json -print0 | xargs -0 -n 1 -P 8 bash -c 'put "$1"' _
# 2. group and array metadata below the root
find . -type f -name zarr.json ! -path ./zarr.json -print0 | xargs -0 -n 1 -P 8 bash -c 'put "$1"' _
# 3. root metadata, last
put ./zarr.json

If an upload fails in phase 1 or 2, xargs exits with 123. Then set -e stops the script before the next phase.

The R2 upload script

scripts/upload_stores.sh uploads to R2 in the same three phases. It stops before the next phase when an upload fails.

It reads the store from data/stores/<aoi>/$STORE. Set STORE=chronozarr-4 for a versioned prefix.

It sets Cache-Control for each object:

  • Every chunk and shard gets immutable, except the objects that an append rewrites.
  • The objects that an append rewrites get max-age=300. The cache classes list them.
  • --trailing-ttl SECONDS changes the 300.
  • --dry-run prints the class of every object and uploads nothing.