Squid Labs

Set Up a Distributed MinIO Cluster for Testing using Docker Compose

Kev's photo
Kev
·

4 min read

MinIO is an open-source object storage server that is simple, scalable, and fast. It is compatible with the Amazon S3 API, making it easy to use with various tools and applications.

Distributed MinIO combines multiple instances to form a single, logical object storage server, improving performance and reliability. Distributed MinIO also improves reliability by storing data redundantly across multiple instances, ensuring data availability even if one instance fails. It is a useful tool for applications that need to scale and handle large amounts of data with high performance and reliability.

Setting Up

Create a new directory for the project: This will be the root directory for your distributed MinIO cluster.

Create a docker-compose.yml file in the root directory. This file will define the configuration for the distributed MinIO cluster.

In the docker-compose.yml file, define a new service for each MinIO instance that you want to include in the cluster and a load balancer to acces the service from a single entrypoint. Here is an example configuration that creates two MinIO instances as a distributed cluster and an nginx load balancer to access the cluster, replace the parameters with your wanted admin login and password in the file
MINIO_ROOT_USER=

MINIO_ROOT_PASSWORD=

version: '3.7'

# configuration source https://github.com/minio/minio/blob/master/docs/orchestration/docker-compose/docker-compose.yaml

# Settings and configurations that are common for all containers
x-minio-common: &minio-common
  image: quay.io/minio/minio:RELEASE.2023-05-18T00-05-36Z
  command: server --console-address ":9001" http://minio{1...2}/data{1...2}
  expose:
    - "9000"
    - "9001"
  environment:
    - MINIO_ROOT_USER=
    - MINIO_ROOT_PASSWORD=
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
    interval: 30s
    timeout: 20s
    retries: 3

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    <<: *minio-common
    hostname: minio1
    volumes:
      - data1-1:/data1
      - data1-2:/data2

  minio2:
    <<: *minio-common
    hostname: minio2
    volumes:
      - data2-1:/data1
      - data2-2:/data2

  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"
    depends_on:
      - minio1
      - minio2

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:

create a file named nginx.conf. This file will define the configuration of the nginx load balancer used to distribute the load for the distributed MinIO cluster.

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9000;
        server minio2:9000;
    }

    upstream console {
        ip_hash;
        server minio1:9001;
        server minio2:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }

    server {
        listen       9001;
        listen  [::]:9001;
        server_name  localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;

            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;

            proxy_connect_timeout 300;

            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";

            chunked_transfer_encoding off;

            proxy_pass http://console;
        }
    }
}

Start the distributed MinIO cluster: To start the distributed MinIO cluster, navigate to the root directory of the project and run the following command 

docker-compose up -d

This will start the MinIO instances in detached mode, meaning they will run in the background.

You can verify the the health of the distributed MinIO cluster by going to http://localhost:9001/tools/metrics

You might be required to input the username and password

You can create a bucket in the bucket tab. You can upload a file and notice that it has been replicated into 4 areas and specify in the
/data1-1 /data1-2/ /data2-1 /data2-2

You can find the volume in /var/lib/docker/volumes

#minioDocker compose