Minio is a S3 compatible object storage server which can be run with a variety of storage backends, including NFS, GlusterFS, and other distributed storage or cloud storage backends. You can opt for multiple load-balanced Minio servers against a single storage backend, or multiple Minio servers with data spanned across multiple drives – with erasure coding. If your applications consume S3 object storage but you don’t wish to use Amazon S3, typically because your architecture is hosted in a public or private cloud outside of AWS, Minio might be the right solution.
A Minio server, or a load balancer in front of multiple Minio servers, serves as a S3 endpoint that any application requiring S3 compatible object storage can consume. One common use case of Minio is as a gateway to other non-Amazon object storage services, such as Azure Blob Storage, Google Cloud Storage, or BackBlaze B2.
Using Minio as a gateway to BackBlaze B2 is particularly popular because of BackBlaze’s low per-gigabyte storage cost ($0.005/GB/mo) and membership in the CloudFlare-led Bandwidth Alliance. Because of this unique agreement, any static assets stored in a BackBlaze B2 bucket can be served over the CloudFlare CDN using a CNAME record, with no egress bandwidth costs billed to the B2 bucket owner.
Popular open-source applications that consume S3 compatible cloud storage include media storage for WordPress or RocketChat. Note that however using BackBlaze B2 through Minio as primary or external storage for NextCloud is no longer supported, as NextCloud version 14 and above use a MultipartUpload S3 API call which is translated to an a large file upload API call that the B2 API requires at least 2 parts for. Facilitating the separation of concerns between user data and application code is a simple way to make conventional apps more “cloud native.”
Minio is backed by Docker alumni including Steve Singh, former CEO of Docker, Inc. so it’s not a surprise that Minio runs as a Docker container. The container image is available on the Docker Hub at minio/minio.
This article will describe how to set up Minio as a S3 to B2 gateway and use the s3cmd command line tool to interact with the BackBlaze B2 bucket using Amazon S3 commands. If you require consulting or support with using Minio as part of your cloud architecture, please contact our Minio consultants.
It assumes that you want to set up a single Minio container on localhost mapped to the BackBlaze B2 cloud storage service. Setting up on localhost means that its safe to connect to the Minio endpoint, in the below example localhost:9001 without SSL from applications hosted on the same server.
Configuring Minio on the Server-Side Using Minio Docker Container
The Minio access key and secret key should be identical to your B2 access and secret key, which you can obtain from the App Keys section of the dashboard after you sign in to your B2 Cloud Storage account.
The named volume, minio-data, is used to persist Minio’s configuration and application data (such as authentication tokens) when the container is redeployed.
docker volume create minio-data
docker run -d --name b2-s3 --restart on-failure -p 127.0.0.1:9001:9000 -v minio-data:/data -e "MINIO_ACCESS_KEY=1111111111111111111111111" -e "MINIO_SECRET_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" minio/minio gateway b2
With a few minor changes to the docker run command, it’s also possible to set up Minio in gateway mode with Azure Blob Storage or Google Cloud Storage, by passing the respective credentials for these services.
docker volume create minio-data
docker run -d --name azure-s3 --restart on-failure-p 127.0.0.1:9001:9000-v minio-data:/data -e "MINIO_ACCESS_KEY=azurestorageaccountname" -e "MINIO_SECRET_KEY=azurestorageaccountkey" minio/minio gateway azure
docker volume create minio-data
docker run -d --name gcs-s3 --restart on-failure -p 127.0.0.1:9001:9000 \
-v minio-data:/data
-v /path/to/credentials.json:/credentials.json \
-e "GOOGLE_APPLICATION_CREDENTIALS=/credentials.json" \
-e "MINIO_ACCESS_KEY=minioaccountname" \
-e "MINIO_SECRET_KEY=minioaccountkey" \
minio/minio gateway gcs yourprojectid
docker volume create minio-data
docker run --name nas-s3 --restart on-failure -p 127.0.0.1:9001:9000 \
-v minio-data:/data \
-v /shared/nasvol:/container/vol \
-e "MINIO_ACCESS_KEY=minio" \
-e "MINIO_SECRET_KEY=minio123" \
minio/minio gateway nas /container/vol
If you wish to connect to your Minio gateway over an untrusted network such as the open Internet, you would certainly need to set up a TLS reverse proxy (such as nginx) with a valid SSL certificate to prevent the data from being intercepted in-transit. You could use a free certificate authority, such as Let’s Encrypt, to obtain the certificate.
Setting up a reverse proxy also has the benefit of allowing you to securely access Minio’s web interface from a browser, so that you can browse, upload, download, and delete files from the Minio backend in just a few clicks.
Install nginx from your package manager, either sudo apt install nginx (for Ubuntu) or sudo yum install epel-release && sudo yum update && yum install nginx (for CentOS).
Enable and start the nginx service, sudo systemctl enable nginx && sudo systemctl start nginx.
Also, get the Certbot client for your distro, sudo add-apt-repository ppa:certbot/certbot && sudo apt-get update && sudo apt-get install certbot python-certbot-nginx (for Ubuntu) or sudo yum install certbot python2-certbot-nginx (for CentOS).
Create an A record pointing the fully qualified domain name (FQDN) for your Minio endpoint to the IP address of your Minio server. Then, run the following command to obtain a certificate, where minio.example.com is your domain or subdomain.
sudo certbot certonly --webroot -w /usr/share/nginx/html -d minio.example.com
Provide an email address for renewal reminders (you shouldn’t get any if automatic renewal is set up correctly in a later step) and agree to the Let’s Encrypt TOS. If you did everything correctly so far, you should have a valid certificate and private key located at /etc/letsencrypt/live/minio.example.com/fullchain.pem and /etc/letsencrypt/live/minio.example.com/privkey.pem respectively.
Now go to the directory where configuration files for Nginx are stored (/etc/nginx/sites-available/ for Ubuntu, /etc/nginx/conf.d/ for CentOS), and create a new configuration file called minio.conf.
This configuration forces all connections to the Minio endpoint to use HTTPS, it sets the client_max_body_size value to 0so objects of an unlimited size can be uploaded, and the client and proxy timeouts are set to high values so larger uploaded chunks don’t time out. It also implements the recommendations by Cipherli.st to earn an A on Qualys SSL Test (A+ if HSTS is enabled).
server {
listen 80;
listen [::]:80;
server_name minio.example.com;
location /.well-known {
alias /usr/share/nginx/html/.well-known;
}
location / {
# enforce https
return 301 https://$server_name:443$request_uri;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name minio.example.com;
ssl_certificate /etc/letsencrypt/live/minio.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/minio.example.com/privkey.pem;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/dhparam.pem;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_ecdh_curve secp384r1;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 1.0.0.1 valid=300s;
resolver_timeout 5s;
# add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
fastcgi_hide_header X-Powered-By;
client_max_body_size 0;
client_header_timeout 60;
client_body_timeout 86400;
fastcgi_read_timeout 86400;
proxy_connect_timeout 60;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
send_timeout 86400;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://minio;
}
}
If using Ubuntu, create a symlink from the sites-enabled folder, like so, cd /etc/nginx/sites-enabled && ln -s /etc/nginx/sites-available/minio.conf minio.conf to enable the config.
Generate the dhparam.pem file for Nginx using this OpenSSL command. This ensures a secure TLS connection with randomly generated Diffie-Hellman values.openssl dhparam -out /etc/nginx/dhparam.pem 4096
Test the config with the nginx -t command, and if there are no errors, you’re good to restart the Nginx service to have the changes take effect.
sudo systemctl restart nginx
As root, add the following line to the system-wide crontab, /etc/crontab, to facilitate certificate renewal. Test whether the renewal would succeed by running certbot renew --dry-run at the terminal.
0 2 * * * root certbot renew > /dev/null 2>&1
Visit minio.example.com in any web browser, provide the access and secret keys, and you will be redirected to the Minio Browser where you create new buckets, list the contents of buckets, put (upload), get (download), and delete objects.
Configuring s3cmd client to connect to Minio
Install the s3cmd client, a command line based tool which can be used to access any S3 compatible endpoint, including Minio endpoints.
Ubuntu: sudo apt install s3cmd
CentOS: sudo yum install s3cmd
Create the configuration file in your home directory, ~/.s3cfg
If accessing a Minio endpoint, the bucket_location (i.e. region) should typically be blank. Even if you are accessing a cloud storage service on the backend where a region is designated (e.g. us-east-1), Minio itself is regionless.
The access_key and secret_key should correspond to the MINIO_ACCESS_KEY and MINIO_SECRET_KEY environment variables specified when starting the Minio container. If using Minio as a gateway for an external cloud storage service, these keys should correspond to the credentials issued by that service.
If using Minio with NAS, the access and secret keys may correspond to a specific Minio user created through the mc admin create user command. Creating a Minio user through the Minio Client tool allows you to assign a unique access key by user, so their privileges can be limited by a custom or canned policy, defined in a JSON format compatible with AWS IAM permissions for S3 (e.g. writeonly, readonly, readwrite).
# Setup endpoint
host_base = minio.example.com
host_bucket = bucket-name
bucket_location =
use_https = True
# Setup access keys
access_key = 1111111111111111111111111
secret_key = AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# Enable S3 v4 signature APIs
signature_v2 = False
Now you can list the buckets and their contents, put, get, and delete objects on the underlying NAS or cloud storage service through Minio using the following S3 commands.
List buckets: s3cmd ls s3://
Create bucket: s3cmd mb s3://bucket-name
List bucket contents: s3cmd ls s3://bucket-name
Put object to bucket: s3cmd -v put /path/to/local_file.bin s3://bucket-name/remote_file.bin --multipart-chunk-size-mb=15
Get object from bucket: s3cmd -v get s3://bucket-name/remote_file.bin /path/to/local_file.bin
Remove object from bucket: s3cmd rm s3://bucket-name/remote_file.bin
Connecting a Web Application such as RocketChat to Minio
Below is an example of how RocketChat can be connected to a Minio server, which uses Backblaze B2 as backend storage. Note that the “Force Path Style” option should always be enabled, as your Nginx load balancer is not set up to resolve the subdomains of the bucket names. Therefore you want your application to put/get objects at https://minio.example.com/bucket-name (path style enabled), not https://bucket-name.minio.example.com (path style disabled).
FAQs
What is MinIO S3 gateway? ›
What is MinIO? MinIO is a Kubernetes native object storage suite that offers S3-compatible object storage and multi-cloud configuration. MinIO Gateway for S3 is compatible for configuration and use with Filebase through a MinIO docker container that can be quickly deployed on a variety of different systems.
Why use MinIO instead of S3? ›Easy setup. While the Minio app for UCS simply exposes a local directory through the S3 API, Minio itself can easily be configured to protect against disk failures and data corruption through distributing data between multiple disks and multiple nodes hosting Minio.
How does MinIO Gateway work? ›Deployment scenarios
NAS Gateway mode: MinIO Gateway adds Amazon S3 compatibility to NAS storage. You can run multiple Minio instances on the same shared NAS volume as a distributed object gateway. For using Minio as NAS Gateway, you need a PV that runs with ReadWriteMany-supported volume plug-ins.
Amazon S3 provides a simple web service interface that you can use to store and retrieve any amount of data, at any time, from anywhere. Using this service, you can easily build applications that make use of cloud native storage.
Is MinIO faster than S3? ›In conclusion, MinIO is significantly faster than AWS S3 when used to store ClickHouse table data.
What are the types of storage gateways in S3? ›Introducing AWS Storage Gateway
To support these use cases, the service provides four different types of gateways – Tape Gateway, Amazon S3 File Gateway, Amazon FSx File Gateway, and Volume Gateway – that seamlessly connect on-premises applications to cloud storage, caching data locally for low-latency access.
At $11,510 MinIO and Seagate represent the best economics - by a considerable amount. The combination is 47% less expensive per month than standard S3 and 33% less than S3 with Intelligent Tiering.
Can I use MinIO commercially? ›The commercial MinIO license gives you the rights to create software on commercial terms without any Open Source license obligations. With the commercial license you also have access to the SUBNET portal and MinIO's premium support experience.
How do I make a S3 bucket with MinIO? ›- SSH to instance as ${USER}.
- Check the minio service is up and running. docker ps | grep minio exit.
- Enter the login credentials. Get the Access Key and Secret Key from your System Administrator. ...
- Click button to create a new bucket. Give a name.
MinIO is unique in its ability to support its claim of S3 compatibility. With tens of thousands of customers and open source users, our S3 API compatibility is the most widely tested and implemented in the world - covering millions of combinations of hardware, software and applications.
Can we host MinIO on our server? ›
If you want to access the MinIO server via your server's IP address only, generate a certificate for it using the following command: sudo certgen -host your-server-ip.
What is MinIO good for? ›MinIO offers a unique suite of features to protect data within and across clouds - both public and private. MinIO's enterprise data lifecycle management tools, including versioning, object locking and the various derivative components, satisfying multiple use cases.
Can you use S3 as a data warehouse? ›You can store structured data (such as relational data), semi-structured data (such as JSON, XML, and CSV files), and unstructured data (such as images or media files). This feature makes Amazon S3 the appropriate storage solution for your cloud data lake.
Can we use S3 as data warehouse? ›Easily load your Amazon S3 data into your data warehouse
S3 is easy to use and built to store and retrieve any amount of data from anywhere. S3 is commonly used as a company's data lake and the data you can have in S3, vary from dumps of files to exports from other services.
Amazon Glacier, also known as Amazon Simple Storage Service (S3) Glacier, is a low-cost cloud storage service for data with longer retrieval times offered by Amazon Web Services (AWS). Amazon Glacier provides storage for data archiving and backup of cold data.
Does MinIO have a REST API? ›Management APIs implement remote administrative operations over HTTP/REST. This guide is intended for SDK developers of package like madmin . If you are an enduser please take a look at mc admin CLI interface.
Where does MinIO store data? ›MinIO Object Storage uses buckets to organize objects. A bucket is similar to a folder or directory in a filesystem, where each bucket can hold an arbitrary number of objects. MinIO buckets provide the same functionality as AWS S3 buckets. For example, consider an application that hosts a web blog.
Is MinIO a database? ›Minio can be classified as a tool in the "Cloud Storage" category, while MySQL is grouped under "Databases". Minio and MySQL are both open source tools.
What are two gateways offered by AWS Storage Gateway? ›Storage Gateway provides three different types of gateways: File Gateway, Volume Gateway, and Tape Gateway. For more information, including use cases, customer stories, and helpful videos, check out the AWS Storage Gateway product page.
What is the difference between S3 file gateway and volume gateway? ›Unlike File Gateways which are used for accessing objects, Volume Gateways present your on-premise application with the iSCSI block storage instead. Volume Gateways allow you to have point-in-time backups of your volumes stored as EBS snapshots, and come in two different operational modes: stored and cached.
How does S3 file gateway work? ›
Amazon S3 File Gateway provides a seamless way to connect to the cloud in order to store application data files and backup images as durable objects in Amazon S3 cloud storage. Amazon S3 File Gateway offers SMB or NFS-based access to data in Amazon S3 with local caching.
Is MinIO strongly consistent? ›Consistency Guarantees
MinIO follows strict read-after-write and list-after-write consistency model for all i/o operations both in distributed and standalone modes. This consistency model is only guaranteed if you use disk filesystems such as xfs, zfs or btrfs etc.. for distributed setup.
MinIO follows strict consistency within the data center and eventual-consistency across data centers to protect the data. Notification functionality to push replication failure events.
Does MinIO compress data? ›MinIO offers the best-in-class compression scheme that allows for fully transparent compression of data on disk. This can in many scenarios lead to a reduction in storage costs, simply by enabling compression.
Is MinIO fast? ›The benchmark results demonstrate that MinIO is able to achieve 10.81 GB/sec read perfor- mance and 8.57 GB/sec write performance from a 16 node MinIO Object Server.
How do I use AWS CLI with MinIO? ›- Prerequisites. Install MinIO Server from here.
- Installation. Install AWS CLI from https://aws.amazon.com/cli/
- Configuration. To configure AWS CLI, type aws configure and specify the MinIO key information. ...
- Commands. To list your buckets.
Step 2: Open the application and select 'Amazon S3' from the list of cloud services available. Step 3: Type in your credentials such as the 'Access Key' and the 'Secret Key' for your Amazon S3 server. In this step you can also specify the bucket. Step 4: Provide the server endpoint as well.
How many buckets can be created in S3 per account? ›A bucket is a container for objects stored in Amazon S3. You can store any number of objects in a bucket and can have up to 100 buckets in your account. To request an increase, visit the Service Quotas Console .
How many S3 buckets can I have per account by default? ›By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your account bucket limit to a maximum of 1,000 buckets by submitting a service limit increase.
What type of storage is MinIO? ›MinIO is a high performance, distributed object storage system. It is software-defined, runs on industry standard hardware and is 100% open source with the dominant license being GNU AGPL v3. MinIO is different in that it was designed from its inception to be the standard in private/hybrid cloud object storage.
What is the difference between s3 and s3api? ›
The main difference between the s3 and s3api commands is that the s3 commands are not solely driven by the JSON models. Rather, the s3 commands are built on top of the operations found in the s3api commands. As a result, these commands allow for higher-level features that are not provided by the s3api commands.
What is alternative for s3? ›We have compiled a list of solutions that reviewers voted as the best overall alternatives and competitors to Amazon Simple Storage Service (S3), including Google Cloud Storage, Azure Blob Storage, DigitalOcean Spaces, and Zadara.
What is the minimum number of servers for MinIO? ›Minimum Nodes per Deployment
MinIO recommends a minimum of 4 host servers per deployment with 4 locally attached drives per server.
A MinIO deployment consists of a set of storage and compute resources running one or more minio server nodes that together act as a single object storage repository. A standalone instance of MinIO consists of a single Server Pool with a single minio server node.
What protocols does MinIO support? ›Minio is written in Go, comes with a command line client plus a browser interface, and supports simple queuing service for Advanced Message Queuing Protocol (AMQP), Elasticsearch, Redis, NATS, and PostgreSQL targets.
Is MinIO secure? ›MinIO uses an authentication encryption scheme (AEAD) to en/decrypt objects as they are written to or read from object storage. MinIO AEAD encryption supports industry standard encryption protocols such as AES-256-GCM and ChaCha20-Poly1305 to secure object data.
What is the best practice for storing data in S3? ›- Block public S3 buckets at the organization level. ...
- Use bucket policies to verify all access granted is restricted and specific. ...
- Ensure that any identity-based policies don't use wildcard actions. ...
- Enable S3 protection in GuardDuty to detect suspicious activities.
S3 is often the core of a big data solution on AWS. S3 offers near-unlimited scalability, is very cost-effective (compared to other storage solutions on AWS like EBS), and tight integration with AWS's other big data tools.
What is the difference between S3 data lake and data warehouse? ›While data warehouses store structured data, a lake is a centralized repository that allows you to store any data at any scale. A data lake offers more storage options, has more complexity, and has different use cases compared to a data warehouse.
Can S3 be used as key value store? ›AWS S3 is a key-value store, one of the major categories of NoSQL databases used for accumulating voluminous, mutating, unstructured, or semistructured data. Uploaded objects are referenced by a unique key, which can be any string.
Can S3 store JSON data? ›
Amazon S3 Select works on objects stored in CSV, JSON, or Apache Parquet format. It also works with objects that are compressed with GZIP or BZIP2 (for CSV and JSON objects only) and server-side encrypted objects.
Can S3 store SQL database? ›With SQL Server 2022, the Backup to URL feature now supports S3-compatible object storage. S3 compatibility means the storage solution uses the Amazon S3 API to provide an S3-compliant interface.
Does S3 have unlimited storage? ›Amazon S3 is an object-based cloud storage solution that gives our customers virtually unlimited storage. It's at the heart of most AWS offerings storing data for enterprise applications, websites, mobile applications, IoT devices, and big data analytics.
Does S3 have cloud storage? ›Amazon Simple Storage Service (Amazon S3) is a scalable, high-speed, web-based cloud storage service. The service is designed for online backup and archiving of data and applications on Amazon Web Services (AWS).
What is S3 gateway endpoint? ›Gateway endpoints provide reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for your VPC. Gateway endpoints do not enable AWS PrivateLink. There is no additional charge for using gateway endpoints. Amazon S3 supports both gateway endpoints and interface endpoints.
How do I connect my MinIO to my S3? ›- Step 1: Download Minio. Download Minio binary. ...
- Step 2: Prepare Object Storage disk. ...
- Step 3: Start Minio service. ...
- Step 4: Configure Firewall. ...
- Step 5: Access Minio Web interface. ...
- Step 6: Test Minio with mc client.
What is Minio? Minio is an open source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality. It is the best server which is suited for storing unstructured data such as photos, videos, log files, backups, and container.
What is MinIO used for? ›MinIO is a High Performance Object Storage released under GNU Affero General Public License v3. 0. It is API compatible with the Amazon S3 cloud storage service. It can handle unstructured data such as photos, videos, log files, backups, and container images with a current maximum supported object size of 5TB.
How do I access my S3 bucket using gateway Endpoint? ›- Open the Amazon S3 console.
- Choose the S3 bucket with connectivity issues.
- Choose the Permissions view.
- Choose Bucket Policy.
- Be sure that the bucket policy allows access from the gateway VPC endpoint and the VPC that you want to connect.
An end point indicates a specific location for accessing a service using a specific protocol and data format. GateWay: An service Gateway provides a central access point for managing, monitoring, and securing access to your publicly exposed web services.
How to create S3 endpoint gateway? ›
- In the navigation pane, choose Endpoints.
- Choose Create endpoint.
- For Service category, choose AWS services.
- For Services, add the filter Type: Gateway and select com. ...
- For VPC, select the VPC in which to create the endpoint.
- For Route tables, select the route tables to be used by the endpoint.
MinIO Object Storage uses buckets to organize objects. A bucket is similar to a folder or directory in a filesystem, where each bucket can hold an arbitrary number of objects. MinIO buckets provide the same functionality as AWS S3 buckets. For example, consider an application that hosts a web blog.