Compatible Providers

Overview

Over the last couple of years, we have seen tremendous growth in this area, with a large number of providers now offering storage services "compatible" with AWS S3. When these providers refer to their services as "S3 compatible", what they usually claim is that their API covers a subset of the AWS S3 API, and is binary identical. In essence, what that means is that a software tool that was originally designed to work with the industry standard AWS S3, would be able to integrate seamlessly with the compatible service, because the API is essentially the same.

We at Yarkon have integrated with a number of these providers. So far, we have not encountered any compatible provider API that is 100% identical to that of AWS S3, though some of them are indeed very close - and should definitely work just fine with software tools that work with AWS S3. Naturally, you will not get S3 features that are not supported, such as "Storage Class", "Acceleration", "Tagging", "Versioning", etc., but for many practical and economical implementations, that should not be a blocker.

Prerequisites

Yarkon is an enterprise grade tool. Thus, it relies on a few security and privacy features that require services which are not storage related, such as AWS IAM and STS. IAM (Identity Access Management) is used for granular user permissions management. STS (Session Token Service) is used for session based credentials.

We strongly believe that there is no practical, manageable and secure way to provide web based access to end users without implementing a subset of these aforementioned services. Therefore, Yarkon requires any compatible provider to implement at least these API endpoints:

  1. STS:GetSessionToken
  2. IAM:AssumeRole

A full integration with Yarkon would require all IAM:Get* and IAM:List* actions as well.

An S3 compatible provider that does not support these API end points will not work with Yarkon. Before attempting to integrate Yarkon with any compatible provider, make sure to check their API and ensure that these end points are supported.

Integration

Once you confirmed that your compatible storage provider does indeed support the required API subset, you can follow these steps to get your storage integrated with Yarkon. In this manual we demonstrate integration with Wasabi, a top-tier compatible provider, but the steps should be the same for any other provider that meets the prerequisites mentioned before.

Depending on the capabilities of your storage provider, some of the features available to AWS S3 users may not be available when using a compatible provider.

Provider Details

The first step is to set up the proper connection information for your provider. Use the Yarkon Admin Console, Administration page, Provider tab to specify the below details:

  • Check the S3 Compatible radio option.
  • Enter a Name for your provider. It can be any name, and will be displayed to your end users in the client interface.
  • Select the Region from the drop down. If your provider does not require a region, use the default one, which is us-east-1.
  • Optionally add an Image for your provider. It will be shown to the end users in the client interface. Acceptable image format is any public URL, or a CSS/data:image compatible format such as svg+xml etc.
  • Specify the End Point for your provider. This would be the URL the client application would use for storage API access.
  • Optionally specify the IAM End Point and the STS End Point of your provider. This is required if any of them is different than the End Point you entered above. If they are the same, you can skip these.

The below image shows how the Yarkon Admin Console page would look when set up with Wasabi, a popular compatible S3 provider.

Provider

Permissions

Follow the instructions provided by your provider to set up the access policies. At the very least, the access policy must be similar to this generic one used with AWS S3:

{
    "Version": "2012-10-17",
    "Statement": [{
            "Sid": "AllowAllS3Actions",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*"
        }, {
            "Sid": "AllowUIToDisplayIAMOptions",
            "Effect": "Allow",
            "Action": [
                "iam:List*",
                "iam:Get*"
            ],
            "Resource": "arn:aws:iam::<account-number>:*"
        }, {
            "Sid": "AllowTheRoleToGetPermissions",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<account-number>:role/yarkons3-console-role"
        }, {
            "Sid": "AllowTheRoleToFederate",
            "Effect": "Allow",
            "Action": [
                "sts:GetFederationToken"
            ],
            "Resource": "arn:aws:sts::<account-number>:*"
        }
    ]
}

Make sure to replace the <account-number> with your account number.

Details (see the Sid attributes for reference):

AllowAllS3Actions - allows the Yarkon full access to S3. If you want to limit the usage of Yarkon in your organization to a predefined set of buckets, replace the statement with the below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowServerToIterateBuckets",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "AllowServerToAccessSpecificBuckets",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::yarkons3-finance",
                "arn:aws:s3:::yarkons3-sales"
            ]
        },
        {
            "Sid": "AllowUserActionsLimitedToSpecificBuckets",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::yarkons3-finance/*",
                "arn:aws:s3:::yarkons3-sales/*"
            ]
        }
    ]
}

AllowUIToDisplayIAMOptions - only required when using Federated or Integrated security models. Yarkon does not need IAM access when set to use the Shared security model. This setting has no impact on the permissions granted to end users. If you only intend to use the Shared model, you can remove it.

AllowTheRoleToGetPermissions - only required when using the Integrated security model. You can remove it if using any of the other models. Also, the role name specified, yarkons3-console-role assumes this is the name you'd be using for the IAM role required (see below). If you choose a different name, make sure to update here.

AllowTheRoleToFederate - only required when using the Federated security model. You can remove it if using any of the other models.

For more information about the access policies used by Yarkon, please review this document Set up the IAM role and policies.

As an example, at the bottom of the page we provide a walk through of steps required on the provider side when using Wasabi.

Access Key

Follow the instructions provider by your provider to get the API Access Key.

Once you have the Access Key, enter it into the Access page in the Yarkon Admin Console.

Access Key

As an example, at the bottom of the page we provide a walk through of steps required on the provider side when using Wasabi.

Features

As mentioned above, some of the features available to users of AWS S3, might not be available when using other storage providers. Use the Features page of the Yarkon Admin Console to turn off features that are not supported.

Features

Example Set Up

As an example for how to set up Yarkon, the following policy are known to work with Wasabi. Other providers might require some tweaks, depending on how closely compatible they are with the AWS S3 spec.

The most generic policy - that is, one that would work for all security models - is:

{
    "Version": "2012-10-17",
    "Statement": [{
            "Sid": "AllowAllS3Actions",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*"
        }, {
            "Sid": "AllowUIToDisplayIAMOptions",
            "Effect": "Allow",
            "Action": [
                "iam:List*",
                "iam:Get*"
            ],
            "Resource": "*"
        }, {
            "Sid": "AllowTheRoleToGetPermissions",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<account-number>:role/yarkons3-console-role"
        }
    ]
}

When using the Integrated Security model, a Trust Policy is required. For more details, review the policies document. The Trust Policy for Wasabi should be set like this:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AllowAssumeRole",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<account-number>:root"
        },
        "Action": "sts:AssumeRole"
    }]
}