Seamless Cloud Storage Integration: Abstracting AWS S3 and Azure Blob Storage in C#

Abstracting AWS S3 and Azure Blob Storage

Cloud storage solutions like AWS S3 and Azure Blob Storage are widely used for storing data at scale. Whether you’re handling a small app or a large enterprise system, integrating both cloud providers can be useful, but managing two APIs can get tricky.

In this post, we’re going to walk through how we can make our lives easier by abstracting the core blob storage operations (like adding, retrieving, and deleting objects) using a common interface in C#. This will allow us to switch between AWS S3 and Azure Blob Storage seamlessly without tying ourselves to a single provider.

Let’s dive into this!

Step 1: Setting Up the SDKs

Before we write any code, we need the SDKs for AWS S3 and Azure Blob Storage. If you’re unfamiliar with these, don’t worry! Let’s get them installed.

• AWS SDK for .NET (S3)
  First, we’ll install the AWS SDK for S3. You can easily do this using NuGet:

Install-Package AWSSDK.S3

For more details on what this SDK can do, you can check the official docs here: AWS SDK for .NET.

• Azure SDK for .NET (Blob Storage)
  Next, for Azure Blob Storage, we’ll install the Azure SDK:

Install-Package Azure.Storage.Blobs

You can find more info on Azure Blob Storage and its SDK here: Azure .NET SDK.

Let’s start by defining an interface that will represent our blob storage service. This interface will expose three core operations that we need: uploading an object, retrieving an object, and deleting an object.

public interface IBlobStorageService
{
    Task UploadObjectAsync(string containerName, string blobName, Stream data);
    Task<Stream> GetObjectAsync(string containerName, string blobName);
    Task DeleteObjectAsync(string containerName, string blobName);
}

With this in place, we now have a blueprint for interacting with any blob storage provider.

Implement AWS S3 Blob Storage Service

Now, we need to implement this interface for AWS S3. Don’t worry, it’s simpler than it sounds. We’ll use the AWS SDK we installed earlier.

using Amazon.S3;
using Amazon.S3.Model;
using Amazon;
using System.IO;
using System.Threading.Tasks;

public class AwsS3StorageService : IBlobStorageService
{
    private readonly IAmazonS3 _s3Client;

    public AwsS3StorageService(string awsAccessKey, string awsSecretKey, RegionEndpoint region)
    {
        _s3Client = new AmazonS3Client(awsAccessKey, awsSecretKey, region);
    }

    public async Task UploadObjectAsync(string bucketName, string objectKey, Stream data)
    {
        var request = new PutObjectRequest
        {
            BucketName = bucketName,
            Key = objectKey,
            InputStream = data
        };
        await _s3Client.PutObjectAsync(request);
    }

    public async Task<Stream> GetObjectAsync(string bucketName, string objectKey)
    {
        var request = new GetObjectRequest
        {
            BucketName = bucketName,
            Key = objectKey
        };
        var response = await _s3Client.GetObjectAsync(request);
        return response.ResponseStream;
    }

    public async Task DeleteObjectAsync(string bucketName, string objectKey)
    {
        var request = new DeleteObjectRequest
        {
            BucketName = bucketName,
            Key = objectKey
        };
        await _s3Client.DeleteObjectAsync(request);
    }
}

Implement Azure Blob Storage Service

Similarly, we need to implement the interface for Azure Blob Storage using its SDK. Let’s do that:

using Azure.Storage.Blobs;
using System.IO;
using System.Threading.Tasks;

public class AzureBlobStorageService : IBlobStorageService
{
    private readonly BlobServiceClient _blobServiceClient;

    public AzureBlobStorageService(string connectionString)
    {
        _blobServiceClient = new BlobServiceClient(connectionString);
    }

    public async Task UploadObjectAsync(string containerName, string blobName, Stream data)
    {
        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        var blobClient = containerClient.GetBlobClient(blobName);
        await blobClient.UploadAsync(data, true);
    }

    public async Task<Stream> GetObjectAsync(string containerName, string blobName)
    {
        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        var blobClient = containerClient.GetBlobClient(blobName);
        var response = await blobClient.DownloadAsync();
        return response.Value.Content;
    }

    public async Task DeleteObjectAsync(string containerName, string blobName)
    {
        var containerClient = _blobServiceClient.GetBlobContainerClient(containerName);
        var blobClient = containerClient.GetBlobClient(blobName);
        await blobClient.DeleteAsync();
    }
}

Using the Abstraction with Sample Resources

Now that we’ve created the two implementations, let’s look at how we can use them. You’ll need to configure the appropriate resources to connect to AWS S3 or Azure Blob Storage.

For AWS S3, you need:

string awsAccessKey = "your-aws-access-key";
string awsSecretKey = "your-aws-secret-key";
string bucketName = "your-s3-bucket";
RegionEndpoint region = RegionEndpoint.USEast1;  // Example region

For Azure Blob Storage, you need:

string azureConnectionString = "your-azure-connection-string";
string containerName = "your-container";

Now, depending on which provider you want to use, you can switch the storage service accordingly.

Here’s an example of how we can use this abstraction in our code:

class Program
{
    static async Task Main(string[] args)
    {
        IBlobStorageService storageService;

        // Choose between AWS and Azure
        if (useAws)
        {
            storageService = new AwsS3StorageService(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);
        }
        else
        {
            storageService = new AzureBlobStorageService(azureConnectionString);
        }

        // Upload an object
        using (var stream = File.OpenRead("path/to/your/file"))
        {
            await storageService.UploadObjectAsync("your-container-or-bucket", "your-blob-or-object", stream);
        }

        // Retrieve an object
        var retrievedStream = await storageService.GetObjectAsync("your-container-or-bucket", "your-blob-or-object");

        // Delete an object
        await storageService.DeleteObjectAsync("your-container-or-bucket", "your-blob-or-object");
    }
}

And there we have it! A clean and simple way to interact with both AWS S3 and Azure Blob Storage using the same interface.

Conclusion

In this post, we’ve walked through how to abstract the interaction with AWS S3 and Azure Blob Storage using a common interface, which helps us manage both cloud providers seamlessly. By implementing this abstraction, we simplify our codebase, making it easy to swap out providers without affecting core functionality

This approach not only decouples your application from specific storage providers but also promotes cleaner, more testable code. Whether you’re working with AWS or Azure, this abstraction allows you to handle blob storage operations (upload, download, delete) in a consistent way, reducing complexity as your application grows. By following these principles, you can build a robust and cloud-agnostic solution that is future-proof for evolving requirements.