Associated Code
The code associated with this post can be found at the link below (you can also download a .zip file of the code from this link).
https://github.com/floatingfrisbee/amazonfileupload
15th February 2013 Update
There is a limitation in the current implementation in that it does not allow for the FileId to be set to the actual name of the file, and instead uses a pre-generated GUID as the FileId. This was pointed out to me by “DE” in the comments below.
The reason is that the signature is generated before the view is rendered and a file is selected. If the FileId is updated later the upload will fail because the signature will not match the signature that Amazon will compute. This can be remedied (I think) by capturing the onchange event on the file input in the view, and make an ajax call to generate the signature at that time instead of pre-computing it. I’ll work on it if I find time, but anyone is free to fork the Git repo and make that fix!
18th January 2013 Update
A lot of people have asked me if I have the associated code, and I finally was able to find some time to put it up on Git. Here is the link
https://github.com/floatingfrisbee/amazonfileupload
Remember to open web.config and put in your AWS key, secret and bucket id. Besides that it should just run as long as you can run an ASP.NET MVC 4 application.
Original Post
Amazon S3 is a service where you can store large (or small) files. It is organized in terms of “buckets” so before you can store any thing under your account, you have to create a bucket first (or have access to an existing one). Then you can specify the bucket you want to upload the file into, along with other required pieces of information and even your own metadata.
That is a great solution for a lot of web applications, specially those that are already utilizing Amazon’s web services. And being able to upload directly to Amazon S3 from the user’s browser is important because it saves bandwidth and CPU cycles for you as the application owner and time for everyone involved.
Obviously there are a couple of issues here!
1. How do you get a user’s browser to bypass your web application, and upload a file directly to Amazon S3’s storage.
2. How do you ensure that the user’s browser communicates only with Amazon S3 and not an impersonator.
3. How do you ensure that none of the parameters of the upload; things like bucket name, the file’s ACL on S3, and other metadata you might specify, are changed by the user or any intermediary during this process (since the upload is not going though your application).
Fortunately there are solutions for each of the above and really at the end of the day it boils down to the same ol’ trick that most cryptography relies on: symmetric and asymmetric keys. I’ll present the code I used to implement this using an ASP.NET MVC application using C# right after I explain the solution in the far more complicated language of English.
Solutions
1. This one is simple. An HTML form that has a “method” of “post” and an “action” that points to the correct Amazon S3 URI for your bucket will do the trick. Off course the form also needs to have an “input” of type “file” and an “input” of type “submit”. Besides this all the other parameters of the upload; both those that are required by Amazon S3 and others that you might want to store for your purposes, are added as hidden “input” fields to the form.
2. This one is simple too. Just ensure that the form’s action is specified with “https” as the protocol.
Something like this
https://<your bucket name>.s3.amazonaws.com
This will cause the browser to use SSL to communicate with Amazon and the usual handshake, certificate lookup and encryption that goes along with it will ensue.
3. The solution for this is more complicated and judging from the buzz on the web is a source of frustration for many a web developers, and more so for ASP.NET developers because there are not many good code examples available online. The solution boils down this; create a string that has all the parameters of the upload, and then hash that string using your Amazon S3 private key and the HMAC SHA1 algorithm. Upload both that string with the upload parameters and the hash generated using the private key with the HTML form created in step 1, as hidden “input” fields. When Amazon S3 receives the form, it will de-hash the hash (since they have access to your private key), and compare it to the upload parameter string, and to the actual upload parameters. If they match up, they know that the upload parameters were not tampered with, otherwise, they will reject the upload. No one else can generate the correct hash because no one else has your Amazon S3 private key (well, at least that you know about!).
There you have it. Now…
Lets move on to the ASP.NET MVC implementation
I was using MVC3 and Razor as my view engine. If you’re new to ASP.NET MVC, you probably want to spend a little time understanding the basics.
First the ViewModel
The members of my ViewModel class are basically the things that the view needs to create the form with it hidden and visible fields, and the form action, method and encapsulation type.
public class FileUploadViewModel
{
public string FormAction { get; set; }
public string FormMethod { get; set; }
public string FormEnclosureType { get; set; }
public string Bucket { get; set; }
public string FileId { get; set; }
public string AWSAccessKey { get; set; }
public string RedirectUrl { get; set; }
public string Acl { get; set; }
public string Base64Policy { get; set; }
public string Signature { get; set; }
}
Next the code to generate an instance of the ViewModel
You could package this code as a part of another library or as a part of the helpers in your MVC solution but for this example, I’m not going to get into that.
The key part here is off course the generation of the policy string and then generating the signature, which involves taking the hash of the policy string using your Amazon S3 private key and the HMAC SHA1 hashing algorithm. Having a foundational understanding of encodings and string formats is helpful as always. As Joel and many others have said, you should familiarize yourself with base64 and also what a strings and “encoding” mean, specially in .NET.
First lets look at the policy string. The policy string (also called the policy document or the “Access Control” document) must adhere to the rules specified in Amazon S3’s documentation. You should familiarize yourself with the syntax of the policy string described in it. In the end for my case it ended up looking like like this.
{
"expiration": "2011-04-20T11:54:21.032Z",
"conditions": [ ["eq", "acl", "private"], ["eq", "bucket": "myas3bucket"], ["eq", "$key", "myfilename.jpg"], ["content-length-range", 0, 20971520], ["eq", "$redirect", "myredirecturl"]]
}
I used a class marked with the[DataContract] attribute and used the DataContractJsonSerializer to serialize it into the right format. Beats constructing the string in code.
Now lets look at how you would generate the signature. You will probably store the private key in web.config like this
<appSettings> <add key="AWSAccessKey" value="MyAmazonS3AccessKey"/> <add key="AWSSecretKey" value="MyAmazonS3SecretKey"/> <add key="AWSBucket" value="MyAS3Bucket"/> </appSettings>
and retrieve it like this before passing it to the method to create the signature.
string publicKey = ConfigurationManager.AppSettings["AWSAccessKey"]; string secretKey = ConfigurationManager.AppSettings["AWSSecretKey"]; string bucketName = ConfigurationManager.AppSettings["AWSBucket"];
Now let’s look at the “Signature”. In Amazon S3’s documentation it is defined as
Signature is the HMAC of the Base64 encoding of the policy
The function below creates the “Signature”. The parameters are “normal” C# strings. Meaning no additional encoding information is being implied. I have made the function more verbose than normal to clarify the steps involved.
private string CreateSignature(string secretKey, string policy)
{
var encoding = new ASCIIEncoding();
var policyBytes = encoding.GetBytes(policy);
var base64Policy = Convert.ToBase64String(policyBytes);
var secretKeyBytes = encoding.GetBytes(secretKey);
var hmacsha1 = new HMACSHA1(secretKeyBytes);
var base64PolicyBytes = encoding.GetBytes(base64Policy);
var signatureBytes = hmacsha1.ComputeHash(base64PolicyBytes);
return Convert.ToBase64String(signatureBytes);
}
Let’s walk through the function. First I am converting the policy string into a series of bytes. To do that you must choose an encoding (because different encodings lead to a different series of bytes for the same string). I choose ASCII because that’s what I found was being used in some of the PHP and Ruby samples I came across. Apparently that is correct because this works.
Once you have the series of bytes for the policy string, you need to convert it into a base64 string, which really means reinterpreting the series of bytes in 6 bit chunks to ensure each chunk stays within a certain range. Anyways, .NET provides the handy Convert.ToBase64String function to deal with that.
Once you have the base64 encoded policy string, you need to hash it. To do that you create an instance of the .NET HMAC SHA1 hasher, supplying the bytes of the secret key as a parameter to its constructor. Again we get the bytes of the private key using the ASCII encoder.
Now we use that hasher to hash the policy bytes. The result of that is the set of bytes of the signature which can then be converted into the base64 encoded string and returned.
Base64 encoding has to be used to transmit both the policy string and the generated hash because base64 is a way to guarantee that any kind of characters will be first converted into the ASCII range. That, as I have read, is a fairly common practice.
So in the end the code to generate the view model ends up looking like this
public FileUploadViewModel GenerateViewModel(string publicKey, string secretKey, string bucketName, string fileName, string redirectUrl)
{
var fileUploadVM = new FileUploadViewMode();
fileUploadVM.FormAction = string.Format(“https://{0}.s3.amazonaws.com/”, bucketName);
fileUploadVM.FormMethod = “post”;
fileUploadVM.FormEnclosureType = “multipart/form-data”;
fileUploadVM.Bucket = bucketName;
fileUploadVM.FileId = fileName;
fileUploadVM.AWSAccessKey = publicKey;
fileUploadVM.RedirectUrl = redirectUrl; // one of private, public-read, public-read-write, or authenticated-read
fileUploadVM.Acl = “private”; // Do what you have to to create the policy string here
var policy = CreatePolicy();
ASCIIEncoding encoding = new ASCIIEncoding();
fileUploadVM.Base64Policy = Convert.ToBase64String(encoding.GetBytes(policy));
fileUploadVM.Signature = CreateSignature(secretKey, policy);
}
Finally the View
The view, like I said is using the Razor engine, and simply lays out a “form” that has all the relevant fields Amazon S3 requires and optionally others you care about.
@model TheApp.ViewModels.FileUploadViewModel
@{
ViewBag.Title = “Upload File”;
Layout = “~/Views/Shared/_Layout.cshtml”;
}
<div id=”fileuploaddiv” class=”fileuploaddivclass”>
<form action=”@Model.FormAction” method=”@Model.FormMethod” enctype=”@Model.FormEnclosureType”>
<input type=”hidden” name=”key” value=”@Model.FileId” />
<input type=”hidden” name=”AWSAccessKeyId” value=”@Model.AWSAccessKey” />
<input type=”hidden” name=”acl” value=”@Model.Acl” />
<input type=”hidden” name=”policy” value=”@Model.Base64EncodedPolicy” />
<input type=”hidden” name=”signature” value=”@Model.Signature” />
<input type=”hidden” name=”redirect” value=”@Model.RedirectUrl” />
<div> Please specify a file for upload: <input type=”file” name=”file” size=”100″ /> </div>
<input type=”submit” value=”Upload” />
</form>
</div>
Hopefully this helps. Feel free to leave feedback and questions.
