Search by Tags

How to use Cloud S3 Library

 

Article updated at 25 May 2022
Compare with Revision




Attention: this is a legacy library and thus not supported by Toradex anymore. We recommend that you use the new libraries for all Toradex modules. Please see the Toradex CE Libraries and Code Samples for up-to-date information.

Amazon Simple Storage Service (Amazon S3)

As part of the AWS Free Usage Tier, you can get started with Amazon S3 for free. Upon sign-up a new AWS customer receives 5GB of Amazon S3 storage, 20,000 Get Requests, 2,000 Put Requests and 15GB of data transfer out each month for one year.

Notes:

  • Number of buckets is limited to 100 per region.
  • Use short length bucket / file / directory names for fast response time.
  • Use only alphanumeric / underscore / hyphen / period characters to construct bucket / file / directory names and avoid using special characters.
  • The system time of Colibri Module should not be deviated by more than 15 minutes otherwise AWS specific functions will return error.

For more information please refer this link: http://aws.amazon.com/s3/

Opening the S3 Handle

Before calling any other function, call S3Open() with the input parameters S3Handle, AWS Access Id, AWS Secret Key and AWS S3 Region / Path.

The call will return TRUE if successful or FALSE if an error has occurred.

Use GetLastError() to get extended error details.

An example is described below (example for Singapore region S3 path is "s3-ap-southeast-1.amazonaws.com"):

// declare HS3 Handle to be used in all S3 based Function calls.
HS3 S3Account1 = NULL; 
// temporary variable to store return of a function call
BOOL result = 0;
 
result = S3Open(S3Account1,
                “20CharacterAWSaccessID”,
                “40CharacterAWSSecretKey”,
                “s3Path”);
if (result == TRUE)
{
    printf(“S3Open Successful\n”
}
else
{
    printf(“S3Open Error\n”
}

The S3Account1 handle returned needs to be passed to all subsequent function calls.

Getting the List of All the Buckets in a Region

The S3ListAllBuckets() function is used to get the list of all buckets in a region, which will be stored in a local file.

  • 1st parameter to the function is the S3Handle returned from the S3Open() function call.
  • 2nd parameter should always be 0 or NULL.
  • 3rd parameter is the local file path in which the function will store the list of buckets data.
  • 4th parameter is the local file size.

The function will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

result = S3ListAllBuckets(s3Account1,
                          0,
                          "\\Temp\\AllbucketList.txt",
                          &s3ListALLBucketsRawDatabytes);
if (result ==  TRUE)
{
    printf(“S3ListAllBuckets Successful\n”
}
else
{
    printf(“S3ListAllBuckets Error\n”
}

Find a Bucket’s Details from the List of all Buckets

The FindFirstBucket() function is used to get a particular bucket’s information from the list of all buckets in a region, which is stored in a local file by a call to function S3ListAllBuckets().

  • 1st parameter to the function is the path of the local file in which the list of buckets is stored.
  • 2nd parameter is size of the file in bytes, returned from the S3ListAllBuckets() call.
  • 3rd parameter is bucket data returned by the function FindFirstBucket(), it will have the Bucket Name and Creation Date information. The programmer needs to allocate sufficient memory and needs to clear that memory before calling this function.
  • 4th parameter is the number of bytes allocated for the bucket data.
  • 5th parameter is a variable pointer to the next bucket details. For the 1st call set this variable to 0. The next bucket details pointer will be returned in this parameter on each subsequent call.
  • The function will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

DWORD nextBucketPointer = 0 ; //Set to zero to get 1st bucket details.
s3bucketData = (char *) malloc(1024); // Allocate memory to hold bucket details
memset(s3bucketData, 0, 1024); // Clear the memory
result = FindFirstBucket("\\Temp\\allbklist.txt",
                         s3ListALLBucketsRawDatabytes,
                         s3bucketData,
                         1024,
                         &nextBucketPointer); 

To get the next bucket details from the list stored in the local file, call the same function FindFirstBucket() with nextBucketPointer variable as 5th input parameter returned from previous call to FindFirstBucket().

memset(s3bucketData, 0 , 1024); // Clear the memory
result = FindFirstBucket("\\Temp\\allbklist.txt",
                         3ListALLBucketsRawDatabytes,
                         s3bucketData,
                         1024, 
                         &nextBucketPointer); 

Get List of All the Files in a Bucket

The S3ListBucket() function is used to get the list of all files in a bucket of a region, which will be stored in a local file.

This function works in a similar way as S3ListAllBuckets() works.

  • 1st parameter is the S3Handle returned from the S3Open() function call.
  • 2nd parameter is the bucket path (Example : “/BucketName/”, don’t forget to use forward slashes).
  • 3rd parameter is the local file path in which the function will store the list data.
  • 4th parameter is the length of the file stored, in bytes.
  • The function will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

DWORD s3ListBucketRawDatabytes = 0;
result = S3ListBucket(s3Account1,
                      “/BucketName/”,
                      "\\Temp\\bklist.txt",
                      &s3ListBucketRawDatabytes);
if (result == TRUE)
{
    printf(“S3ListBucket Successful\n”
}
else
{
    printf(“S3ListBucket Error\n”
}

Find File Details from the List Stored in a Local File

The FindFirstFileInBucket() function is used to get a file detail from the list of all files in a bucket of a region, stored in a local file by previous call to S3ListBucket().

  • 1st parameter is path of the local file in which the List of files in a bucket is stored.
  • 2nd parameter is size of the file in bytes, returned by call to S3ListBucket() in 4th parameter.
  • 3rd parameter is the variable in which file data will be returned by the function FindFirstFileInBucket(). It will have File/Directory name and File/Directory size information, the programmer needs to allocate sufficient memory and clear that memory before calling this function.
  • 4th parameter is the number of bytes allocated for 3rd parameter.
  • 5th parameter is the variable pointer to the next file details, for the 1st call put this variable to NULL, the next file details pointer will be returned in this parameter, and this will be used to get next file details.
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

DWORD nextFilePointer = 0 ; //Set to zero to get 1st file details.
s3FileInBucketData = (char *)malloc(1024); // Allocate memory to hold file details
memset(s3FileInBucketData, 0 , 1024); // Clear the memory
result = FindFirstFileInBucket("\\Temp\\bklist.txt",
                               s3ListBucketRawDatabytes,
                               s3FileInBucketData,
                               1024,
                               &nextFilePointer);

To get the next file details from the list stored in the local file, call the same function: FindFirstFileInBucket() with nextFilePointer variable as 5th input parameter returned from previous call to FindFirstFileInBucket()

memset(s3FileInBucketData, 0 , 1024); // Clear the memory
result = FindFirstFileInBucket("\\Temp\\bklist.txt",
                               s3ListBucketRawDatabytes,
                               s3FileInBucketData,
                               1024,
                               &nextFilePointer);

Create New Directory in a Bucket

The S3MakeDirectory() function is used to create a new directory in an existing bucket.

  • 1st parameter is S3Handle returned from S3Open() function call.
  • 2nd parameter is the path of the directory in a bucket. If bucket does not exist, the call to S3MakeDirectory() will fail. Don’t forget to use forward slashes in the path.
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

result = S3MakeDirectory(s3Account1, “/BucketName/DirectoryName/);
if (result == FALSE)
{
	printf("S3MakeDirectory Fails: Error Code:");
	printf("%d\n", GetLastError());
}
else
{
	printf("S3MakeDirectory Success\n");
}

Delete a directory in a Bucket

The S3RemoveDirectory() function is used to delete a directory in an existing bucket.

  • 1st parameter is the S3Handle returned from the S3Open() function call.
  • 2nd parameter is the path of directory in a bucket (if the bucket does not exist, a call to S3RemoveDirectory() will still be successful). Don’t forget to use forward slashes in the path.
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

result = S3RemoveDirectory(s3Account1, “/BucketName/DirectoryName/);
if (result == FALSE)
{
	printf("S3RemoveDirectory Fails: Error Code:");
	printf("%d\n", GetLastError());
}
else
{
	printf("S3RemoveDirectory Success\n");
}

Delete a File in a Bucket

The S3DeleteFile() function is used to delete a file in an existing bucket.

  • 1st parameter is the S3Handle returned from S3Open() function call.
  • 2nd parameter is the path of the file in a bucket (if bucket does not exist, call to S3DeleteFile() will still be successful).
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

result = S3DeleteFile (s3Account1, “/BucketName/FileName.txt);
if (result == FALSE)
{
	printf("S3DeleteFile Fails: Error Code:");
	printf("%d\n", GetLastError());
}
else
{
	printf("S3DeleteFile Success\n");
}

Download a File

The S3DownloadFile() function is used to download a file from the S3 Bucket.

  • 1st parameter is the S3Handle returned from S3Open() function call.
  • 2nd parameter is the S3 file path in a bucket. If bucket does not exists, call to S3DownloadFile() will fail.
  • 3rd parameter is the local file path where the downloaded file will be saved.
  • 4th parameter is the size of the file to be downloaded, this can be found out by a call to function S3ListBucket() and searching for the file name and size.
  • 5th parameter is the variable in which the function will put the number of bytes downloaded.
  • 6th parameter is the pointer to call back function. With the help of the call back function the programmer can get the progress of the download. The function will be called after every 32KB of data is downloaded.
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

// User need to check if Destination File Already Exists or not to be safe
if (S3DownloadFile(s3Account1, 
                   “/BucketName/File1.txt”,
                   “\file.txt”,
                   10234,
                   &bytesDownloaded,
                   &ProgressCallback))
{
	printf("S3DownloadFile Successful\n");
}
else
{
	printf("S3DownloadFile Fails: Error Code: ");
	printf("%d\n", GetLastError());
} 
 
// Call back function definition
void ProgressCallback(DWORD bytesDone, DWORD totalBytes)
{
    printf("> %d bytes, %d \n", bytesDone, totalBytes);
}

Upload a File

The S3UploadFile() function is used to upload a local file to the S3 bucket.

  • 1st parameter is the S3Handle returned from S3Open() function call.
  • 2nd parameter is the local file path which needs to be uploaded.
  • 3rd parameter is the size of the file to be uploaded.
  • 4th parameter is the S3 file path in a bucket. If bucket does not exists, the call to S3UploadFile() will fail.
  • 5th parameter is the variable in which the function will put number of bytes uploaded.
  • 6th parameter is the pointer to call back function. With the help of the call back function the programmer can get the progress of the upload. The function will be called after every 32KB of data is uploaded.
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.

if (!S3UploadFile(s3Account1,
                  “\LocalFile.txt”,
                  uploadfilesize,
                  “/BucketName/S3File.txt”,
                  &bytesUploaded,
                  &ProgressCallback))
{
	printf("UploadFile Fail\n");
}
else
{
	printf("UploadFile Success\n");
}
 
// Call back function definition
void ProgressCallback(DWORD bytesDone, DWORD totalBytes)
{
    printf("> %d bytes, %d \n", bytesDone, totalBytes);
}

Close S3 Handle

The S3Close() function closes S3Handle opened by the call to S3Open() and frees all the resource allocated.

  • The input parameter is the Handle opened by call to S3Open().
  • It will return TRUE if successful or FALSE if the call fails. Use GetLastError() to get extended error details.
    S3Close(&s3Account1);