Question

BIX extract to Amazon S3 Bucket from Pega Cloud

Hi

My client would like some case data extracted and we are planning on using BIX. We are using Pega Cloud version 7.4 and the BIX documentation states that you have to write the files to an SFTP sever. However the client wants us to write the files to an Amazon S3 Bucket. Has anybody done this before and can suggest an approach?

Comments

Keep up to date on this post and subscribe to comments

Pega
January 9, 2019 - 10:22am

The files are written to S3 bucket before getting to the SFTP server but don't believe there is an option to specify a S3 bucket of your choice. This would be an enhancement on Pega cloud.

September 23, 2019 - 6:36pm

Hi,

 

This might be useful for others who have a similar requirement. Below Java code will send a file from your pega server to amazon s3 bucket. Best practice is to build a function and call it inside your custom activity (refer pzFTPService if you want to use this in conjunction with BIX)

 

The AWS Java SDK jar file needs to loaded to your pega environment. The version of this JAR file will be based on the java version your pega environment.

 

The JAVA code that sends the files to s3 bucket also uses the below JAR files. However, they are already part of OOTB PEGA product and should not be touched due to versioning issues.

1. Apache Commons Logging
2. HttpComponents 
3. jackson annotations
4. Jackson Core
5. Jackson Databind
6. Joda Time

 

Below classes needs to be imported by your function or library.

1. java.io.File
2. java.io.IOException
3. java.util.Date
4. com.amazonaws.AmazonClientException
5. com.amazonaws.AmazonServiceException
6. com.amazonaws.auth.AWSCredentials
7. com.amazonaws.auth.AWSStaticCredentialsProvider
8. com.amazonaws.auth.BasicAWSCredentials
9. com.amazonaws.services.s3.AmazonS3
10. com.amazonaws.services.s3.AmazonS3ClientBuilder
11. com.amazonaws.services.s3.model.ListObjectsRequest
12. com.amazonaws.services.s3.model.ObjectListing
13. com.amazonaws.services.s3.model.PutObjectRequest
14. com.amazonaws.services.s3.model.S3ObjectSummary

 

JAVA CODE:

  /* * The following properties should be coming from a DSS.

    * Make sure the Key & Secret are stored encrypted in the DSS, and never in plain text

    */

        String awsAccessKey = "aws Access key";

        String awsSecretKey = "aws secret key";

        String s3BucketName = "S3 bucket name";

        String awsRegion = "aws region";

        String filePath = "the complete filepath of the source file";

        

   

        AWSCredentials credentials = new BasicAWSCredentials(awsAccessKey,awsSecretKey);

        

    //Server side encryption should be configured at the Bucket level

        AmazonS3 s3 = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).withRegion(awsRegion).build();

        

        try {

            //Upload the file to the S3 bucket

            File fileToUpload = new File(filePath);

            if(fileToUpload.exists() && fileToUpload.isFile()) {

            String s3ObjectKey = "File Name" + "Extension"; //Extension depends on the type of your file .txt, .xml, .zip etc..

                s3.putObject(new PutObjectRequest(s3BucketName, s3ObjectKey, fileToUpload));

            }

            

            //List all the files in the bucket. we can disable this code once the code is completed.

oLog.error("Listing objects");

            ObjectListing objectListing = s3.listObjects(new ListObjectsRequest().withBucketName(s3BucketName));

            for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {

                oLog.error(" - " + objectSummary.getKey() + "  (size = " + objectSummary.getSize() + ")");

            }

          return true;

        } catch (AmazonServiceException ase) {

            System.out.println("Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason.");

            oLog.error("Error Message:    " + ase.getMessage());

            oLog.error("HTTP Status Code: " + ase.getStatusCode());

            oLog.error("AWS Error Code:   " + ase.getErrorCode());

            oLog.error("Error Type:       " + ase.getErrorType());

            oLog.error("Request ID:       " + ase.getRequestId());

          return false;

        } catch (AmazonClientException ace) {

            oLog.error("Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.");

            oLog.error("Error Message: " + ace.getMessage());

          return false;

        }

 

Regards,

Naveen.