Salesforce + AWS Lambdas

In this article, I’d like to share one of my go-to patterns for developing on the Salesforce Platform, a microservices-inspired structure, with the hope of inspiring other, fellow developers in their processes.

I’ve spent a great deal of time with the Salesforce Platform, building all different kinds of interesting services. And I’ve also had a lot of fun doing it. With excellent automation and integration possibilities, I’ve found that you can build some truly cool and innovative stuff for your organization.

Let me also mention that I’ve been working with Trustpilot for several years. It’s a great place with a lively culture that really encourages innovation. Lots of hackathons, 20% projects, and more.

But with rapid prototyping and experimenting, you’ll need a good approach to help keep your org’s Salesforce Platform clean and tidy. Hopefully, this article can help with that challenge.

The Premise

Salesforce uses a Multi-Tenant Architecture. This means that resources are shared among the system’s users and processes, and we can’t go crazy with creative services running in the background. We have to be mindful not to hog all the bandwidth from our users, as well as stay within the Governor Limits that Salesforce imposes.

One might think that it quickly becomes boring with all these limitations, but it doesn’t have to be!

What we like to do is simply offload the heavy lifting. This can easily be done with AWS (our cloud computing platform of choice).

Let’s have a quick look at the general idea, and then a deeper dive into the concept.

Concept Overview

The concept looks like this:

1. An event is triggered in Salesforce (this could be a user action or a scheduled event).
2. Salesforce informs AWS about the event (sends an Outbound Message).
3. AWS Compute Service, like a Lambda, performs all the work (long-running calculations, data aggregation, etc.).
4. AWS reports back to Salesforce with the results, and updates data accordingly.

This integration-pattern is also called “Fire and Forget, Remote Process Invocation”:
https://developer.salesforce.com/docs/atlas.en-us.integration_patterns_and_practices.meta/integration_patterns_and_practices/integ_pat_remote_process_invocation_fire_forget.htm

About AWS Lambdas

I love AWS Lambdas. They fit fantastically into this setup because:

1. They’re serverless, so there’s no burden of maintenance or monitoring uptime. You also don’t pay for when they’re not running, so no idle time.
2. They offer programming language flexibility. This means you can use different programming languages such as Python, for example, which has tons of community-built libraries.
3. As a bonus, they’re great for easy collaboration. You can include other engineers in your project who might be less familiar with Apex and the Salesforce Platform.

Let’s move on and have a look at the concept in greater detail.

Concept Details

Let’s zoom in on what happens on the AWS side. The entire architecture could look something like this: I’ll explain each part.

The Message Receiver

Salesforce Outbound Messages need an endpoint, so we need an AWS API Gateway. We connected the Gateway with Lambda, which will work as our Message Receiver.

Salesforce has several integration capabilities. I like Outbound Messages best because of Salesforce’s built-in retry mechanism, and because Outbound Messages doesn’t consume API Calls (Salesforce has limitations, one being the number of API Calls an org can perform every 24 hours rolling).

The Message Receiver Lambda simply forwards the message to the AWS SQS (a queue service) and confirms that it was received by sending an acknowledgement back to Salesforce.

You might think: Why bother with the SQS? Why not simply handle all the work then and there?

I find having a message queue is super handy, as it adds an enormous amount of flexibility. Let me elaborate.

Advantages of the Message Queue

The first advantage is the asynchronous processing. We can handle spikes in traffic without any problems. Messages simply enter the queue and are processed at our own pace, and no other systems are hanging while waiting for a response.

The second advantage is the Dead Letter Queue (SQS DLQ) that we can connect to, which works like a safety net for our service. It’s a parking lot for messages we fail to process. If anything unexpected or unhandled happens in the Message Processor, we throw an exception and the message goes to the DLQ.

Nothing is lost and nothing is stuck. The Message Processor can simply move on to the next message in the SQS. Meanwhile, we can inspect and debug the messages in the DLQ. Once we have deployed a fix, we can simply push the message from the DLQ back into the SQS, and they’re processed like nothing ever happened.

A side note: while striving to be agile, I find utilizing the DLQ helps in this regard. We can build the main functionality first and worry about the edge cases later (once they appear in the DLQ). It saves a lot of time, and lets you move faster.

The Message Processor

The Message Processor is the actual service where all the work is done. AWS Lambdas can be configured to automatically pick from an SQS whenever new messages are available. Lambdas run up to 15 minutes, which should be enough time to finish the job. Otherwise, consider composing multiple Lambdas with step functions or a Docker instance.

When the job is done, the service simply calls the Salesforce API and updates the relevant records. You’ll need to authenticate, and the best approach for this is to create a Connected App in Salesforce. You’ll also have the option to include a Session Id in your Outbound Message, which works as an access token for callback. However, I dislike this approach for two reasons:

1. Security concerns: the session token is based on the user who triggered the Outbound Message, and gives the same full set of privileges. However, with a Connect App, you can specify only the minimum security policies needed.
2. Session Ids expire: so messages parked in the DLQ for too long become invalid.

In the next section, I’ll give an example of a service we’ve built recently.

Service Example

Let’s go over one of our current services where we use the pattern presented above.

A quick disclaimer: I’ve chosen a fairly simple example to showcase, for the sake of brevity. This may seem misleading, as the concept was about offloading complexity from Salesforce to AWS. However, this could just as easily have been more advanced. We also might showcase a more complex example in the future.

Use Case Context

I’ll be showcasing our Company Registration Number Validator service.

Salesforce holds all the Customer Profiles. That is, businesses you would see on Trustpilot.

We’d like to verify these businesses, and one way to do this is by looking them up in the respective government databases. We’ll be focusing on the UK here.

Here’s the link to their API:

https://developer-specs.company-information.service.gov.uk/companies-house-public-data-api/reference/company-profile/company-profile

Given a UK Company Registration Number, you can perform a lookup by doing a GET request on:

http://api.company.information.service.gov.uk/company/{companyNumber}

You can try the API yourself, but a heads up, you’ll need an API key for authorization.

On the Salesforce Side

In Salesforce, we’ve added a custom field to the Account object: Company_Registration_Number__c.

We also added an outbound message that’s triggered whenever the value of that field changes (as long as it still contains a value). The formula looks like this:

(ISNEW() || ISCHANGED(Company_Registration_Number__c)) &&
!ISBLANK(Company_Registration_Number__c)

*Remember to whitelist the endpoint of your AWS API Gateway in the Salesforce Remote Site Settings

On the AWS Side

The Message Receiver simply forwards the message, so we’ll skip that part and go straight to the Message Processor.

I’ll share some code snippets written in Python. It consists of a few simple steps:

• Parsing the incoming message
• Validate message (check it’s not outdated)
• Perform Company Registration Lookup
• Update in Salesforce

Notice that for AWS Lambdas, the message will be deleted automatically as long as the Lambda function doesn’t run into an error. If running an AWS EC2 instance, however, you’d have to manually delete the message at the end.

Before processing anything, I like to do an upfront check to validate the request. Maybe the message has been sitting in the DLQ and is now outdated, or perhaps the flow was accidentally triggered by the user. Whatever it might be, we check if the message can be skipped.

In this example, we simply want to be sure that the Salesforce Account exists, and that the CRN value has changed in the meantime. Otherwise, we can skip the message, ie. continue to the next message

For Python, simple-salesforce (https://github.com/simple-salesforce/simple-salesforce) is a great package when working with the Salesforce APIs.

The UK Companies House API returns a lot of information. We’ll keep it simple for now and just look at the values of company_name and company_status. Hopefully the name matches and the status is active.

Finally, update the Salesforce Account with the newly-found values. Done! The message is now removed from the message queue.

In Conclusion

I’ve presented an alternative approach on how to develop a service for the Salesforce Platform that relies on offloading and asynchronous processing. In my experience, this approach adds a lot of flexibility to the development process, and is advantageous for agile and iteration driven development.

This approach also has certain advantages when it comes to scaling. It’s easy to introduce new services, and it’s quick to clean them up again. It’s great for experimenting and handling high traffic, but it doesn’t cost you when idle.

As for everything in IT, it’s all about the tradeoffs. So it must be said that this approach probably isn’t an ideal fit for every solution. Use it where it makes sense.

Keep an eye out for part two in the future, showcasing even more about this interesting concept. Take care!