In this episode, Robert and I dig deep into Integration Events. What are they? Where did they come from? How can I use them? Should I use them at all? All your questions are answered!

I know what you’re thinking – haven’t you mentioned integration events before? Sure, but just at a high level. And we’ve both been getting some pretty interesting questions from customers and partners who are starting to leverage this feature. So I thought “hey, why not try to address a whole bunch of those in a podcast?”  Here we go…

What are integration events?

We mentioned these in our R17 podcast. Integration events are essentially our answer to an outbound web service call. We don’t enable a call to an external api directly from the system, but customers need a way to initiate an action outside the system based on some activity in CRMOD.

An example might be someone creating a new Account or updating a Contact e-mail address. You’d want those kinds of things to kick off a process that syncs those changes back to another system. In the old days you would use a polling process that would query, say, for all contacts modified since the last time you polled. You’d get a bunch of records back and process them.

The problem is that this is very inefficient – you had no way of know if the field or fields you cared about was actually touched. Plus you may be scanning a whole lot of records and that could be pretty slow. So integration events give us a better way.

Way better, in fact. The cool thing is that the event itself is just a simple xml message – the lingua franca of the web, naturally – that just gives you what you need to know. Like the row ID of the record, the value of the changed field, and any fields marked for auditing.

How do we use them?

OK, so that’s a good high level, let’s get into detail and approach this as we would in an implementation. First, let’s use your example as our use case – we need to track changes to Contact e-mails so that we can populate those changes in a separate marketing database. The basic flow is a user updates a contact and if (and only if) they changed the e-mail address, we want it to trigger a message that this address needs to be updated in the other system.

But first…

We need to do a little planning first and make sure we’re ready to implement integration events.

First, you may need to have Integration Events enabled for your company. You can see this as a checkbox in the Company Profile. If the “Integration Event Enabled” checkbox is not checked, contact support to have it enabled.

Now that you have integration events enabled, we may want to create a named queue. Companies can just use the default queue or they can create new ones. Queue is a britishism for “line”. But in this context, we’re talking about a named container for your events. Having separate named queues can make managing multiple integrations a bit easier.

Back to our use case…

Alright, so we have a few pieces to handle. First, we need to know when a specific field on a specific type of record (Contact) is changed. Hey, isn’t workflow good for that?

So we first need a workflow rule to check for a change in the field. This will be a “When Modified record saved” Trigger Event. We’ll put a condition on the workflow to compare the current value of Email to the previous value (PRE)

Then we set up an action of type “Integration Event”. We’re done, right?

Almost…  We can set up a few options on the Event. First, we assign it to one or more Queues – this is required.  Once the action is saved, you’ll see a “Configure” option in the drop-down thingy next to it. This is where we see our Field Tracking screen. And we want to check off the boxes next to the fields that we want to track – so in this case, we’d be sure to select “Email”.

Using the new event

OK, that was easy! Whenever a contact record is modified, our new workflow rule will fire and evaluate the rule condition to see if the email address has changed. If so, the action will be invoked and an integration event will be added to our named queue.

So what can we do with that? Well, as we mentioned at the top of the ‘cast, the event is a little package of information in xml format. It’s going to contain a few things:

1. The field data that we marked as “tracked” (IF that field data changed)

2. All fields that are marked as required for the record type (not page layout level)

3. All fields that are included in the Audit trail.

And this event just sits in the queue. All alone. Until other events come and join it, I guess.

So what we need is another, external application to come and check the queue to see if anything is in it. Via web services, natch. This client application will log-in to CRMOD using the Service API and Integration Event WSDL (you’ll see these in your Web Services Administration page). With integration event web services there are only two things you can do:

GetEvents – this pulls down the events in the queue. Your options here are to specify how many events you want and which named queue you want them from.

DeleteEvents – this removes the events from the queue. Again, you can specify how many to remove and from which queue.

Now we have a process where our external application wakes up every so often – how often depends on how frequently you expect events to be generated – and checks the queue. It Gets Events, then Deletes those same events, freeing the queue for new events.

Is that all? Pretty much. The setup in CRMOD is pretty straightforward. Of course, we do get a lot of questions…

What’s the maximum queue size?

You can have an unlimited number of named queues, but there is a maximum number of events allowed per company. Once your queue reaches the limit for your company, no more events will be added. You will never lose events, but new changes will just stop being glommed on top.

And each named queue gets a portion of that company total limit. So if your company has 1000 event limit and you set up 4 queues, you can parcel out that amount – 250 each, or whatever you choose. This is all done via the Integration Event administration page.

What if my queue fills up?

A couple of things. First, for each named queue you can set up an alert  threshold. So an e-mail will be sent to whomever you designate when the queue reaches a certain size – which you set. That can help to give you a heads-up that something might be going on.

Second, a good integration practice is to prevent the queue from filling up to begin with. Ideally the queue never reaches your alert threshold or even 50%. The key is to poll the queue and clear it regularly – you may need to do some fine tuning to get it just right, but ultimately you want a good balance.

Can I get a bigger event limit?

Maybe. If you have a good reason to think your queues will need more than the default allotment, you can submit an SR and request more. This is part of sizing – space is allocated based on the maximum – so such requests are taken seriously.

And if you ask for something unreasonable – like 500,000 – without adequate justification is likely to be shot down. Most customers get by with 1000 or so just fine. I’ve seen up to 10,000 with good justification.

How can I stop integration events from being generated?

Typical scenarios would be during import or another inbound integration. If you don’t want integration events to be generated from imports, you can add an ExcludeChannel condition to your workflow. You can do the same for integrations, but there is also an “Echo” parameter you can include in your web services – if this is set to “off” then the insert / update via web services won’t result in an integration event.

Are there performance implications to using Events?

Most often integration events solve performance problems. They allow you to get direct information about changes / inserts without polling the entire db. That’s much more efficient.

Keep in mind that there may be some overhead with workflow – if you have a lot of workflows firing and creating a bunch of integration events, it might slow things down. A good practice is to employ a rule condition so events only get created when you really want them to.

I also get questions about the frequency of polling the queue. Generally I haven’t seen many issues with polling down to every 5 minutes or so. Some customers I’ve worked with do every minute without issue. Best is to start with the least frequent interval that achieves the business need and keeps the queue at a low level – then increase if needed.

What if the event package doesn’t contain the field I want?

This happens sometimes – in our use case, let’s say you needed to know something more about the Contact’s Account. That information may not be directly in the event package. This would require a “query back” – another web service call based on the event contents to get the additional information.

So not quite as efficient as just getting everything in the event, but still pretty fast since you will always get the record Row ID in the event package to create a quick query.

Other best practices?

One last thought. Error handling. Poor error handling is the number one reason we see integration event queues fill up and those alerts go out. You need to budget the time and resources to do testing for a variety of conditions – such as an unexpected bit of xml in the event package. Make sure your event processor can gracefully deal with whatever it finds in the queue – save the data and move on.

Wrap Up

Hopefully we’ve covered all of YOUR questions about integration events. No? Then leave us a comment, we love comments! Adios until next time….

Share and Enjoy:
  • Digg
  • LinkedIn
  • del.icio.us
  • Facebook
  • Mixx
  • Google Bookmarks
  • Twitter
  • StumbleUpon
  • Technorati
  1. Satish on 27 May 11 4:59 am

    Hi.

    Excellent podcast.

    Can you plz come with new podcast on data import to CRM OD using web services integration. Looking forward for the same. TIA

    Rgds
    Satish.

  2. Udayvir Singh on 27 May 11 7:56 am

    Hi Louis Peters,

    I have faced one issue regarding clearing the Integration Event Queue (IEQ). Let me give you an example to better understand the issue.

    Suppose, I have to process a records that are moved to queue using workflow. When I query an IEQ, I can specify number of records to query as a parameter (i.e. “EventCount”).

    For instance, I query 20 records at once and process them. If all the records are processed successfully, I delete them all (last 20 records queried) using “LastEventId”, that I get as a result of IEQ query. And then I query next set of 20 records and process them.

    In this scenario, if some records fail (while processing) due to any reason, I do not see any way of selective deletion of records from IEQ, as CRMOD does not provide Event Id for each IEQ record, if records are queried in bulk. Instead CRMOD provides “LastEventId”, using which one can delete all the records queried (in last IEQ query).

    It will be great, if you could respond with some thing that can handle this scenario.

    Appreciate your help. Looking forward to your reply.

    Thanks and Regards,
    -Udayvir Singh

  3. Satish on 06 Jun 11 2:55 am

    Hi Louis,

    Can you please explain in detail how the workflows work in CRM on Demand in comparision with On Premise.

    Appreciate your help …. Looking forward to your reply.

    Rgds,
    Satish

  4. Pooja on 01 Jul 11 7:05 am

    I am trying to use integration events with PHP.
    I am not getting exactly how to call the integrationevents.wsdl in the file and where to define the schema files.

    Please be a bit more elaborative as to where and how to include these wsdl files.

    Regards,

    Pooja