Remote Event Receivers – you’re all doing it wrong

Remote Event Receivers are a powerful way to integrate custom code into your SharePoint Online environment.  Essentially a Remote Event Receiver is a hook that allows you to execute your code in response to an event that occurs in SharePoint.  There are several techniques for responding to events in SharePoint Online, but the Remote Event Receiver is the most powerful. It offers dozens of different events to attach to and allows you to configure synchronously or asynchronously. It is also very easy to attach, develop, deploy, test, and maintain. Is also very misunderstood.

Remote Event Receivers were introduced along with SharePoint 2013 and the arrival of the App model. Microsoft provided tooling with Visual Studio to create Remote Event Receivers, but unfortunately the only way to expose this tooling was in the context of a Provider-Hosted App.

This is unfortunate because Remote Event Receivers have nothing to do with Provider-Hosted Apps. In order to develop a Remote Event Receiver using the Microsoft-provided tooling, a developer had to create a Provider-Hosted App project and deploy their RER along with it. This added a great deal of complexity to the development effort, and made packaging and deployment a painful and tedious experience.

To further complicate things, Microsoft decided to wire up a WCF service as the endpoint in its RER tooling. This is sheer lunacy, even back in 2013. A web API project would have been simpler and would be more in line with Microsoft development tooling efforts. The opacity and complexity of WCF made RER development even more cumbersome. Actually, I believe they decided to use the WCF service so the development experience would be similar to that of old-school Event Receivers, with the ability to use a deserialized Event Properties object.

Building a Remote Event Receiver is Easy

In truth, it is remarkably simple to configure, develop, deploy, and maintain a Remote Event Receiver, but in order to do so you must completely abandon the Microsoft tooling and just set up the pieces yourself. Luckily there are really only two components to a Remote Event Receiver:

  • The endpoint
  • The registration

The registration is where you tell SharePoint, “call this endpoint every time this event occurs”. The CSOM provides mechanisms for adding Remote Event Receivers, but the details depend somewhat on the type of event receiver being deployed. The  PnP PowerShell library provides the ability to register a Remote Event Receiver in a one-liner. For example, to set up an RER that is invoked every time an item is updated on a list, execute:

Add-PnPEventReceiver -List "Tasks" -Name "TasksRER" -Url https://my-rer.azurewebsites.net/Service1.svc -EventReceiverType ItemAdded -Synchronization Asynchronous

More about registering Remote Event Receivers

The endpoint is just a web service listening at a certain URL, and you have lots of options for this. A Web API project would work great for this. Azure Functions are also a very compelling option. You are also free to write services in Java, Node or whatever other technology you can think of. In the example below, we’ll use the canonical WCF Service you’d get with the Visual Studio item template, but we’re going to sidestep the template and wire things up ourselves. It’s actually easier this way.

 

Creating the Remote Event Receiver shell

In Visual Studio,

  1.  create an empty ASP.NET web application
  2. Add the Nuget Package ”AppForSharePointOnlineWebToolkit”.
  3. Add a new item of type WCF Service to the web app.
  4. Get rid of the IService reference and set your service to implement IRemoteEventService, which lives in the Microsoft.SharePoint.Client.EventReceivers namespace. This namespace came into the project with the Nuget packages we added earlier. Resolve the squiggly to implement the interface stubs.

Your service class should look something like this:

RER-stub

F5 your project and navigate to the service to make sure it’s accepting requests. Take note of the port number.  We’ll need that to set up our proxy for local debugging.

Locally debugging your remote event receiver

To test this event receiver locally, we’ll use ngrok. According to its documentation, “ngrok is a reverse proxy that creates a secure tunnel from a public endpoint to a locally running web service.”  We will use it to map an Internet endpoint to our local machine so we can intercept and debug requests coming from SharePoint Online.

Assuming you have installed Node.JS and ngrok, create your proxy by executing the following. 56754 is the port number hosting my local WCF service.

ngrok-1

Once it connects it’ll output some data, including the public URL of our proxy connection:

ngrok-2

Next, open up a browser and navigate to the ngrok URL, append the service endpoint, and you should be able to see that the ngrok URL is returning your service.

ngrok-3

 

Next we’ll attach our event receiver to a SharePoint list. Using the PnP PowerShell cmdlet shown above, we’ll add a Remote Event Receiver to our site. Make sure to open a new PowerShell session for this and leave the ngrok session running in its window. The proxy will be released when the window is closed.

Now, set a breakpoint in your ProcessOneWayEvent method, add a task to the list and make an edit to it.  If all goes well your local web service will be called and the breakpoint will hit:

rer-2

Make sure you closely inspect the properties object in the debugger, and get a feel for all the data that’s in there. For this particular event we’ll want to check out ItemEventProperties and ItemEventProperties.AfterProperties for some useful metadata that gets passed into the service.

Deploying your Remote Event Receiver to Azure

When you are ready to deploy to the Internet you can deploy to Azure just like a normal web application. You’ll want to run Add-PnPEventReceiver using the Internet URL to register your event receiver for real, of course.

 

64 thoughts on “Remote Event Receivers – you’re all doing it wrong

  1. Thank you for this article.

    Question tho.
    how can i have the ClientContext?

    my code inside “using (ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties))”

    is not firing because the properties.ContextToken is null.

    How can i do this?

    Like

      • Hi, thank you for the reference.

        I’ve used it as my event-receiver. What were aiming is to use the app-only access token. Sadly it throws an error saying that “Token request failed.”

        Like

      • Hi Derek. Thanks for the article, it was very helpful. I’m wondering if you can shed some light on an issue I’m having. I have a remote event receiver deployed to a Windows Server 2012 box with IIS 8. I have an SP 2013 on-prem install, and I’ve attached an ItemAdded receiver to a library there. I’ve confirmed that the receiver is firing. I’m using your Helpers.GetAuthenticatedContext method to try to grab the context. It seems to blow up for me when it tries to get the context (ClientContext ctx = new ClientContext(siteUrl);). I tried to catch the exception but nothing is showing. Not sure if you have any on-prem experience, but would you know what might be going on here?

        Like

      • Yep, it should work exactly the same in SP 2013. Make sure your site url and credentials are correct. If it’s blowing up at that specific line then I’d guess the URL is improperly formatted. Make sure it’s just the site url with no pages or lists in the url.

        Another thing to consider with on-prem would be firewall issues. Can the event receiver reach out to SharePoint? Do they both exist on the same machine? If not, can you ping the SharePoint server from the event receiver server? Also if the on-prem server has non standard authentication setups like ADFS then it might not work.

        Like

  2. Thanks! The receiver server and the SP server are on different boxes, but I was able to ping SP from the receiver server. I did format the url differently and was able to grab the context. I can get list/list item properties, but now I’m blowing up on the ExecuteQuery method. I’m not seeing a specific error in my logs, but I’m guessing this is a permissions issue, would you agree?

    Like

    • There’s probably an error in your code. Try to get your code working in a console app before porting over to the event receiver, and start with a simple concept, then flesh it out until you have it all working.

      Like

  3. It does work fine in a console app. It’s a very simple action. I have it narrowed down to the ExecuteQuery method. That works fine in the console app, but in the receiver (running on the remote server), it fails.

    Like

  4. Hi,

    Thanks for your article. I also think this would be a Nice way to create an RER. I have followed your exsamle and every thing work great until I publish the WCF to Azure. I have change the url in Sharepoint so the event Will hit the Azure address. No matter what I do, the RER is not triggered. Is there Any additional settings there needs to be done in the Web.config like endpoint bindings?

    Like

  5. We are currently working on a demo project to deploy a python flask application as an SER endpoint. We registered the endpoint with the PnP command as shown in your article but we aren’t receiving any packets. Maybe flask doesn’t recognize the WCF protocol. Our next step is to print out the incoming stream of bytes if there is a problem with registering the SER. Do you have any thoughts on that? Especially swapping out ASP.NET with python.

    Like

    • What you are trying should work just fine. There are no special protocols involved. I’ve used a Microsoft Flow as an event receiver and also a .NET MVC endpoint, and there’s a guy online who did it with Azure Functions. A few troubleshooting steps: Make sure the RER is actually registered and attached to the event you’re expecting (Get-PnpEventReciever or something like that). Next, make sure your Python app is exposed to the internet because the requests are actually coming from SharePoint Online. So if your application is inside a firewall it won’t work. If all that’s wired up you should at least be able to see the traffic coming in. Good luck!

      Like

  6. Hi derek, thanks for your quick reply. We discovered, that it is only possible to wire an event receiver to an endpoint listening on port 80. The problem was that we previously tried to wire the RER to our endpoint listening on port 5000, ofc publicly available. Tested with SharePoint Online.

    Sadly SharePoint Online communicates via SOAP. I don’t come from a microsoft background but I read that WCF 4.0 (https://en.wikipedia.org/wiki/Windows_Communication_Foundation#Interoperability) supports the JSON format compared to this non-modern XML syntax. Do you know if it is possible to force SharePoint Online communication via JSON instead of XML?

    If not the next logical step for us is to parse the SOAP XML Envelope. Do you know any (parsing?) libraries (doesn’t matter if .NET based or not) which make life easier? Thanks for your help.

    Like

    • Yeah, the port 80 restriction makes sense but I never really though about it. There’s no way to make SharePoint change the way it posts its RER data but I’d love to be proven wrong on that. WCF/XML was in interesting architectural decision for sure. As far as parsing goes, I’d probably use XmlSerializer in .NET or xml2js in Node. I wouldn’t know what to do in Python. You might also want to look into WebHooks if you haven’t already.

      Like

      • Yeah, I already looked into webhooks. Looks more developer friendly… but we would like to leverage the ProcessTwoWayEvent, basically the sync functionality of the RER. So there is no other choice.

        Actually, my company is using SharePoint since ~2000 and I wrote/maintain the one or other farm solution for SP 2010/2013. We have multiple customers using different customer-specific farm solutions. Microsoft’s focus will clearly be at its cloud business in the future. It’s a matter of time customers have to update their 2010/2013 on-premise versions due to the EOL. Some of our customers even have SharePoint Online + the on-premise version with their farm-solutions.

        Over the last years, it felt tedious developing SP solutions. So I convinced my boss to create new projects from scratch with other, open source technologies. Basically, a 12factor distributed system with containerized services – a docker/linux/python/django/rest/nginx/postgres stack. Oh boy, what a dream compared to the tedious work years before… while working on such projects the wind changed: microsoft announced msql/pwsh for linux, .NET core, etc. And now that time has come and even my boss questions and fears the future of on-premise/farm solutions we are evaluating SharePoint Online programming models. In my opinion finally a step in the right direction.

        Ofc, I am a bit biased because of my work with declarative frameworks in the python universe and the established CI/CD workflow we now have in other projects. So my expectations are high. But what a disappointment. The SharePoint/PnP-PowerShell module is installable on linux, but throws an error when connecting, there are no official libraries for python/php/node, the CSOM libraries are not .NET core compatible ( there is a work in progress badge on the feature request site), the last time someone commented on the PnP issue list was 3 weeks ago, I am dealing with 404 links in the docs, the collapsible navigation menu on msdn gives me ulcer :(, the majority of blog post (yours seems to be an exception) uses screenshots like click here and click there without any explanation. I don’t want to complain too much. I just want to find a good compromise between the SharePoint world, which has a good OOTB functionality, and a developer’s vivid needs.

        The future clearly points to the cloud and it’s a matter of time some of our customers decide to abandon on-premise versions. Right now it seems that the only hassle free option is to use SharePoint’s REST API. For other functionality we have to use the RER, lets see how my demo projects envolve in the future. But when I think about that I have to RDP into a windows machine to interact with my sharepoint online resources and browse around not well maintained official repositories at github I already start to question again if this is really the way to go and how the future should look like. Thanks for your help and sorry for my grammar.

        Like

  7. Hi Derek, I’ve managed to publish my RER to azure and attached it to my SPO project, it works fine when im using the http protocol.

    but when we host it in our server that is using https it doesn’t work as expected. do you have any experience on this?

    Thanks!

    Like

    • Make sure the RER is registered using an https url. Assuming you’ve done that it sounds like a misconfiguration or certificate issue on your server. But assuming it’s reachable from the Internet it should work. What do you mean, it “doesn’t work”? Your’re getting an error? What does it say?

      Like

  8. It is reachable on the internet via https and i registered it in my list using the https url as well. “doesn’t work” by means of it seems that its not getting trough the ProcessOneWayEvent because i didn’t see the changes i expect.

    i’ve also checked our IIS logs and see “2018-12-11 05:33:20 POST /addin/autotag/pbpo_autotag.svc – 443 – – – 404 0 0 206”

    Liked by 1 person

    • I am facing the same problem like Justin Randelle Gallardo. It is reachable on the internet via https and i registered it in my list using the https url as well. “doesn’t work” by means of it seems that its not getting trough the ProcessOneWayEvent because i didn’t see the changes i expect.

      Like

      • Check that your RER is registered properly and that it’s Asynchronous – a Synchronous RER would fire ProcessEvent instead. Check your logging on your web server to ascertain whether the request is actually making it to the server. Finally check your code’s logging. If it’s reaching your code and throwing an exception you should be able to see that in the logs.

        If you’re hosting this yourself you might have certificate issues as well.

        So in summary it could be: incorrect registration, connectivity from SPO to your server, certificate issues, or error in your code. You should be able to figure out which it is and go from there.

        Like

  9. Hi Derek, Thank you for this wonderful post and thank you for your prompt reply. Situation is our wcf service is deployed as Azure App Service (Secured HTTPS way) and the RER is registered as Asynchronous and registered service url is secured url “https”. I also tried placing a log inside ProcessOneWayEvent and checked if its logging anything in Azure. But Not found anything.

    When i disabled HTTPS on Azure App Service and registered RER again with non-secure “http” url, RER worked like a charm and also I can find the log as well.

    Certainly this is something to do with HTTPS. Can you please guide further? Do we need to do some extra things in Azure (like add certificate etc) ?

    wcf service(https) opens in browser without any problem over internet.

    Like

    • Hi Shriram, Azure Web apps using https have always “just worked” for me out of the box. I’ve never had to do any sort of configuration for them to work. If this is a dev site, maybe try deleting and recreating the web app and see if the issue persists. I don’t know what else I can suggest.

      Like

  10. Hi Derek, I was able to fix the issue with wcf service using Azure Web apps using https. I had to add this endpoint to the wcf service web.config and then published again to Azure. Now RER service(https) working great with SP List.

    Like

  11. Hey Derek,

    Really nice article. Instead of RER i feel to use workflow / Flow or nintext workflow to achieve similar stuff. this makes less maintenance, easy to migrate and maintain and more important time saving. Let me know what you think. Thanks.

    Like

    • If Flow can meet your requirement, I’d say go with that. It’ll be simpler to support and debug. But there are limitations to Flow. You can’t easily version Flows.If you break it you can’t roll back a change. You can’t easily use Flow in a dev/test/production scenario. Flow gets unmanageable when there are a lot of logic branches or loops. Also Flows don’t scale easily. If you need to capture events on a thousand different lists, and RER can do that but Flows have issues with that. Good question, and maybe a good blog topic. I don’t know much about Nintex but suspect those might have similar considerations as Flow. Good luck!

      Like

  12. Hi Derek,

    Thanks for that excellent article. I am trying to get RER working with Azure Functions and have referred to the post by Sergei Sergeev. MY RER is attached to a library in the host web. I can see it has been added but it is not firing when a document is added to the list. It seems to me as if what is missing is the registration step – navigation to AppInv.aspx and doing the registration as Sergei-Sergeev has done. However I am trying to attach this RER as part of an automated process (the automated process provisions and customises the site where the library exists). Is there a programmatic equivalent of AppInv.aspx?

    Cheers,

    Raj

    Like

  13. Hi Derek,

    If I understand correctly, since ContextToken is null in the RER, we can’t validate if a request really coming from our SharePoint Online site. Also the web service needs to be anonymous.

    With the SharePointOnlineCredentials or an app-only technique, I guess anyone on the internet would be able to call our web service (with the right Header) and the code would execute without any restriction.

    How could we secure the whole thing? I’m thinking about limiting access to our web service to SharePoint IPs/domains?

    Thanks,
    Chris

    Like

    • Hi Chris, you’re correct in that the RER deployed in this fashion could theoretically be spoofed, but I would put the actual probability of this happening at zero, unless the perpetrator had detailed inside information – in which case they could probably effect the same change with much less effort using other methods.

      If you’re really concerned with this risk, I think it’d be pretty straightforward to call back into SharePoint to query whether the trigger had actually taken place – for example, on an ItemUpdated, call the item in question and verify it was actually updated as such.

      You could probably filter IP addresses in code – maybe in a global.asax – although I’ve never tried this. You’d have to validate a wide array of IPs though I’d suspect since the calls would be coming from SharePoint Online so any valid Azure IP address might be a valid one.

      Like

  14. Hi All,
    I am getting below error while calling RER service from on premises to Sharepoint online
    could you please give your suggestion to fix this issue.

    The remote event receiver callout failed.
    Details: The request channel timed out while waiting for a reply
    after 00:00:29.9062494. Increase the timeout value passed to the
    call to Request or increase the SendTimeout value on the Binding.
    The time allotted to this operation may have been a portion of a longer timeout.

    Regards
    Pradeep pal

    Like

    • Looks like an issue with your code, or a connectivity issue. Are you trying to access things behind a firewall? It’s not really clear to me what “from on premises to Sharepoint online” means.

      Like

      • Hey Derek, thanks for quick reply
        In rer we have an service part and an app par, service part I have hosted in local iis, and app part(.app )file deployed in SharePoint online. I have total 2720 library where is need to apply list added , and item added and item deleted events I wrote code and fetch all list and using for loop, I am attaching events one by one. it is working fine about for 630 list and attaching events but after 630 list it’s give error of time out during app installation. Error is mentioned in my previous comment.
        I have generating text log in each step so there is no problem in connectivity.
        Thanks and hoping for solution
        Regards
        Pradeep

        Like

  15. I love when a random googling brings me to an old colleague’s expert blog post! Also, another cogent reminder why I don’t do event receivers… or workflows… why did I choose SharePoint development in the first place??!? Time for another existential crisis.. nice seeing you, Derek!

    Like

  16. Hi Derek, thanks for this great post, I had a question:
    If the WCf service is hosted internally (intranet), will it be called when registered to a Sharepoint online list?

    Like

  17. Hi Drek,
    i tried your code and i download the sample code from @ https://github.com/dgusoff/RemoteEventReceiver.Starter. but i have these questions:-

    1. Inside one of the comments you mentioned the following “//if using App Only Context, use this method, and make sure ClientId and ClientSecret are specified in AppSettings”, but it is not clear where i should add those variables? as inside your sample web.config file i can not find any values for clientid and ClientSecret? and should we generate those from the site collection first using @ _layouts/15/appregnew.aspx?

    2. I tried to call the GetAppOnlyContext(string siteUrl) but it raised the following exception “Token request failed.”

    Can you advice on these 2 questions? and thanks for your code and valuable article.

    Like

    • First, you need to generate a client ID and client secret. Client ID can be an Azure AD application registration, although you can also use appregnew. To create a client secret, go to _layouts/AppInv.aspx. If you want a tenant-scoped app, use the _layouts/AppInv.aspx on the _admin site collection. And yeeah, if the sample code does not have those app settings in its config, you’ll need to add them. Good luck! Creating an app-only client id and secret could probably be a blog post on its own.

      Like

  18. ok thanks for your reply. but is there a sample code of how i need to modify the `public static ClientContext GetAppOnlyContext(string siteUrl)` method to read the clientId and clientsecret from the web.config?
    Thanks

    Like

  19. Hi Drek,

    Now inside the sample project which you provided it define the following method for the GetAppOnlyContext:-

    public static ClientContext GetAppOnlyContext(string siteUrl)
    {
    try
    {
    Uri siteUri = new Uri(siteUrl);
    string realm = TokenHelper.GetRealmFromTargetUrl(siteUri);
    string accessToken = TokenHelper.GetAppOnlyAccessToken(TokenHelper.SharePointPrincipal, siteUri.Authority, realm).AccessToken;

    return TokenHelper.GetClientContextWithAccessToken(siteUri.ToString(), accessToken);
    }

    catch (Exception ex)
    {
    Trace.TraceInformation(“GetAppOnlyContext failed. {0}”, ex.Message);
    }
    return null;
    }

    so i can not find where inside this method it will read the Clientid and ClientSecret from the web.config? or i am missing something?
    Thanks

    Like

  20. Hi Derek,

    Thanks for your valuable replies i can locate the related code now. So I added this code, which get fired when adding an announcement item inside a development site:-

    public void ProcessOneWayEvent(SPRemoteEventProperties properties)

    {

    var prop = properties;

    var itemId = properties.ItemEventProperties.ListItemId;
    var listTitle = properties.ItemEventProperties.ListTitle;
    using (ClientContext clientContext = Helpers.GetAppOnlyContext(properties.ItemEventProperties.WebUrl))
    {
    //clientContext.Credentials = new NetworkCredential(this.UserName, this.Password, this.Domain);
    var list = clientContext.Site.RootWeb.Lists.GetByTitle(“testlist”);
    var item = list.GetItemById(properties.ItemEventProperties.ListItemId);
    item[“Title”] = “updated by me”;
    item.Update();
    clientContext.Load(item);
    clientContext.ExecuteQuery();
    // throw new NotImplementedException();

    }
    }

    now on the `clientContext.ExecuteQuery();` i got this exception:-

    Exception thrown: ‘Microsoft.SharePoint.Client.ServerUnauthorizedAccessException’ in Microsoft.SharePoint.Client.Runtime.dll
    An exception of type ‘Microsoft.SharePoint.Client.ServerUnauthorizedAccessException’ occurred in Microsoft.SharePoint.Client.Runtime.dll but was not handled in user code
    Access denied. You do not have permission to perform this action or access this resource.

    Do you have any idea why i got this error?
    Thanks

    Like

  21. Pingback: #StackBounty: #sharepoint-online #development #azure #remote-event-receiver Remote event receiver stop working after deploying them ins… – TechUtils.in

  22. Pingback: #StackBounty: #sharepoint-online #development #azure #remote-event-receiver Remote event reciever is not firing after deploying it to a… – TechUtils.in

  23. Hi Derek
    Is there a PnP powershell command to deploy an ItemAdding event receiver for all document libraries?

    You can do this in VS2013 with an elements xml file that specifies a list type (101 I think for all document libraries) so you do not specify a particular list i.e. the event receiver triggers for all document librarians.

    Like

Leave a comment