How to be able to process a large JSON response?












1















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question

























  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    Mar 25 at 18:05











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    Mar 25 at 18:13











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    Mar 25 at 18:14
















1















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question

























  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    Mar 25 at 18:05











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    Mar 25 at 18:13











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    Mar 25 at 18:14














1












1








1








We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?










share|improve this question
















We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?







json rest governorlimits






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 25 at 18:13







Sander de Jong

















asked Mar 25 at 17:48









Sander de JongSander de Jong

1,39921538




1,39921538













  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    Mar 25 at 18:05











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    Mar 25 at 18:13











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    Mar 25 at 18:14



















  • Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

    – Jayant Das
    Mar 25 at 18:05











  • @JayantDas I have added clarification to my question.

    – Sander de Jong
    Mar 25 at 18:13











  • Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

    – Jayant Das
    Mar 25 at 18:14

















Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

– Jayant Das
Mar 25 at 18:05





Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?

– Jayant Das
Mar 25 at 18:05













@JayantDas I have added clarification to my question.

– Sander de Jong
Mar 25 at 18:13





@JayantDas I have added clarification to my question.

– Sander de Jong
Mar 25 at 18:13













Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

– Jayant Das
Mar 25 at 18:14





Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.

– Jayant Das
Mar 25 at 18:14










2 Answers
2






active

oldest

votes


















4














There's just no way to cram that much data into Salesforce as-is.



I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






share|improve this answer
























  • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    Mar 25 at 18:11













  • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    Mar 25 at 18:14






  • 1





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    Mar 25 at 21:06



















0














In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






share|improve this answer
























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "459"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f255209%2fhow-to-be-able-to-process-a-large-json-response%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    4














    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer
























    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      Mar 25 at 18:11













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      Mar 25 at 18:14






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      Mar 25 at 21:06
















    4














    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer
























    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      Mar 25 at 18:11













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      Mar 25 at 18:14






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      Mar 25 at 21:06














    4












    4








    4







    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs






    share|improve this answer













    There's just no way to cram that much data into Salesforce as-is.



    I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).



    As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.



    Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs







    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered Mar 25 at 17:58









    Derek FDerek F

    20.7k52353




    20.7k52353













    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      Mar 25 at 18:11













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      Mar 25 at 18:14






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      Mar 25 at 21:06



















    • I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

      – Sander de Jong
      Mar 25 at 18:11













    • I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

      – Sander de Jong
      Mar 25 at 18:14






    • 1





      @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

      – Derek F
      Mar 25 at 21:06

















    I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    Mar 25 at 18:11







    I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.

    – Sander de Jong
    Mar 25 at 18:11















    I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    Mar 25 at 18:14





    I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.

    – Sander de Jong
    Mar 25 at 18:14




    1




    1





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    Mar 25 at 21:06





    @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.

    – Derek F
    Mar 25 at 21:06













    0














    In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






    share|improve this answer




























      0














      In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






      share|improve this answer


























        0












        0








        0







        In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.






        share|improve this answer













        In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Mar 25 at 17:56









        sfdcfoxsfdcfox

        262k12209454




        262k12209454






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Salesforce Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f255209%2fhow-to-be-able-to-process-a-large-json-response%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Plaza Victoria

            Puebla de Zaragoza

            Musa