Import Contact Issue












2














I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.



Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.



I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?



I verified the processor, free memory, file size, etc are all way under utilized.



Rememdy is kill all user/application processes, repair database and continue forward.



But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.



Thx










share|improve this question



























    2














    I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.



    Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.



    I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?



    I verified the processor, free memory, file size, etc are all way under utilized.



    Rememdy is kill all user/application processes, repair database and continue forward.



    But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.



    Thx










    share|improve this question

























      2












      2








      2







      I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.



      Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.



      I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?



      I verified the processor, free memory, file size, etc are all way under utilized.



      Rememdy is kill all user/application processes, repair database and continue forward.



      But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.



      Thx










      share|improve this question













      I am importing CSV files, about 7000 contacts per import, about 20 fields per contact, about 800kb sile size. DB size is about 55k, issue arose at about 20k. Since then, it times out after 2000ish records. The importation ultimately executes with fields and data appearing to correctly function thus far through testing.



      Issue is the importation process freezes. It seems to run about 4 to 5 minutes, then apparently times out. Server demonstrates mysql processes, usually about 6-10, in sleep state when "time out" occurs. Checking table in phpmyadmin, I can quickly see where to edit csv file to continue.



      I am not deduping, though I am validating address data. I suspect the address validation is causing the issue. Does the USPS address validator have a "counter" on it that is now limiting my "per day/then/per session" usage?



      I verified the processor, free memory, file size, etc are all way under utilized.



      Rememdy is kill all user/application processes, repair database and continue forward.



      But, if I can adjust a setting to enable better processing, without sacrificing security, pls share.



      Thx







      import






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 2 hours ago









      drummondislebsd

      311




      311






















          1 Answer
          1






          active

          oldest

          votes


















          1














          You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.



          Past that:



          One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.



          Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).



          I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.



          If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.






          share|improve this answer





















            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "605"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            noCode: true, onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcivicrm.stackexchange.com%2fquestions%2f27859%2fimport-contact-issue%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.



            Past that:



            One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.



            Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).



            I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.



            If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.






            share|improve this answer


























              1














              You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.



              Past that:



              One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.



              Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).



              I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.



              If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.






              share|improve this answer
























                1












                1








                1






                You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.



                Past that:



                One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.



                Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).



                I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.



                If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.






                share|improve this answer












                You have several options, which I detail below - but I think the most important thing to do is check your CiviCRM log. It's likely to tell you what your problem is.



                Past that:



                One option is to adjust your php.ini settings to change the max_execution_time to zero. You can also do this in civicrm.settings.php if you prefer.



                Another is to import your data on the command line using the API CSV import tool - but this requires breaking up your data into separate entities (e.g. Contact, Address, Email, etc.). This is very efficient on large imports but not worth it unless you can script it (e.g. with Pentaho Data Integration).



                I agree that it's very likely that either geocoding or USPS address validation is likely a culprit - but the easiest way to test that is to disable, do a test import and check the result.



                If you're a programmer and don't like using Pentaho, you could also try writing an importer that uses the advimport extension.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered 1 hour ago









                Jon G - Megaphone Tech

                25.6k11767




                25.6k11767






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to CiviCRM Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcivicrm.stackexchange.com%2fquestions%2f27859%2fimport-contact-issue%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Plaza Victoria

                    Puebla de Zaragoza

                    Musa