Errors and residuals in linear regression











up vote
0
down vote

favorite












I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.



Setting:
Given simple linear regression,
$$ y_i = beta_0 + beta_1 x_i + epsilon_i$$
with the error terms being normally distributed, i.e., $epsilon_i$ ~ $N(0, sigma^2)$. In this setting we consider the $y_i$ and the errors $epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $beta_0$ and $beta_1$ are unknown and fixed.
Therefore we have $E[y_i] = beta_0 + beta_1 x_i$ and $Var[y_i] = sigma^2$, which means the $y_i$ are also normally distributed according to $y_i$~$N(beta_0 + beta_1 x_i, sigma^2)$.



The errors $epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[y_i] = beta_0 + beta_1 x_i$, i.e.
$$ y_i - E[y_i] = (beta_0 + beta_1 x_i + epsilon_i)-(beta_0 + beta_1 x_i) = epsilon_i.$$



We now want to estimate the model $E[y_i] = beta_0 + beta_1 x_i$, as the coefficients $beta_0$ and $beta_1$ are unkown. We do so by
$$ hat{y} = hat{beta_0} + hat{beta_1} x_i.$$



The residuals, lets call them $delta_i$, are definded as the deviations of the observations $y_i$ from the estimated model $hat{y_i}$, i.e.
$$delta_i = y_i - hat{y_i}.$$



Now the questions:



1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $sum epsilon_i^2$, and some reduce the residual sum of square (RSS), $sum delta_i^2$, which is obviously not the same thing. Some even write they reduce $sum left(y_i - hat{y_i}right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?



2.) How comes that there are so many contradicting definitions of residuals, errors and least squares in the text books, and is there any one book that is precise and consistent?



3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance?



4.) Some authors also write the estimated fromula as $ hat{y} = hat{beta_0} + hat{beta_1} x_i + hat{epsilon_i}$, where they depict the residuals $delta_i$ as $hat{epsilon_i}$. But I think this is a wrong formula because then $y_i - hat{y_i}$, which is the defintion of the residuals, leads to result that is different from the residuals. Is there any justification why some would write the residuals into the estimated formula?










share|cite|improve this question









New contributor




guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
























    up vote
    0
    down vote

    favorite












    I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.



    Setting:
    Given simple linear regression,
    $$ y_i = beta_0 + beta_1 x_i + epsilon_i$$
    with the error terms being normally distributed, i.e., $epsilon_i$ ~ $N(0, sigma^2)$. In this setting we consider the $y_i$ and the errors $epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $beta_0$ and $beta_1$ are unknown and fixed.
    Therefore we have $E[y_i] = beta_0 + beta_1 x_i$ and $Var[y_i] = sigma^2$, which means the $y_i$ are also normally distributed according to $y_i$~$N(beta_0 + beta_1 x_i, sigma^2)$.



    The errors $epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[y_i] = beta_0 + beta_1 x_i$, i.e.
    $$ y_i - E[y_i] = (beta_0 + beta_1 x_i + epsilon_i)-(beta_0 + beta_1 x_i) = epsilon_i.$$



    We now want to estimate the model $E[y_i] = beta_0 + beta_1 x_i$, as the coefficients $beta_0$ and $beta_1$ are unkown. We do so by
    $$ hat{y} = hat{beta_0} + hat{beta_1} x_i.$$



    The residuals, lets call them $delta_i$, are definded as the deviations of the observations $y_i$ from the estimated model $hat{y_i}$, i.e.
    $$delta_i = y_i - hat{y_i}.$$



    Now the questions:



    1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $sum epsilon_i^2$, and some reduce the residual sum of square (RSS), $sum delta_i^2$, which is obviously not the same thing. Some even write they reduce $sum left(y_i - hat{y_i}right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?



    2.) How comes that there are so many contradicting definitions of residuals, errors and least squares in the text books, and is there any one book that is precise and consistent?



    3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance?



    4.) Some authors also write the estimated fromula as $ hat{y} = hat{beta_0} + hat{beta_1} x_i + hat{epsilon_i}$, where they depict the residuals $delta_i$ as $hat{epsilon_i}$. But I think this is a wrong formula because then $y_i - hat{y_i}$, which is the defintion of the residuals, leads to result that is different from the residuals. Is there any justification why some would write the residuals into the estimated formula?










    share|cite|improve this question









    New contributor




    guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






















      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.



      Setting:
      Given simple linear regression,
      $$ y_i = beta_0 + beta_1 x_i + epsilon_i$$
      with the error terms being normally distributed, i.e., $epsilon_i$ ~ $N(0, sigma^2)$. In this setting we consider the $y_i$ and the errors $epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $beta_0$ and $beta_1$ are unknown and fixed.
      Therefore we have $E[y_i] = beta_0 + beta_1 x_i$ and $Var[y_i] = sigma^2$, which means the $y_i$ are also normally distributed according to $y_i$~$N(beta_0 + beta_1 x_i, sigma^2)$.



      The errors $epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[y_i] = beta_0 + beta_1 x_i$, i.e.
      $$ y_i - E[y_i] = (beta_0 + beta_1 x_i + epsilon_i)-(beta_0 + beta_1 x_i) = epsilon_i.$$



      We now want to estimate the model $E[y_i] = beta_0 + beta_1 x_i$, as the coefficients $beta_0$ and $beta_1$ are unkown. We do so by
      $$ hat{y} = hat{beta_0} + hat{beta_1} x_i.$$



      The residuals, lets call them $delta_i$, are definded as the deviations of the observations $y_i$ from the estimated model $hat{y_i}$, i.e.
      $$delta_i = y_i - hat{y_i}.$$



      Now the questions:



      1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $sum epsilon_i^2$, and some reduce the residual sum of square (RSS), $sum delta_i^2$, which is obviously not the same thing. Some even write they reduce $sum left(y_i - hat{y_i}right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?



      2.) How comes that there are so many contradicting definitions of residuals, errors and least squares in the text books, and is there any one book that is precise and consistent?



      3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance?



      4.) Some authors also write the estimated fromula as $ hat{y} = hat{beta_0} + hat{beta_1} x_i + hat{epsilon_i}$, where they depict the residuals $delta_i$ as $hat{epsilon_i}$. But I think this is a wrong formula because then $y_i - hat{y_i}$, which is the defintion of the residuals, leads to result that is different from the residuals. Is there any justification why some would write the residuals into the estimated formula?










      share|cite|improve this question









      New contributor




      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      I think in common literature about statstics the authors are often very imprecise when it comes to residuals and errors. So far, I could not work that difference out completely and therefore have several questions.



      Setting:
      Given simple linear regression,
      $$ y_i = beta_0 + beta_1 x_i + epsilon_i$$
      with the error terms being normally distributed, i.e., $epsilon_i$ ~ $N(0, sigma^2)$. In this setting we consider the $y_i$ and the errors $epsilon_i$ to be random variables, and the independent variables $x_i$ to be nonrandom variables. The parameters $beta_0$ and $beta_1$ are unknown and fixed.
      Therefore we have $E[y_i] = beta_0 + beta_1 x_i$ and $Var[y_i] = sigma^2$, which means the $y_i$ are also normally distributed according to $y_i$~$N(beta_0 + beta_1 x_i, sigma^2)$.



      The errors $epsilon_i$ are now defined as the deviations of the observations $y_i$ from the "true" (deterministic model) $E[y_i] = beta_0 + beta_1 x_i$, i.e.
      $$ y_i - E[y_i] = (beta_0 + beta_1 x_i + epsilon_i)-(beta_0 + beta_1 x_i) = epsilon_i.$$



      We now want to estimate the model $E[y_i] = beta_0 + beta_1 x_i$, as the coefficients $beta_0$ and $beta_1$ are unkown. We do so by
      $$ hat{y} = hat{beta_0} + hat{beta_1} x_i.$$



      The residuals, lets call them $delta_i$, are definded as the deviations of the observations $y_i$ from the estimated model $hat{y_i}$, i.e.
      $$delta_i = y_i - hat{y_i}.$$



      Now the questions:



      1.) In least squares estimation some authors reduce the squared sum of errors (SSE), $sum epsilon_i^2$, and some reduce the residual sum of square (RSS), $sum delta_i^2$, which is obviously not the same thing. Some even write they reduce $sum left(y_i - hat{y_i}right)^2$ but then they reduce the SSE, which is not even consistent within their own chosen framework. What is now the correct procedure of least square estimation, reducing SSE or RSS?



      2.) How comes that there are so many contradicting definitions of residuals, errors and least squares in the text books, and is there any one book that is precise and consistent?



      3.) Can we say anything about the distribution of the residuals, i.e. are they also normally distributed and if so with which mean and variance?



      4.) Some authors also write the estimated fromula as $ hat{y} = hat{beta_0} + hat{beta_1} x_i + hat{epsilon_i}$, where they depict the residuals $delta_i$ as $hat{epsilon_i}$. But I think this is a wrong formula because then $y_i - hat{y_i}$, which is the defintion of the residuals, leads to result that is different from the residuals. Is there any justification why some would write the residuals into the estimated formula?







      statistics least-squares linear-regression






      share|cite|improve this question









      New contributor




      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|cite|improve this question









      New contributor




      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|cite|improve this question




      share|cite|improve this question








      edited 17 hours ago





















      New contributor




      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked yesterday









      guest1

      123




      123




      New contributor




      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      guest1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.



























          active

          oldest

          votes











          Your Answer





          StackExchange.ifUsing("editor", function () {
          return StackExchange.using("mathjaxEditing", function () {
          StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
          StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
          });
          });
          }, "mathjax-editing");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "69"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          noCode: true, onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });






          guest1 is a new contributor. Be nice, and check out our Code of Conduct.










           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2996575%2ferrors-and-residuals-in-linear-regression%23new-answer', 'question_page');
          }
          );

          Post as a guest





































          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          guest1 is a new contributor. Be nice, and check out our Code of Conduct.










           

          draft saved


          draft discarded


















          guest1 is a new contributor. Be nice, and check out our Code of Conduct.













          guest1 is a new contributor. Be nice, and check out our Code of Conduct.












          guest1 is a new contributor. Be nice, and check out our Code of Conduct.















           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2996575%2ferrors-and-residuals-in-linear-regression%23new-answer', 'question_page');
          }
          );

          Post as a guest




















































































          Popular posts from this blog

          Plaza Victoria

          Puebla de Zaragoza

          Musa