what is the behavior of `>` `>` in shell
I know the basis of these three. And the reason why cat < file > file clears the content of file. But I don't understand why cat < file >> file would instead of writing the content twice, it concatenates the content like in an endless while loop.
shell
add a comment |
I know the basis of these three. And the reason why cat < file > file clears the content of file. But I don't understand why cat < file >> file would instead of writing the content twice, it concatenates the content like in an endless while loop.
shell
add a comment |
I know the basis of these three. And the reason why cat < file > file clears the content of file. But I don't understand why cat < file >> file would instead of writing the content twice, it concatenates the content like in an endless while loop.
shell
I know the basis of these three. And the reason why cat < file > file clears the content of file. But I don't understand why cat < file >> file would instead of writing the content twice, it concatenates the content like in an endless while loop.
shell
shell
asked Jan 13 at 1:56
TiinaTiina
6271614
6271614
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Well, my cat (in Kubuntu) says input file is output file and exits. I understand your cat is not as smart (or as lazy).
What you observe is quite easy to explain. cat reads and writes in chunks. Suppose the first chunk doesn't reach the end of file (EOF) and its content gets concatenated at the end of the same file. Reading cat advanced by some amount of bytes but the end of the file "drifted away" by the same amount. In effect cat is again as far from EOF as it was at the beginning. And it goes on and on and on.
As if you ran and for every 100 meters the finish line was moved additional 100 meters away from you. You would never reach it and the distance traveled would grow with no end.
Unless the original distance is 100 meters or less. In this case you would get to the finish line before it's relocated. Analogously cat might finish if the file is small enough, but I really doubt it. I expect cat "knows" it got to EOF only when it tries to read again and is able to read 0 bytes, no more. This means it already read until EOF in the previous step, when it read more than 0 bytes. But in your case, if it read more than 0 bytes in the previous step, then the file grew immediately after and there is still data to read.
I predict two cases so your cat will stop:
- when the file is empty (0 bytes long),
- when the file doesn't exist and you change the order of redirections (i.e.
cat >> file < filewill work,cat < file >> filewill throw an error; this is because redirections are "served" from left to right,>>creates the file if needed,<requires the file to exist).
In general using the same file with stdin and stdout (or/and stderr) redirection in a single command is asking for trouble. Still, if you need to do this, there is sponge utility (e.g. in moreutils package for Ubuntu, Debian).
spongereads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
If no output file is specified,
spongeoutputs to stdout.
Instead of cat < file >> file run sponge < file >> file. It will do what you expected from cat (unless the file is huge enough to deplete your resources while being soaked up, I think).
Another way is < file cat | cat >> file. Due to buffering between two cat-s the first one will really hit EOF and exit before the second one expands the file, if the file is small enough. Repeat this command few times starting with a small (but not empty) file and the file will eventually grow large enough to make the command behave like your original command with one cat. Note < file cat | unbuffer cat >> file gets rid of the buffer and loops even for very small files.
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,catwould handle it properly since it loads every trunk until EOF.
– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,catwill wait. Very simple to observe: run justcat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.
– Kamil Maciorowski
Jan 13 at 3:43
add a comment |
Well, basically this works as follows:
>is used to override the content of a file. E.g:echo "HELLO WORLD" > file.txt.
>>is used to append content to a file. E.g:echo "New line" >> file.txt.
<you can use a file as input and redirect to another file. E.g:cat < file.txt > file2.txt. A practical example isftp myftpbox.com < commands.txt, this would connect to a FTP and execute a list of commands place in the filecommands.txt.
NOTE: Be careful when you use > because you can override an existing file without any confirmation, you can alter this by running set -o noclobber.
I hope this helps.
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,set -o noclobberis beyond the basis, granted). What is more important, you didn't address the real issue at all.
– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393655%2fwhat-is-the-behavior-of-in-shell%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Well, my cat (in Kubuntu) says input file is output file and exits. I understand your cat is not as smart (or as lazy).
What you observe is quite easy to explain. cat reads and writes in chunks. Suppose the first chunk doesn't reach the end of file (EOF) and its content gets concatenated at the end of the same file. Reading cat advanced by some amount of bytes but the end of the file "drifted away" by the same amount. In effect cat is again as far from EOF as it was at the beginning. And it goes on and on and on.
As if you ran and for every 100 meters the finish line was moved additional 100 meters away from you. You would never reach it and the distance traveled would grow with no end.
Unless the original distance is 100 meters or less. In this case you would get to the finish line before it's relocated. Analogously cat might finish if the file is small enough, but I really doubt it. I expect cat "knows" it got to EOF only when it tries to read again and is able to read 0 bytes, no more. This means it already read until EOF in the previous step, when it read more than 0 bytes. But in your case, if it read more than 0 bytes in the previous step, then the file grew immediately after and there is still data to read.
I predict two cases so your cat will stop:
- when the file is empty (0 bytes long),
- when the file doesn't exist and you change the order of redirections (i.e.
cat >> file < filewill work,cat < file >> filewill throw an error; this is because redirections are "served" from left to right,>>creates the file if needed,<requires the file to exist).
In general using the same file with stdin and stdout (or/and stderr) redirection in a single command is asking for trouble. Still, if you need to do this, there is sponge utility (e.g. in moreutils package for Ubuntu, Debian).
spongereads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
If no output file is specified,
spongeoutputs to stdout.
Instead of cat < file >> file run sponge < file >> file. It will do what you expected from cat (unless the file is huge enough to deplete your resources while being soaked up, I think).
Another way is < file cat | cat >> file. Due to buffering between two cat-s the first one will really hit EOF and exit before the second one expands the file, if the file is small enough. Repeat this command few times starting with a small (but not empty) file and the file will eventually grow large enough to make the command behave like your original command with one cat. Note < file cat | unbuffer cat >> file gets rid of the buffer and loops even for very small files.
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,catwould handle it properly since it loads every trunk until EOF.
– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,catwill wait. Very simple to observe: run justcat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.
– Kamil Maciorowski
Jan 13 at 3:43
add a comment |
Well, my cat (in Kubuntu) says input file is output file and exits. I understand your cat is not as smart (or as lazy).
What you observe is quite easy to explain. cat reads and writes in chunks. Suppose the first chunk doesn't reach the end of file (EOF) and its content gets concatenated at the end of the same file. Reading cat advanced by some amount of bytes but the end of the file "drifted away" by the same amount. In effect cat is again as far from EOF as it was at the beginning. And it goes on and on and on.
As if you ran and for every 100 meters the finish line was moved additional 100 meters away from you. You would never reach it and the distance traveled would grow with no end.
Unless the original distance is 100 meters or less. In this case you would get to the finish line before it's relocated. Analogously cat might finish if the file is small enough, but I really doubt it. I expect cat "knows" it got to EOF only when it tries to read again and is able to read 0 bytes, no more. This means it already read until EOF in the previous step, when it read more than 0 bytes. But in your case, if it read more than 0 bytes in the previous step, then the file grew immediately after and there is still data to read.
I predict two cases so your cat will stop:
- when the file is empty (0 bytes long),
- when the file doesn't exist and you change the order of redirections (i.e.
cat >> file < filewill work,cat < file >> filewill throw an error; this is because redirections are "served" from left to right,>>creates the file if needed,<requires the file to exist).
In general using the same file with stdin and stdout (or/and stderr) redirection in a single command is asking for trouble. Still, if you need to do this, there is sponge utility (e.g. in moreutils package for Ubuntu, Debian).
spongereads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
If no output file is specified,
spongeoutputs to stdout.
Instead of cat < file >> file run sponge < file >> file. It will do what you expected from cat (unless the file is huge enough to deplete your resources while being soaked up, I think).
Another way is < file cat | cat >> file. Due to buffering between two cat-s the first one will really hit EOF and exit before the second one expands the file, if the file is small enough. Repeat this command few times starting with a small (but not empty) file and the file will eventually grow large enough to make the command behave like your original command with one cat. Note < file cat | unbuffer cat >> file gets rid of the buffer and loops even for very small files.
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,catwould handle it properly since it loads every trunk until EOF.
– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,catwill wait. Very simple to observe: run justcat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.
– Kamil Maciorowski
Jan 13 at 3:43
add a comment |
Well, my cat (in Kubuntu) says input file is output file and exits. I understand your cat is not as smart (or as lazy).
What you observe is quite easy to explain. cat reads and writes in chunks. Suppose the first chunk doesn't reach the end of file (EOF) and its content gets concatenated at the end of the same file. Reading cat advanced by some amount of bytes but the end of the file "drifted away" by the same amount. In effect cat is again as far from EOF as it was at the beginning. And it goes on and on and on.
As if you ran and for every 100 meters the finish line was moved additional 100 meters away from you. You would never reach it and the distance traveled would grow with no end.
Unless the original distance is 100 meters or less. In this case you would get to the finish line before it's relocated. Analogously cat might finish if the file is small enough, but I really doubt it. I expect cat "knows" it got to EOF only when it tries to read again and is able to read 0 bytes, no more. This means it already read until EOF in the previous step, when it read more than 0 bytes. But in your case, if it read more than 0 bytes in the previous step, then the file grew immediately after and there is still data to read.
I predict two cases so your cat will stop:
- when the file is empty (0 bytes long),
- when the file doesn't exist and you change the order of redirections (i.e.
cat >> file < filewill work,cat < file >> filewill throw an error; this is because redirections are "served" from left to right,>>creates the file if needed,<requires the file to exist).
In general using the same file with stdin and stdout (or/and stderr) redirection in a single command is asking for trouble. Still, if you need to do this, there is sponge utility (e.g. in moreutils package for Ubuntu, Debian).
spongereads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
If no output file is specified,
spongeoutputs to stdout.
Instead of cat < file >> file run sponge < file >> file. It will do what you expected from cat (unless the file is huge enough to deplete your resources while being soaked up, I think).
Another way is < file cat | cat >> file. Due to buffering between two cat-s the first one will really hit EOF and exit before the second one expands the file, if the file is small enough. Repeat this command few times starting with a small (but not empty) file and the file will eventually grow large enough to make the command behave like your original command with one cat. Note < file cat | unbuffer cat >> file gets rid of the buffer and loops even for very small files.
Well, my cat (in Kubuntu) says input file is output file and exits. I understand your cat is not as smart (or as lazy).
What you observe is quite easy to explain. cat reads and writes in chunks. Suppose the first chunk doesn't reach the end of file (EOF) and its content gets concatenated at the end of the same file. Reading cat advanced by some amount of bytes but the end of the file "drifted away" by the same amount. In effect cat is again as far from EOF as it was at the beginning. And it goes on and on and on.
As if you ran and for every 100 meters the finish line was moved additional 100 meters away from you. You would never reach it and the distance traveled would grow with no end.
Unless the original distance is 100 meters or less. In this case you would get to the finish line before it's relocated. Analogously cat might finish if the file is small enough, but I really doubt it. I expect cat "knows" it got to EOF only when it tries to read again and is able to read 0 bytes, no more. This means it already read until EOF in the previous step, when it read more than 0 bytes. But in your case, if it read more than 0 bytes in the previous step, then the file grew immediately after and there is still data to read.
I predict two cases so your cat will stop:
- when the file is empty (0 bytes long),
- when the file doesn't exist and you change the order of redirections (i.e.
cat >> file < filewill work,cat < file >> filewill throw an error; this is because redirections are "served" from left to right,>>creates the file if needed,<requires the file to exist).
In general using the same file with stdin and stdout (or/and stderr) redirection in a single command is asking for trouble. Still, if you need to do this, there is sponge utility (e.g. in moreutils package for Ubuntu, Debian).
spongereads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
If no output file is specified,
spongeoutputs to stdout.
Instead of cat < file >> file run sponge < file >> file. It will do what you expected from cat (unless the file is huge enough to deplete your resources while being soaked up, I think).
Another way is < file cat | cat >> file. Due to buffering between two cat-s the first one will really hit EOF and exit before the second one expands the file, if the file is small enough. Repeat this command few times starting with a small (but not empty) file and the file will eventually grow large enough to make the command behave like your original command with one cat. Note < file cat | unbuffer cat >> file gets rid of the buffer and loops even for very small files.
edited Jan 13 at 3:53
answered Jan 13 at 3:05
Kamil MaciorowskiKamil Maciorowski
27.3k155982
27.3k155982
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,catwould handle it properly since it loads every trunk until EOF.
– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,catwill wait. Very simple to observe: run justcat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.
– Kamil Maciorowski
Jan 13 at 3:43
add a comment |
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,catwould handle it properly since it loads every trunk until EOF.
– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,catwill wait. Very simple to observe: run justcat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.
– Kamil Maciorowski
Jan 13 at 3:43
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,
cat would handle it properly since it loads every trunk until EOF.– Tiina
Jan 13 at 3:22
does this mean if I have a file that is endless and eventually larger than memory, no matter how slow data arrives, only if it does not write EOF,
cat would handle it properly since it loads every trunk until EOF.– Tiina
Jan 13 at 3:22
@Tiina If you have a data stream arriving slowly,
cat will wait. Very simple to observe: run just cat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.– Kamil Maciorowski
Jan 13 at 3:43
@Tiina If you have a data stream arriving slowly,
cat will wait. Very simple to observe: run just cat, it will wait for your input. Type few lines, then it will wait again. Hit Ctrl+D to send a signal that will be interpreted as if EOF was reached.– Kamil Maciorowski
Jan 13 at 3:43
add a comment |
Well, basically this works as follows:
>is used to override the content of a file. E.g:echo "HELLO WORLD" > file.txt.
>>is used to append content to a file. E.g:echo "New line" >> file.txt.
<you can use a file as input and redirect to another file. E.g:cat < file.txt > file2.txt. A practical example isftp myftpbox.com < commands.txt, this would connect to a FTP and execute a list of commands place in the filecommands.txt.
NOTE: Be careful when you use > because you can override an existing file without any confirmation, you can alter this by running set -o noclobber.
I hope this helps.
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,set -o noclobberis beyond the basis, granted). What is more important, you didn't address the real issue at all.
– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
add a comment |
Well, basically this works as follows:
>is used to override the content of a file. E.g:echo "HELLO WORLD" > file.txt.
>>is used to append content to a file. E.g:echo "New line" >> file.txt.
<you can use a file as input and redirect to another file. E.g:cat < file.txt > file2.txt. A practical example isftp myftpbox.com < commands.txt, this would connect to a FTP and execute a list of commands place in the filecommands.txt.
NOTE: Be careful when you use > because you can override an existing file without any confirmation, you can alter this by running set -o noclobber.
I hope this helps.
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,set -o noclobberis beyond the basis, granted). What is more important, you didn't address the real issue at all.
– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
add a comment |
Well, basically this works as follows:
>is used to override the content of a file. E.g:echo "HELLO WORLD" > file.txt.
>>is used to append content to a file. E.g:echo "New line" >> file.txt.
<you can use a file as input and redirect to another file. E.g:cat < file.txt > file2.txt. A practical example isftp myftpbox.com < commands.txt, this would connect to a FTP and execute a list of commands place in the filecommands.txt.
NOTE: Be careful when you use > because you can override an existing file without any confirmation, you can alter this by running set -o noclobber.
I hope this helps.
Well, basically this works as follows:
>is used to override the content of a file. E.g:echo "HELLO WORLD" > file.txt.
>>is used to append content to a file. E.g:echo "New line" >> file.txt.
<you can use a file as input and redirect to another file. E.g:cat < file.txt > file2.txt. A practical example isftp myftpbox.com < commands.txt, this would connect to a FTP and execute a list of commands place in the filecommands.txt.
NOTE: Be careful when you use > because you can override an existing file without any confirmation, you can alter this by running set -o noclobber.
I hope this helps.
answered Jan 13 at 2:47
Manuel FlorianManuel Florian
1595
1595
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,set -o noclobberis beyond the basis, granted). What is more important, you didn't address the real issue at all.
– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
add a comment |
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,set -o noclobberis beyond the basis, granted). What is more important, you didn't address the real issue at all.
– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,
set -o noclobber is beyond the basis, granted). What is more important, you didn't address the real issue at all.– Kamil Maciorowski
Jan 13 at 3:30
-1. Despite the OP says "I know the basis of these three", your answer gives the basis (well,
set -o noclobber is beyond the basis, granted). What is more important, you didn't address the real issue at all.– Kamil Maciorowski
Jan 13 at 3:30
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
@KamilMaciorowski +1 for the effor >.<
– Tiina
Jan 13 at 5:19
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
I guess I misunderstood last night. Your feedbacks are really appreciated here. Thanks
– Manuel Florian
Jan 13 at 9:40
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393655%2fwhat-is-the-behavior-of-in-shell%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown