Creating and splitting large multipage TIFF images
I need to both create and split multipage TIFF images, ranging from 2 to almost 100 pages (A4, 300 dpi, 2500×3500 px). The job is performed periodically by a script on an x64 Linux server. Currently I'm using Imagemagick. The smaller cases do not pose any problems, but the larger ones do.
I need to radically reduce amount of memory used during the operation.
For example, this:
convert *.jpg -compress lzw output.tif
(70 jpeg files) consumes about 4.6 GB of RAM, even though each input is less than 2MB the resulting file is less than 250MB.
The reverse operation:
convert input.tif output-%04d.png
has similar issues.
From what I have read, this happens because Imagemagick first loads and decodes all the input images and only after that it starts encoding them into the output file.
How can I create and split multipage TIFF images without such huge memory footprint? I don't have to necessarily use ImageMagick, any other free tool will be fine.
imagemagick tiff image-manipulation
asked Aug 6 '14 at 12:35
Karol S
17817
add a comment |
I need to both create and split multipage TIFF images, ranging from 2 to almost 100 pages (A4, 300 dpi, 2500×3500 px). The job is performed periodically by a script on an x64 Linux server. Currently I'm using Imagemagick. The smaller cases do not pose any problems, but the larger ones do.
I need to radically reduce amount of memory used during the operation.
For example, this:
convert *.jpg -compress lzw output.tif
(70 jpeg files) consumes about 4.6 GB of RAM, even though each input is less than 2MB the resulting file is less than 250MB.
The reverse operation:
convert input.tif output-%04d.png
has similar issues.
From what I have read, this happens because Imagemagick first loads and decodes all the input images and only after that it starts encoding them into the output file.
How can I create and split multipage TIFF images without such huge memory footprint? I don't have to necessarily use ImageMagick, any other free tool will be fine.
imagemagick tiff image-manipulation
asked Aug 6 '14 at 12:35
Karol S
17817
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
The first exampleconvertabove creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".
– Hannu
Mar 28 '16 at 8:52
add a comment |
I need to both create and split multipage TIFF images, ranging from 2 to almost 100 pages (A4, 300 dpi, 2500×3500 px). The job is performed periodically by a script on an x64 Linux server. Currently I'm using Imagemagick. The smaller cases do not pose any problems, but the larger ones do.
I need to radically reduce amount of memory used during the operation.
For example, this:
convert *.jpg -compress lzw output.tif
(70 jpeg files) consumes about 4.6 GB of RAM, even though each input is less than 2MB the resulting file is less than 250MB.
The reverse operation:
convert input.tif output-%04d.png
has similar issues.
From what I have read, this happens because Imagemagick first loads and decodes all the input images and only after that it starts encoding them into the output file.
How can I create and split multipage TIFF images without such huge memory footprint? I don't have to necessarily use ImageMagick, any other free tool will be fine.
imagemagick tiff image-manipulation
asked Aug 6 '14 at 12:35
Karol S
17817
I need to both create and split multipage TIFF images, ranging from 2 to almost 100 pages (A4, 300 dpi, 2500×3500 px). The job is performed periodically by a script on an x64 Linux server. Currently I'm using Imagemagick. The smaller cases do not pose any problems, but the larger ones do.
I need to radically reduce amount of memory used during the operation.
For example, this:
convert *.jpg -compress lzw output.tif
(70 jpeg files) consumes about 4.6 GB of RAM, even though each input is less than 2MB the resulting file is less than 250MB.
The reverse operation:
convert input.tif output-%04d.png
has similar issues.
From what I have read, this happens because Imagemagick first loads and decodes all the input images and only after that it starts encoding them into the output file.
How can I create and split multipage TIFF images without such huge memory footprint? I don't have to necessarily use ImageMagick, any other free tool will be fine.
imagemagick tiff image-manipulation
imagemagick tiff image-manipulation
asked Aug 6 '14 at 12:35
Karol S
17817
asked Aug 6 '14 at 12:35
Karol S
17817
asked Aug 6 '14 at 12:35
Karol S
17817
asked Aug 6 '14 at 12:35
Karol S
17817
asked Aug 6 '14 at 12:35
Karol S
17817
17817
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
The first exampleconvertabove creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".
– Hannu
Mar 28 '16 at 8:52
add a comment |
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
The first exampleconvertabove creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".
– Hannu
Mar 28 '16 at 8:52
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
The first example
convert above creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".– Hannu
Mar 28 '16 at 8:52
The first example
convert above creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".– Hannu
Mar 28 '16 at 8:52
add a comment |
2 Answers
2
active
oldest
votes
I had the same problem today while trying to split a 1700 image, 1G tif file. 16G of memory wasn't enough, then tried having it cache on disk, but that was slow and it easily exhausted more than 100G on the harddrive without accomplishing anything (this was probably a bug).
But apparently Imagemagick can extract a specific tif from the original file without loading it completely, so was able to split the bigger file with a simple bash script:
END=2000
for ((i=1;i<=END;i++));do
echo $i
convert bigassfile.tif[$i] -scene 1 split/smallerfile_$i.tif
done
No idea though how to create a big file without running out of memory, so maybe this is half an answer?
answered Dec 19 '16 at 16:08
tarikki
1112
add a comment |
I find @tarikki's answer one of the best, because it really doesn't hang the server nor does it eat RAM and disk space, and it's fast.
Some improvements that helped me:
1. replace END=2000 by END=$(identify -format "%n" bigassfile.tif)
2. The TIF index is 0-based, so the looping should start at 0 and use < instead of <= : for((i=0;i<END;i++))
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
answered Dec 7 at 17:51
MrMacvos
11
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f793219%2fcreating-and-splitting-large-multipage-tiff-images%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
I had the same problem today while trying to split a 1700 image, 1G tif file. 16G of memory wasn't enough, then tried having it cache on disk, but that was slow and it easily exhausted more than 100G on the harddrive without accomplishing anything (this was probably a bug).
But apparently Imagemagick can extract a specific tif from the original file without loading it completely, so was able to split the bigger file with a simple bash script:
END=2000
for ((i=1;i<=END;i++));do
echo $i
convert bigassfile.tif[$i] -scene 1 split/smallerfile_$i.tif
done
No idea though how to create a big file without running out of memory, so maybe this is half an answer?
answered Dec 19 '16 at 16:08
tarikki
1112
add a comment |
I had the same problem today while trying to split a 1700 image, 1G tif file. 16G of memory wasn't enough, then tried having it cache on disk, but that was slow and it easily exhausted more than 100G on the harddrive without accomplishing anything (this was probably a bug).
But apparently Imagemagick can extract a specific tif from the original file without loading it completely, so was able to split the bigger file with a simple bash script:
END=2000
for ((i=1;i<=END;i++));do
echo $i
convert bigassfile.tif[$i] -scene 1 split/smallerfile_$i.tif
done
No idea though how to create a big file without running out of memory, so maybe this is half an answer?
answered Dec 19 '16 at 16:08
tarikki
1112
add a comment |
I had the same problem today while trying to split a 1700 image, 1G tif file. 16G of memory wasn't enough, then tried having it cache on disk, but that was slow and it easily exhausted more than 100G on the harddrive without accomplishing anything (this was probably a bug).
But apparently Imagemagick can extract a specific tif from the original file without loading it completely, so was able to split the bigger file with a simple bash script:
END=2000
for ((i=1;i<=END;i++));do
echo $i
convert bigassfile.tif[$i] -scene 1 split/smallerfile_$i.tif
done
No idea though how to create a big file without running out of memory, so maybe this is half an answer?
answered Dec 19 '16 at 16:08
tarikki
1112
I had the same problem today while trying to split a 1700 image, 1G tif file. 16G of memory wasn't enough, then tried having it cache on disk, but that was slow and it easily exhausted more than 100G on the harddrive without accomplishing anything (this was probably a bug).
But apparently Imagemagick can extract a specific tif from the original file without loading it completely, so was able to split the bigger file with a simple bash script:
END=2000
for ((i=1;i<=END;i++));do
echo $i
convert bigassfile.tif[$i] -scene 1 split/smallerfile_$i.tif
done
No idea though how to create a big file without running out of memory, so maybe this is half an answer?
answered Dec 19 '16 at 16:08
tarikki
1112
answered Dec 19 '16 at 16:08
tarikki
1112
answered Dec 19 '16 at 16:08
tarikki
1112
answered Dec 19 '16 at 16:08
tarikki
1112
1112
add a comment |
add a comment |
I find @tarikki's answer one of the best, because it really doesn't hang the server nor does it eat RAM and disk space, and it's fast.
Some improvements that helped me:
1. replace END=2000 by END=$(identify -format "%n" bigassfile.tif)
2. The TIF index is 0-based, so the looping should start at 0 and use < instead of <= : for((i=0;i<END;i++))
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
answered Dec 7 at 17:51
MrMacvos
11
add a comment |
I find @tarikki's answer one of the best, because it really doesn't hang the server nor does it eat RAM and disk space, and it's fast.
Some improvements that helped me:
1. replace END=2000 by END=$(identify -format "%n" bigassfile.tif)
2. The TIF index is 0-based, so the looping should start at 0 and use < instead of <= : for((i=0;i<END;i++))
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
answered Dec 7 at 17:51
MrMacvos
11
add a comment |
I find @tarikki's answer one of the best, because it really doesn't hang the server nor does it eat RAM and disk space, and it's fast.
Some improvements that helped me:
1. replace END=2000 by END=$(identify -format "%n" bigassfile.tif)
2. The TIF index is 0-based, so the looping should start at 0 and use < instead of <= : for((i=0;i<END;i++))
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
answered Dec 7 at 17:51
MrMacvos
11
I find @tarikki's answer one of the best, because it really doesn't hang the server nor does it eat RAM and disk space, and it's fast.
Some improvements that helped me:
1. replace END=2000 by END=$(identify -format "%n" bigassfile.tif)
2. The TIF index is 0-based, so the looping should start at 0 and use < instead of <= : for((i=0;i<END;i++))
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
answered Dec 7 at 17:51
MrMacvos
11
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
edited Dec 7 at 19:18
Pimp Juice IT
23k113869
23k113869
answered Dec 7 at 17:51
MrMacvos
11
answered Dec 7 at 17:51
MrMacvos
11
answered Dec 7 at 17:51
MrMacvos
11
11
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f793219%2fcreating-and-splitting-large-multipage-tiff-images%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
To put some perspective on it: EACH 2500×3500 pixel image will take up 2500×3500×3 bytes at least as it resides in memory.That is 26250000 bytes per image, 1837500000 bytes total for 70 images. Then you create a DUPLICATE of that in the TIF, total 3675000000. Then you request to save it using lzw compression; some buffers is probably required for that. Maybe add buffers for writing... Handling 70-100 page files isn't easy, especially if the pages are nothing but bitmaps.
– Hannu
Aug 6 '14 at 12:59
@Hannu Not easy for who? The real world says that there's a concept of streaming transforms, and unpacking a huge image stack in memory simultaneously is lame and fugly.
– polkovnikov.ph
Mar 25 '16 at 0:13
The first example
convertabove creates a single PAGED tiff. Depending on how Imagemagick works internally, you MIGHT indeed have a "huge image stack in memory".– Hannu
Mar 28 '16 at 8:52