Old 20th October 2008, 16:51   #1
Jekano
Junior Member
 
Join Date: Oct 2008
Posts: 1
NSIS Installer for big files (more 4 GB and zip)

Hello,

I want to make a installer with one big file. The size of data more then 4GB. But NSIS can do file only <2GB now.

I try installer based on zip file. But its size more 4GB. When I try open file .zip I get "Error openning zip file". What kind error - I don't know.

Can I create one file installer with size more then 4GB? How do this?

Thanks!
Jekano is offline   Reply With Quote
Old 20th October 2008, 17:51   #2
Afrow UK
Moderator
 
Afrow UK's Avatar
 
Join Date: Nov 2002
Location: Surrey, England
Posts: 8,434
No you can't.

Stu
Afrow UK is offline   Reply With Quote
Old 11th November 2008, 10:43   #3
atomjack
Junior Member
 
Join Date: Sep 2008
Posts: 4
I looked at this a few months back. Basically there is no good solution currently for installing large files. I ended up distributing a small installer together with a set of .tbz2 files and using the 'untgz' plugin to decompress the files: the 'untgz' plugin was the only one that I found that could handle files that were larger than 4GB.

If your individual files are smaller than 2GB then you can compress them into individual zip files (so long as no zip file holds more than 2GB of data you're OK). Then there are various unzip plugins which will work. However, there's no solution that will let you distribute a single >4GB installer.
atomjack is offline   Reply With Quote
Old 8th July 2016, 13:49   #4
digthegoat
Junior Member
 
Join Date: Jul 2016
Posts: 1
Now that it is 8 years later, any way to do this? I have a file that is encrypted with a 10-15 character password and lost the password. Total file is something like 8 GB.
digthegoat is offline   Reply With Quote
Old 8th July 2016, 15:00   #5
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,420
Quote:
Originally Posted by digthegoat View Post
Now that it is 8 years later, any way to do this? I have a file that is encrypted with a 10-15 character password and lost the password. Total file is something like 8 GB.
You can use https://sourceforge.net/projects/nsisbi/ but I don't see how that is related to you losing a password.

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 2nd August 2016, 01:17   #6
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
Does the NSISBI work together with lzma2 for multicore support?
(without it creating installers will be very slow - as lzma supports only single core - it would be great to have this working in Nsis 3 together with a modern version of 7zip supporting lzma2 so creating the installer will be much quicker).
extremecarver is offline   Reply With Quote
Old 2nd August 2016, 03:51   #7
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 916
No, my NSISBI fork for external files does not add compressors, if I did I'll be adding lz4 for speedy compression because the output file can be huge.

The LZMA compression module has a special exemption from the 7-Zip developer for use in NSIS, I doubt lzma2 will be added any time soon.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile

Last edited by JasonFriday13; 2nd August 2016 at 06:49.
JasonFriday13 is offline   Reply With Quote
Old 2nd August 2016, 11:16   #8
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
okay - thanks. Would have been nice. I know most users don't mind. It's just that I create about 600GB of NSIS installers per week - and just running lzma for packing adds up lots of time (well I serialized it - but better would be to have it running on 8 cores instead of 1 core and running several compressors in parallel because that slows down the hdd more).
extremecarver is offline   Reply With Quote
Old 2nd August 2016, 12:11   #9
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,420
Quote:
Originally Posted by JasonFriday13 View Post
The LZMA compression module has a special exemption from the 7-Zip developer for use in NSIS, I doubt lzma2 will be added any time soon.
The lzma license has changed and you don't need the exemption now (IANAL)...

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 31st August 2018, 13:00   #10
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
Just to pop this up once again. Any chance of including lzma2 into Nsisbi (well it would be nice in normal nsis too). Also I think the lzma module should be much quicker even using singlecore - guess it has not been updated long long time either.


For my testcase I get:
Compressor off 548MB
LZ4 489MB (just a tiny bit slower then compress off)
LZ4 465MB with SetCompressorDictSize 1: Very Slow - Comparable to BZIP2 speed. Really Useless Mode.
Zlib 420MB (a lot slower - but still way faster than lzma, a bit slower than simply compressing the whole data lzma2 with 7zip on a 4 Core processor with Hyperthreading - but yeah it's also single core use only)
BZIP 2: 420MB -- very slow
LZMA 377MB. very very slow.

So lz4 without Dictionary is kinda nice if compression/uncompression speed is really important - and as it's nearly as fast as no compression at all - it is attractive. On my dataset ZLIB is not doing too bad - considering it is still pretty fast - and gets the size down again a lot from LZ4.
LZMA is king however about compression ratio - and if NSIS/NSISBI could move to lzma2 that would at least on the compression side make the speed pretty attractive.


Note - on NSISBI there is still a problem in the log with choosing lz4 as Compressor:
SetCompressor: lz4
warning 8026: SetCompressorDictSize: compressor is not set to LZMA. Effectively ignored.




Another problem is - I cannot unextract/open installers created by Nsisbi with 7zip. That's really not good as my installers also serve as container for linux users to download and just unpack the data. It does not matter which compressor I choose - the problem exists as soon as I use NSISBI - if I go back to NSIS I can again unpack all installers.
extremecarver is offline   Reply With Quote
Old 8th September 2018, 19:21   #11
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 916
That's because I have changed the first header in the exehead to support external files, so no you can't unpack them unless 7zip changes their code.

Also, SetCompressorDictSize is just two lines of code, one is the warning and the other just sets the value. All compressors get passed this value, only lzma uses it and I decided to use it for a different purpose (lz4 high compression).

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 20th October 2019, 20:59   #12
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
I still think lzma2 (or xz with multithreading or any other lzma based compiler with multithreading support) would be great. Much faster packing and unpacking.
Yes lz4 is super fast - but for many people downloading files over 2GB is still slow - so lzma2 would be the best compromise for speed vs size (also unpacking of lzma2 is much faster than lzma). Maybe even replace lzma with lzma2 in nsisbi?
extremecarver is offline   Reply With Quote
Old 29th May 2021, 20:19   #13
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
Just popping this up again. Any chance of moving to lzma2 (both Nsis and Nsisbi). Even most notebooks are now quadcore, 8 threads, and Nsis installers are painfully slow.

On my notebook the speed of unpacking via nsis7z for external files is about 10x the speed of nsis unpacking lzma. Packing is about 5x slower. I really cannot understand why NSIS and NSISBI cannot move to lzma2. Multi Processor support is soo soo needed. Also as it saves battery on notebooks. Notebooks are way more efficient using multicore over singlecore..

(I create every week about 400GB of nsis installers. and external files are just much more cumbersome compared to offering all in one file). For an average download speed of 50Mbit lzma2 is overall way faster than lz4 (time downloading and time unpacking combined). Yes if you have a 1000mbit line then lz4 is the answer. But very few people do (and it creates much more load on the server).

The exception to use lzma2 /lzma is not needed anymore AFAIK - so it would be great to upgrade from lzma to lzma2.
extremecarver is offline   Reply With Quote
Old 31st May 2021, 15:12   #14
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,420
As most things in a open source project, lack of developer time and interest would be a big reason.

Inno Setup supports lzma2 AFAIK.

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 1st June 2021, 15:08   #15
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 916
Yep. Anyone is welcome to have a crack at it and implement it, it doesn't have to be me or the nsis devs alone.

On a side note, there are multi-threaded versions of all 3 codecs used in nsis. But the hard bit is abstracting the threading out and putting it into a generic threading class for all three compressors to use (I have briefly looked at it in the past, as I'm running out of things to do on nsisbi).

But yes, time is my factor. Currently working 60 hours a week, and all I want to do on my days off is rest (I currently work in a hot factory, not an air-conditioned office like alot of programmers do).

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 1st June 2021, 21:36   #16
extremecarver
Junior Member
 
Join Date: Apr 2013
Posts: 37
well thanks for the answer. Good that you consider multithreading an important point. I thought that lzma2 the main advantage over lzma was exactly that - multithreading. In the last years CPUs have not improved a lot on singlethreaded speeds, but got way more threads/cores so speed improvements are largely depending on multithreading. If there is multithreading for lzma I would not think there is a need for lzma2. But I think lzma2 will be way easier concerning multi threading as that was the goal for creating it.
extremecarver is offline   Reply With Quote
Old 3rd June 2021, 03:23   #17
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 916
Nsis isn't really designed for 24/7 continuous install making, most projects only create an installer once so the speed doesn't matter much. Like I said, nothing stopping you from writing it yourself, nothing stopped me from making nsisbi in the first place .

On a side note, dealing with codecs is a bit "out of my league". The only reason LZ4 exists in nsisbi is because it has a wrapper layer which just calls the LZ4 api, and converts it back to nsis to make it compatible.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Reply
Go Back   Winamp & Shoutcast Forums > Developer Center > NSIS Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump