0% found this document useful (0 votes)
186 views9 pages

Deflate: From Wikipedia, The Free Encyclopedia

The document discusses the DEFLATE algorithm which uses LZ77 and Huffman coding for data compression. It works by replacing repeated strings with pointers and assigning shorter bit sequences to more common symbols. The algorithm is used in formats like gzip, PNG and ZIP. Common implementations include zlib, 7-Zip and those in programming languages like Java.

Uploaded by

Rakesh Inani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
186 views9 pages

Deflate: From Wikipedia, The Free Encyclopedia

The document discusses the DEFLATE algorithm which uses LZ77 and Huffman coding for data compression. It works by replacing repeated strings with pointers and assigning shorter bit sequences to more common symbols. The algorithm is used in formats like gzip, PNG and ZIP. Common implementations include zlib, 7-Zip and those in programming languages like Java.

Uploaded by

Rakesh Inani
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

DEFLATE
FromWikipedia,thefreeencyclopedia

Incomputing,deflateisadatacompressionalgorithmthatusesacombinationoftheLZ77algorithmand
Huffmancoding.ItwasoriginallydefinedbyPhilKatzforversion2ofhisPKZIParchivingtoolandwas
laterspecifiedinRFC1951.[1]
TheoriginalalgorithmasdesignedbyKatzwaspatentedasU.S.Patent5,051,745
(https://www.google.com/patents/US5051745)andassignedtoPKWARE,Inc.[2][3]AsstatedintheRFC
document,Deflateiswidelythoughttobeimplementableinamannernotcoveredbypatents.[1]Thishas
ledtoitswidespreaduse,forexampleingzipcompressedfiles,PNGimagefilesandthe.ZIPfileformat
forwhichKatzoriginallydesignedit.

Contents
1Streamformat
1.1Duplicatestringelimination
1.2Bitreduction
2Encoder/compressor
2.1Deflate64/EnhancedDeflate
3UsingDeflateinnewsoftware
3.1Encoderimplementations
3.2Hardwareencoders
4Decoder/decompressor
4.1Inflateonlyimplementations
4.2Hardwaredecoders
5Seealso
6References
7Externallinks

Streamformat
ADeflatestreamconsistsofaseriesofblocks.Eachblockisprecededbya3bitheader:
Firstbit:Lastblockinstreammarker:
1:thisisthelastblockinthestream.
0:therearemoreblockstoprocessafterthisone.

Secondandthirdbits:Encodingmethodusedforthisblocktype:
http://en.wikipedia.org/wiki/DEFLATE

1/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

00:astored/raw/literalsection,between0and65,535bytesinlength.
01:astaticHuffmancompressedblock,usingapreagreedHuffmantree.
10:acompressedblockcompletewiththeHuffmantablesupplied.
11:reserved,don'tuse.

Mostblockswillendupbeingencodedusingmethod10,thedynamicHuffmanencoding,whichproduces
anoptimisedHuffmantreecustomisedforeachblockofdataindividually.Instructionstogeneratethe
necessaryHuffmantreeimmediatelyfollowtheblockheader.
Compressionisachievedthroughtwosteps
Thematchingandreplacementofduplicatestringswithpointers.
Replacingsymbolswithnew,weightedsymbolsbasedonfrequencyofuse.

Duplicatestringelimination
Withincompressedblocks,ifaduplicateseriesofbytesisspotted(arepeatedstring),thenabackreference
isinserted,linkingtothepreviouslocationofthatidenticalstringinstead.Anencodedmatchtoanearlier
stringconsistsofalength(3258bytes)andadistance(132,768bytes).Relativebackreferencescanbe
madeacrossanynumberofblocks,aslongasthedistanceappearswithinthelast32kBofuncompressed
datadecoded(termedtheslidingwindow).

Bitreduction
Thesecondcompressionstageconsistsofreplacingcommonlyusedsymbolswithshorterrepresentations
andlesscommonlyusedsymbolswithlongerrepresentations.ThemethodusedisHuffmancodingwhich
createsanunprefixedtreeofnonoverlappingintervals,wherethelengthofeachsequenceisinversely
proportionaltotheprobabilityofthatsymbolneedingtobeencoded.Themorelikelyasymbolhastobe
encoded,theshorteritsbitsequencewillbe.
Atreeiscreated,containingspacefor288symbols:
0255:representtheliteralbytes/symbols0255.
256:endofblockstopprocessingiflastblock,otherwisestartprocessingnextblock.
257285:combinedwithextrabits,amatchlengthof3258bytes.
286,287:notused,reservedandillegalbutstillpartofthetree.
Amatchlengthcodewillalwaysbefollowedbyadistancecode.Basedonthedistancecoderead,further
"extra"bitsmaybereadinordertoproducethefinaldistance.Thedistancetreecontainsspacefor32
symbols:
03:distances14
45:distances58,1extrabit
67:distances916,2extrabits
http://en.wikipedia.org/wiki/DEFLATE

2/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

89:distances1732,3extrabits
...
2627:distances8,19316,384,12extrabits
2829:distances16,38532,768,13extrabits
3031:notused,reservedandillegalbutstillpartofthetree.
Notethatforthematchdistancesymbols229,thenumberofextrabitscanbecalculatedas

Encoder/compressor
Duringthecompressionstage,itistheencoderthatchoosestheamountoftimespentlookingformatching
strings.Thezlib/gzipreferenceimplementationallowstheusertoselectfromaslidingscaleoflikely
resultingcompressionlevelvs.speedofencoding.Optionsrangefrom0(donotattemptcompression,just
storeuncompressed)to9representingthemaximumcapabilityofthereferenceimplementationin
zlib/gzip.
OtherDeflateencodershavebeenproduced,allofwhichwillalsoproduceacompatiblebitstreamcapable
ofbeingdecompressedbyanyexistingDeflatedecoder.Differingimplementationswilllikelyproduce
variationsonthefinalencodedbitstreamproduced.Thefocuswithnonzlibversionsofanencoderhas
normallybeentoproduceamoreefficientlycompressedandsmallerencodedstream.

Deflate64/EnhancedDeflate
Deflate64,specifiedbyPKWare,isaproprietaryvariantoftheDeflateprocedure.Thefundamental
mechanismsremainthesame.Whathaschangedistheincreaseindictionarysizefrom32kBto64kB,an
additionof14bitstothedistancecodessothattheymayaddressarangeof64kB,andthelengthcodehas
beenextendedby16bitssothatitmaydefinelengthsof3to65538bytes.[4]ThisleadstoDeflate64having
aslightlyhighercompressionratioandaslightlylowercompressiontimethanDeflate.[5]Severalfree
and/oropensourceprojectssupportDeflate64,suchas7Zip,[6]whileothers,suchaszlib,donot,asa
resultoftheproprietarynatureoftheprocedure[7]andtheverymodestperformanceincreaseover
Deflate.[8]

UsingDeflateinnewsoftware
ImplementationsofDeflatearefreelyavailableinmanylanguages.Cprogramstypicallyusethezlib
library(undertheoldBSDlicensewithoutadvertisingclause).ProgramswrittenusingtheBorlanddialects
ofPascalcanusepaszlibaC++libraryisincludedaspartof7Zip/AdvanceCOMP.Javaincludessupport
aspartofthestandardlibrary(injava.util.zip).Microsoft.NETFramework2.0baseclasslibrarysupportsit
intheSystem.IO.Compressionnamespace.

Encoderimplementations
PKZIP:thefirstimplementation,originallydonebyPhilKatzaspartofPKZip.
http://en.wikipedia.org/wiki/DEFLATE

3/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

zlib/gzip:standardreferenceimplementationusedinahugeamountofsoftware,owingtopublic
availabilityofthesourcecodeandalicenseallowinginclusionintoothersoftware.
jzlib(http://www.jcraft.com/jzlib/):Rewrite/reimplementation/portofthezlibencoderinto
pureJavaanddistributedunderaBSDlicense.(Fullyfeaturedreplacementforjava.util.zip).
PasZLIB(http://www.nomssi.de/paszlib/paszlib.html):Translation/portofthezlibcodeinto
PascalsourcecodebyJacquesNomssiNzali.
gziplite(http://sourceforge.net/projects/gziplite/):Minimalistreworkofgzip/gunzipwith
minimalmemoryrequirement,alsosupportingontheflydatacompression/decompression(no
needtobufferizeallinput)andinput/outputto/frommemory.
pako(https://github.com/nodeca/pako):fullfeaturedzlibporttoJavaScript,optimizedforhigh
speed.Worksinbrowsersandnode.js.
miniz(http://code.google.com/p/miniz/)PublicdomainDeflate/Inflateimplementationwithazlib
compatibleAPIinasingle.Csourcefile
lodepng(http://lodev.org/lodepng/)byLodeVandevenne.ABSDlicensedsinglefilePNGfilereader
withbuiltinC++Inflateimplementationandnoexternaldependencies.
KZIP(http://advsys.net/ken/utils.htm#kzip)/PNGOUT:anencoderbythegameprogrammerKen
Silvermanusing"anexhaustivesearchofallpatterns"and"[an]advancedblocksplitter".
PuZip(http://www.cs.tut.fi/~albert/Dev/puzip/):designedforCommodore64/C128computers.PuZip
islimitedtoan8kBLZ77windowsize,withonlythestore(type00)andfixedHuffman(type01)
methods.
BigSpeedDeflate(http://www.bigspeed.net/index.php?page=bsdefdll):"Tinyinmemorycompression
library"availableasaMSWindowsDLLlimitedto32kBblocksatatimeandthreecompression
settings.
BJWFlate&DeflOpt(http://www.walbeehm.com/download/)/DeflOpt:BenJosWalbeehm'sutilities
"designedtoattempttosqueezeeverypossiblebyteoutofthefilesitcompresses".Notethatthe
authorhasstoppeddevelopmentonBJWFlate(butnotDeflOpt)inMarch2004.
Crypto++:containsapublicdomainimplementationinC++aimedmainlyatreducingpotential
securityvulnerabilities.Theauthor,WeiDaistates"Thiscodeislessclever,buthopefullymore
understandableandmaintainable[thanzlib]".
DeflateStream(http://msdn.microsoft.com/enus/library/system.io.compression.deflatestream.aspx)
animplementationofastreamthatperformsDEFLATEcompression,itispackagedwiththeBase
ClassLibraryincludedwiththe.NETFramework.
ParallelDeflateOutputStream(http://cheeso.members.winisp.net/DotNetZipHelp/html/26cbdba2
021accf1a9c9b7ae55f6ecb8.htm)anopensourcestreamthatimplementsaparallel(multithread)
deflatingstream,forusein.NETprograms.
7Zip/AdvanceCOMP:writtenbyIgorPavlovinC++,thisversionisfreelylicensedandtendsto
http://en.wikipedia.org/wiki/DEFLATE

4/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

achievehighercompressionthanzlibattheexpenseofCPUusage.Hasanoptiontousethe
DEFLATE64storageformat.
deflate.s7i(http://seed7.sourceforge.net/libraries/deflate.htm)/gzip.s7i
(http://seed7.sourceforge.net/libraries/gzip.htm),apureSeed7implementationofDeflateandgzip
compression,byThomasMertes.MadeavailableundertheGNULGPLlicense.
PuTTY`sshzlib.c`:astandaloneimplementation,capableoffulldecode,butstatictreeonlycreation,
bySimonTatham.MITlicensed.
Halibut(http://www.chiark.greenend.org.uk/~sgtatham/halibut/)`deflate.c`:astandalone
implementationcapableoffulldecode.ForkedfromPuTTY's`sshzlib.c`,butextendedtowrite
dynamicHuffmantreesandprovidesAdler32andCRC32checksumsupport.
Plan9fromBellLabsoperatingsystem'slibflate(http://plan9.bell
labs.com/sources/plan9/sys/src/libflate/)implementsdeflatecompression.
Hyperbac:usesitsownproprietarylosslesscompressionlibrary(writteninC++andAssembly)with
anoptiontoimplementtheDEFLATE64storageformat.
zip.js(http://gildaslormeau.github.com/zip.js/):JavaScriptimplementation.
Zopfli:publicdomainCimplementationbyGooglethatachieveshighestcompressionattheexpense
ofCPUusage.
AdvanceCOMPusesthehighercompressionratioversionofDeflateasimplementedby7Zip(or
optionallyZopfliinrecentversions)toenablerecompressionofgzip,PNG,MNGandZIPfileswiththe
possibilityofachievingsmallerfilesizesthanzlibisabletoatmaximumsettings.Anevenmoreeffective
(butalsomoreuserinputdemandingandCPUintensive)DeflateencoderisemployedinsideKen
Silverman'sKZIPandPNGOUTutilities,althoughrecentversionsofAdvanceCOMPhavesurpassedKZIP
andPNGOUTwhenusingAdvanceCOMP'sZopflimode.

Hardwareencoders
AHA361PCIX/AHA362PCIXfromComtechAHA(http://www.aha.com/).Comtechproduceda
PCIXcard(PCIID:193f:0001)capableofcompressingstreamsusingDeflateatarateofupto3.0
Gbit/s(375MB/s)forincominguncompresseddata.AccompanyingtheLinuxkerneldriverforthe
AHA361PCIXisan"ahagzip"utilityandcustomised"mod_deflate_aha"capableofusingthe
hardwarecompressionfromApache.ThehardwareisbasedonaXilinxVirtexFPGAandfour
customAHA3601ASICs.TheAHA361/AHA362boardsarelimitedtoonlyhandlingstaticHuffman
blocksandrequiresoftwaretobemodifiedtoaddsupportthecardswerenotabletosupportthe
fullDeflatespecification,meaningtheycouldonlyreliablydecodetheirownoutput(astreamthat
didnotcontainanydynamicHuffmantype2blocks).
StorCompress300(http://www.indranetworks.com/SC300.html)/MX3
(http://www.indranetworks.com/SCMX3.html)fromIndraNetworks
(http://www.indranetworks.com/).ThisisarangeofPCI(PCIID:17b4:0011)orPCIXcards
http://en.wikipedia.org/wiki/DEFLATE

5/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

featuringbetweenoneandsixcompressionengineswithclaimedprocessingspeedsofupto3.6
Gbit/s(450MB/s).AversionofthecardsareavailablewiththeseparatebrandWebEnhance
specificallydesignedforwebservinguseratherthanSANorbackupuseaPCIerevision,theMX4E
(http://www.indranetworks.com/SCMX4E.html)isalsoproduced.
AHA363PCIe(http://www.aha.com/show_prod.php?id=36)/AHA364PCIe
(http://www.aha.com/show_prod.php?id=37)/AHA367PCIe(http://www.aha.com/show_prod.php?
id=38).In2008,ComtechstartedproducingtwoPCIecards(PCIID:193f:0363/193f:0364)witha
newhardwareAHA3610encoderchip.Thenewchipwasdesignedtobecapableofasustained
2.5Gbit/s.Usingtwoofthesechips,theAHA363PCIeboardcanprocessDeflateatarateofupto
5.0Gbit/s(625MB/s)usingthetwochannels(twocompressionandtwodecompression).The
AHA364PCIevariantisanencodeonlyversionofthecarddesignedforoutgoingloadbalancers
andinsteadhasmultipleregistersetstoallow32independentvirtualcompressionchannelsfeeding
twophysicalcompressionengines.Linux,MicrosoftWindows,andOpenSolariskerneldevice
driversareavailableforbothofthenewcards,alongwithamodifiedzlibsystemlibrarysothat
dynamicallylinkedapplicationscanautomaticallyusethehardwaresupportwithoutinternal
modification.TheAHA367PCIeboard(PCIID:193f:0367)issimilartotheAHA363PCIebutuses
fourAHA3610chipsforasustainedcompressionrateof10Gbit/s(1250MB/s).Unlikethe
AHA362PCIX,thedecompressionenginesontheAHA363PCIeandAHA367PCIeboardsare
fullydeflatecompliant.

Decoder/decompressor
InflateisthedecodingprocessthattakesaDeflatebitstreamfordecompressionandcorrectlyproducesthe
originalfullsizedataorfile.

Inflateonlyimplementations
ThenormalintentwithanalternativeInflateimplementationishighlyoptimiseddecodingspeed,or
extremelypredictableRAMusageformicrocontrollerembeddedsystems.
Assembly
6502inflate(https://github.com/pfusik/zlib6502),writtenbyPiotrFusikin6502assembly
language.
ElektronikaMK90inflate(http://www.pisi.com.pl/piotr433/mk90mc1e.htm#inflate),the
above6502programportedbyPiotrPiatektothePDP11architecture.
SAMflate(http://sourceforge.net/projects/samflate/),writtenbyAndrewCollierinZ80
assemblylanguagewithoptionalmemorypagingsupportfortheSAMCoup,andmade
availableundertheBSD/GPL/LGPL/DFSGlicenses.
http://en.wikipedia.org/wiki/DEFLATE

6/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

C/C++
kunzip(http://www.mikekohn.net/file_formats/kunzip.php)byMichaelKohnandunrelatedto
"KZIP".ComeswithCsourcecodeundertheGNULGPLlicense.UsedintheGIMPinstaller.
puff.c(zlib),asmall,unencumbered,singlefilereferenceimplementationincludedinthe
/contrib/puffdirectoryofthezlibdistribution.
tinf(http://www.ibsensoftware.com/download.html)writtenbyJrgenIbseninANSICand
comeswithzliblicense.Addsabout2kcode.
tinfl.c(http://code.google.com/p/miniz/source/browse/trunk/tinfl.c)(miniz
(http://code.google.com/p/miniz/)),PublicdomainInflateimplementationcontainedentirelyin
asingleCfunction.
PCDEZIP,BobFlandersandMichaelHolmes,publishedinPCMagazine19940111.

inflate.cl(http://opensource.franz.com/deflate/)byJohnFoderaro.SelfstandingCommonLisp
decoderdistributedwithaGNULGPLlicense.
inflate.s7i(http://seed7.sourceforge.net/libraries/inflate.htm)/gzip.s7i
(http://seed7.sourceforge.net/libraries/gzip.htm),apureSeed7implementationofDeflateandgzip
decompression,byThomasMertes.MadeavailableundertheGNULGPLlicense.
pyflate(http://www.paul.sladen.org/projects/pyflate/),apurePythonstandaloneDeflate(gzip)and
bzip2decoderbyPaulSladen.Writtenforresearch/prototypingandmadeavailableunderthe
BSD/GPL/LGPL/DFSGlicenses.
deflatelua(http://luausers.org/wiki/ModuleCompressDeflateLua),apureLuaimplementationof
Deflateandgzip/zlibdecompression,byDavidManura.
inflate(https://github.com/chrisdickinson/inflate)apureJavascriptimplementationofInflateby
ChrisDickinson
pako(https://github.com/nodeca/pako):JavaScriptspeedoptimizedportofzlib.Containsseparate
buildwithinflateonly.

Hardwaredecoders
SerialInflateGPU(http://www.bitsim.com/en/badge.htm)fromBitSim.Hardwareimplementationof
Inflate.PartofBitSim'sBADGE(BitsimAcceleratedDisplayGraphicsEngine)controlleroffering
forembeddedsystems.

Seealso
Listofarchiveformats
Listoffilearchivers
http://en.wikipedia.org/wiki/DEFLATE

7/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

Comparisonoffilearchivers

References
1. ^abL.PeterDeutsch(May1996).DEFLATECompressedDataFormatSpecificationversion1.3
(https://tools.ietf.org/html/rfc1951#sectionAbstract).IETF.p.1.sec.Abstract.RFC1951.
https://tools.ietf.org/html/rfc1951#sectionAbstract.Retrieved23April2014.
2. ^USpatent5051745(http://worldwide.espacenet.com/textdoc?DB=EPODOC&IDX=US5051745),Katz,Phillip
W.,"Stringsearcher,andcompressorusingsame",published24September1991,issued24September1991
3. ^David,Salomon(2007).DataCompression:TheCompleteReference(http://books.google.com/books?
id=ujnQogzx_2EC&pg=PA241)(4ed.).Springer.p.241.ISBN9781846286025.
4. ^BinaryEssenceDeflate64(http://www.binaryessence.com/dct/imp/en000225.htm)
5. ^BinaryEssence"CalgaryCorpus"compressioncomparisons
(http://www.binaryessence.com/dct/apc/en000263.htm)
6. ^7ZipManualandDocumentationcompressionMethod
(http://sevenzip.sourceforge.jp/chm/cmdline/switches/method.htm)
7. ^HistoryofLosslessDataCompressionAlgorithmsDeflate64
(http://ieeeghn.org/wiki/index.php/History_of_Lossless_Data_Compression_Algorithms#DEFLATE64)
8. ^zlibFAQDoeszlibsupportthenew"Deflate64"formatintroducedbyPKWare?
(http://www.zlib.net/zlib_faq.html#faq40)

Externallinks
PKWARE,Inc.'sappnote.txt,.ZIPFileFormatSpecification
(http://www.pkware.com/documents/casestudies/APPNOTE.TXT)Section10,X.DeflatingMethod
8.
RFC1951DeflateCompressedDataFormatSpecificationversion1.3
zlibHomePage(http://www.zlib.net)
AnExplanationoftheDeflateAlgorithm(http://zlib.net/feldspar.html)byAntaeusFeldspar.
ExtendedApplicationofSuffixTreestoDataCompression
(http://www.larsson.dogma.net/dccpaper.pdf)AnexcellentalgorithmtoimplementDeflatebyJesper
Larsson
Retrievedfrom"http://en.wikipedia.org/w/index.php?title=DEFLATE&oldid=623213190"
Categories: Losslesscompressionalgorithms
Thispagewaslastmodifiedon28August2014at19:19.
TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmay
http://en.wikipedia.org/wiki/DEFLATE

8/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

apply.Byusingthissite,youagreetotheTermsofUseandPrivacyPolicy.Wikipediaisa
registeredtrademarkoftheWikimediaFoundation,Inc.,anonprofitorganization.

http://en.wikipedia.org/wiki/DEFLATE

9/9

You might also like