Audializer:ConvertingPicturestoSound SignalsandSystems:ProjectFinalReport BrendanRitter,BrendanQuinlivan,&GregEdelston Spring2013 ProjectOverview Thegoalofourprojectwastocreateanimagetoaudioconverter.Wesetouttodevelopa programthattookinanimagefileasinput,andproducedanaudiofilewhich,whenanalyzed throughaspectrumanalyser,showedtheoriginalimage.Ideally,theprogramwouldhave producedwonderfulsoundingaudio. Inordertoaccomplishthistask,ourteaminitiallysetouttodeveloptheentireprogramin MATLAB.However,afterrunningintosomeissueswithMATLABssoundpackages,wedecided tocompletetheimageanalysisinMATLAB,andthencallaPythonscriptfromMATLABto completethesoundgeneration. Withthisproject,ourteamgoalscenteredarounddevelopinganexcitingapplicationofsignals andsystemswhichwoulddeepenourintuitiveunderstandingofthecontent.

Thiswasespecially importanttousbecausethefocusoftheearlierportionoftheclasswasontheory.Additionally, inthisprojectwewantedtodrawconnectionsbetweensignalprocessingandotherdisciplines (inthiscasewechosemusic). ImageAnalysis OurprogrambeginsbytakinganimageintoMATLABinanyofseveraldifferentfileformats,and convertingthatimageintogreyscale.Afterthisconversion,theimageisdividedintovertical slivers,eachonepixelwide.AnexampleofthiscanbeseeninFigures1and2.

Figure1.Agreyscaleimageofadiagonallinewheretheverticalpositionofeachpixelandthe greyscalevaluecorrespondtothefrequencyandamplitudeofthesoundwavesproduced. Additionally,thehorizontalpositionofeachpixeldeterminethetimeatwhichthesoundwave occursinthesoundclip.

Figure2.AverticalsliverofthediagonallinefoundinFigure1thatisonepixelwide(imagenotto scale). Withthephotodividedintoslivers,theprogrammapseachpixelinthatslivertoalogscaleof frequenciesourlaptopspeakerscouldproduce(80Hzto5,500Hz)basedonthepixelsheightin thesliver.Itismappedsuchthatthepixelsatthebottomoftheimagecorrespondtolow frequenciesandthoseatthetopcorrespondtohighfrequencies. Finally,eachpixelnotonlyhasanassociatedfrequencybasedonitsverticalpositioninthe image,butalsoanassociatedamplitudebasedonthegreyscalevalueofthepixel.Darkerpixels denotehigheramplitudes,whilelighterpixelsareloweramplitudes.Foreachslivertheprogram mapsthefrequenciestothecorrespondingamplitudesasshowninthegraphinFigure3.

Figure3.ThenormalizedamplitudesandfrequenciesfortheslivershowninFigure2.Notice thatatapproximately250Hzthereisaspikeintheamplitudewhichcorrespondstotheblack pixelsfoundinFigure2. Thus,theprogramhasbrokeneachverticalsliverdownintoaseriesoffrequenciesand correspondingamplitudes. SoundGeneration Aftereachsliverisanalyzed,alistofallofthefrequenciesandamplitudesforallofthesliversis passedtoaPythonscriptthroughaCSVfile.Eachsliveristhenmadeintoapythondictionaryof frequenciesandtheirassociatedamplitudes. Asinewaveintime(witharbitraryduration)isgeneratedforeveryfrequencyandamplitudein thisdictionarywiththeincorporationofafrequencyscalingconstant.Thus,theoutcomeofthis processismultiplesinewaves.Thesewavesarethenaddedtomakeachordandnormalized basedonhowmanysinewaveswereaddedsotheamplitudedoesntgetdistorted.Theresultis asoundbyterepresentingoneparticularsliveroftheimage. Whenthisprocessisrepeatedforallthesliversinthepictureandtheircorrespondingsound bytesareconcatenated,anaudializedversionofthepictureiscreated.

HowweusedSignalsandSystems Whenweanalyzeanimage,wecreateasignalinthefrequencydomainwhichmustthen processintothetimedomain.However,sincewearealsooperatinginadiscretespace,the

signalbehavesasasampledfunctionwithdeltasasthevaluesateachpoint. WeusethelinearityoftheFouriertransformtorealizethattheinversetransformofthesumof impulsesisequivalenttothesumoftheinversetransformsoftheimpulses. Wealsorealizethatinorderforthetimedomainsignaltobereal,thefrequencydomainsignal mustbesymmetricalacrosstheyaxistherefore,wemirrorthefrequencies.Foragiven frequency 0 ,thiscreatesthefrequencysignal ( + 0) + ( 0) .FromSignalsand Systems,weknowthatthisequationistheFouriertransformofacosinefunctiontherefore, eachimpulseinthefrequencydomainmapstoasinusoidalfunctioninthetimedomain. Itshouldbenotedthatasinefunctioncanbeinterchangedwithacosinefunction,sincethey equivalent,saveforaphaseshift. SignalsandSystemshelpedustounderstandwhatwasgoingonbehindthescenes.Itwould havebeenpossibletoblindlyacceptthatasinewaveproducesapuretoneandthereforemakes sensetoproduce.However,usingouraccumulatedknowledgefromthesemester,we understoodeverythingthathappened,andwereabletoproducemusicwithfullknowledgeof whyitworks. Conclusions Overall,wewerecapableofmakingapicturetosoundconverter.However,therewereseveral areasofourprojectthatcouldhavebeenimproved. Onefeaturethatwouldhavemadeourprojectmorecohesivewouldhavebeenonlyperforming operationsinMATLAB.Wedecidedoriginallynottodothisduetocomplicationswiththesound producingabilitiesofMATLAB.However,givenmoretimewecouldhavestraightenedthese problemsout. Inaddition,anotheroptionavailabletouswastouseapieceofcodecreatedinMATLABthat producedguitarlikenotesusingsomesortofdynamicfilter.Althougheffortwasputinto deconstructingitsmethodofsoundcreation,ourcurrentmethodwasfinalizedbeforewecould fullyunderstandtheguitarcode.Thus,afurtherdeconstructionwouldhaveallowedustostayin theMATLABdomain,andincludeevenmoresignalsandsystems. Afurtherareaofinvestigationcouldhavebeenthecapabilitiesofourlaptopsspeakers.Itwasat timesunclearwhetherourprogrammingmethodswereincorrect,orwhetherweweregetting distortionbyrunningupagainstthehardwarelimitofourspeakers.Afurtherlookintothehard limitsmighthaveactuallyvalidatedpreviousMATLABattemptstomakesound.

AreasforFutureWork Thereisthepotentialforalotoffutureworkonthisproject.Onepotentialimprovementcouldbe convertingthefrequenciestoapentatonicscale.Thiswouldhavethebenefitofmakingevery imagesoundpleasing.However,thismodificationwouldmaketheprocessirreversiblebecause therewouldnolongerbeaonetoonemappingofverticalpositiontofrequencies. Alongthosesamelines,afutureprojectcouldbereversetheaudializeprocessandmakea pictureoutofasoundfile.Basedonourcurrentprogramitwouldbetrivialtoconvertthe amplitudesandfrequenciesassociatedwitheachpixelbackintoanimage.However,the processofdissectingtheconstructedaudiofileintotheseamplitudesandfrequencies associatedwitheachpixelwouldrequireasignificantamountofwork. Wecanalsoforeseeapplicationsforoursoftwarewithencryption.Ifthereisaparticular algorithmformappingagivenpixeltoafrequency,amplitude,andtimemorecomplexthana simplelogarithmicmappingthenanimagecouldbesecurelyencodedintoasoundfile.