Hadoop

Tuesday,18November2014

ShareItwithyourfriend(s)

Whatisunstructureddata?
Unstructureddatareferstoanydatathatcannotbeidentifiedeasily.Itcouldbeintheformof
images,videos,documents,email,logsorintheformofrandomtext.Itisnotintheformofrowsand
columns.

Advertisement

PostedbyPradeepKumarNandaat20:17 Nocomments:

Contactmeinmymailid
[email protected]'s
overhere.

ContactForm

BlogArchive

Name

2014(16)
November(
16)

Whatisstructureddata?
Structureddataisthedatathatiseasilyidentifiableasitisorganizedinastructure.
Themostcommonformofstructureddataisadatabasewherespecificinformationisstoredin
tables,i.erowsandcolumns.

PostedbyPradeepKumarNandaat20:14 Nocomments:

Howthehadoopshellcommandworksinternally?
TheHadoopshellcommandsareaconvenientwrapperaroundtheHDFSFileSystemAPI.Infact,
callingthehadoopshellscriptandpassingitthefsoptionsetstheJavaapplicationentrypointtothe
org.apache.hadoop.fs.FsShellclass.TheFsShellclasstheninstantiatesan
org.apache.hadoop.fs.FileSystemobjectandmapsthefilesystem'smethodstothefscommand
linearguments.
Forexample,
hadoopfsmkdir/data/pknanda,isequivalenttoFileSystem.mkdirs(new
Path("/data/pknanda")).
Similarly,hadoopfscopyFromLocalfile.txt/data/pknandaisequivalentto
FileSystem.copyFromLocal(newPath("file.txt"),newPath("/data/pknanda")).
ThesameappliestocopyingthedatafromHDFStothelocalfilesystem.ThecopyToLocalHadoop
shellcommandisequivalenttoFileSystem.copyToLocal(newPath("/data/pknanda/file.txt"),new
Path("./file.txt")).
ThemkdircommandtakesthegeneralformofhadoopfsmkdirPATH1PATH2.Forexample,
hadoopfsmkdir/data/pknandas/file1/data/pknanda/file2wouldcreatetwofoldersinHDFS:
/data/pknanda/file1and/data/pknanda/file2,respectively.

hadoopfsmkdir/data/pknanda/file1/data/pknanda/file2
hadoopfsls/data/pknanda

Whatis
unstructure
ddata?
Unstructure
ddataref...
Whatis
structured
data?
Structured
dataisthe
d...

Send

AboutMe

Monday,17November2014

Themkdircommandreturns0onsuccessand1onerror:

Message*

WhatisthebasicdifferencebetweentraditionalRDBMSandHadoop?
1.TraditionalRDBMSisusedfortransactionalsystemstoreportandarchivethedata,whereas
Hadoopisanapproachtostorehugeamountofdatainthedistributedfilesystemandprocessit.
2.RDBMSwillbeusefulwhenyouwanttoseekonerecordfromBigdata,whereas,Hadoopwillbe
usefulwhenyouwantBigdatainoneshotandperformanalysisonthatlater.

PostedbyPradeepKumarNandaat20:11 Nocomments:

Email*

PradeepKumar
Nanda
Viewmycomplete
profile

Whatisthe
basic
difference
between
traditional
...
Howthe
hadoop
shell
command
works
internally?
...
Whatareall
HDFSshell
commands
available?
1.co...
Howtoget
more
information
abouta
particular
ha...
Howtolistall
hadoopfile
system
commands
?bi...
Whydowe
need
Hadoop?
Everydaya
large
amounto...
Howanalysis
ofBigData
isusefulfor
organizati...
Whatarethe
threemain
charactersti
csof
Hadoop...

HowBigis
BigData?
Withtime,
data
volumeis...

Similarly,allhadoopshellcommandsareworkinginternally.

PostedbyPradeepKumarNandaat22:54 2comments:

WhatareallHDFSshellcommandsavailable?
1.copyFromLocal(CopyafileordirectoryfromLocaltoHDFS)
hadoopfscopyToLocal/home/training/Local/file1/home/training/hdfs
Note:Canbeusedforcopyingmultiplefiles,similarpatternfiles,allthefiles,adirectory
2.moveFromLocal(CopyafileordirectoryfromLocaltoHDFS)
hadoopfsmoveToLocal/home/training/Local/file1/home/training/hdfs
3.copyToLocal(CopyafileordirectoryfromHDFStoLocal)
hadoopfscopyToLocal/home/training/hdfs/file1/home/training/Local
4.moveToLocal(Notyetimplemented)
5.cp(copyafromonelocationtoanotherlocationinsideHDFS)
hadoopfscp/home/training/hdfs/file1/home/training/hdfs/hdfs1
6.mv(moveafromonelocationtoanotherlocationinsideHDFS)
hadoopfsmv/home/training/hdfs/file1/home/training/hdfs/hdfs1
7.put(SimilartocopyFromLocal)
hadoopfsput/home/training/Local/file1/home/training/hdfs
8.get(SimilartocopyToLocal)
hadoopfsget/home/training/hdfs/file1/home/training/Local
9.getmerge(writesmultiplefilecontentsintoasinglefileinLocalFilesystem)
hadoopfsget/home/training/hdfs/file1/home/training/hdfs/file2/home/training/Local
10.mkdir(Createadirectory)
hadoopfsmkdir/home/training/Dir1
11.touchz(cancreatenno:ofemptyfilesinHDFS)
hadoopfstouchz/home/training/hdfs/file1
12.rm(Removeafile)
hadoopfsrm/home/training/hdfs/file1
13.rmr(CanbeusedforremovingafileorDirectoryrecursively)
hadoopfsrmr/home/training/hdfs/file
hadoopfsrmr/home/training/hdfs/Dir1
Note:Canbeusedtoremovesimilarpatternfiles(*.sh,*.txtetc),allthefiles(*)
14.ls(Listsallthefiles&directories)
hadoopfsls/home/training/hdfs
15.ls|tailn(TailoptionwithList)
hadoopfsls/home/training/hdfs|tail10
16.ls|headn(headoptionwithList)
hadoopfsls/home/training/hdfs|head10
17.cat(Displaysthecontentofafile)
hadoopfscat/home/training/hdfs/file
18.text(Displaysthecontentofzippedfiles)
hadoopfstext/home/training/hdfs/file.gz
19.cat|tailn(Displaybottomnlinesofafile)
hadoopfscat/home/training/hdfs/file|tail10
20.cat|headn(Displaytopnlinesofafile)
hadoopfscat/home/training/hdfs/file|tail10
21.cat|wcl(Countstheno:oflinesinafile)

WhatisBIG
DATA?Big
Datais
nothingbut
anasso...
HowHDFS
interactwith
theFile
system
shellinter...
Whatis
HDFS?
HDFS
standsfor
Hadoop
Distributed
...
Whatis
Hadoop?
Hadoopis
awayof
organising
o...
Ourpeople
makeIT
Vision,IT
visionby
our
people...

hadoopfscat/user/mdpuser/hdfs10/file1|wcl
22.cat|wcw(Countstheno:ofwordsinafile)
hadoopfscat/user/mdpuser/hdfs10/file1|wcw
23.cat|wcc(Countstheno:ofCharactersinafile)
hadoopfscat/user/mdpuser/hdfs10/file1|wcc
24.du(DiskUsageofafileordirectory)
hadoopfsdu/home/training/hdfs
25.duh(formats&showsfileordirectorysizeinhumanreadableformat)
hadoopfsdu/home/training/hdfs
26.dus(showssummaryofthedirectoriesinsteadofeachfile)
hadoopfsdus/home/training/hdfs
27.df(Diskusageoftheentirefilesystem)
hadoopfsdf
O/P:
Filesystem

Size

Used

Available

Use%

hdfs://nameservice1 328040332591104 102783556870823 210750795833344 31%

28.dfh(Formats&showsinthehumanreadableformat)
hadoopfsdfh
O/P:
Filesystem

Size

Used Available Use%

hdfs://nameservice1 298.4T 93.5T 191.7T

31%

29.count(CountsalltheDirectories&Filesinthegivenpath)
hadoopfscount/home/training/hdfs
30.fsck(Tocheckfilesystemhealth)
hadoopfsck/home/training/hdfs
31.fsckfilesblocks(DisplayscorrespondingFiles&theirblocklevelinfo)
hadoopfsck/home/training/hdfsfilesblocks
32.fsckfilesblockslocations(Displaysfiles&blocklevelinfoincludingtheblocklocation)
33.hadoopfsck/home/training/hdfsfilesblocks=locations
34.setrep(usedtochangethereplicationfactorafileoradirectory)
hadoopfssetrep5/home/training/hdfs/file1
35.Controllingblocksizeatfilelevelwithoutchangingtheblocksizeinhdfssite.xml
HadoopfsDdfs.block.size=134217728putsource_pathdestination_path
36.Controllingreplicationatfilelevelirrespectiveofthedefaultreplicationsetto3
HadoopfsDdfs.replication=2putsource_pathdestination_path

PostedbyPradeepKumarNandaat22:15 Nocomments:

Howtogetmoreinformationaboutaparticularhadoopshellcommand?
usethehelpoption.
bin/hadoopfshelpls

PostedbyPradeepKumarNandaat22:07 Nocomments:

Howtolistallhadoopfilesystemcommands?
bin/hadoopfs

PostedbyPradeepKumarNandaat22:01 Nocomments:

Home

OlderPosts

Subscribeto:Posts(Atom)

[email protected].