Hadoop Introduction
Hadoop Introduction
================================================================
1. Introduction of hdfs
============================================ hdfs
===================================================================================
==
[[email protected] /home/out-rajpoot-b]# hdfs
Usage: hdfs [--config confdir] COMMAND
where COMMAND is one of:
dfs run a filesystem command on the file systems supported in
Hadoop.
namenode -format format the DFS filesystem
secondarynamenode run the DFS secondary namenode
namenode run the DFS namenode
journalnode run the DFS journalnode
zkfc run the ZK Failover Controller daemon
datanode run a DFS datanode
dfsadmin run a DFS admin client
diskbalancer Distributes data evenly among disks on a given node
haadmin run a DFS HA admin client
fsck run a DFS filesystem checking utility
balancer run a cluster balancing utility
jmxget get JMX exported values from NameNode or DataNode.
mover run a utility to move block replicas across
storage types
oiv apply the offline fsimage viewer to an fsimage
oiv_legacy apply the offline fsimage viewer to an legacy fsimage
oev apply the offline edits viewer to an edits file
fetchdt fetch a delegation token from the NameNode
getconf get config values from configuration
groups get the groups which users belong to
snapshotDiff diff two snapshots of a directory or diff the
current directory contents with a snapshot
lsSnapshottableDir list all snapshottable dirs owned by the current user
Use -help to see options
portmap run a portmap service
nfs3 run an NFS version 3 gateway
cacheadmin configure the HDFS cache
crypto configure HDFS encryption zones
storagepolicies list/get/set block storage policies
version print the version
===================================================================================
===================================================================================
====
air command
===================================================================================
===================================================================================
======
-root <repository-path>
Specifies the path to the EME technical
repository. The repository path may also be
specified via the AB_AIR_ROOT configuration
variable.
===================================================================================
===========================================================================
air - sibling command
===================================================================================
=================================================================================
air annotation - Lists category annotations
--------------------------------------------
air branch - Administers branches in the EME technical repository
These commands must be called with the repository shut down and can only
be executed locally. They are repository-wide.
-----------------------------------------------------------------------------------
-----
Usage: air project <subcommand>
These commands read information from the filesystem and add it to the
repository.
air sandbox
ab-compatibility <file> [-show] [-update] [-set <version>]
air sandbox
add <file> [-type <type>]
air sandbox
analysis-level <pset> {-show | -set <level>}
air sandbox
attach <sandbox-path>
<project-path>
[-published-eme published-name]
air sandbox checkin <sandbox-path> [<sandbox-path> ...]
[-dry-run]
[-force]
[-lock]
[-auto-add]
[-no-read-only]
[-noanalysis]
[-import-commons]
[-mimeobj-max-size <size limit>]
[-tag <tag-name> ]
[-comment <comment>]
air sandbox convert <path>
[-format <sandbox-format>]
air sandbox create <path>
[-template <path> | -type {public | private |
supporting}]
[-related-private <private sandbox>]
[-rel-loc <relative location>]
[-prefix <prefix>]
[-replace <s1> <s2>]
[-location <location>]
[-nodirectories]
air sandbox dataset - Administers reusable datasets
air sandbox detach [-q] [<path>]
air sandbox diff {<path> |
<path1> <path2> |
-version <v> <path> |
-version <v1> -version <v2> <path>}
[-terse | -verbose | -wizard]
[-ignore-param-order]
[-external-files]
[-ignore-dep-analysis]
[-text]
[-all [-no-diff] [-same]]
[-original-format]
[-dml]
[-m]
air sandbox eval <sandbox-path> [<sub-object>] [-details [-v]]
(<parameter> [-expression | -type | -show] |
-eval-expression <expression> [-position <parameter>])
[<parameter>=<value> ...]
air sandbox export <sandbox-path> [<arguments>]
air sandbox field-lineage <rpath>
[ [ -dep <var> [-upstream | -downstream] |
[-statements -ports -datasets] ] |
-outputs-to-inputs |
-inputs-to-outputs |
-datasets |
-vars ]
[-m]
air sandbox files <sandbox-path> [<arguments>]
air sandbox find [<path>] [-up | -down]
[-relative] [-project]
air sandbox find-text [-replace <replacement> [-dry-run]
[-lock [-break-locks]] [-force] |
[-linked [-all-linked]] [-external]]
[-regex | [-exact] [-ignore-case]]
[-file-limit <limit>] [-result-limit <limit>]
[[-object-type <type>] ...]
[[-attribute <attribute>] ...]
[[-task-type <type>] ...]
[[-path-filter <filter>] ...]
[[-component-type <type>] ...]
[[-component-label <label>] ...]
[[-task-label <label>] ...] [-param <name>]
[-all-file-types] [-m] <query> <path> [<path> ...]
air sandbox get-required-files [-force] [<path> ...]
air sandbox import <sandbox-path> [<arguments>]
[-dry-run] [-fast-dry-run]
[-force] [-lock]
[-auto-add] [-add-files <file> ...]
[-exclude-files <file> ...]
[-no-read-only]
[-files <file> ...]
[-find-required-files]
[-tag <tag-name> -move-tag {-tag-all |-tag-project|-tag-
exact}]
[-comment <comment>]
[-published-eme <published-name>]
[-analyze-referencing-files]
[-noanalysis]
[-nodiscover-datasets]
[-rollup-errors | -parsable-errors | -legacy-errors | -
individual-rollup-errors]
[-max-warnings [all | <num-warnings>]]
[-import-commons] [-quiet]
[-mimeobj-max-size <size limit>]
air sandbox index - Manage and query a sandbox index.
air sandbox info [-basic] <path> ...
air sandbox integrate <sandbox-path>
[-source-root <source root>]
[-source-branch <source branch>]
[-source-version <source version>]
[-source-project <source project path>]
[-base-root <base root>]
[-base-branch <base branch>]
[-base-version <base version>]
[-base-project <base project path>]
[-no-lock] [-dry-run]
[-files <file> ...]
[-quiet] [-leave-unresolved]
air sandbox lock { <path> | -parameters [<sandbox-path>] }
[-set | -release | -break | -reset]
[-nocheck]
[-force] [-unbreakable] [-manual-release]
air sandbox merge <base> <source> <target> <destination>
[-resolutions <url> | -b | -s | -t]
[-text-b | -text-s | -text-t | -text-none]
[-nopropagate] [-nolegend] [-ignore-ws]
[-diffs] [-mdiff] [-rdiff] [-force-text]
air sandbox move <from_file> <to_file>
air sandbox package-for-support <path> [<path>...]
[-name <package name>]
air sandbox parameter [<pset-spec>|<sandbox-spec>] <parameter>
[<interpretation>] [<value>] [<flag>]
[<pset-spec>|<sandbox-spec>] <parameter> -choice
<choice> ...
[<pset-spec>|<sandbox-spec>] <parameter> -dependent
<choice-parameter>
[<pset-spec>|<sandbox-spec>] <parameter> -case
<choice> [<interpretation>] <value>
[<pset-spec>|<sandbox-spec>] <parameter> -common
<common-sandbox-path> [-relative-common]
[<pset-spec>|<sandbox-spec>] <parameter> -comment
<comment-value>
[<pset-spec>|<sandbox-spec>] <parameter> -type <type-
value>
[<pset-spec>|<sandbox-spec>] <parameter> -dml-type
<dml-type-expression>
[<pset-spec>|<sandbox-spec>] -show <parameter>
[<pset-spec>|<sandbox-spec>] -remove <parameter>
[<pset-spec>|<sandbox-spec>] -eval <parameter>
[<parameter>=<value> ...]
[<pset-spec>|<sandbox-spec>] -eval-expression
<expression> [-position <parameter>]
[<pset-spec>|<sandbox-spec>] -expression <parameter>
air sandbox parameterize-path <scope-path> <path>
[-graph]
[-exported-only]
[-no-brace]
air sandbox precompile-graph <pset>
[-silent | -verbose] [-ignore-analysis-level]
[<positional-input-value> ...]
[-<input-parm-name> <value>] ...
air sandbox pset <pset-path>
[ -prototype <path> |
-print-prototypes |
-base <base-path> |
-insert-base <base-path> |
-replace-base <base-path> |
-remove-base |
-print-bases |
-split-runtime [-overwrite-existing] ]
air sandbox remove <file> [-force] [-no-delete]
air sandbox replace-subgraph -graph-path <filesystem_path>.mp
-path-to-subgraph-source <filesystem_path>.mp
-implementation-path <filesystem_path_impl>.mp
[-interface-path <filesystem_path>.mgi]
[-port-name-change <subgraph-port> <interface-
port> ...]
[-allow-overrides-of-non-interface-parameters]
[-walk-into-linked-subgraphs]
air sandbox resolve <sandbox-file>
[-merge | -take-edited | -take-source | -take-base | -
take-target]
[-b | -s | -t]
[-dry-run] [-force]
air sandbox revert [-basedir <sandbox>]
<path> ...
-latest
[-no-backup]
[-set <name> <value>] ...
[-common <rpath> <common-sandbox-dir>] ...
[-lock]
air sandbox rollback [<project-name>]
[-basedir <sandbox>]
-version <version>
[-force]
[-files <path> {path ...}]
air sandbox run <path>
[-resolve-from <sandbox>]
[-script-only] [-force-dynamic] [-force-fold]
[-background-mode] [-help-inputs] [-yarn-estimate-only]
[<positional-input-value> ...]
[-<input-parm-name> <value>] ...
[<plan-recovery-option>]
[<plan-debugging-options>]
[<other-plan-options>]
air sandbox scripts [<path>]
-inner <sbox-path>
<define_mode> <start_mode>
[-<param_name> <value>] ...
air sandbox search parameter <path> <paramname>
air sandbox show-common [<sandbox-path>]
[-exact [-param]]
air sandbox status <path>
[-R]
air sandbox update <path> ...
{-from <component-path> | -all
| -user-defined
| -top-level-ai-components-only}
[-interface]
[-no-lock]
[-continue]
[-force]
[-dry-run]
[-no-check]
[-verbose]
air sandbox validate-graph <path>
[-continuous]
[[ -analysis [-eme] [-field-warning]] |
[-complexity | -complexity-details]]
[-all]
[-rollup-errors | -m | -legacy-errors]
[-no-validation-extensions]
[-report-suppressions]
[-show-suppressed-validation-extension-errors]
[-interface-only]
-----------------------------------------------------------------------------------
-----
air system - Maintains system objects in an EME technical repository