4 * Licensed to the Apache Software Foundation (ASF) under one
5 * or more contributor license agreements. See the NOTICE file
6 * distributed with this work for additional information
7 * regarding copyright ownership. The ASF licenses this file
8 * to you under the Apache License, Version 2.0 (the
9 * "License"); you may not use this file except in compliance
10 * with the License. You may obtain a copy of the License at
12 * http://www.apache.org/licenses/LICENSE-2.0
14 * Unless required by applicable law or agreed to in writing, software
15 * distributed under the License is distributed on an "AS IS" BASIS,
16 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17 * See the License for the specific language governing permissions and
18 * limitations under the License.
32 <<quickstart,Quickstart>> will get you up and running on a single-node, standalone instance of HBase.
35 == Quick Start - Standalone HBase
37 This section describes the setup of a single-node standalone HBase.
38 A _standalone_ instance has all HBase daemons -- the Master, RegionServers,
39 and ZooKeeper -- running in a single JVM persisting to the local filesystem.
40 It is our most basic deploy profile. We will show you how
41 to create a table in HBase using the `hbase shell` CLI,
42 insert rows into the table, perform put and scan operations against the
43 table, enable or disable the table, and start and stop HBase.
45 Apart from downloading HBase, this procedure should take less than 10 minutes.
47 === JDK Version Requirements
49 HBase requires that a JDK be installed.
50 See <<java,Java>> for information about supported JDK versions.
52 === Get Started with HBase
54 .Procedure: Download, Configure, and Start HBase in Standalone Mode
55 . Choose a download site from this list of link:https://www.apache.org/dyn/closer.lua/hbase/[Apache Download Mirrors].
56 Click on the suggested top link.
57 This will take you to a mirror of _HBase Releases_.
58 Click on the folder named _stable_ and then download the binary file that looks like
59 _hbase-<version>-bin.tar.gz_.
61 . Extract the downloaded file and change to the newly-created directory.
64 $ tar xzvf hbase-<version>-bin.tar.gz
68 . Set the `JAVA_HOME` environment variable in _conf/hbase-env.sh_.
69 First, locate the installation of `java` on your machine. On Unix systems, you can use the
70 _whereis java_ command. Once you have the location, edit _conf/hbase-env.sh_ file, found inside
71 the extracted _hbase-<version>_ directory, uncomment the line starting with `#export JAVA_HOME=`,
72 and then set it to your Java installation path.
74 .Example extract from _conf/hbase-env.sh_ where `JAVA_HOME` is set
75 # Set environment variables here.
76 # The java implementation to use.
77 export JAVA_HOME=/usr/jdk64/jdk1.8.0_112
80 . Optionally set the <<hbase.tmp.dir,`hbase.tmp.dir`>> property in _conf/hbase-site.xml_.
81 At this time, you may consider changing the location on the local filesystem where HBase writes
82 its application data and the data written by its embedded ZooKeeper instance. By default, HBase
83 uses paths under <<hbase.tmp.dir,`hbase.tmp.dir`>> for these directories.
85 NOTE: On most systems, this is a path created under _/tmp_. Many system periodically delete the
86 contents of _/tmp_. If you start working with HBase in this way, and then return after the
87 cleanup operation takes place, you're likely to find strange errors. The following
88 configuration will place HBase's runtime data in a _tmp_ directory found inside the extracted
89 _hbase-<version>_ directory, where it will be safe from this periodic cleanup.
91 Open _conf/hbase-site.xml_ and paste the `<property>` tags between the empty `<configuration>`
94 .Example _hbase-site.xml_ for Standalone HBase
100 <name>hbase.tmp.dir</name>
107 You do not need to create the HBase _tmp_ directory; HBase will do this for you.
109 NOTE: When unconfigured, HBase uses <<hbase.tmp.dir,`hbase.tmp.dir`>> as a starting point for many
110 important configurations. Notable among them are <<hbase.rootdir,`hbase.rootdir`>>, the path under
111 which HBase stores its data. You can specify values for this configuration directly, as you'll see
112 in the subsequent sections.
114 NOTE: In this example, HBase is running on Hadoop's `LocalFileSystem`. That abstraction doesn't
115 provide the durability promises that HBase needs to operate safely. This is most likely acceptable
116 for local development and testing use cases. It is not appropriate for production deployments;
117 eventually you will lose data. Instead, ensure your production deployment sets
118 <<hbase.rootdir,`hbase.rootdir`>> to a durable `FileSystem` implementation.
120 . The _bin/start-hbase.sh_ script is provided as a convenient way to start HBase.
121 Issue the command, and if all goes well, a message is logged to standard output showing that HBase started successfully.
122 You can use the `jps` command to verify that you have one running process called `HMaster`.
123 In standalone mode HBase runs all daemons within this single JVM, i.e.
124 the HMaster, a single HRegionServer, and the ZooKeeper daemon.
125 Go to _http://localhost:16010_ to view the HBase Web UI.
127 NOTE: Java needs to be installed and available.
128 If you get an error indicating that Java is not installed,
129 but it is on your system, perhaps in a non-standard location,
130 edit the _conf/hbase-env.sh_ file and modify the `JAVA_HOME`
131 setting to point to the directory that contains _bin/java_ on your system.
135 .Procedure: Use HBase For the First Time
138 Connect to your running instance of HBase using the `hbase shell` command, located in the [path]_bin/_ directory of your HBase install.
139 In this example, some usage and version information that is printed when you start HBase Shell has been omitted.
140 The HBase Shell prompt ends with a `>` character.
148 . Display HBase Shell Help Text.
150 Type `help` and press Enter, to display some basic usage information for HBase Shell, as well as several example commands.
151 Notice that table names, rows, columns all must be enclosed in quote characters.
155 Use the `create` command to create a new table.
156 You must specify the table name and the ColumnFamily name.
159 hbase(main):001:0> create 'test', 'cf'
160 0 row(s) in 0.4170 seconds
162 => Hbase::Table - test
165 . List Information About your Table
167 Use the `list` command to confirm your table exists
170 hbase(main):002:0> list 'test'
173 1 row(s) in 0.0180 seconds
179 Now use the `describe` command to see details, including configuration defaults
182 hbase(main):003:0> describe 'test'
183 Table test is ENABLED
185 COLUMN FAMILIES DESCRIPTION
186 {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
187 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'f
188 alse', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE
194 . Put data into your table.
196 To put data into your table, use the `put` command.
199 hbase(main):003:0> put 'test', 'row1', 'cf:a', 'value1'
200 0 row(s) in 0.0850 seconds
202 hbase(main):004:0> put 'test', 'row2', 'cf:b', 'value2'
203 0 row(s) in 0.0110 seconds
205 hbase(main):005:0> put 'test', 'row3', 'cf:c', 'value3'
206 0 row(s) in 0.0100 seconds
209 Here, we insert three values, one at a time.
210 The first insert is at `row1`, column `cf:a`, with a value of `value1`.
211 Columns in HBase are comprised of a column family prefix, `cf` in this example, followed by a colon and then a column qualifier suffix, `a` in this case.
213 . Scan the table for all data at once.
215 One of the ways to get data from HBase is to scan.
216 Use the `scan` command to scan the table for data.
217 You can limit your scan, but for now, all data is fetched.
220 hbase(main):006:0> scan 'test'
222 row1 column=cf:a, timestamp=1421762485768, value=value1
223 row2 column=cf:b, timestamp=1421762491785, value=value2
224 row3 column=cf:c, timestamp=1421762496210, value=value3
225 3 row(s) in 0.0230 seconds
228 . Get a single row of data.
230 To get a single row of data at a time, use the `get` command.
233 hbase(main):007:0> get 'test', 'row1'
235 cf:a timestamp=1421762485768, value=value1
236 1 row(s) in 0.0350 seconds
241 If you want to delete a table or change its settings, as well as in some other situations, you need to disable the table first, using the `disable` command.
242 You can re-enable it using the `enable` command.
245 hbase(main):008:0> disable 'test'
246 0 row(s) in 1.1820 seconds
248 hbase(main):009:0> enable 'test'
249 0 row(s) in 0.1770 seconds
252 Disable the table again if you tested the `enable` command above:
255 hbase(main):010:0> disable 'test'
256 0 row(s) in 1.1820 seconds
261 To drop (delete) a table, use the `drop` command.
264 hbase(main):011:0> drop 'test'
265 0 row(s) in 0.1370 seconds
268 . Exit the HBase Shell.
270 To exit the HBase Shell and disconnect from your cluster, use the `quit` command.
271 HBase is still running in the background.
274 .Procedure: Stop HBase
275 . In the same way that the _bin/start-hbase.sh_ script is provided to conveniently start all HBase daemons, the _bin/stop-hbase.sh_ script stops them.
279 $ ./bin/stop-hbase.sh
280 stopping hbase....................
284 . After issuing the command, it can take several minutes for the processes to shut down.
285 Use the `jps` to be sure that the HMaster and HRegionServer processes are shut down.
287 The above has shown you how to start and stop a standalone instance of HBase.
288 In the next sections we give a quick overview of other modes of hbase deploy.
290 [[quickstart_pseudo]]
291 === Pseudo-Distributed Local Install
293 After working your way through the <<quickstart,quickstart>> using standalone mode, you can
294 re-configure HBase to run in pseudo-distributed mode. Pseudo-distributed mode means that HBase
295 still runs completely on a single host, but each HBase daemon (HMaster, HRegionServer, and
296 ZooKeeper) runs as a separate process. Previously in <<quickstart,standalone mode>>, all these
297 daemons ran in a single jvm process, and your data was stored under
298 <<hbase.tmp.dir,`hbase.tmp.dir`>>. In this walk-through, your data will be stored in in HDFS
299 instead, assuming you have HDFS available. This is optional; you can skip the HDFS configuration
300 to continue storing your data in the local filesystem.
302 .Hadoop Configuration
303 NOTE: This procedure assumes that you have configured Hadoop and HDFS on your local system and/or a
304 remote system, and that they are running and available. It also assumes you are using Hadoop 2.
306 link:https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html[Setting up a Single Node Cluster]
307 in the Hadoop documentation is a good starting point.
309 . Stop HBase if it is running.
311 If you have just finished <<quickstart,quickstart>> and HBase is still running, stop it.
312 This procedure will create a totally new directory where HBase will store its data, so any databases you created before will be lost.
316 Edit the _hbase-site.xml_ configuration.
317 First, add the following property which directs HBase to run in distributed mode, with one JVM instance per daemon.
323 <name>hbase.cluster.distributed</name>
328 Next, add a configuration for `hbase.rootdir` so that it points to the address of your HDFS instance, using the `hdfs:////` URI syntax.
329 In this example, HDFS is running on the localhost at port 8020.
335 <name>hbase.rootdir</name>
336 <value>hdfs://localhost:8020/hbase</value>
340 You do not need to create the directory in HDFS; HBase will do this for you.
341 If you create the directory, HBase will attempt to do a migration, which is not what you want.
343 Finally, remove the configuration for `hbase.tmp.dir`.
346 Use the _bin/start-hbase.sh_ command to start HBase.
347 If your system is configured correctly, the `jps` command should show the HMaster and HRegionServer processes running.
349 . Check the HBase directory in HDFS.
351 If everything worked correctly, HBase created its directory in HDFS.
352 In the configuration above, it is stored in _/hbase/_ on HDFS.
353 You can use the `hadoop fs` command in Hadoop's _bin/_ directory to list this directory.
357 $ ./bin/hadoop fs -ls /hbase
359 drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/.tmp
360 drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/WALs
361 drwxr-xr-x - hbase users 0 2014-06-25 18:48 /hbase/corrupt
362 drwxr-xr-x - hbase users 0 2014-06-25 18:58 /hbase/data
363 -rw-r--r-- 3 hbase users 42 2014-06-25 18:41 /hbase/hbase.id
364 -rw-r--r-- 3 hbase users 7 2014-06-25 18:41 /hbase/hbase.version
365 drwxr-xr-x - hbase users 0 2014-06-25 21:49 /hbase/oldWALs
368 . Create a table and populate it with data.
370 You can use the HBase Shell to create a table, populate it with data, scan and get values from it, using the same procedure as in <<shell_exercises,shell exercises>>.
372 . Start and stop a backup HBase Master (HMaster) server.
374 NOTE: Running multiple HMaster instances on the same hardware does not make sense in a production environment, in the same way that running a pseudo-distributed cluster does not make sense for production.
375 This step is offered for testing and learning purposes only.
377 The HMaster server controls the HBase cluster.
378 You can start up to 9 backup HMaster servers, which makes 10 total HMasters, counting the primary.
379 To start a backup HMaster, use the `local-master-backup.sh`.
380 For each backup master you want to start, add a parameter representing the port offset for that master.
381 Each HMaster uses two ports (16000 and 16010 by default). The port offset is added to these ports, so using an offset of 2, the backup HMaster would use ports 16002 and 16012.
382 The following command starts 3 backup servers using ports 16002/16012, 16003/16013, and 16005/16015.
386 $ ./bin/local-master-backup.sh start 2 3 5
389 To kill a backup master without killing the entire cluster, you need to find its process ID (PID). The PID is stored in a file with a name like _/tmp/hbase-USER-X-master.pid_.
390 The only contents of the file is the PID.
391 You can use the `kill -9` command to kill that PID.
392 The following command will kill the master with port offset 1, but leave the cluster running:
396 $ cat /tmp/hbase-testuser-1-master.pid |xargs kill -9
399 . Start and stop additional RegionServers
401 The HRegionServer manages the data in its StoreFiles as directed by the HMaster.
402 Generally, one HRegionServer runs per node in the cluster.
403 Running multiple HRegionServers on the same system can be useful for testing in pseudo-distributed mode.
404 The `local-regionservers.sh` command allows you to run multiple RegionServers.
405 It works in a similar way to the `local-master-backup.sh` command, in that each parameter you provide represents the port offset for an instance.
406 Each RegionServer requires two ports, and the default ports are 16020 and 16030.
407 Since HBase version 1.1.0, HMaster doesn't use region server ports, this leaves 10 ports (16020 to 16029 and 16030 to 16039) to be used for RegionServers.
408 For supporting additional RegionServers, set environment variables HBASE_RS_BASE_PORT and HBASE_RS_INFO_BASE_PORT to appropriate values before running script `local-regionservers.sh`.
409 e.g. With values 16200 and 16300 for base ports, 99 additional RegionServers can be supported, on a server.
410 The following command starts four additional RegionServers, running on sequential ports starting at 16022/16032 (base ports 16020/16030 plus 2).
414 $ .bin/local-regionservers.sh start 2 3 4 5
417 To stop a RegionServer manually, use the `local-regionservers.sh` command with the `stop` parameter and the offset of the server to stop.
420 $ .bin/local-regionservers.sh stop 3
425 You can stop HBase the same way as in the <<quickstart,quickstart>> procedure, using the _bin/stop-hbase.sh_ command.
428 [[quickstart_fully_distributed]]
429 === Advanced - Fully Distributed
431 In reality, you need a fully-distributed configuration to fully test HBase and to use it in real-world scenarios.
432 In a distributed configuration, the cluster contains multiple nodes, each of which runs one or more HBase daemon.
433 These include primary and backup Master instances, multiple ZooKeeper nodes, and multiple RegionServer nodes.
435 This advanced quickstart adds two more nodes to your cluster.
436 The architecture will be as follows:
438 .Distributed Cluster Demo Architecture
439 [cols="1,1,1,1", options="header"]
441 | Node Name | Master | ZooKeeper | RegionServer
442 | node-a.example.com | yes | yes | no
443 | node-b.example.com | backup | yes | yes
444 | node-c.example.com | no | yes | yes
447 This quickstart assumes that each node is a virtual machine and that they are all on the same network.
448 It builds upon the previous quickstart, <<quickstart_pseudo>>, assuming that the system you configured in that procedure is now `node-a`.
449 Stop HBase on `node-a` before continuing.
451 NOTE: Be sure that all the nodes have full access to communicate, and that no firewall rules are in place which could prevent them from talking to each other.
452 If you see any errors like `no route to host`, check your firewall.
454 [[passwordless.ssh.quickstart]]
455 .Procedure: Configure Passwordless SSH Access
457 `node-a` needs to be able to log into `node-b` and `node-c` (and to itself) in order to start the daemons.
458 The easiest way to accomplish this is to use the same username on all hosts, and configure password-less SSH login from `node-a` to each of the others.
460 . On `node-a`, generate a key pair.
462 While logged in as the user who will run HBase, generate a SSH key pair, using the following command:
469 If the command succeeds, the location of the key pair is printed to standard output.
470 The default name of the public key is _id_rsa.pub_.
472 . Create the directory that will hold the shared keys on the other nodes.
474 On `node-b` and `node-c`, log in as the HBase user and create a _.ssh/_ directory in the user's home directory, if it does not already exist.
475 If it already exists, be aware that it may already contain other keys.
477 . Copy the public key to the other nodes.
479 Securely copy the public key from `node-a` to each of the nodes, by using the `scp` or some other secure means.
480 On each of the other nodes, create a new file called _.ssh/authorized_keys_ _if it does
481 not already exist_, and append the contents of the _id_rsa.pub_ file to the end of it.
482 Note that you also need to do this for `node-a` itself.
485 $ cat id_rsa.pub >> ~/.ssh/authorized_keys
488 . Test password-less login.
490 If you performed the procedure correctly, you should not be prompted for a password when you SSH from `node-a` to either of the other nodes using the same username.
492 . Since `node-b` will run a backup Master, repeat the procedure above, substituting `node-b` everywhere you see `node-a`.
493 Be sure not to overwrite your existing _.ssh/authorized_keys_ files, but concatenate the new key onto the existing file using the `>>` operator rather than the `>` operator.
495 .Procedure: Prepare `node-a`
497 `node-a` will run your primary master and ZooKeeper processes, but no RegionServers. Stop the RegionServer from starting on `node-a`.
499 . Edit _conf/regionservers_ and remove the line which contains `localhost`. Add lines with the hostnames or IP addresses for `node-b` and `node-c`.
501 Even if you did want to run a RegionServer on `node-a`, you should refer to it by the hostname the other servers would use to communicate with it.
502 In this case, that would be `node-a.example.com`.
503 This enables you to distribute the configuration to each node of your cluster any hostname conflicts.
506 . Configure HBase to use `node-b` as a backup master.
508 Create a new file in _conf/_ called _backup-masters_, and add a new line to it with the hostname for `node-b`.
509 In this demonstration, the hostname is `node-b.example.com`.
511 . Configure ZooKeeper
513 In reality, you should carefully consider your ZooKeeper configuration.
514 You can find out more about configuring ZooKeeper in <<zookeeper,zookeeper>> section.
515 This configuration will direct HBase to start and manage a ZooKeeper instance on each node of the cluster.
517 On `node-a`, edit _conf/hbase-site.xml_ and add the following properties.
522 <name>hbase.zookeeper.quorum</name>
523 <value>node-a.example.com,node-b.example.com,node-c.example.com</value>
526 <name>hbase.zookeeper.property.dataDir</name>
527 <value>/usr/local/zookeeper</value>
531 . Everywhere in your configuration that you have referred to `node-a` as `localhost`, change the reference to point to the hostname that the other nodes will use to refer to `node-a`.
532 In these examples, the hostname is `node-a.example.com`.
534 .Procedure: Prepare `node-b` and `node-c`
536 `node-b` will run a backup master server and a ZooKeeper instance.
538 . Download and unpack HBase.
540 Download and unpack HBase to `node-b`, just as you did for the standalone and pseudo-distributed quickstarts.
542 . Copy the configuration files from `node-a` to `node-b`.and `node-c`.
544 Each node of your cluster needs to have the same configuration information.
545 Copy the contents of the _conf/_ directory to the _conf/_ directory on `node-b` and `node-c`.
548 .Procedure: Start and Test Your Cluster
549 . Be sure HBase is not running on any node.
551 If you forgot to stop HBase from previous testing, you will have errors.
552 Check to see whether HBase is running on any of your nodes by using the `jps` command.
553 Look for the processes `HMaster`, `HRegionServer`, and `HQuorumPeer`.
554 If they exist, kill them.
558 On `node-a`, issue the `start-hbase.sh` command.
559 Your output will be similar to that below.
564 node-c.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-c.example.com.out
565 node-a.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-a.example.com.out
566 node-b.example.com: starting zookeeper, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-zookeeper-node-b.example.com.out
567 starting master, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-master-node-a.example.com.out
568 node-c.example.com: starting regionserver, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-regionserver-node-c.example.com.out
569 node-b.example.com: starting regionserver, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-regionserver-node-b.example.com.out
570 node-b.example.com: starting master, logging to /home/hbuser/hbase-0.98.3-hadoop2/bin/../logs/hbase-hbuser-master-nodeb.example.com.out
573 ZooKeeper starts first, followed by the master, then the RegionServers, and finally the backup masters.
575 . Verify that the processes are running.
577 On each node of the cluster, run the `jps` command and verify that the correct processes are running on each server.
578 You may see additional Java processes running on your servers as well, if they are used for other purposes.
580 .`node-a` `jps` Output
588 .`node-b` `jps` Output
597 .`node-c` `jps` Output
605 .ZooKeeper Process Name
608 The `HQuorumPeer` process is a ZooKeeper instance which is controlled and started by HBase.
609 If you use ZooKeeper this way, it is limited to one instance per cluster node and is appropriate for testing only.
610 If ZooKeeper is run outside of HBase, the process is called `QuorumPeer`.
611 For more about ZooKeeper configuration, including using an external ZooKeeper instance with HBase, see <<zookeeper,zookeeper>> section.
614 . Browse to the Web UI.
619 In HBase newer than 0.98.x, the HTTP ports used by the HBase Web UI changed from 60010 for the
620 Master and 60030 for each RegionServer to 16010 for the Master and 16030 for the RegionServer.
623 If everything is set up correctly, you should be able to connect to the UI for the Master
624 `http://node-a.example.com:16010/` or the secondary master at `http://node-b.example.com:16010/`
626 If you can connect via `localhost` but not from another host, check your firewall rules.
627 You can see the web UI for each of the RegionServers at port 16030 of their IP addresses, or by
628 clicking their links in the web UI for the Master.
630 . Test what happens when nodes or services disappear.
632 With a three-node cluster you have configured, things will not be very resilient.
633 You can still test the behavior of the primary Master or a RegionServer by killing the associated processes and watching the logs.
638 The next chapter, <<configuration,configuration>>, gives more information about the different HBase run modes, system requirements for running HBase, and critical configuration areas for setting up a distributed HBase cluster.