HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942)

[hbase.git] / src / main / asciidoc / _chapters / architecture.adoc

////
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
////

= Architecture
:doctype: book
:numbered:
:toc: left
:icons: font
:experimental:
:toc: left
:source-language: java

[[arch.overview]]
== Overview

[[arch.overview.nosql]]
=== NoSQL?

HBase is a type of "NoSQL" database.
"NoSQL" is a general term meaning that the database isn't an RDBMS which supports SQL as its primary access language, but there are many types of NoSQL databases: BerkeleyDB is an example of a local NoSQL database, whereas HBase is very much a distributed database.
Technically speaking, HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc.

However, HBase has many features which supports both linear and modular scaling.
HBase clusters expand by adding RegionServers that are hosted on commodity class servers.
If a cluster expands from 10 to 20 RegionServers, for example, it doubles both in terms of storage and as well as processing capacity.
An RDBMS can scale well, but only up to a point - specifically, the size of a single database
server - and for the best performance requires specialized hardware and storage devices.
HBase features of note are:

* Strongly consistent reads/writes:  HBase is not an "eventually consistent" DataStore.
  This makes it very suitable for tasks such as high-speed counter aggregation.
* Automatic sharding: HBase tables are distributed on the cluster via regions, and regions are automatically split and re-distributed as your data grows.
* Automatic RegionServer failover
* Hadoop/HDFS Integration: HBase supports HDFS out of the box as its distributed file system.
* MapReduce: HBase supports massively parallelized processing via MapReduce for using HBase as both source and sink.
* Java Client API: HBase supports an easy to use Java API for programmatic access.
* Thrift/REST API: HBase also supports Thrift and REST for non-Java front-ends.
* Block Cache and Bloom Filters: HBase supports a Block Cache and Bloom Filters for high volume query optimization.
* Operational Management: HBase provides build-in web-pages for operational insight as well as JMX metrics.

[[arch.overview.when]]
=== When Should I Use HBase?

HBase isn't suitable for every problem.

First, make sure you have enough data.
If you have hundreds of millions or billions of rows, then HBase is a good candidate.
If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle.

Second, make sure you can live without all the extra features that an RDBMS provides (e.g., typed columns, secondary indexes, transactions, advanced query languages, etc.)  An application built against an RDBMS cannot be "ported" to HBase by simply changing a JDBC driver, for example.
Consider moving from an RDBMS to HBase as a complete redesign as opposed to a port.

Third, make sure you have enough hardware.
Even HDFS doesn't do well with anything less than 5 DataNodes (due to things such as HDFS block replication which has a default of 3), plus a NameNode.

HBase can run quite well stand-alone on a laptop - but this should be considered a development configuration only.

[[arch.overview.hbasehdfs]]
=== What Is The Difference Between HBase and Hadoop/HDFS?

link:https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html[HDFS] is a distributed file system that is well suited for the storage of large files.
Its documentation states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files.
HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables.
This can sometimes be a point of conceptual confusion.
HBase internally puts your data in indexed "StoreFiles" that exist on HDFS for high-speed lookups.
See the <<datamodel>> and the rest of this chapter for more information on how HBase achieves its goals.

[[arch.catalog]]
== Catalog Tables

The catalog table `hbase:meta` exists as an HBase table and is filtered out of the HBase shell's `list` command, but is in fact a table just like any other.

[[arch.catalog.meta]]
=== hbase:meta

The `hbase:meta` table (previously called `.META.`) keeps a list of all regions in the system, and the location of `hbase:meta` is stored in ZooKeeper.

The `hbase:meta` table structure is as follows:

.Key

* Region key of the format (`[table],[region start key],[region id]`)

.Values

* `info:regioninfo` (serialized link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HRegionInfo.html[HRegionInfo] instance for this region)
* `info:server` (server:port of the RegionServer containing this region)
* `info:serverstartcode` (start-time of the RegionServer process containing this region)

When a table is in the process of splitting, two other columns will be created, called `info:splitA` and `info:splitB`.
These columns represent the two daughter regions.
The values for these columns are also serialized HRegionInfo instances.
After the region has been split, eventually this row will be deleted.

.Note on HRegionInfo
[NOTE]
====
The empty key is used to denote table start and table end.
A region with an empty start key is the first region in a table.
If a region has both an empty start and an empty end key, it is the only region in the table
====

In the (hopefully unlikely) event that programmatic processing of catalog metadata
is required, see the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/RegionInfo.html#parseFrom-byte:A-[RegionInfo.parseFrom] utility.

[[arch.catalog.startup]]
=== Startup Sequencing

First, the location of `hbase:meta` is looked up in ZooKeeper.
Next, `hbase:meta` is updated with server and startcode values.

For information on region-RegionServer assignment, see <<regions.arch.assignment>>.

[[architecture.client]]
== Client

The HBase client finds the RegionServers that are serving the particular row range of interest.
It does this by querying the `hbase:meta` table.
See <<arch.catalog.meta>> for details.
After locating the required region(s), the client contacts the RegionServer serving that region, rather than going through the master, and issues the read or write request.
This information is cached in the client so that subsequent requests need not go through the lookup process.
Should a region be reassigned either by the master load balancer or because a RegionServer has died, the client will requery the catalog tables to determine the new location of the user region.

See <<master.runtime>> for more information about the impact of the Master on HBase Client communication.

Administrative functions are done via an instance of link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Admin.html[Admin]

[[client.connections]]
=== Cluster Connections

The API changed in HBase 1.0. For connection configuration information, see <<client_dependencies>>.

==== API as of HBase 1.0.0

It's been cleaned up and users are returned Interfaces to work against rather than particular types.
In HBase 1.0, obtain a `Connection` object from `ConnectionFactory` and thereafter, get from it instances of `Table`, `Admin`, and `RegionLocator` on an as-need basis.
When done, close the obtained instances.
Finally, be sure to cleanup your `Connection` instance before exiting.
`Connections` are heavyweight objects but thread-safe so you can create one for your application and keep the instance around.
`Table`, `Admin` and `RegionLocator` instances are lightweight.
Create as you go and then let go as soon as you are done by closing them.
See the link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/package-summary.html[Client Package Javadoc Description] for example usage of the new HBase 1.0 API.

==== API before HBase 1.0.0

Instances of `HTable` are the way to interact with an HBase cluster earlier than 1.0.0. _link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html[Table] instances are not thread-safe_. Only one thread can use an instance of Table at any given time.
When creating Table instances, it is advisable to use the same link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HBaseConfiguration[HBaseConfiguration] instance.
This will ensure sharing of ZooKeeper and socket instances to the RegionServers which is usually what you want.
For example, this is preferred:

[source,java]
----
HBaseConfiguration conf = HBaseConfiguration.create();
HTable table1 = new HTable(conf, "myTable");
HTable table2 = new HTable(conf, "myTable");
----

as opposed to this:

[source,java]
----
HBaseConfiguration conf1 = HBaseConfiguration.create();
HTable table1 = new HTable(conf1, "myTable");
HBaseConfiguration conf2 = HBaseConfiguration.create();
HTable table2 = new HTable(conf2, "myTable");
----

For more information about how connections are handled in the HBase client, see link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/ConnectionFactory.html[ConnectionFactory].

[[client.connection.pooling]]
===== Connection Pooling

For applications which require high-end multithreaded access (e.g., web-servers or application servers that may serve many application threads in a single JVM), you can pre-create a `Connection`, as shown in the following example:

.Pre-Creating a `Connection`
====
[source,java]
----
// Create a connection to the cluster.
Configuration conf = HBaseConfiguration.create();
try (Connection connection = ConnectionFactory.createConnection(conf);
     Table table = connection.getTable(TableName.valueOf(tablename))) {
  // use table as needed, the table returned is lightweight
}
----
====

.`HTablePool` is Deprecated
[WARNING]
====
Previous versions of this guide discussed `HTablePool`, which was deprecated in HBase 0.94, 0.95, and 0.96, and removed in 0.98.1, by link:https://issues.apache.org/jira/browse/HBASE-6580[HBASE-6580], or `HConnection`, which is deprecated in HBase 1.0 by `Connection`.
Please use link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Connection.html[Connection] instead.
====

[[client.writebuffer]]
=== WriteBuffer and Batch Methods

In HBase 1.0 and later, link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable] is deprecated in favor of link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html[Table]. `Table` does not use autoflush. To do buffered writes, use the BufferedMutator class.

In HBase 2.0 and later, link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HTable.html[HTable] does not use BufferedMutator to execute the ``Put`` operation. Refer to link:https://issues.apache.org/jira/browse/HBASE-18500[HBASE-18500] for more information.

For additional information on write durability, review the link:/acid-semantics.html[ACID semantics] page.

For fine-grained control of batching of ``Put``s or ``Delete``s, see the link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Table.html#batch-java.util.List-java.lang.Object:A-[batch] methods on Table.

[[async.client]]
=== Asynchronous Client ===

It is a new API introduced in HBase 2.0 which aims to provide the ability to access HBase asynchronously.

You can obtain an `AsyncConnection` from `ConnectionFactory`, and then get a asynchronous table instance from it to access HBase. When done, close the `AsyncConnection` instance(usually when your program exits).

For the asynchronous table, most methods have the same meaning with the old `Table` interface, expect that the return value is wrapped with a CompletableFuture usually. We do not have any buffer here so there is no close method for asynchronous table, you do not need to close it. And it is thread safe.

There are several differences for scan:

* There is still a `getScanner` method which returns a `ResultScanner`. You can use it in the old way and it works like the old `ClientAsyncPrefetchScanner`.
* There is a `scanAll` method which will return all the results at once. It aims to provide a simpler way for small scans which you want to get the whole results at once usually.
* The Observer Pattern. There is a scan method which accepts a `ScanResultConsumer` as a parameter. It will pass the results to the consumer.

Notice that `AsyncTable` interface is templatized. The template parameter specifies the type of `ScanResultConsumerBase` used by scans, which means the observer style scan APIs are different. The two types of scan consumers are - `ScanResultConsumer` and `AdvancedScanResultConsumer`.

`ScanResultConsumer` needs a separate thread pool which is used to execute the callbacks registered to the returned CompletableFuture. Because the use of separate thread pool frees up RPC threads, callbacks are free to do anything. Use this if the callbacks are not quick, or when in doubt.

`AdvancedScanResultConsumer` executes callbacks inside the framework thread. It is not allowed to do time consuming work in the callbacks else it will likely block the framework threads and cause very bad performance impact. As its name, it is designed for advanced users who want to write high performance code. See `org.apache.hadoop.hbase.client.example.HttpProxyExample` for how to write fully asynchronous code with it.

[[async.admin]]
=== Asynchronous Admin ===

You can obtain an `AsyncConnection` from `ConnectionFactory`, and then get a `AsyncAdmin` instance from it to access HBase. Notice that there are two `getAdmin` methods to get a `AsyncAdmin` instance. One method has one extra thread pool parameter which is used to execute callbacks. It is designed for normal users. Another method doesn't need a thread pool and all the callbacks are executed inside the framework thread so it is not allowed to do time consuming works in the callbacks. It is designed for advanced users.

The default `getAdmin` methods will return a `AsyncAdmin` instance which use default configs. If you want to customize some configs, you can use `getAdminBuilder` methods to get a `AsyncAdminBuilder` for creating `AsyncAdmin` instance. Users are free to only set the configs they care about to create a new `AsyncAdmin` instance.

For the `AsyncAdmin` interface, most methods have the same meaning with the old `Admin` interface, expect that the return value is wrapped with a CompletableFuture usually.

For most admin operations, when the returned CompletableFuture is done, it means the admin operation has also been done. But for compact operation, it only means the compact request was sent to HBase and may need some time to finish the compact operation. For `rollWALWriter` method, it only means the rollWALWriter request was sent to the region server and may need some time to finish the `rollWALWriter` operation.

For region name, we only accept `byte[]` as the parameter type and it may be a full region name or a encoded region name. For server name, we only accept `ServerName` as the parameter type. For table name, we only accept `TableName` as the parameter type. For `list*` operations, we only accept `Pattern` as the parameter type if you want to do regex matching.

[[client.external]]
=== External Clients

Information on non-Java clients and custom protocols is covered in <<external_apis>>

[[client.masterregistry]]

=== Master Registry (new as of 2.3.0)
Client internally works with a _connection registry_ to fetch the metadata needed by connections.
This connection registry implementation is responsible for fetching the following metadata.

* Active master address
* Current meta region(s) locations
* Cluster ID (unique to this cluster)

This information is needed as a part of various client operations like connection set up, scans,
gets, etc. Traditionally, the connection registry implementation has been based on ZooKeeper as the
source of truth and clients fetched the metadata directly from the ZooKeeper quorum. HBase 2.3.0
introduces a new connection registry implementation based on direct communication with the Masters.
With this implementation, clients now fetch required metadata via master RPC end points instead of
maintaining connections to ZooKeeper. This change was done for the following reasons.

* Reduce load on ZooKeeper since that is critical for cluster operation.
* Holistic client timeout and retry configurations since the new registry brings all the client
operations under HBase rpc framework.
* Remove the ZooKeeper client dependency on HBase client library.

This means:

* At least a single active or stand by master is needed for cluster connection setup. Refer to
<<master.runtime>> for more details.
* Master can be in a critical path of read/write operations, especially if the client metadata cache
is empty or stale.
* There is higher connection load on the masters that before since the clients talk directly to
HMasters instead of ZooKeeper ensemble`

To reduce hot-spotting on a single master, all the masters (active & stand-by) expose the needed
service to fetch the connection metadata. This lets the client connect to any master (not just active).
Both ZooKeeper-based and Master-based connection registry implementations are available in 2.3+. For
2.3 and earlier, the ZooKeeper-based implementation remains the default configuration.
The Master-based implementation becomes the default in 3.0.0.

Change the connection registry implementation by updating the value configured for
`hbase.client.registry.impl`. To explicitly enable the ZooKeeper-based registry, use

[source, xml]
<property>
    <name>hbase.client.registry.impl</name>
    <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
 </property>

To explicitly enable the Master-based registry, use

[source, xml]
<property>
    <name>hbase.client.registry.impl</name>
    <value>org.apache.hadoop.hbase.client.MasterRegistry</value>
 </property>

==== MasterRegistry RPC hedging

MasterRegistry implements hedging of connection registry RPCs across active and stand-by masters.
This lets the client make the same request to multiple servers and which ever responds first is
returned back to the client immediately. This improves performance, especially when a subset of
servers are under load. The hedging fan out size is configurable, meaning the number of requests
that are hedged in a single attempt, using the configuration key
_hbase.client.master_registry.hedged.fanout_ in the client configuration. It defaults to 2. With
this default, the RPCs are tried in batches of 2. The hedging policy is still primitive and does not
adapt to any sort of live rpc performance metrics.

==== Additional Notes

* Clients hedge the requests in a randomized order to avoid hot-spotting a single master.
* Cluster internal connections (masters &lt;-&gt; regionservers) still use ZooKeeper based connection
registry.
* Cluster internal state is still tracked in Zookeeper, hence ZK availability requirements are same
as before.
* Inter cluster replication still uses ZooKeeper based connection registry to simplify configuration
management.

For more implementation details, please refer to the https://github.com/apache/hbase/tree/master/dev-support/design-docs[design doc] and
https://issues.apache.org/jira/browse/HBASE-18095[HBASE-18095].

'''
NOTE: (Advanced) In case of any issues with the master based registry, use the following
configuration to fallback to the ZooKeeper based connection registry implementation.
[source, xml]
<property>
    <name>hbase.client.registry.impl</name>
    <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
 </property>

[[client.filter]]
== Client Request Filters

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html[Get] and link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html[Scan] instances can be optionally configured with link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/Filter.html[filters] which are applied on the RegionServer.

Filters can be confusing because there are many different types, and it is best to approach them by understanding the groups of Filter functionality.

[[client.filter.structural]]
=== Structural

Structural Filters contain other Filters.

[[client.filter.structural.fl]]
==== FilterList

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/FilterList.html[FilterList] represents a list of Filters with a relationship of `FilterList.Operator.MUST_PASS_ALL` or `FilterList.Operator.MUST_PASS_ONE` between the Filters.
The following example shows an 'or' between two Filters (checking for either 'my value' or 'my other value' on the same attribute).

[source,java]
----
FilterList list = new FilterList(FilterList.Operator.MUST_PASS_ONE);
SingleColumnValueFilter filter1 = new SingleColumnValueFilter(
  cf,
  column,
  CompareOperator.EQUAL,
  Bytes.toBytes("my value")
  );
list.add(filter1);
SingleColumnValueFilter filter2 = new SingleColumnValueFilter(
  cf,
  column,
  CompareOperator.EQUAL,
  Bytes.toBytes("my other value")
  );
list.add(filter2);
scan.setFilter(list);
----

[[client.filter.cv]]
=== Column Value

[[client.filter.cv.scvf]]
==== SingleColumnValueFilter

A SingleColumnValueFilter (see:
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.html)
can be used to test column values for equivalence (`CompareOperaor.EQUAL`),
inequality (`CompareOperaor.NOT_EQUAL`), or ranges (e.g., `CompareOperaor.GREATER`). The following is an
example of testing equivalence of a column to a String value "my value"...

[source,java]
----
SingleColumnValueFilter filter = new SingleColumnValueFilter(
  cf,
  column,
  CompareOperaor.EQUAL,
  Bytes.toBytes("my value")
  );
scan.setFilter(filter);
----

[[client.filter.cv.cvf]]
==== ColumnValueFilter

Introduced in HBase-2.0.0 version as a complementation of SingleColumnValueFilter, ColumnValueFilter
gets matched cell only, while SingleColumnValueFilter gets the entire row
(has other columns and values) to which the matched cell belongs. Parameters of constructor of
ColumnValueFilter are the same as SingleColumnValueFilter.
[source,java]
----
ColumnValueFilter filter = new ColumnValueFilter(
  cf,
  column,
  CompareOperaor.EQUAL,
  Bytes.toBytes("my value")
  );
scan.setFilter(filter);
----

Note. For simple query like "equals to a family:qualifier:value", we highly recommend to use the
following way instead of using SingleColumnValueFilter or ColumnValueFilter:
[source,java]
----
Scan scan = new Scan();
scan.addColumn(Bytes.toBytes("family"), Bytes.toBytes("qualifier"));
ValueFilter vf = new ValueFilter(CompareOperator.EQUAL,
  new BinaryComparator(Bytes.toBytes("value")));
scan.setFilter(vf);
...
----
This scan will restrict to the specified column 'family:qualifier', avoiding scan of unrelated
families and columns, which has better performance, and `ValueFilter` is the condition used to do
the value filtering.

But if query is much more complicated beyond this book, then please make your good choice case by case.

[[client.filter.cvp]]
=== Column Value Comparators

There are several Comparator classes in the Filter package that deserve special mention.
These Comparators are used in concert with other Filters, such as <<client.filter.cv.scvf>>.

[[client.filter.cvp.rcs]]
==== RegexStringComparator

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/RegexStringComparator.html[RegexStringComparator] supports regular expressions for value comparisons.

[source,java]
----
RegexStringComparator comp = new RegexStringComparator("my.");   // any value that starts with 'my'
SingleColumnValueFilter filter = new SingleColumnValueFilter(
  cf,
  column,
  CompareOperaor.EQUAL,
  comp
  );
scan.setFilter(filter);
----

See the Oracle JavaDoc for link:http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html[supported RegEx patterns in Java].

[[client.filter.cvp.substringcomparator]]
==== SubstringComparator

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html[SubstringComparator] can be used to determine if a given substring exists in a value.
The comparison is case-insensitive.

[source,java]
----

SubstringComparator comp = new SubstringComparator("y val");   // looking for 'my value'
SingleColumnValueFilter filter = new SingleColumnValueFilter(
  cf,
  column,
  CompareOperaor.EQUAL,
  comp
  );
scan.setFilter(filter);
----

[[client.filter.cvp.bfp]]
==== BinaryPrefixComparator

See link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/BinaryPrefixComparator.html[BinaryPrefixComparator].

[[client.filter.cvp.bc]]
==== BinaryComparator

See link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/BinaryComparator.html[BinaryComparator].

[[client.filter.cvp.bcc]]
==== BinaryComponentComparator

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/BinaryComponentComparator.html[BinaryComponentComparator] can be used to compare specific value at specific location with in the cell value. The comparison can be done for both ascii and binary data.

[source,java]
----
byte[] partialValue = Bytes.toBytes("partial_value");
    int partialValueOffset =
    Filter partialValueFilter = new ValueFilter(CompareFilter.CompareOp.GREATER,
            new BinaryComponentComparator(partialValue,partialValueOffset));
----
See link:https://issues.apache.org/jira/browse/HBASE-22969[HBASE-22969] for other use cases and details.

[[client.filter.kvm]]
=== KeyValue Metadata

As HBase stores data internally as KeyValue pairs, KeyValue Metadata Filters evaluate the existence of keys (i.e., ColumnFamily:Column qualifiers) for a row, as opposed to values the previous section.

[[client.filter.kvm.ff]]
==== FamilyFilter

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/FamilyFilter.html[FamilyFilter] can be used to filter on the ColumnFamily.
It is generally a better idea to select ColumnFamilies in the Scan than to do it with a Filter.

[[client.filter.kvm.qf]]
==== QualifierFilter

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/QualifierFilter.html[QualifierFilter] can be used to filter based on Column (aka Qualifier) name.

[[client.filter.kvm.cpf]]
==== ColumnPrefixFilter

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.html[ColumnPrefixFilter] can be used to filter based on the lead portion of Column (aka Qualifier) names.

A ColumnPrefixFilter seeks ahead to the first column matching the prefix in each row and for each involved column family.
It can be used to efficiently get a subset of the columns in very wide rows.

Note: The same column qualifier can be used in different column families.
This filter returns all matching columns.

Example: Find all columns in a row and family that start with "abc"

[source,java]
----
Table t = ...;
byte[] row = ...;
byte[] family = ...;
byte[] prefix = Bytes.toBytes("abc");
Scan scan = new Scan(row, row); // (optional) limit to one row
scan.addFamily(family); // (optional) limit to one family
Filter f = new ColumnPrefixFilter(prefix);
scan.setFilter(f);
scan.setBatch(10); // set this if there could be many columns returned
ResultScanner rs = t.getScanner(scan);
for (Result r = rs.next(); r != null; r = rs.next()) {
  for (Cell cell : result.listCells()) {
    // each cell represents a column
  }
}
rs.close();
----

[[client.filter.kvm.mcpf]]
==== MultipleColumnPrefixFilter

link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.html[MultipleColumnPrefixFilter] behaves like ColumnPrefixFilter but allows specifying multiple prefixes.

Like ColumnPrefixFilter, MultipleColumnPrefixFilter efficiently seeks ahead to the first column matching the lowest prefix and also seeks past ranges of columns between prefixes.
It can be used to efficiently get discontinuous sets of columns from very wide rows.

Example: Find all columns in a row and family that start with "abc" or "xyz"

[source,java]
----
Table t = ...;
byte[] row = ...;
byte[] family = ...;
byte[][] prefixes = new byte[][] {Bytes.toBytes("abc"), Bytes.toBytes("xyz")};
Scan scan = new Scan(row, row); // (optional) limit to one row
scan.addFamily(family); // (optional) limit to one family
Filter f = new MultipleColumnPrefixFilter(prefixes);
scan.setFilter(f);
scan.setBatch(10); // set this if there could be many columns returned
ResultScanner rs = t.getScanner(scan);
for (Result r = rs.next(); r != null; r = rs.next()) {
  for (Cell cell : result.listCells()) {
    // each cell represents a column
  }
}
rs.close();
----

[[client.filter.kvm.crf]]
==== ColumnRangeFilter

A link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/ColumnRangeFilter.html[ColumnRangeFilter] allows efficient intra row scanning.

A ColumnRangeFilter can seek ahead to the first matching column for each involved column family.
It can be used to efficiently get a 'slice' of the columns of a very wide row.
i.e.
you have a million columns in a row but you only want to look at columns bbbb-bbdd.

Note: The same column qualifier can be used in different column families.
This filter returns all matching columns.

Example: Find all columns in a row and family between "bbbb" (inclusive) and "bbdd" (inclusive)

[source,java]
----
Table t = ...;
byte[] row = ...;
byte[] family = ...;
byte[] startColumn = Bytes.toBytes("bbbb");
byte[] endColumn = Bytes.toBytes("bbdd");
Scan scan = new Scan(row, row); // (optional) limit to one row
scan.addFamily(family); // (optional) limit to one family
Filter f = new ColumnRangeFilter(startColumn, true, endColumn, true);
scan.setFilter(f);
scan.setBatch(10); // set this if there could be many columns returned
ResultScanner rs = t.getScanner(scan);
for (Result r = rs.next(); r != null; r = rs.next()) {
  for (Cell cell : result.listCells()) {
    // each cell represents a column
  }
}
rs.close();
----

Note:  Introduced in HBase 0.92

[[client.filter.row]]
=== RowKey

[[client.filter.row.rf]]
==== RowFilter

It is generally a better idea to use the startRow/stopRow methods on Scan for row selection, however link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/RowFilter.html[RowFilter] can also be used.

You can supplement a scan (both bounded and unbounded) with RowFilter constructed from link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/BinaryComponentComparator.html[BinaryComponentComparator] for further filtering out or filtering in rows. See link:https://issues.apache.org/jira/browse/HBASE-22969[HBASE-22969] for use cases and other details.

[[client.filter.utility]]
=== Utility

[[client.filter.utility.fkof]]
==== FirstKeyOnlyFilter

This is primarily used for rowcount jobs.
See link:https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.html[FirstKeyOnlyFilter].

[[architecture.master]]
== Master

`HMaster` is the implementation of the Master Server.
The Master server is responsible for monitoring all RegionServer instances in the cluster, and is the interface for all metadata changes.
In a distributed cluster, the Master typically runs on the <<arch.hdfs.nn>>.
J Mohamed Zahoor goes into some more detail on the Master Architecture in this blog posting, link:http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/[HBase HMaster Architecture ].

[[master.startup]]
=== Startup Behavior

If run in a multi-Master environment, all Masters compete to run the cluster.
If the active Master loses its lease in ZooKeeper (or the Master shuts down), then the remaining Masters jostle to take over the Master role.

[[master.runtime]]
=== Runtime Impact

A common dist-list question involves what happens to an HBase cluster when the Master goes down. This information has changed starting 3.0.0.

==== Up until releases 2.x.y
Because the HBase client talks directly to the RegionServers, the cluster can still function in a "steady state". Additionally, per <<arch.catalog>>, `hbase:meta` exists as an HBase table and is not resident in the Master.
However, the Master controls critical functions such as RegionServer failover and completing region splits.
So while the cluster can still run for a short time without the Master, the Master should be restarted as soon as possible.

==== Staring release 3.0.0
As mentioned in section <<client.masterregistry>>, the default connection registry for clients is now based on master rpc end points. Hence the requirements for
masters' uptime are even tighter starting this release.

- At least one active or stand by master is needed for a connection set up, unlike before when all the clients needed was a ZooKeeper ensemble.
- Master is now in critical path for read/write operations. For example, if the meta region bounces off to a different region server, clients
need master to fetch the new locations. Earlier this was done by fetching this information directly from ZooKeeper.
- Masters will now have higher connection load than before. So, the server side configuration might need adjustment depending on the load.

Overall, the master uptime requirements, when this feature is enabled, are even higher for the client operations to go through.

[[master.api]]
=== Interface

The methods exposed by `HMasterInterface` are primarily metadata-oriented methods:

* Table (createTable, modifyTable, removeTable, enable, disable)
* ColumnFamily (addColumn, modifyColumn, removeColumn)
* Region (move, assign, unassign) For example, when the `Admin` method `disableTable` is invoked, it is serviced by the Master server.

[[master.processes]]
=== Processes

The Master runs several background threads:

[[master.processes.loadbalancer]]
==== LoadBalancer

Periodically, and when there are no regions in transition, a load balancer will run and move regions around to balance the cluster's load.
See <<balancer_config>> for configuring this property.

See <<regions.arch.assignment>> for more information on region assignment.

[[master.processes.catalog]]
==== CatalogJanitor

Periodically checks and cleans up the `hbase:meta` table.
See <<arch.catalog.meta>> for more information on the meta table.

[[master.wal]]
=== MasterProcWAL

_MasterProcWAL is replaced in hbase-2.3.0 by an alternate Procedure Store implementation; see
<<in-master-procedure-store-region>>. This section pertains to hbase-2.0.0 through hbase-2.2.x_
 
HMaster records administrative operations and their running states, such as the handling of a crashed server,
table creation, and other DDLs, into a Procedure Store. The Procedure Store WALs are stored under the
MasterProcWALs directory. The Master WALs are not like RegionServer WALs. Keeping up the Master WAL allows
us to run a state machine that is resilient across Master failures. For example, if a HMaster was in the
middle of creating a table encounters an issue and fails, the next active HMaster can take up where
the previous left off and carry the operation to completion. Since hbase-2.0.0, a
new AssignmentManager (A.K.A AMv2) was introduced and the HMaster handles region assignment
operations, server crash processing, balancing, etc., all via AMv2 persisting all state and
transitions into MasterProcWALs rather than up into ZooKeeper, as we do in hbase-1.x.

See <<amv2>> (and <<pv2>> for its basis) if you would like to learn more about the new
AssignmentManager.

[[master.wal.conf]]
==== Configurations for MasterProcWAL
Here are the list of configurations that effect MasterProcWAL operation.
You should not have to change your defaults.

[[hbase.procedure.store.wal.periodic.roll.msec]]
*`hbase.procedure.store.wal.periodic.roll.msec`*::
+
.Description
Frequency of generating a new WAL
+
.Default
`1h (3600000 in msec)`

[[hbase.procedure.store.wal.roll.threshold]]
*`hbase.procedure.store.wal.roll.threshold`*::
+
.Description
Threshold in size before the WAL rolls. Every time the WAL reaches this size or the above period, 1 hour, passes since last log roll, the HMaster will generate a new WAL.
+
.Default
`32MB (33554432 in byte)`

[[hbase.procedure.store.wal.warn.threshold]]
*`hbase.procedure.store.wal.warn.threshold`*::
+
.Description
If the number of WALs goes beyond this threshold, the following message should appear in the HMaster log with WARN level when rolling.

 procedure WALs count=xx above the warning threshold 64. check running procedures to see if something is stuck.

+
.Default
`64`

[[hbase.procedure.store.wal.max.retries.before.roll]]
*`hbase.procedure.store.wal.max.retries.before.roll`*::
+
.Description
Max number of retry when syncing slots (records) to its underlying storage, such as HDFS. Every attempt, the following message should appear in the HMaster log.

 unable to sync slots, retry=xx

+
.Default
`3`

[[hbase.procedure.store.wal.sync.failure.roll.max]]
*`hbase.procedure.store.wal.sync.failure.roll.max`*::
+
.Description
After the above 3 retrials, the log is rolled and the retry count is reset to 0, thereon a new set of retrial starts. This configuration controls the max number of attempts of log rolling upon sync failure. That is, HMaster is allowed to fail to sync 9 times in total. Once it exceeds, the following log should appear in the HMaster log.

 Sync slots after log roll failed, abort.
+
.Default
`3`

[[regionserver.arch]]
== RegionServer

`HRegionServer` is the RegionServer implementation.
It is responsible for serving and managing regions.
In a distributed cluster, a RegionServer runs on a <<arch.hdfs.dn>>.

[[regionserver.arch.api]]
=== Interface

The methods exposed by `HRegionRegionInterface` contain both data-oriented and region-maintenance methods:

* Data (get, put, delete, next, etc.)
* Region (splitRegion, compactRegion, etc.) For example, when the `Admin` method `majorCompact` is invoked on a table, the client is actually iterating through all regions for the specified table and requesting a major compaction directly to each region.

[[regionserver.arch.processes]]
=== Processes

The RegionServer runs a variety of background threads:

[[regionserver.arch.processes.compactsplit]]
==== CompactSplitThread

Checks for splits and handle minor compactions.

[[regionserver.arch.processes.majorcompact]]
==== MajorCompactionChecker

Checks for major compactions.

[[regionserver.arch.processes.memstore]]
==== MemStoreFlusher

Periodically flushes in-memory writes in the MemStore to StoreFiles.

[[regionserver.arch.processes.log]]
==== LogRoller

Periodically checks the RegionServer's WAL.

=== Coprocessors

Coprocessors were added in 0.92.
There is a thorough link:https://blogs.apache.org/hbase/entry/coprocessor_introduction[Blog Overview of CoProcessors] posted.
Documentation will eventually move to this reference guide, but the blog is the most current information available at this time.

[[block.cache]]
=== Block Cache

HBase provides two different BlockCache implementations to cache data read from HDFS:
the default on-heap `LruBlockCache` and the `BucketCache`, which is (usually) off-heap.
This section discusses benefits and drawbacks of each implementation, how to choose the
appropriate option, and configuration options for each.

.Block Cache Reporting: UI
[NOTE]
====
See the RegionServer UI for detail on caching deploy.
See configurations, sizings, current usage, time-in-the-cache, and even detail on block counts and types.
====

==== Cache Choices

`LruBlockCache` is the original implementation, and is entirely within the Java heap.
`BucketCache` is optional and mainly intended for keeping block cache data off-heap, although `BucketCache` can also be a file-backed cache.
 In file-backed we can either use it in the file mode or the mmaped mode.
 We also have pmem mode where the bucket cache resides on the persistent memory device.

When you enable BucketCache, you are enabling a two tier caching system. We used to describe the
tiers as "L1" and "L2" but have deprecated this terminology as of hbase-2.0.0. The "L1" cache referred to an
instance of LruBlockCache and "L2" to an off-heap BucketCache. Instead, when BucketCache is enabled,
all DATA blocks are kept in the BucketCache tier and meta blocks -- INDEX and BLOOM blocks -- are on-heap in the `LruBlockCache`.
Management of these two tiers and the policy that dictates how blocks move between them is done by `CombinedBlockCache`.

[[cache.configurations]]
==== General Cache Configurations

Apart from the cache implementation itself, you can set some general configuration options to control how the cache performs.
See link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/hfile/CacheConfig.html[CacheConfig].
After setting any of these options, restart or rolling restart your cluster for the configuration to take effect.
Check logs for errors or unexpected behavior.

See also <<blockcache.prefetch>>, which discusses a new option introduced in link:https://issues.apache.org/jira/browse/HBASE-9857[HBASE-9857].

[[block.cache.design]]
==== LruBlockCache Design

The LruBlockCache is an LRU cache that contains three levels of block priority to allow for scan-resistance and in-memory ColumnFamilies:

* Single access priority: The first time a block is loaded from HDFS it normally has this priority and it will be part of the first group to be considered during evictions.
  The advantage is that scanned blocks are more likely to get evicted than blocks that are getting more usage.
* Multi access priority: If a block in the previous priority group is accessed again, it upgrades to this priority.
  It is thus part of the second group considered during evictions.
* In-memory access priority: If the block's family was configured to be "in-memory", it will be part of this priority disregarding the number of times it was accessed.
  Catalog tables are configured like this.
  This group is the last one considered during evictions.
+
To mark a column family as in-memory, call

[source,java]
----
HColumnDescriptor.setInMemory(true);
----

if creating a table from java, or set `IN_MEMORY => true` when creating or altering a table in the shell: e.g.

[source]
----
hbase(main):003:0> create  't', {NAME => 'f', IN_MEMORY => 'true'}
----

For more information, see the LruBlockCache source

[[block.cache.usage]]
==== LruBlockCache Usage

Block caching is enabled by default for all the user tables which means that any read operation will load the LRU cache.
This might be good for a large number of use cases, but further tunings are usually required in order to achieve better performance.
An important concept is the link:http://en.wikipedia.org/wiki/Working_set_size[working set size], or WSS, which is: "the amount of memory needed to compute the answer to a problem". For a website, this would be the data that's needed to answer the queries over a short amount of time.

The way to calculate how much memory is available in HBase for caching is:

[source]
----
number of region servers * heap size * hfile.block.cache.size * 0.99
----

The default value for the block cache is 0.4 which represents 40% of the available heap.
The last value (99%) is the default acceptable loading factor in the LRU cache after which eviction is started.
The reason it is included in this equation is that it would be unrealistic to say that it is possible to use 100% of the available memory since this would make the process blocking from the point where it loads new blocks.
Here are some examples:

* One region server with the heap size set to 1 GB and the default block cache size will have 405 MB of block cache available.
* 20 region servers with the heap size set to 8 GB and a default block cache size will have 63.3 GB of block cache.
* 100 region servers with the heap size set to 24 GB and a block cache size of 0.5 will have about 1.16 TB of block cache.

Your data is not the only resident of the block cache.
Here are others that you may have to take into account:

Catalog Tables::
  The `hbase:meta` table is forced into the block cache and have the in-memory priority which means that they are harder to evict.

NOTE: The hbase:meta tables can occupy a few MBs depending on the number of regions.

HFiles Indexes::
  An _HFile_ is the file format that HBase uses to store data in HDFS.
  It contains a multi-layered index which allows HBase to seek the data without having to read the whole file.
  The size of those indexes is a factor of the block size (64KB by default), the size of your keys and the amount of data you are storing.
  For big data sets it's not unusual to see numbers around 1GB per region server, although not all of it will be in cache because the LRU will evict indexes that aren't used.

Keys::
  The values that are stored are only half the picture, since each value is stored along with its keys (row key, family qualifier, and timestamp). See <<keysize>>.

Bloom Filters::
  Just like the HFile indexes, those data structures (when enabled) are stored in the LRU.

Currently the recommended way to measure HFile indexes and bloom filters sizes is to look at the region server web UI and checkout the relevant metrics.
For keys, sampling can be done by using the HFile command line tool and look for the average key size metric.
Since HBase 0.98.3, you can view details on BlockCache stats and metrics in a special Block Cache section in the UI.

It's generally bad to use block caching when the WSS doesn't fit in memory.
This is the case when you have for example 40GB available across all your region servers' block caches but you need to process 1TB of data.
One of the reasons is that the churn generated by the evictions will trigger more garbage collections unnecessarily.
Here are two use cases:

* Fully random reading pattern: This is a case where you almost never access the same row twice within a short amount of time such that the chance of hitting a cached block is close to 0.
  Setting block caching on such a table is a waste of memory and CPU cycles, more so that it will generate more garbage to pick up by the JVM.
  For more information on monitoring GC, see <<trouble.log.gc>>.
* Mapping a table: In a typical MapReduce job that takes a table in input, every row will be read only once so there's no need to put them into the block cache.
  The Scan object has the option of turning this off via the setCacheBlocks method (set it to false). You can still keep block caching turned on on this table if you need fast random read access.
  An example would be counting the number of rows in a table that serves live traffic, caching every block of that table would create massive churn and would surely evict data that's currently in use.

[[data.blocks.in.fscache]]
===== Caching META blocks only (DATA blocks in fscache)

An interesting setup is one where we cache META blocks only and we read DATA blocks in on each access.
If the DATA blocks fit inside fscache, this alternative may make sense when access is completely random across a very large dataset.
To enable this setup, alter your table and for each column family set `BLOCKCACHE => 'false'`.
You are 'disabling' the BlockCache for this column family only. You can never disable the caching of META blocks.
Since link:https://issues.apache.org/jira/browse/HBASE-4683[HBASE-4683 Always cache index and bloom blocks], we will cache META blocks even if the BlockCache is disabled.

[[offheap.blockcache]]
==== Off-heap Block Cache

[[enable.bucketcache]]
===== How to Enable BucketCache

The usual deployment of BucketCache is via a managing class that sets up two caching tiers:
an on-heap cache implemented by LruBlockCache and a second  cache implemented with BucketCache.
The managing class is link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.html[CombinedBlockCache] by default.
The previous link describes the caching 'policy' implemented by CombinedBlockCache.
In short, it works by keeping meta blocks -- INDEX and BLOOM in the on-heap LruBlockCache tier -- and DATA blocks are kept in the BucketCache tier.

====
Pre-hbase-2.0.0 versions::
Fetching will always be slower when fetching from BucketCache in pre-hbase-2.0.0,
as compared to the native on-heap LruBlockCache. However, latencies tend to be less
erratic across time, because there is less garbage collection when you use BucketCache since it is managing BlockCache allocations, not the GC.
If the BucketCache is deployed in off-heap mode, this memory is not managed by the GC at all.
This is why you'd use BucketCache in pre-2.0.0, so your latencies are less erratic,
to mitigate GCs and heap fragmentation, and so you can safely use more memory.
See Nick Dimiduk's link:http://www.n10k.com/blog/blockcache-101/[BlockCache 101] for comparisons running on-heap vs off-heap tests.
Also see link:https://people.apache.org/~stack/bc/[Comparing BlockCache Deploys] which finds that if your dataset fits inside your LruBlockCache deploy, use it otherwise if you are experiencing cache churn (or you want your cache to exist beyond the vagaries of java GC), use BucketCache.
+
In pre-2.0.0,
one can configure the BucketCache so it receives the `victim` of an LruBlockCache eviction.
All Data and index blocks are cached in L1 first. When eviction happens from L1, the blocks (or `victims`) will get moved to L2.
Set `cacheDataInL1` via `(HColumnDescriptor.setCacheDataInL1(true)` or in the shell, creating or amending column families setting `CACHE_DATA_IN_L1` to true: e.g.
[source]
----
hbase(main):003:0> create 't', {NAME => 't', CONFIGURATION => {CACHE_DATA_IN_L1 => 'true'}}
----

hbase-2.0.0+ versions::
HBASE-11425 changed the HBase read path so it could hold the read-data off-heap avoiding copying of cached data on to the java heap.
See <<regionserver.offheap.readpath>>. In hbase-2.0.0, off-heap latencies approach those of on-heap cache latencies with the added
benefit of NOT provoking GC.
+
From HBase 2.0.0 onwards, the notions of L1 and L2 have been deprecated. When BucketCache is turned on, the DATA blocks will always go to BucketCache and INDEX/BLOOM blocks go to on heap LRUBlockCache. `cacheDataInL1` support has been removed.
====

[[bc.deloy.modes]]
====== BucketCache Deploy Modes
The BucketCache Block Cache can be deployed _offheap_, _file_ or _mmaped_ file mode.

You set which via the `hbase.bucketcache.ioengine` setting.
Setting it to `offheap` will have BucketCache make its allocations off-heap, and an ioengine setting of `file:PATH_TO_FILE` will direct BucketCache to use file caching (Useful in particular if you have some fast I/O attached to the box such as SSDs). From 2.0.0, it is possible to have more than one file backing the BucketCache. This is very useful especially when the Cache size requirement is high. For multiple backing files, configure ioengine as `files:PATH_TO_FILE1,PATH_TO_FILE2,PATH_TO_FILE3`. BucketCache can be configured to use an mmapped file also. Configure ioengine as `mmap:PATH_TO_FILE` for this.

It is possible to deploy a tiered setup where we bypass the CombinedBlockCache policy and have BucketCache working as a strict L2 cache to the L1 LruBlockCache.
For such a setup, set `hbase.bucketcache.combinedcache.enabled` to `false`.
In this mode, on eviction from L1, blocks go to L2.
When a block is cached, it is cached first in L1.
When we go to look for a cached block, we look first in L1 and if none found, then search L2.
Let us call this deploy format, _Raw L1+L2_.
NOTE: This L1+L2 mode is removed from 2.0.0. When BucketCache is used, it will be strictly the DATA cache and the LruBlockCache will cache INDEX/META blocks.

Other BucketCache configs include: specifying a location to persist cache to across restarts, how many threads to use writing the cache, etc.
See the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/hfile/CacheConfig.html[CacheConfig.html] class for configuration options and descriptions.

To check it enabled, look for the log line describing cache setup; it will detail how BucketCache has been deployed.
Also see the UI. It will detail the cache tiering and their configuration.

[[bc.example]]
====== BucketCache Example Configuration
This sample provides a configuration for a 4 GB off-heap BucketCache with a 1 GB on-heap cache.

Configuration is performed on the RegionServer.

Setting `hbase.bucketcache.ioengine` and `hbase.bucketcache.size` > 0 enables `CombinedBlockCache`.
Let us presume that the RegionServer has been set to run with a 5G heap: i.e. `HBASE_HEAPSIZE=5g`.


. First, edit the RegionServer's _hbase-env.sh_ and set `HBASE_OFFHEAPSIZE` to a value greater than the off-heap size wanted, in this case, 4 GB (expressed as 4G). Let's set it to 5G.
  That'll be 4G for our off-heap cache and 1G for any other uses of off-heap memory (there are other users of off-heap memory other than BlockCache; e.g.
  DFSClient in RegionServer can make use of off-heap memory). See <<direct.memory>>.
+
[source]
----
HBASE_OFFHEAPSIZE=5G
----

. Next, add the following configuration to the RegionServer's _hbase-site.xml_.
+
[source,xml]
----
<property>
  <name>hbase.bucketcache.ioengine</name>
  <value>offheap</value>
</property>
<property>
  <name>hfile.block.cache.size</name>
  <value>0.2</value>
</property>
<property>
  <name>hbase.bucketcache.size</name>
  <value>4196</value>
</property>
----

. Restart or rolling restart your cluster, and check the logs for any issues.


In the above, we set the BucketCache to be 4G.
We configured the on-heap LruBlockCache have 20% (0.2) of the RegionServer's heap size (0.2 * 5G = 1G). In other words, you configure the L1 LruBlockCache as you would normally (as if there were no L2 cache present).

link:https://issues.apache.org/jira/browse/HBASE-10641[HBASE-10641] introduced the ability to configure multiple sizes for the buckets of the BucketCache, in HBase 0.98 and newer.
To configurable multiple bucket sizes, configure the new property `hbase.bucketcache.bucket.sizes` to a comma-separated list of block sizes, ordered from smallest to largest, with no spaces.
The goal is to optimize the bucket sizes based on your data access patterns.
The following example configures buckets of size 4096 and 8192.

[source,xml]
----
<property>
  <name>hbase.bucketcache.bucket.sizes</name>
  <value>4096,8192</value>
</property>
----

[[direct.memory]]
.Direct Memory Usage In HBase
[NOTE]
====
The default maximum direct memory varies by JVM.
Traditionally it is 64M or some relation to allocated heap size (-Xmx) or no limit at all (JDK7 apparently). HBase servers use direct memory, in particular short-circuit reading (See <<perf.hdfs.configs.localread>>), the hosted DFSClient will allocate direct memory buffers. How much the DFSClient uses is not easy to quantify; it is the number of open HFiles * `hbase.dfs.client.read.shortcircuit.buffer.size` where `hbase.dfs.client.read.shortcircuit.buffer.size` is set to 128k in HBase -- see _hbase-default.xml_ default configurations.
If you do off-heap block caching, you'll be making use of direct memory.
The RPCServer uses a ByteBuffer pool. From 2.0.0, these buffers are off-heap ByteBuffers.
Starting your JVM, make sure the `-XX:MaxDirectMemorySize` setting in _conf/hbase-env.sh_ considers off-heap BlockCache (`hbase.bucketcache.size`), DFSClient usage, RPC side ByteBufferPool max size. This has to be bit higher than sum of off heap BlockCache size and max ByteBufferPool size. Allocating an extra of 1-2 GB for the max direct memory size has worked in tests. Direct memory, which is part of the Java process heap, is separate from the object heap allocated by -Xmx.
The value allocated by `MaxDirectMemorySize` must not exceed physical RAM, and is likely to be less than the total available RAM due to other memory requirements and system constraints.

You can see how much memory -- on-heap and off-heap/direct -- a RegionServer is configured to use and how much it is using at any one time by looking at the _Server Metrics: Memory_ tab in the UI.
It can also be gotten via JMX.
In particular the direct memory currently used by the server can be found on the `java.nio.type=BufferPool,name=direct` bean.
Terracotta has a link:http://terracotta.org/documentation/4.0/bigmemorygo/configuration/storage-options[good write up] on using off-heap memory in Java.
It is for their product BigMemory but a lot of the issues noted apply in general to any attempt at going off-heap. Check it out.
====

.hbase.bucketcache.percentage.in.combinedcache
[NOTE]
====
This is a pre-HBase 1.0 configuration removed because it was confusing.
It was a float that you would set to some value between 0.0 and 1.0.
Its default was 0.9.
If the deploy was using CombinedBlockCache, then the LruBlockCache L1 size was calculated to be `(1 - hbase.bucketcache.percentage.in.combinedcache) * size-of-bucketcache`  and the BucketCache size was `hbase.bucketcache.percentage.in.combinedcache * size-of-bucket-cache`.
where size-of-bucket-cache itself is EITHER the value of the configuration `hbase.bucketcache.size` IF it was specified as Megabytes OR `hbase.bucketcache.size` * `-XX:MaxDirectMemorySize` if `hbase.bucketcache.size` is between 0 and 1.0.

In 1.0, it should be more straight-forward.
Onheap LruBlockCache size is set as a fraction of java heap using `hfile.block.cache.size setting` (not the best name) and BucketCache is set as above in absolute Megabytes.
====

==== Compressed BlockCache

link:https://issues.apache.org/jira/browse/HBASE-11331[HBASE-11331] introduced lazy BlockCache decompression, more simply referred to as compressed BlockCache.
When compressed BlockCache is enabled data and encoded data blocks are cached in the BlockCache in their on-disk format, rather than being decompressed and decrypted before caching.

For a RegionServer hosting more data than can fit into cache, enabling this feature with SNAPPY compression has been shown to result in 50% increase in throughput and 30% improvement in mean latency while, increasing garbage collection by 80% and increasing overall CPU load by 2%. See HBASE-11331 for more details about how performance was measured and achieved.
For a RegionServer hosting data that can comfortably fit into cache, or if your workload is sensitive to extra CPU or garbage-collection load, you may receive less benefit.

The compressed BlockCache is disabled by default. To enable it, set `hbase.block.data.cachecompressed` to `true` in _hbase-site.xml_ on all RegionServers.

[[regionserver_splitting_implementation]]
=== RegionServer Splitting Implementation

As write requests are handled by the region server, they accumulate in an in-memory storage system called the _memstore_. Once the memstore fills, its content are written to disk as additional store files. This event is called a _memstore flush_. As store files accumulate, the RegionServer will <<compaction,compact>> them into fewer, larger files. After each flush or compaction finishes, the amount of data stored in the region has changed. The RegionServer consults the region split policy to determine if the region has grown too large or should be split for another policy-specific reason. A region split request is enqueued if the policy recommends it.

Logically, the process of splitting a region is simple. We find a suitable point in the keyspace of the region where we should divide the region in half, then split the region's data into two new regions at that point. The details of the process however are not simple.  When a split happens, the newly created _daughter regions_ do not rewrite all the data into new files immediately. Instead, they create small files similar to symbolic link files, named link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/Reference.html[Reference files], which point to either the top or bottom part of the parent store file according to the split point. The reference file is used just like a regular data file, but only half of the records are considered. The region can only be split if there are no more references to the immutable data files of the parent region. Those reference files are cleaned gradually by compactions, so that the region will stop referring to its parents files, and can be split further.

Although splitting the region is a local decision made by the RegionServer, the split process itself must coordinate with many actors. The RegionServer notifies the Master before and after the split, updates the `.META.` table so that clients can discover the new daughter regions, and rearranges the directory structure and data files in HDFS. Splitting is a multi-task process. To enable rollback in case of an error, the RegionServer keeps an in-memory journal about the execution state. The steps taken by the RegionServer to execute the split are illustrated in <<regionserver_split_process_image>>. Each step is labeled with its step number. Actions from RegionServers or Master are shown in red, while actions from the clients are shown in green.

[[regionserver_split_process_image]]
.RegionServer Split Process
image::region_split_process.png[Region Split Process]

. The RegionServer decides locally to split the region, and prepares the split. *THE SPLIT TRANSACTION IS STARTED.* As a first step, the RegionServer acquires a shared read lock on the table to prevent schema modifications during the splitting process. Then it creates a znode in zookeeper under `/hbase/region-in-transition/region-name`, and sets the znode's state to `SPLITTING`.
. The Master learns about this znode, since it has a watcher for the parent `region-in-transition` znode.
. The RegionServer creates a sub-directory named `.splits` under the parent’s `region` directory in HDFS.
. The RegionServer closes the parent region and marks the region as offline in its local data structures. *THE SPLITTING REGION IS NOW OFFLINE.* At this point, client requests coming to the parent region will throw `NotServingRegionException`. The client will retry with some backoff. The closing region is flushed.
. The RegionServer creates region directories under the `.splits` directory, for daughter
regions A and B, and creates necessary data structures. Then it splits the store files,
in the sense that it creates two Reference files per store file in the parent region.
Those reference files will point to the parent region's files.
. The RegionServer creates the actual region directory in HDFS, and moves the reference files for each daughter.
. The RegionServer sends a `Put` request to the `.META.` table, to set the parent as offline in the `.META.` table and add information about daughter regions. At this point, there won’t be individual entries in `.META.` for the daughters. Clients will see that the parent region is split if they scan `.META.`, but won’t know about the daughters until they appear in `.META.`. Also, if this `Put` to `.META`. succeeds, the parent will be effectively split. If the RegionServer fails before this RPC succeeds, Master and the next Region Server opening the region will clean dirty state about the region split. After the `.META.` update, though, the region split will be rolled-forward by Master.
. The RegionServer opens daughters A and B in parallel.
. The RegionServer adds the daughters A and B to `.META.`, together with information that it hosts the regions. *THE SPLIT REGIONS (DAUGHTERS WITH REFERENCES TO PARENT) ARE NOW ONLINE.* After this point, clients can discover the new regions and issue requests to them. Clients cache the `.META.` entries locally, but when they make requests to the RegionServer or `.META.`, their caches will be invalidated, and they will learn about the new regions from `.META.`.
. The RegionServer updates znode `/hbase/region-in-transition/region-name` in ZooKeeper to state `SPLIT`, so that the master can learn about it. The balancer can freely re-assign the daughter regions to other region servers if necessary. *THE SPLIT TRANSACTION IS NOW FINISHED.*
. After the split, `.META.` and HDFS will still contain references to the parent region. Those references will be removed when compactions in daughter regions rewrite the data files. Garbage collection tasks in the master periodically check whether the daughter regions still refer to the parent region's files. If not, the parent region will be removed.

[[wal]]
=== Write Ahead Log (WAL)

[[purpose.wal]]
==== Purpose

The _Write Ahead Log (WAL)_ records all changes to data in HBase, to file-based storage.
Under normal operations, the WAL is not needed because data changes move from the MemStore to StoreFiles.
However, if a RegionServer crashes or becomes unavailable before the MemStore is flushed, the WAL ensures that the changes to the data can be replayed.
If writing to the WAL fails, the entire operation to modify the data fails.

HBase uses an implementation of the link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/wal/WAL.html[WAL] interface.
Usually, there is only one instance of a WAL per RegionServer. An exception
is the RegionServer that is carrying _hbase:meta_; the _meta_ table gets its
own dedicated WAL.
The RegionServer records Puts and Deletes to its WAL, before recording them
these Mutations <<store.memstore>> for the affected <<store>>.

.The HLog
[NOTE]
====
Prior to 2.0, the interface for WALs in HBase was named `HLog`.
In 0.94, HLog was the name of the implementation of the WAL.
You will likely find references to the HLog in documentation tailored to these older versions.
====

The WAL resides in HDFS in the _/hbase/WALs/_ directory, with subdirectories per region.

For more general information about the concept of write ahead logs, see the Wikipedia
link:http://en.wikipedia.org/wiki/Write-ahead_logging[Write-Ahead Log] article.


[[wal.providers]]
==== WAL Providers
In HBase, there are a number of WAL implementations (or 'Providers'). Each is known
by a short name label (that unfortunately is not always descriptive). You set the provider in
_hbase-site.xml_ passing the WAL provider short-name as the value on the
_hbase.wal.provider_ property (Set the provider for _hbase:meta_ using the
_hbase.wal.meta_provider_ property, otherwise it uses the same provider configured
by _hbase.wal.provider_).

 * _asyncfs_: The *default*. New since hbase-2.0.0 (HBASE-15536, HBASE-14790). This _AsyncFSWAL_ provider, as it identifies itself in RegionServer logs, is built on a new non-blocking dfsclient implementation. It is currently resident in the hbase codebase but intent is to move it back up into HDFS itself. WALs edits are written concurrently ("fan-out") style to each of the WAL-block replicas on each DataNode rather than in a chained pipeline as the default client does. Latencies should be better. See link:https://www.slideshare.net/HBaseCon/apache-hbase-improvements-and-practices-at-xiaomi[Apache HBase Improvements and Practices at Xiaomi] at slide 14 onward for more detail on implementation.
 * _filesystem_: This was the default in hbase-1.x releases. It is built on the blocking _DFSClient_ and writes to replicas in classic _DFSCLient_ pipeline mode. In logs it identifies as _FSHLog_ or _FSHLogProvider_.
 * _multiwal_: This provider is made of multiple instances of _asyncfs_ or  _filesystem_. See the next section for more on _multiwal_.

Look for the lines like the below in the RegionServer log to see which provider is in place (The below shows the default AsyncFSWALProvider):

----
2018-04-02 13:22:37,983 INFO  [regionserver/ve0528:16020] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider
----

NOTE: As the _AsyncFSWAL_ hacks into the internal of DFSClient implementation, it will be easily broken by upgrading the hadoop dependencies, even for a simple patch release. So if you do not specify the wal provider explicitly, we will first try to use the _asyncfs_, if failed, we will fall back to use _filesystem_. And notice that this may not always work, so if you still have problem starting HBase due to the problem of starting _AsyncFSWAL_, please specify _filesystem_ explicitly in the config file.

NOTE: EC support has been added to hadoop-3.x, and it is incompatible with WAL as the EC output stream does not support hflush/hsync. In order to create a non-EC file in an EC directory, we need to use the new builder-based create API for _FileSystem_, but it is only introduced in hadoop-2.9+ and for HBase we still need to support hadoop-2.7.x. So please do not enable EC for the WAL directory until we find a way to deal with it.

==== MultiWAL
With a single WAL per RegionServer, the RegionServer must write to the WAL serially, because HDFS files must be sequential. This causes the WAL to be a performance bottleneck.

HBase 1.0 introduces support MultiWal in link:https://issues.apache.org/jira/browse/HBASE-5699[HBASE-5699]. MultiWAL allows a RegionServer to write multiple WAL streams in parallel, by using multiple pipelines in the underlying HDFS instance, which increases total throughput during writes. This parallelization is done by partitioning incoming edits by their Region. Thus, the current implementation will not help with increasing the throughput to a single Region.

RegionServers using the original WAL implementation and those using the MultiWAL implementation can each handle recovery of either set of WALs, so a zero-downtime configuration update is possible through a rolling restart.

.Configure MultiWAL
To configure MultiWAL for a RegionServer, set the value of the property `hbase.wal.provider` to `multiwal` by pasting in the following XML:

[source,xml]
----
<property>
  <name>hbase.wal.provider</name>
  <value>multiwal</value>
</property>
----

Restart the RegionServer for the changes to take effect.

To disable MultiWAL for a RegionServer, unset the property and restart the RegionServer.


[[wal_flush]]
==== WAL Flushing

TODO (describe).

==== WAL Splitting

A RegionServer serves many regions.
All of the regions in a region server share the same active WAL file.
Each edit in the WAL file includes information about which region it belongs to.
When a region is opened, the edits in the WAL file which belong to that region need to be replayed.
Therefore, edits in the WAL file must be grouped by region so that particular sets can be replayed to regenerate the data in a particular region.
The process of grouping the WAL edits by region is called _log splitting_.
It is a critical process for recovering data if a region server fails.

Log splitting is done by the HMaster during cluster start-up or by the ServerShutdownHandler as a region server shuts down.
So that consistency is guaranteed, affected regions are unavailable until data is restored.
All WAL edits need to be recovered and replayed before a given region can become available again.
As a result, regions affected by log splitting are unavailable until the process completes.

.Procedure: Log Splitting, Step by Step
. The _/hbase/WALs/<host>,<port>,<startcode>_ directory is renamed.
+
Renaming the directory is important because a RegionServer may still be up and accepting requests even if the HMaster thinks it is down.
If the RegionServer does not respond immediately and does not heartbeat its ZooKeeper session, the HMaster may interpret this as a RegionServer failure.
Renaming the logs directory ensures that existing, valid WAL files which are still in use by an active but busy RegionServer are not written to by accident.
+
The new directory is named according to the following pattern:
+
----
/hbase/WALs/<host>,<port>,<startcode>-splitting
----
+
An example of such a renamed directory might look like the following:
+
----
/hbase/WALs/srv.example.com,60020,1254173957298-splitting
----

. Each log file is split, one at a time.
+
The log splitter reads the log file one edit entry at a time and puts each edit entry into the buffer corresponding to the edit's region.
At the same time, the splitter starts several writer threads.
Writer threads pick up a corresponding buffer and write the edit entries in the buffer to a temporary recovered edit file.
The temporary edit file is stored to disk with the following naming pattern:
+
----
/hbase/<table_name>/<region_id>/recovered.edits/.temp
----
+
This file is used to store all the edits in the WAL log for this region.
After log splitting completes, the _.temp_ file is renamed to the sequence ID of the first log written to the file.
+
To determine whether all edits have been written, the sequence ID is compared to the sequence of the last edit that was written to the HFile.
If the sequence of the last edit is greater than or equal to the sequence ID included in the file name, it is clear that all writes from the edit file have been completed.

. After log splitting is complete, each affected region is assigned to a RegionServer.
+
When the region is opened, the _recovered.edits_ folder is checked for recovered edits files.
If any such files are present, they are replayed by reading the edits and saving them to the MemStore.
After all edit files are replayed, the contents of the MemStore are written to disk (HFile) and the edit files are deleted.


===== Handling of Errors During Log Splitting

If you set the `hbase.hlog.split.skip.errors` option to `true`, errors are treated as follows:

* Any error encountered during splitting will be logged.
* The problematic WAL log will be moved into the _.corrupt_ directory under the hbase `rootdir`,
* Processing of the WAL will continue

If the `hbase.hlog.split.skip.errors` option is set to `false`, the default, the exception will be propagated and the split will be logged as failed.
See link:https://issues.apache.org/jira/browse/HBASE-2958[HBASE-2958 When
hbase.hlog.split.skip.errors is set to false, we fail the split but that's it].
We need to do more than just fail split if this flag is set.

====== How EOFExceptions are treated when splitting a crashed RegionServer's WALs

If an EOFException occurs while splitting logs, the split proceeds even when `hbase.hlog.split.skip.errors` is set to `false`.
An EOFException while reading the last log in the set of files to split is likely, because the RegionServer was likely in the process of writing a record at the time of a crash.
For background, see link:https://issues.apache.org/jira/browse/HBASE-2643[HBASE-2643 Figure how to deal with eof splitting logs]

===== Performance Improvements during Log Splitting

WAL log splitting and recovery can be resource intensive and take a long time, depending on the number of RegionServers involved in the crash and the size of the regions. <<distributed.log.splitting>> was developed to improve performance during log splitting.

[[distributed.log.splitting]]
.Enabling or Disabling Distributed Log Splitting

Distributed log processing is enabled by default since HBase 0.92.
The setting is controlled by the `hbase.master.distributed.log.splitting` property, which can be set to `true` or `false`, but defaults to `true`.

==== WAL splitting based on procedureV2
After HBASE-20610, we introduce a new way to do WAL splitting coordination by procedureV2 framework. This can simplify the process of WAL splitting and no need to connect zookeeper any more.

[[background]]
.Background
Currently, splitting WAL processes are coordinated by zookeeper. Each region server are trying to grab tasks from zookeeper. And the burden becomes heavier when the number of region server increase.

[[implementation.on.master.side]]
.Implementation on Master side
During ServerCrashProcedure, SplitWALManager will create one SplitWALProcedure for each WAL file which should be split. Then each SplitWALProcedure will spawn a SplitWalRemoteProcedure to send the request to region server.
SplitWALProcedure is a StateMachineProcedure and here is the state transfer diagram.

.WAL_splitting_coordination
image::WAL_splitting.png[]

[[implementation.on.region.server.side]]
.Implementation on Region Server side
Region Server will receive a SplitWALCallable and execute it, which is much more straightforward than before. It will return null if success and return exception if there is any error.

[[preformance]]
.Performance
According to tests on a cluster which has 5 regionserver and 1 master.
procedureV2 coordinated WAL splitting has a better performance than ZK coordinated WAL splitting no master when restarting the whole cluster or one region server crashing.

[[enable.this.feature]]
.Enable this feature
To enable this feature, first we should ensure our package of HBase already contains these code. If not, please upgrade the package of HBase cluster without any configuration change first.
Then change configuration 'hbase.split.wal.zk.coordinated' to false. Rolling upgrade the master with new configuration. Now WAL splitting are handled by our new implementation.
But region server are still trying to grab tasks from zookeeper, we can rolling upgrade the region servers with the new configuration to stop that.

* steps as follows:
** Upgrade whole cluster to get the new Implementation.
** Upgrade Master with new configuration 'hbase.split.wal.zk.coordinated'=false.
** Upgrade region server to stop grab tasks from zookeeper.

[[wal.compression]]
==== WAL Compression ====

The content of the WAL can be compressed using LRU Dictionary compression.
This can be used to speed up WAL replication to different datanodes.
The dictionary can store up to 2^15^ elements; eviction starts after this number is exceeded.

To enable WAL compression, set the `hbase.regionserver.wal.enablecompression` property to `true`.
The default value for this property is `false`.
By default, WAL tag compression is turned on when WAL compression is enabled.
You can turn off WAL tag compression by setting the `hbase.regionserver.wal.tags.enablecompression` property to 'false'.

A possible downside to WAL compression is that we lose more data from the last block in the WAL if it is ill-terminated
mid-write. If entries in this last block were added with new dictionary entries but we failed persist the amended
dictionary because of an abrupt termination, a read of this last block may not be able to resolve last-written entries.

[[wal.durability]]
==== Durability
It is possible to set _durability_ on each Mutation or on a Table basis. Options include:

 * _SKIP_WAL_: Do not write Mutations to the WAL (See the next section, <<wal.disable>>).
 * _ASYNC_WAL_: Write the WAL asynchronously; do not hold-up clients waiting on the sync of their write to the filesystem but return immediately. The edit becomes visible. Meanwhile, in the background, the Mutation will be flushed to the WAL at some time later. This option currently may lose data. See HBASE-16689.
 * _SYNC_WAL_: The *default*. Each edit is sync'd to HDFS before we return success to the client.
 * _FSYNC_WAL_: Each edit is fsync'd to HDFS and the filesystem before we return success to the client.

Do not confuse the _ASYNC_WAL_ option on a Mutation or Table with the _AsyncFSWAL_ writer; they are distinct
options unfortunately closely named

[[arch.custom.wal.dir]]
==== Custom WAL Directory
HBASE-17437 added support for specifying a WAL directory outside the HBase root directory or even in a different FileSystem since 1.3.3/2.0+. Some FileSystems (such as Amazon S3) don’t support append or consistent writes, in such scenario WAL directory needs to be configured in a different FileSystem to avoid loss of writes.

Following configurations are added to accomplish this:

. `hbase.wal.dir`
+
This defines where the root WAL directory is located, could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset.

. `hbase.rootdir.perms`
+
Configures FileSystem permissions to set on the root directory. This is '700' by default.

. `hbase.wal.dir.perms`
+
Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.

NOTE: While migrating to custom WAL dir (outside the HBase root directory or a different FileSystem) existing WAL files must be copied manually to new WAL dir, otherwise it may lead to data loss/inconsistency as HMaster has no information about previous WAL directory.

[[wal.disable]]
==== Disabling the WAL

It is possible to disable the WAL, to improve performance in certain specific situations.
However, disabling the WAL puts your data at risk.
The only situation where this is recommended is during a bulk load.
This is because, in the event of a problem, the bulk load can be re-run with no risk of data loss.

The WAL is disabled by calling the HBase client field `Mutation.writeToWAL(false)`.
Use the `Mutation.setDurability(Durability.SKIP_WAL)` and Mutation.getDurability() methods to set and get the field's value.
There is no way to disable the WAL for only a specific table.

WARNING: If you disable the WAL for anything other than bulk loads, your data is at risk.


[[regions.arch]]
== Regions

Regions are the basic element of availability and distribution for tables, and are comprised of a Store per Column Family.
The hierarchy of objects is as follows:

----
Table                    (HBase table)
    Region               (Regions for the table)
        Store            (Store per ColumnFamily for each Region for the table)
            MemStore     (MemStore for each Store for each Region for the table)
            StoreFile    (StoreFiles for each Store for each Region for the table)
                Block    (Blocks within a StoreFile within a Store for each Region for the table)
----

For a description of what HBase files look like when written to HDFS, see <<trouble.namenode.hbase.objects>>.

[[arch.regions.size]]
=== Considerations for Number of Regions

In general, HBase is designed to run with a small (20-200) number of relatively large (5-20Gb) regions per server.
The considerations for this are as follows:

[[too_many_regions]]
==== Why should I keep my Region count low?

Typically you want to keep your region count low on HBase for numerous reasons.
Usually right around 100 regions per RegionServer has yielded the best results.
Here are some of the reasons below for keeping region count low:

. MSLAB (MemStore-local allocation buffer) requires 2MB per MemStore (that's 2MB per family per region). 1000 regions that have 2 families each is 3.9GB of heap used, and it's not even storing data yet.
  NB: the 2MB value is configurable.
. If you fill all the regions at somewhat the same rate, the global memory usage makes it that it forces tiny flushes when you have too many regions which in turn generates compactions.
  Rewriting the same data tens of times is the last thing you want.
  An example is filling 1000 regions (with one family) equally and let's consider a lower bound for global MemStore usage of 5GB (the region server would have a big heap). Once it reaches 5GB it will force flush the biggest region, at that point they should almost all have about 5MB of data so it would flush that amount.
  5MB inserted later, it would flush another region that will now have a bit over 5MB of data, and so on.
  This is currently the main limiting factor for the number of regions; see <<ops.capacity.regions.count>> for detailed formula.
. The master as is is allergic to tons of regions, and will take a lot of time assigning them and moving them around in batches.
  The reason is that it's heavy on ZK usage, and it's not very async at the moment (could really be improved -- and has been improved a bunch in 0.96 HBase).
. In older versions of HBase (pre-HFile v2, 0.90 and previous), tons of regions on a few RS can cause the store file index to rise, increasing heap usage and potentially creating memory pressure or OOME on the RSs

Another issue is the effect of the number of regions on MapReduce jobs; it is typical to have one mapper per HBase region.
Thus, hosting only 5 regions per RS may not be enough to get sufficient number of tasks for a MapReduce job, while 1000 regions will generate far too many tasks.

See <<ops.capacity.regions>> for configuration guidelines.

[[regions.arch.assignment]]
=== Region-RegionServer Assignment

This section describes how Regions are assigned to RegionServers.

[[regions.arch.assignment.startup]]
==== Startup

When HBase starts regions are assigned as follows (short version):

. The Master invokes the `AssignmentManager` upon startup.
. The `AssignmentManager` looks at the existing region assignments in `hbase:meta`.
. If the region assignment is still valid (i.e., if the RegionServer is still online) then the assignment is kept.
. If the assignment is invalid, then the `LoadBalancerFactory` is invoked to assign the region.
  The load balancer (`StochasticLoadBalancer` by default in HBase 1.0) assign the region to a RegionServer.
. `hbase:meta` is updated with the RegionServer assignment (if needed) and the RegionServer start codes (start time of the RegionServer process) upon region opening by the RegionServer.

[[regions.arch.assignment.failover]]
==== Failover

When a RegionServer fails:

. The regions immediately become unavailable because the RegionServer is down.
. The Master will detect that the RegionServer has failed.
. The region assignments will be considered invalid and will be re-assigned just like the startup sequence.
. In-flight queries are re-tried, and not lost.
. Operations are switched to a new RegionServer within the following amount of time:
+
[source]
----
ZooKeeper session timeout + split time + assignment/replay time
----


[[regions.arch.balancer]]
==== Region Load Balancing

Regions can be periodically moved by the <<master.processes.loadbalancer>>.

[[regions.arch.states]]
==== Region State Transition

HBase maintains a state for each region and persists the state in `hbase:meta`.
The state of the `hbase:meta` region itself is persisted in ZooKeeper.
You can see the states of regions in transition in the Master web UI.
Following is the list of possible region states.

.Possible Region States
* `OFFLINE`: the region is offline and not opening
* `OPENING`: the region is in the process of being opened
* `OPEN`: the region is open and the RegionServer has notified the master
* `FAILED_OPEN`: the RegionServer failed to open the region
* `CLOSING`: the region is in the process of being closed
* `CLOSED`: the RegionServer has closed the region and notified the master
* `FAILED_CLOSE`: the RegionServer failed to close the region
* `SPLITTING`: the RegionServer notified the master that the region is splitting
* `SPLIT`: the RegionServer notified the master that the region has finished splitting
* `SPLITTING_NEW`: this region is being created by a split which is in progress
* `MERGING`: the RegionServer notified the master that this region is being merged with another region
* `MERGED`: the RegionServer notified the master that this region has been merged
* `MERGING_NEW`: this region is being created by a merge of two regions

.Region State Transitions
image::region_states.png[]

.Graph Legend
* Brown: Offline state, a special state that can be transient (after closed before opening), terminal (regions of disabled tables), or initial (regions of newly created tables)
* Palegreen: Online state that regions can serve requests
* Lightblue: Transient states
* Red: Failure states that need OPS attention
* Gold: Terminal states of regions split/merged
* Grey: Initial states of regions created through split/merge

.Transition State Descriptions
. The master moves a region from `OFFLINE` to `OPENING` state and tries to assign the region to a RegionServer.
  The RegionServer may or may not have received the open region request.
  The master retries sending the open region request to the RegionServer until the RPC goes through or the master runs out of retries.
  After the RegionServer receives the open region request, the RegionServer begins opening the region.
. If the master is running out of retries, the master prevents the RegionServer from opening the region by moving the region to `CLOSING` state and trying to close it, even if the RegionServer is starting to open the region.
. After the RegionServer opens the region, it continues to try to notify the master until the master moves the region to `OPEN` state and notifies the RegionServer.
  The region is now open.
. If the RegionServer cannot open the region, it notifies the master.
  The master moves the region to `CLOSED` state and tries to open the region on a different RegionServer.
. If the master cannot open the region on any of a certain number of regions, it moves the region to `FAILED_OPEN` state, and takes no further action until an operator intervenes from the HBase shell, or the server is dead.
. The master moves a region from `OPEN` to `CLOSING` state.
  The RegionServer holding the region may or may not have received the close region request.
  The master retries sending the close request to the server until the RPC goes through or the master runs out of retries.
. If the RegionServer is not online, or throws `NotServingRegionException`, the master moves the region to `OFFLINE` state and re-assigns it to a different RegionServer.
. If the RegionServer is online, but not reachable after the master runs out of retries, the master moves the region to `FAILED_CLOSE` state and takes no further action until an operator intervenes from the HBase shell, or the server is dead.
. If the RegionServer gets the close region request, it closes the region and notifies the master.
  The master moves the region to `CLOSED` state and re-assigns it to a different RegionServer.
. Before assigning a region, the master moves the region to `OFFLINE` state automatically if it is in `CLOSED` state.
. When a RegionServer is about to split a region, it notifies the master.
  The master moves the region to be split from `OPEN` to `SPLITTING` state and add the two new regions to be created to the RegionServer.
  These two regions are in `SPLITTING_NEW` state initially.
. After notifying the master, the RegionServer starts to split the region.
  Once past the point of no return, the RegionServer notifies the master again so the master can update the `hbase:meta` table.
  However, the master does not update the region states until it is notified by the server that the split is done.
  If the split is successful, the splitting region is moved from `SPLITTING` to `SPLIT` state and the two new regions are moved from `SPLITTING_NEW` to `OPEN` state.
. If the split fails, the splitting region is moved from `SPLITTING` back to `OPEN` state, and the two new regions which were created are moved from `SPLITTING_NEW` to `OFFLINE` state.
. When a RegionServer is about to merge two regions, it notifies the master first.
  The master moves the two regions to be merged from `OPEN` to `MERGING` state, and adds the new region which will hold the contents of the merged regions region to the RegionServer.
  The new region is in `MERGING_NEW` state initially.
. After notifying the master, the RegionServer starts to merge the two regions.
  Once past the point of no return, the RegionServer notifies the master again so the master can update the META.
  However, the master does not update the region states until it is notified by the RegionServer that the merge has completed.
  If the merge is successful, the two merging regions are moved from `MERGING` to `MERGED` state and the new region is moved from `MERGING_NEW` to `OPEN` state.
. If the merge fails, the two merging regions are moved from `MERGING` back to `OPEN` state, and the new region which was created to hold the contents of the merged regions is moved from `MERGING_NEW` to `OFFLINE` state.
. For regions in `FAILED_OPEN` or `FAILED_CLOSE` states, the master tries to close them again when they are reassigned by an operator via HBase Shell.

[[regions.arch.locality]]
=== Region-RegionServer Locality

Over time, Region-RegionServer locality is achieved via HDFS block replication.
The HDFS client does the following by default when choosing locations to write replicas:

. First replica is written to local node
. Second replica is written to a random node on another rack
. Third replica is written on the same rack as the second, but on a different node chosen randomly
. Subsequent replicas are written on random nodes on the cluster.
  See _Replica Placement: The First Baby Steps_ on this page: link:https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html[HDFS Architecture]

Thus, HBase eventually achieves locality for a region after a flush or a compaction.
In a RegionServer failover situation a RegionServer may be assigned regions with non-local StoreFiles (because none of the replicas are local), however as new data is written in the region, or the table is compacted and StoreFiles are re-written, they will become "local" to the RegionServer.

For more information, see _Replica Placement: The First Baby Steps_ on this page: link:https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html[HDFS Architecture] and also Lars George's blog on link:http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html[HBase and HDFS locality].

[[arch.region.splits]]
=== Region Splits

Regions split when they reach a configured threshold.
Below we treat the topic in short.
For a longer exposition, see link:http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/[Apache HBase Region Splitting and Merging] by our Enis Soztutar.

Splits run unaided on the RegionServer; i.e. the Master does not participate.
The RegionServer splits a region, offlines the split region and then adds the daughter regions to `hbase:meta`, opens daughters on the parent's hosting RegionServer and then reports the split to the Master.
See <<disable.splitting>> for how to manually manage splits (and for why you might do this).

==== Custom Split Policies
You can override the default split policy using a custom
link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/RegionSplitPolicy.html[RegionSplitPolicy](HBase 0.94+).
Typically a custom split policy should extend HBase's default split policy:
link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html[IncreasingToUpperBoundRegionSplitPolicy].

The policy can set globally through the HBase configuration or on a per-table
basis.

.Configuring the Split Policy Globally in _hbase-site.xml_
[source,xml]
----
<property>
  <name>hbase.regionserver.region.split.policy</name>
  <value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value>
</property>
----

.Configuring a Split Policy On a Table Using the Java API
[source,java]
HTableDescriptor tableDesc = new HTableDescriptor("test");
tableDesc.setValue(HTableDescriptor.SPLIT_POLICY, ConstantSizeRegionSplitPolicy.class.getName());
tableDesc.addFamily(new HColumnDescriptor(Bytes.toBytes("cf1")));
admin.createTable(tableDesc);
----

[source]
.Configuring the Split Policy On a Table Using HBase Shell
----
hbase> create 'test', {METADATA => {'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}},{NAME => 'cf1'}
----

The policy can be set globally through the HBaseConfiguration used or on a per table basis:
[source,java]
----
HTableDescriptor myHtd = ...;
myHtd.setValue(HTableDescriptor.SPLIT_POLICY, MyCustomSplitPolicy.class.getName());
----

NOTE: The `DisabledRegionSplitPolicy` policy blocks manual region splitting.

[[manual_region_splitting_decisions]]
=== Manual Region Splitting

It is possible to manually split your table, either at table creation (pre-splitting), or at a later time as an administrative action.
You might choose to split your region for one or more of the following reasons.
There may be other valid reasons, but the need to manually split your table might also point to problems with your schema design.

.Reasons to Manually Split Your Table
* Your data is sorted by timeseries or another similar algorithm that sorts new data at the end of the table.
  This means that the Region Server holding the last region is always under load, and the other Region Servers are idle, or mostly idle.
  See also <<timeseries>>.
* You have developed an unexpected hotspot in one region of your table.
  For instance, an application which tracks web searches might be inundated by a lot of searches for a celebrity in the event of news about that celebrity.
  See <<perf.one.region,perf.one.region>> for more discussion about this particular scenario.
* After a big increase in the number of RegionServers in your cluster, to get the load spread out quickly.
* Before a bulk-load which is likely to cause unusual and uneven load across regions.

See <<disable.splitting>> for a discussion about the dangers and possible benefits of managing splitting completely manually.

NOTE: The `DisabledRegionSplitPolicy` policy blocks manual region splitting.

==== Determining Split Points

The goal of splitting your table manually is to improve the chances of balancing the load across the cluster in situations where good rowkey design alone won't get you there.
Keeping that in mind, the way you split your regions is very dependent upon the characteristics of your data.
It may be that you already know the best way to split your table.
If not, the way you split your table depends on what your keys are like.

Alphanumeric Rowkeys::
  If your rowkeys start with a letter or number, you can split your table at letter or number boundaries.
  For instance, the following command creates a table with regions that split at each vowel, so the first region has A-D, the second region has E-H, the third region has I-N, the fourth region has O-V, and the fifth region has U-Z.

Using a Custom Algorithm::
  The RegionSplitter tool is provided with HBase, and uses a _SplitAlgorithm_ to determine split points for you.
  As parameters, you give it the algorithm, desired number of regions, and column families.
  It includes three split algorithms.
  The first is the
  `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.HexStringSplit.html[HexStringSplit]`
  algorithm, which assumes the row keys are hexadecimal strings.
  The second is the
  `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.DecimalStringSplit.html[DecimalStringSplit]`
  algorithm, which assumes the row keys are decimal strings in the range 00000000 to 99999999.
  The third,
  `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.UniformSplit.html[UniformSplit]`,
  assumes the row keys are random byte arrays.
  You will probably need to develop your own
  `link:https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/util/RegionSplitter.SplitAlgorithm.html[SplitAlgorithm]`,
  using the provided ones as models.

=== Online Region Merges

Both Master and RegionServer participate in the event of online region merges.
Client sends merge RPC to the master, then the master moves the regions together to the RegionServer where the more heavily loaded region resided. Finally the master sends the merge request to this RegionServer which then runs the merge.
Similar to process of region splitting, region merges run as a local transaction on the RegionServer. It offlines the regions and then merges two regions on the file system, atomically delete merging regions from `hbase:meta` and adds the merged region to `hbase:meta`, opens the merged region on the RegionServer and reports the merge to the Master.

An example of region merges in the HBase shell
[source,bourne]
----
$ hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME'
$ hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true
----
It's an asynchronous operation and call returns immediately without waiting merge completed.
Passing `true` as the optional third parameter will force a merge. Normally only adjacent regions can be merged.
The `force` parameter overrides this behaviour and is for expert use only.

[[store]]
=== Store

A Store hosts a MemStore and 0 or more StoreFiles (HFiles). A Store corresponds to a column family for a table for a given region.

[[store.memstore]]
==== MemStore

The MemStore holds in-memory modifications to the Store.
Modifications are Cells/KeyValues.
When a flush is requested, the current MemStore is moved to a snapshot and is cleared.
HBase continues to serve edits from the new MemStore and backing snapshot until the flusher reports that the flush succeeded.
At this point, the snapshot is discarded.
Note that when the flush happens, MemStores that belong to the same region will all be flushed.

==== MemStore Flush

A MemStore flush can be triggered under any of the conditions listed below.
The minimum flush unit is per region, not at individual MemStore level.

. When a MemStore reaches the size specified by `hbase.hregion.memstore.flush.size`,
  all MemStores that belong to its region will be flushed out to disk.

. When the overall MemStore usage reaches the value specified by
  `hbase.regionserver.global.memstore.upperLimit`, MemStores from various regions
  will be flushed out to disk to reduce overall MemStore usage in a RegionServer.
+
The flush order is based on the descending order of a region's MemStore usage.
+
Regions will have their MemStores flushed until the overall MemStore usage drops
to or slightly below `hbase.regionserver.global.memstore.lowerLimit`.

. When the number of WAL log entries in a given region server's WAL reaches the
  value specified in `hbase.regionserver.max.logs`, MemStores from various regions
  will be flushed out to disk to reduce the number of logs in the WAL.
+
The flush order is based on time.
+
Regions with the oldest MemStores are flushed first until WAL count drops below
`hbase.regionserver.max.logs`.

[[hregion.scans]]
==== Scans

* When a client issues a scan against a table, HBase generates `RegionScanner` objects, one per region, to serve the scan request.
* The `RegionScanner` object contains a list of `StoreScanner` objects, one per column family.
* Each `StoreScanner` object further contains a list of `StoreFileScanner` objects, corresponding to each StoreFile and HFile of the corresponding column family, and a list of `KeyValueScanner` objects for the MemStore.
* The two lists are merged into one, which is sorted in ascending order with the scan object for the MemStore at the end of the list.
* When a `StoreFileScanner` object is constructed, it is associated with a `MultiVersionConcurrencyControl` read point, which is the current `memstoreTS`, filtering out any new updates beyond the read point.

[[hfile]]
==== StoreFile (HFile)

StoreFiles are where your data lives.

===== HFile Format

The _HFile_ file format is based on the SSTable file described in the link:http://research.google.com/archive/bigtable.html[BigTable [2006]] paper and on Hadoop's link:https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/file/tfile/TFile.html[TFile] (The unit test suite and the compression harness were taken directly from TFile). Schubert Zhang's blog post on link:http://cloudepr.blogspot.com/2009/09/hfile-block-indexed-file-format-to.html[HFile: A Block-Indexed File Format to Store Sorted Key-Value Pairs] makes for a thorough introduction to HBase's HFile.
Matteo Bertozzi has also put up a helpful description, link:http://th30z.blogspot.com/2011/02/hbase-io-hfile.html?spref=tw[HBase I/O: HFile].

For more information, see the HFile source code.
Also see <<hfilev2>> for information about the HFile v2 format that was included in 0.92.

[[hfile_tool]]
===== HFile Tool

To view a textualized version of HFile content, you can use the `hbase hfile` tool.
Type the following to see usage:

[source,bash]
----
$ ${HBASE_HOME}/bin/hbase hfile
----
For example, to view the content of the file _hdfs://10.81.47.41:8020/hbase/default/TEST/1418428042/DSMP/4759508618286845475_, type the following:
[source,bash]
----
 $ ${HBASE_HOME}/bin/hbase hfile -v -f hdfs://10.81.47.41:8020/hbase/default/TEST/1418428042/DSMP/4759508618286845475
----
If you leave off the option -v to see just a summary on the HFile.
See usage for other things to do with the `hfile` tool.

NOTE: In the output of this tool, you might see 'seqid=0' for certain keys in places such as 'Mid-key'/'firstKey'/'lastKey'. These are
 'KeyOnlyKeyValue' type instances - meaning their seqid is irrelevant & we just need the keys of these Key-Value instances.

[[store.file.dir]]
===== StoreFile Directory Structure on HDFS

For more information of what StoreFiles look like on HDFS with respect to the directory structure, see <<trouble.namenode.hbase.objects>>.

[[hfile.blocks]]
==== Blocks

StoreFiles are composed of blocks.
The blocksize is configured on a per-ColumnFamily basis.

Compression happens at the block level within StoreFiles.
For more information on compression, see <<compression>>.

For more information on blocks, see the HFileBlock source code.

[[keyvalue]]
==== KeyValue

The KeyValue class is the heart of data storage in HBase.
KeyValue wraps a byte array and takes offsets and lengths into the passed array which specify where to start interpreting the content as KeyValue.

The KeyValue format inside a byte array is:

* keylength
* valuelength
* key
* value

The Key is further decomposed as:

* rowlength
* row (i.e., the rowkey)
* columnfamilylength
* columnfamily
* columnqualifier
* timestamp
* keytype (e.g., Put, Delete, DeleteColumn, DeleteFamily)

KeyValue instances are _not_ split across blocks.
For example, if there is an 8 MB KeyValue, even if the block-size is 64kb this KeyValue will be read in as a coherent block.
For more information, see the KeyValue source code.

[[keyvalue.example]]
===== Example

To emphasize the points above, examine what happens with two Puts for two different columns for the same row:

* Put #1: `rowkey=row1, cf:attr1=value1`
* Put #2: `rowkey=row1, cf:attr2=value2`

Even though these are for the same row, a KeyValue is created for each column:

Key portion for Put #1:

* `rowlength ------------> 4`
* `row ------------------> row1`
* `columnfamilylength ---> 2`
* `columnfamily ---------> cf`
* `columnqualifier ------> attr1`
* `timestamp ------------> server time of Put`
* `keytype --------------> Put`

Key portion for Put #2:

* `rowlength ------------> 4`
* `row ------------------> row1`
* `columnfamilylength ---> 2`
* `columnfamily ---------> cf`
* `columnqualifier ------> attr2`
* `timestamp ------------> server time of Put`
* `keytype --------------> Put`

It is critical to understand that the rowkey, ColumnFamily, and column (aka columnqualifier) are embedded within the KeyValue instance.
The longer these identifiers are, the bigger the KeyValue is.

[[compaction]]
==== Compaction

.Ambiguous Terminology
* A _StoreFile_ is a facade of HFile.
  In terms of compaction, use of StoreFile seems to have prevailed in the past.
* A _Store_ is the same thing as a ColumnFamily.
  StoreFiles are related to a Store, or ColumnFamily.
* If you want to read more about StoreFiles versus HFiles and Stores versus ColumnFamilies, see link:https://issues.apache.org/jira/browse/HBASE-11316[HBASE-11316].

When the MemStore reaches a given size (`hbase.hregion.memstore.flush.size`), it flushes its contents to a StoreFile.
The number of StoreFiles in a Store increases over time. _Compaction_ is an operation which reduces the number of StoreFiles in a Store, by merging them together, in order to increase performance on read operations.
Compactions can be resource-intensive to perform, and can either help or hinder performance depending on many factors.

Compactions fall into two categories: minor and major.
Minor and major compactions differ in the following ways.

_Minor compactions_ usually select a small number of small, adjacent StoreFiles and rewrite them as a single StoreFile.
Minor compactions do not drop (filter out) deletes or expired versions, because of potential side effects.
See <<compaction.and.deletes>> and <<compaction.and.versions>> for information on how deletes and versions are handled in relation to compactions.
The end result of a minor compaction is fewer, larger StoreFiles for a given Store.

The end result of a _major compaction_ is a single StoreFile per Store.
Major compactions also process delete markers and max versions.
See <<compaction.and.deletes>> and <<compaction.and.versions>> for information on how deletes and versions are handled in relation to compactions.

[[compaction.and.deletes]]
.Compaction and Deletions
When an explicit deletion occurs in HBase, the data is not actually deleted.
Instead, a _tombstone_ marker is written.
The tombstone marker prevents the data from being returned with queries.
During a major compaction, the data is actually deleted, and the tombstone marker is removed from the StoreFile.
If the deletion happens because of an expired TTL, no tombstone is created.
Instead, the expired data is filtered out and is not written back to the compacted StoreFile.

[[compaction.and.versions]]
.Compaction and Versions
When you create a Column Family, you can specify the maximum number of versions to keep, by specifying `ColumnFamilyDescriptorBuilder.setMaxVersions(int versions)`.
The default value is `1`.
If more versions than the specified maximum exist, the excess versions are filtered out and not written back to the compacted StoreFile.

.Major Compactions Can Impact Query Results
[NOTE]
====
In some situations, older versions can be inadvertently resurrected if a newer version is explicitly deleted.
See <<major.compactions.change.query.results>> for a more in-depth explanation.
This situation is only possible before the compaction finishes.
====

In theory, major compactions improve performance.
However, on a highly loaded system, major compactions can require an inappropriate number of resources and adversely affect performance.
In a default configuration, major compactions are scheduled automatically to run once in a 7-day period.
This is sometimes inappropriate for systems in production.
You can manage major compactions manually.
See <<managed.compactions>>.

Compactions do not perform region merges.
See <<ops.regionmgt.merge>> for more information on region merging.

.Compaction Switch
We can switch on and off the compactions at region servers. Switching off compactions will also
interrupt any currently ongoing compactions. It can be done dynamically using the "compaction_switch"
command from hbase shell. If done from the command line, this setting will be lost on restart of the
server. To persist the changes across region servers modify the configuration hbase.regionserver
.compaction.enabled in hbase-site.xml and restart HBase.


[[compaction.file.selection]]
===== Compaction Policy - HBase 0.96.x and newer

Compacting large StoreFiles, or too many StoreFiles at once, can cause more IO load than your cluster is able to handle without causing performance problems.
The method by which HBase selects which StoreFiles to include in a compaction (and whether the compaction is a minor or major compaction) is called the _compaction policy_.

Prior to HBase 0.96.x, there was only one compaction policy.
That original compaction policy is still available as `RatioBasedCompactionPolicy`. The new compaction default policy, called `ExploringCompactionPolicy`, was subsequently backported to HBase 0.94 and HBase 0.95, and is the default in HBase 0.96 and newer.
It was implemented in link:https://issues.apache.org/jira/browse/HBASE-7842[HBASE-7842].
In short, `ExploringCompactionPolicy` attempts to select the best possible set of StoreFiles to compact with the least amount of work, while the `RatioBasedCompactionPolicy` selects the first set that meets the criteria.

Regardless of the compaction policy used, file selection is controlled by several configurable parameters and happens in a multi-step approach.
These parameters will be explained in context, and then will be given in a table which shows their descriptions, defaults, and implications of changing them.

[[compaction.being.stuck]]
====== Being Stuck

When the MemStore gets too large, it needs to flush its contents to a StoreFile.
However, Stores are configured with a bound on the number StoreFiles,
`hbase.hstore.blockingStoreFiles`, and if in excess, the MemStore flush must wait
until the StoreFile count is reduced by one or more compactions. If the MemStore
is too large and the number of StoreFiles is also too high, the algorithm is said
to be "stuck". By default we'll wait on compactions up to
`hbase.hstore.blockingWaitTime` milliseconds. If this period expires, we'll flush
anyways even though we are in excess of the
`hbase.hstore.blockingStoreFiles` count.

Upping the `hbase.hstore.blockingStoreFiles` count will allow flushes to happen
but a Store with many StoreFiles in will likely have higher read latencies. Try to
figure why Compactions are not keeping up. Is it a write spurt that is bringing
about this situation or is a regular occurance and the cluster is under-provisioned
for the volume of writes?

[[exploringcompaction.policy]]
====== The ExploringCompactionPolicy Algorithm

The ExploringCompactionPolicy algorithm considers each possible set of adjacent StoreFiles before choosing the set where compaction will have the most benefit.

One situation where the ExploringCompactionPolicy works especially well is when you are bulk-loading data and the bulk loads create larger StoreFiles than the StoreFiles which are holding data older than the bulk-loaded data.
This can "trick" HBase into choosing to perform a major compaction each time a compaction is needed, and cause a lot of extra overhead.
With the ExploringCompactionPolicy, major compactions happen much less frequently because minor compactions are more efficient.

In general, ExploringCompactionPolicy is the right choice for most situations, and thus is the default compaction policy.
You can also use ExploringCompactionPolicy along with <<ops.stripe>>.

The logic of this policy can be examined in hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java.
The following is a walk-through of the logic of the ExploringCompactionPolicy.


. Make a list of all existing StoreFiles in the Store.
  The rest of the algorithm filters this list to come up with the subset of HFiles which will be chosen for compaction.
. If this was a user-requested compaction, attempt to perform the requested compaction type, regardless of what would normally be chosen.
  Note that even if the user requests a major compaction, it may not be possible to perform a major compaction.
  This may be because not all StoreFiles in the Column Family are available to compact or because there are too many Stores in the Column Family.
. Some StoreFiles are automatically excluded from consideration.
  These include:
+
* StoreFiles that are larger than `hbase.hstore.compaction.max.size`
* StoreFiles that were created by a bulk-load operation which explicitly excluded compaction.
  You may decide to exclude StoreFiles resulting from bulk loads, from compaction.
  To do this, specify the `hbase.mapreduce.hfileoutputformat.compaction.exclude` parameter during the bulk load operation.

. Iterate through the list from step 1, and make a list of all potential sets of StoreFiles to compact together.
  A potential set is a grouping of `hbase.hstore.compaction.min` contiguous StoreFiles in the list.
  For each set, perform some sanity-checking and figure out whether this is the best compaction that could be done:
+
* If the number of StoreFiles in this set (not the size of the StoreFiles) is fewer than `hbase.hstore.compaction.min` or more than `hbase.hstore.compaction.max`, take it out of consideration.
* Compare the size of this set of StoreFiles with the size of the smallest possible compaction that has been found in the list so far.
  If the size of this set of StoreFiles represents the smallest compaction that could be done, store it to be used as a fall-back if the algorithm is "stuck" and no StoreFiles would otherwise be chosen.
  See <<compaction.being.stuck>>.
* Do size-based sanity checks against each StoreFile in this set of StoreFiles.
** If the size of this StoreFile is larger than `hbase.hstore.compaction.max.size`, take it out of consideration.
** If the size is greater than or equal to `hbase.hstore.compaction.min.size`, sanity-check it against the file-based ratio to see whether it is too large to be considered.
+
The sanity-checking is successful if:
** There is only one StoreFile in this set, or
** For each StoreFile, its size multiplied by `hbase.hstore.compaction.ratio` (or `hbase.hstore.compaction.ratio.offpeak` if off-peak hours are configured and it is during off-peak hours) is less than the sum of the sizes of the other HFiles in the set.

. If this set of StoreFiles is still in consideration, compare it to the previously-selected best compaction.
  If it is better, replace the previously-selected best compaction with this one.
. When the entire list of potential compactions has been processed, perform the best compaction that was found.
  If no StoreFiles were selected for compaction, but there are multiple StoreFiles, assume the algorithm is stuck (see <<compaction.being.stuck>>) and if so, perform the smallest compaction that was found in step 3.

[[compaction.ratiobasedcompactionpolicy.algorithm]]
====== RatioBasedCompactionPolicy Algorithm

The RatioBasedCompactionPolicy was the only compaction policy prior to HBase 0.96, though ExploringCompactionPolicy has now been backported to HBase 0.94 and 0.95.
To use the RatioBasedCompactionPolicy rather than the ExploringCompactionPolicy, set `hbase.hstore.defaultengine.compactionpolicy.class` to `RatioBasedCompactionPolicy` in the _hbase-site.xml_ file.
To switch back to the ExploringCompactionPolicy, remove the setting from the _hbase-site.xml_.

The following section walks you through the algorithm used to select StoreFiles for compaction in the RatioBasedCompactionPolicy.


. The first phase is to create a list of all candidates for compaction.
  A list is created of all StoreFiles not already in the compaction queue, and all StoreFiles newer than the newest file that is currently being compacted.
  This list of StoreFiles is ordered by the sequence ID.
  The sequence ID is generated when a Put is appended to the write-ahead log (WAL), and is stored in the metadata of the HFile.
. Check to see if the algorithm is stuck (see <<compaction.being.stuck>>, and if so, a major compaction is forced.
  This is a key area where <<exploringcompaction.policy>> is often a better choice than the RatioBasedCompactionPolicy.
. If the compaction was user-requested, try to perform the type of compaction that was requested.
  Note that a major compaction may not be possible if all HFiles are not available for compaction or if too many StoreFiles exist (more than `hbase.hstore.compaction.max`).
. Some StoreFiles are automatically excluded from consideration.
  These include:
+
* StoreFiles that are larger than `hbase.hstore.compaction.max.size`
* StoreFiles that were created by a bulk-load operation which explicitly excluded compaction.
  You may decide to exclude StoreFiles resulting from bulk loads, from compaction.
  To do this, specify the `hbase.mapreduce.hfileoutputformat.compaction.exclude` parameter during the bulk load operation.

. The maximum number of StoreFiles allowed in a major compaction is controlled by the `hbase.hstore.compaction.max` parameter.
  If the list contains more than this number of StoreFiles, a minor compaction is performed even if a major compaction would otherwise have been done.
  However, a user-requested major compaction still occurs even if there are more than `hbase.hstore.compaction.max` StoreFiles to compact.
. If the list contains fewer than `hbase.hstore.compaction.min` StoreFiles to compact, a minor compaction is aborted.
  Note that a major compaction can be performed on a single HFile.
  Its function is to remove deletes and expired versions, and reset locality on the StoreFile.
. The value of the `hbase.hstore.compaction.ratio` parameter is multiplied by the sum of StoreFiles smaller than a given file, to determine whether that StoreFile is selected for compaction during a minor compaction.
  For instance, if hbase.hstore.compaction.ratio is 1.2, FileX is 5MB, FileY is 2MB, and FileZ is 3MB:
+
----
5 <= 1.2 x (2 + 3)            or            5 <= 6
----
+
In this scenario, FileX is eligible for minor compaction.
If FileX were 7MB, it would not be eligible for minor compaction.
This ratio favors smaller StoreFile.
You can configure a different ratio for use in off-peak hours, using the parameter `hbase.hstore.compaction.ratio.offpeak`, if you also configure `hbase.offpeak.start.hour` and `hbase.offpeak.end.hour`.

. If the last major compaction was too long ago and there is more than one StoreFile to be compacted, a major compaction is run, even if it would otherwise have been minor.
  By default, the maximum time between major compactions is 7 days, plus or minus a 4.8 hour period, and determined randomly within those parameters.
  Prior to HBase 0.96, the major compaction period was 24 hours.
  See `hbase.hregion.majorcompaction` in the table below to tune or disable time-based major compactions.

[[compaction.parameters]]
====== Parameters Used by Compaction Algorithm

This table contains the main configuration parameters for compaction.
This list is not exhaustive.
To tune these parameters from the defaults, edit the _hbase-default.xml_ file.
For a full list of all configuration parameters available, see <<config.files,config.files>>

`hbase.hstore.compaction.min`::
  The minimum number of StoreFiles which must be eligible for compaction before compaction can run.
  The goal of tuning `hbase.hstore.compaction.min` is to avoid ending up with too many tiny StoreFiles
  to compact. Setting this value to 2 would cause a minor compaction each time you have two StoreFiles
  in a Store, and this is probably not appropriate. If you set this value too high, all the other
  values will need to be adjusted accordingly. For most cases, the default value is appropriate.
  In previous versions of HBase, the parameter `hbase.hstore.compaction.min` was called
  `hbase.hstore.compactionThreshold`.
+
*Default*: 3

`hbase.hstore.compaction.max`::
  The maximum number of StoreFiles which will be selected for a single minor compaction,
  regardless of the number of eligible StoreFiles. Effectively, the value of
  `hbase.hstore.compaction.max` controls the length of time it takes a single
  compaction to complete. Setting it larger means that more StoreFiles are included
  in a compaction. For most cases, the default value is appropriate.
+
*Default*: 10

`hbase.hstore.compaction.min.size`::
  A StoreFile smaller than this size will always be eligible for minor compaction.
  StoreFiles this size or larger are evaluated by `hbase.hstore.compaction.ratio`
  to determine if they are eligible. Because this limit represents the "automatic
  include" limit for all StoreFiles smaller than this value, this value may need
  to be reduced in write-heavy environments where many files in the 1-2 MB range
  are being flushed, because every StoreFile will be targeted for compaction and
  the resulting StoreFiles may still be under the minimum size and require further
  compaction. If this parameter is lowered, the ratio check is triggered more quickly.
  This addressed some issues seen in earlier versions of HBase but changing this
  parameter is no longer necessary in most situations.
+
*Default*:128 MB

`hbase.hstore.compaction.max.size`::
  A StoreFile larger than this size will be excluded from compaction. The effect of
  raising `hbase.hstore.compaction.max.size` is fewer, larger StoreFiles that do not
  get compacted often. If you feel that compaction is happening too often without
  much benefit, you can try raising this value.
+
*Default*: `Long.MAX_VALUE`

`hbase.hstore.compaction.ratio`::
  For minor compaction, this ratio is used to determine whether a given StoreFile
  which is larger than `hbase.hstore.compaction.min.size` is eligible for compaction.
  Its effect is to limit compaction of large StoreFile. The value of
  `hbase.hstore.compaction.ratio` is expressed as a floating-point decimal.
+
* A large ratio, such as 10, will produce a single giant StoreFile. Conversely,
  a value of .25, will produce behavior similar to the BigTable compaction algorithm,
  producing four StoreFiles.
* A moderate value of between 1.0 and 1.4 is recommended. When tuning this value,
  you are balancing write costs with read costs. Raising the value (to something like
  1.4) will have more write costs, because you will compact larger StoreFiles.
  However, during reads, HBase will need to seek through fewer StoreFiles to
  accomplish the read. Consider this approach if you cannot take advantage of <<blooms>>.
* Alternatively, you can lower this value to something like 1.0 to reduce the
  background cost of writes, and use  to limit the number of StoreFiles touched
  during reads. For most cases, the default value is appropriate.
+
*Default*: `1.2F`

`hbase.hstore.compaction.ratio.offpeak`::
  The compaction ratio used during off-peak compactions, if off-peak hours are
  also configured (see below). Expressed as a floating-point decimal. This allows
  for more aggressive (or less aggressive, if you set it lower than
  `hbase.hstore.compaction.ratio`) compaction during a set time period. Ignored
  if off-peak is disabled (default). This works the same as
  `hbase.hstore.compaction.ratio`.
+
*Default*: `5.0F`

`hbase.offpeak.start.hour`::
  The start of off-peak hours, expressed as an integer between 0 and 23, inclusive.
  Set to -1 to disable off-peak.
+
*Default*: `-1` (disabled)

`hbase.offpeak.end.hour`::
  The end of off-peak hours, expressed as an integer between 0 and 23, inclusive.
  Set to -1 to disable off-peak.
+
*Default*: `-1` (disabled)

`hbase.regionserver.thread.compaction.throttle`::
  There are two different thread pools for compactions, one for large compactions
  and the other for small compactions. This helps to keep compaction of lean tables
  (such as `hbase:meta`) fast. If a compaction is larger than this threshold,
  it goes into the large compaction pool. In most cases, the default value is
  appropriate.
+
*Default*: `2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size`
(which defaults to `128`)

`hbase.hregion.majorcompaction`::
  Time between major compactions, expressed in milliseconds. Set to 0 to disable
  time-based automatic major compactions. User-requested and size-based major
  compactions will still run. This value is multiplied by
  `hbase.hregion.majorcompaction.jitter` to cause compaction to start at a
  somewhat-random time during a given window of time.
+
*Default*: 7 days (`604800000` milliseconds)

`hbase.hregion.majorcompaction.jitter`::
  A multiplier applied to hbase.hregion.majorcompaction to cause compaction to
  occur a given amount of time either side of `hbase.hregion.majorcompaction`.
  The smaller the number, the closer the compactions will happen to the
  `hbase.hregion.majorcompaction` interval. Expressed as a floating-point decimal.
+
*Default*: `.50F`

[[compaction.file.selection.old]]
===== Compaction File Selection

.Legacy Information
[NOTE]
====
This section has been preserved for historical reasons and refers to the way compaction worked prior to HBase 0.96.x.
You can still use this behavior if you enable <<compaction.ratiobasedcompactionpolicy.algorithm>>. For information on the way that compactions work in HBase 0.96.x and later, see <<compaction>>.
====

To understand the core algorithm for StoreFile selection, there is some ASCII-art in the Store source code that will serve as useful reference.

It has been copied below:
[source]
----
/* normal skew:
 *
 *         older ----> newer
 *     _
 *    | |   _
 *    | |  | |   _
 *  --|-|- |-|- |-|---_-------_-------  minCompactSize
 *    | |  | |  | |  | |  _  | |
 *    | |  | |  | |  | | | | | |
 *    | |  | |  | |  | | | | | |
 */
----
.Important knobs:
* `hbase.hstore.compaction.ratio` Ratio used in compaction file selection algorithm (default 1.2f).
* `hbase.hstore.compaction.min` (in HBase v 0.90 this is called `hbase.hstore.compactionThreshold`) (files) Minimum number of StoreFiles per Store to be selected for a compaction to occur (default 2).
* `hbase.hstore.compaction.max` (files) Maximum number of StoreFiles to compact per minor compaction (default 10).
* `hbase.hstore.compaction.min.size` (bytes) Any StoreFile smaller than this setting with automatically be a candidate for compaction.
  Defaults to `hbase.hregion.memstore.flush.size` (128 mb).
* `hbase.hstore.compaction.max.size` (.92) (bytes) Any StoreFile larger than this setting with automatically be excluded from compaction (default Long.MAX_VALUE).

The minor compaction StoreFile selection logic is size based, and selects a file for compaction when the `file <= sum(smaller_files) * hbase.hstore.compaction.ratio`.

[[compaction.file.selection.example1]]
====== Minor Compaction File Selection - Example #1 (Basic Example)

This example mirrors an example from the unit test `TestCompactSelection`.

* `hbase.hstore.compaction.ratio` = 1.0f
* `hbase.hstore.compaction.min` = 3 (files)
* `hbase.hstore.compaction.max` = 5 (files)
* `hbase.hstore.compaction.min.size` = 10 (bytes)
* `hbase.hstore.compaction.max.size` = 1000 (bytes)

The following StoreFiles exist: 100, 50, 23, 12, and 12 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12.

Why?

* 100 -> No, because sum(50, 23, 12, 12) * 1.0 = 97.
* 50 -> No, because sum(23, 12, 12) * 1.0 = 47.
* 23 -> Yes, because sum(12, 12) * 1.0 = 24.
* 12 -> Yes, because the previous file has been included, and because this does not exceed the max-file limit of 5
* 12 -> Yes, because the previous file had been included, and because this does not exceed the max-file limit of 5.

[[compaction.file.selection.example2]]
====== Minor Compaction File Selection - Example #2 (Not Enough Files ToCompact)

This example mirrors an example from the unit test `TestCompactSelection`.

* `hbase.hstore.compaction.ratio` = 1.0f
* `hbase.hstore.compaction.min` = 3 (files)
* `hbase.hstore.compaction.max` = 5 (files)
* `hbase.hstore.compaction.min.size` = 10 (bytes)
* `hbase.hstore.compaction.max.size` = 1000 (bytes)

The following StoreFiles exist: 100, 25, 12, and 12 bytes apiece (oldest to newest). With the above parameters, no compaction will be started.

Why?

* 100 -> No, because sum(25, 12, 12) * 1.0 = 47
* 25 -> No, because sum(12, 12) * 1.0 = 24
* 12 -> No. Candidate because sum(12) * 1.0 = 12, there are only 2 files to compact and that is less than the threshold of 3
* 12 -> No. Candidate because the previous StoreFile was, but there are not enough files to compact

[[compaction.file.selection.example3]]
====== Minor Compaction File Selection - Example #3 (Limiting Files To Compact)

This example mirrors an example from the unit test `TestCompactSelection`.

* `hbase.hstore.compaction.ratio` = 1.0f
* `hbase.hstore.compaction.min` = 3 (files)
* `hbase.hstore.compaction.max` = 5 (files)
* `hbase.hstore.compaction.min.size` = 10 (bytes)
* `hbase.hstore.compaction.max.size` = 1000 (bytes)

The following StoreFiles exist: 7, 6, 5, 4, 3, 2, and 1 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 7, 6, 5, 4, 3.

Why?

* 7 -> Yes, because sum(6, 5, 4, 3, 2, 1) * 1.0 = 21.
  Also, 7 is less than the min-size
* 6 -> Yes, because sum(5, 4, 3, 2, 1) * 1.0 = 15.
  Also, 6 is less than the min-size.
* 5 -> Yes, because sum(4, 3, 2, 1) * 1.0 = 10.
  Also, 5 is less than the min-size.
* 4 -> Yes, because sum(3, 2, 1) * 1.0 = 6.
  Also, 4 is less than the min-size.
* 3 -> Yes, because sum(2, 1) * 1.0 = 3.
  Also, 3 is less than the min-size.
* 2 -> No.
  Candidate because previous file was selected and 2 is less than the min-size, but the max-number of files to compact has been reached.
* 1 -> No.
  Candidate because previous file was selected and 1 is less than the min-size, but max-number of files to compact has been reached.

[[compaction.config.impact]]
.Impact of Key Configuration Options

NOTE: This information is now included in the configuration parameter table in <<compaction.parameters>>.

[[ops.date.tiered]]
===== Date Tiered Compaction

Date tiered compaction is a date-aware store file compaction strategy that is beneficial for time-range scans for time-series data.

[[ops.date.tiered.when]]
===== When To Use Date Tiered Compactions

Consider using Date Tiered Compaction for reads for limited time ranges, especially scans of recent data

Don't use it for

* random gets without a limited time range
* frequent deletes and updates
* Frequent out of order data writes creating long tails, especially writes with future timestamps
* frequent bulk loads with heavily overlapping time ranges

.Performance Improvements
Performance testing has shown that the performance of time-range scans improve greatly for limited time ranges, especially scans of recent data.

[[ops.date.tiered.enable]]
====== Enabling Date Tiered Compaction

You can enable Date Tiered compaction for a table or a column family, by setting its `hbase.hstore.engine.class` to `org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine`.

You also need to set `hbase.hstore.blockingStoreFiles` to a high number, such as 60, if using all default settings, rather than the default value of 12). Use 1.5~2 x projected file count if changing the parameters, Projected file count = windows per tier x tier count + incoming window min + files older than max age

You also need to set `hbase.hstore.compaction.max` to the same value as `hbase.hstore.blockingStoreFiles` to unblock major compaction.

.Procedure: Enable Date Tiered Compaction
. Run one of following commands in the HBase shell.
  Replace the table name `orders_table` with the name of your table.
+
[source,sql]
----
alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine', 'hbase.hstore.blockingStoreFiles' => '60', 'hbase.hstore.compaction.min'=>'2', 'hbase.hstore.compaction.max'=>'60'}
alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine', 'hbase.hstore.blockingStoreFiles' => '60', 'hbase.hstore.compaction.min'=>'2', 'hbase.hstore.compaction.max'=>'60'}}
create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine', 'hbase.hstore.blockingStoreFiles' => '60', 'hbase.hstore.compaction.min'=>'2', 'hbase.hstore.compaction.max'=>'60'}
----

. Configure other options if needed.
  See <<ops.date.tiered.config>> for more information.

.Procedure: Disable Date Tiered Compaction
. Set the `hbase.hstore.engine.class` option to either nil or `org.apache.hadoop.hbase.regionserver.DefaultStoreEngine`.
  Either option has the same effect.
  Make sure you set the other options you changed to the original settings too.
+
[source,sql]
----
alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.DefaultStoreEngine'， 'hbase.hstore.blockingStoreFiles' => '12', 'hbase.hstore.compaction.min'=>'6', 'hbase.hstore.compaction.max'=>'12'}}
----

When you change the store engine either way, a major compaction will likely be performed on most regions.
This is not necessary on new tables.

[[ops.date.tiered.config]]
====== Configuring Date Tiered Compaction

Each of the settings for date tiered compaction should be configured at the table or column family level.
If you use HBase shell, the general command pattern is as follows:

[source,sql]
----
alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}}
----

[[ops.date.tiered.config.parameters]]
.Tier Parameters

You can configure your date tiers by changing the settings for the following parameters:

.Date Tier Parameters
[cols="1,1a", frame="all", options="header"]
|===
| Setting
| Notes

|`hbase.hstore.compaction.date.tiered.max.storefile.age.millis`
|Files with max-timestamp smaller than this will no longer be compacted.Default at Long.MAX_VALUE.

| `hbase.hstore.compaction.date.tiered.base.window.millis`
| Base window size in milliseconds. Default at 6 hours.

| `hbase.hstore.compaction.date.tiered.windows.per.tier`
| Number of windows per tier. Default at 4.

| `hbase.hstore.compaction.date.tiered.incoming.window.min`
| Minimal number of files to compact in the incoming window. Set it to expected number of files in the window to avoid wasteful compaction. Default at 6.

| `hbase.hstore.compaction.date.tiered.window.policy.class`
| The policy to select store files within the same time window. It doesn’t apply to the incoming window. Default at exploring compaction. This is to avoid wasteful compaction.
|===

[[ops.date.tiered.config.compaction.throttler]]
.Compaction Throttler

With tiered compaction all servers in the cluster will promote windows to higher tier at the same time, so using a compaction throttle is recommended:
Set `hbase.regionserver.throughput.controller` to `org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController`.

NOTE: For more information about date tiered compaction, please refer to the design specification at https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8
[[ops.stripe]]
===== Experimental: Stripe Compactions

Stripe compactions is an experimental feature added in HBase 0.98 which aims to improve compactions for large regions or non-uniformly distributed row keys.
In order to achieve smaller and/or more granular compactions, the StoreFiles within a region are maintained separately for several row-key sub-ranges, or "stripes", of the region.
The stripes are transparent to the rest of HBase, so other operations on the HFiles or data work without modification.

Stripe compactions change the HFile layout, creating sub-regions within regions.
These sub-regions are easier to compact, and should result in fewer major compactions.
This approach alleviates some of the challenges of larger regions.

Stripe compaction is fully compatible with <<compaction>> and works in conjunction with either the ExploringCompactionPolicy or RatioBasedCompactionPolicy.
It can be enabled for existing tables, and the table will continue to operate normally if it is disabled later.

[[ops.stripe.when]]
===== When To Use Stripe Compactions

Consider using stripe compaction if you have either of the following:

* Large regions.
  You can get the positive effects of smaller regions without additional overhead for MemStore and region management overhead.
* Non-uniform keys, such as time dimension in a key.
  Only the stripes receiving the new keys will need to compact.
  Old data will not compact as often, if at all

.Performance Improvements
Performance testing has shown that the performance of reads improves somewhat, and variability of performance of reads and writes is greatly reduced.
An overall long-term performance improvement is seen on large non-uniform-row key regions, such as a hash-prefixed timestamp key.
These performance gains are the most dramatic on a table which is already large.
It is possible that the performance improvement might extend to region splits.

[[ops.stripe.enable]]
====== Enabling Stripe Compaction

You can enable stripe compaction for a table or a column family, by setting its `hbase.hstore.engine.class` to `org.apache.hadoop.hbase.regionserver.StripeStoreEngine`.
You also need to set the `hbase.hstore.blockingStoreFiles` to a high number, such as 100 (rather than the default value of 10).

.Procedure: Enable Stripe Compaction
. Run one of following commands in the HBase shell.
  Replace the table name `orders_table` with the name of your table.
+
[source,sql]
----
alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}
alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}}
create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}
----

. Configure other options if needed.
  See <<ops.stripe.config>> for more information.
. Enable the table.

.Procedure: Disable Stripe Compaction
. Set the `hbase.hstore.engine.class` option to either nil or `org.apache.hadoop.hbase.regionserver.DefaultStoreEngine`.
  Either option has the same effect.
+
[source,sql]
----
alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'rg.apache.hadoop.hbase.regionserver.DefaultStoreEngine'}
----

. Enable the table.

When you enable a large table after changing the store engine either way, a major compaction will likely be performed on most regions.
This is not necessary on new tables.

[[ops.stripe.config]]
====== Configuring Stripe Compaction

Each of the settings for stripe compaction should be configured at the table or column family level.
If you use HBase shell, the general command pattern is as follows:

[source,sql]
----
alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}}
----

[[ops.stripe.config.sizing]]
.Region and stripe sizing

You can configure your stripe sizing based upon your region sizing.
By default, your new regions will start with one stripe.
On the next compaction after the stripe has grown too large (16 x MemStore flushes size), it is split into two stripes.
Stripe splitting continues as the region grows, until the region is large enough to split.

You can improve this pattern for your own data.
A good rule is to aim for a stripe size of at least 1 GB, and about 8-12 stripes for uniform row keys.
For example, if your regions are 30 GB, 12 x 2.5 GB stripes might be a good starting point.

.Stripe Sizing Settings
[cols="1,1a", frame="all", options="header"]
|===
| Setting
| Notes

|`hbase.store.stripe.initialStripeCount`
|The number of stripes to create when stripe compaction is enabled. You can use it as follows:

* For relatively uniform row keys, if you know the approximate
  target number of stripes from the above, you can avoid some
  splitting overhead by starting with several stripes (2, 5, 10...).
  If the early data is not representative of overall row key
  distribution, this will not be as efficient.

* For existing tables with a large amount of data, this setting
  will effectively pre-split your stripes.

* For keys such as hash-prefixed sequential keys, with more than
  one hash prefix per region, pre-splitting may make sense.


| `hbase.store.stripe.sizeToSplit`
| The maximum size a stripe grows before splitting. Use this in
conjunction with `hbase.store.stripe.splitPartCount` to
control the target stripe size (`sizeToSplit = splitPartsCount * target
stripe size`), according to the above sizing considerations.

| `hbase.store.stripe.splitPartCount`
| The number of new stripes to create when splitting a stripe. The default is 2, which is appropriate for most cases. For non-uniform row keys, you can experiment with increasing the number to 3 or 4, to isolate the arriving updates into narrower slice of the region without additional splits being required.
|===

[[ops.stripe.config.memstore]]
.MemStore Size Settings

By default, the flush creates several files from one MemStore, according to existing stripe boundaries and row keys to flush.
This approach minimizes write amplification, but can be undesirable if the MemStore is small and there are many stripes, because the files will be too small.

In this type of situation, you can set `hbase.store.stripe.compaction.flushToL0` to `true`.
This will cause a MemStore flush to create a single file instead.
When at least `hbase.store.stripe.compaction.minFilesL0` such files (by default, 4) accumulate, they will be compacted into striped files.

[[ops.stripe.config.compact]]
.Normal Compaction Configuration and Stripe Compaction

All the settings that apply to normal compactions (see <<compaction.parameters>>) apply to stripe compactions.
The exceptions are the minimum and maximum number of files, which are set to higher values by default because the files in stripes are smaller.
To control these for stripe compactions, use `hbase.store.stripe.compaction.minFiles` and `hbase.store.stripe.compaction.maxFiles`, rather than `hbase.hstore.compaction.min` and `hbase.hstore.compaction.max`.

[[ops.fifo]]
===== FIFO Compaction

FIFO compaction policy selects only files which have all cells expired. The column family *MUST* have non-default TTL.
Essentially, FIFO compactor only collects expired store files.

Because we don't do any real compaction, we do not use CPU and IO (disk and network) and evict hot data from a block cache. 
As a result, both RW throughput and latency can be improved.

[[ops.fifo.when]]
===== When To Use FIFO Compaction

Consider using FIFO Compaction when your use case is

* Very high volume raw data which has low TTL and which is the source of another data (after additional processing).
* Data which can be kept entirely in a a block cache (RAM/SSD). No need for compaction of a raw data at all.

Do not use FIFO compaction when

* Table/ColumnFamily has MIN_VERSION > 0
* Table/ColumnFamily has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)

[[ops.fifo.enable]]
====== Enabling FIFO Compaction

For Table:

[source,java]
----
HTableDescriptor desc = new HTableDescriptor(tableName);
    desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
      FIFOCompactionPolicy.class.getName());
----

For Column Family:

[source,java]
----
HColumnDescriptor desc = new HColumnDescriptor(family);
    desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
      FIFOCompactionPolicy.class.getName());
----

From HBase Shell:

[source,bash]
----
create 'x',{NAME=>'y', TTL=>'30'}, {CONFIGURATION => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.hstore.blockingStoreFiles' => 1000}}
----

Although region splitting is still supported, for optimal performance it should be disabled, either by setting explicitly `DisabledRegionSplitPolicy` or by setting `ConstantSizeRegionSplitPolicy` and very large max region size.
You will have to increase to a very large number store's blocking file (`hbase.hstore.blockingStoreFiles`) as well.
There is a sanity check on table/column family configuration in case of FIFO compaction and minimum value for number of blocking file is 1000.

[[arch.bulk.load]]
== Bulk Loading

[[arch.bulk.load.overview]]
=== Overview

HBase includes several methods of loading data into tables.
The most straightforward method is to either use the `TableOutputFormat` class from a MapReduce job, or use the normal client APIs; however, these are not always the most efficient methods.

The bulk load feature uses a MapReduce job to output table data in HBase's internal data format, and then directly load the generated StoreFiles into a running cluster.
Using bulk load will use less CPU and network resources than loading via the HBase API.

[[arch.bulk.load.arch]]
=== Bulk Load Architecture

The HBase bulk load process consists of two main steps.

[[arch.bulk.load.prep]]
==== Preparing data via a MapReduce job

The first step of a bulk load is to generate HBase data files (StoreFiles) from a MapReduce job using `HFileOutputFormat2`.
This output format writes out data in HBase's internal storage format so that they can be later loaded efficiently into the cluster.

In order to function efficiently, `HFileOutputFormat2` must be configured such that each output HFile fits within a single region.
In order to do this, jobs whose output will be bulk loaded into HBase use Hadoop's `TotalOrderPartitioner` class to partition the map output into disjoint ranges of the key space, corresponding to the key ranges of the regions in the table.

`HFileOutputFormat2` includes a convenience function, `configureIncrementalLoad()`, which automatically sets up a `TotalOrderPartitioner` based on the current region boundaries of a table.

[[arch.bulk.load.complete]]
==== Completing the data load

After a data import has been prepared, either by using the `importtsv` tool with the "`importtsv.bulk.output`" option or by some other MapReduce job using the `HFileOutputFormat`, the `completebulkload` tool is used to import the data into the running cluster.
This command line tool iterates through the prepared data files, and for each one determines the region the file belongs to.
It then contacts the appropriate RegionServer which adopts the HFile, moving it into its storage directory and making the data available to clients.

If the region boundaries have changed during the course of bulk load preparation, or between the preparation and completion steps, the `completebulkload` utility will automatically split the data files into pieces corresponding to the new boundaries.
This process is not optimally efficient, so users should take care to minimize the delay between preparing a bulk load and importing it into the cluster, especially if other clients are simultaneously loading data through other means.

[[arch.bulk.load.complete.help]]
[source,bash]
----
$ hadoop jar hbase-mapreduce-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/todd/myoutput mytable
----

The `-c config-file` option can be used to specify a file containing the appropriate hbase parameters (e.g., hbase-site.xml) if not supplied already on the CLASSPATH (In addition, the CLASSPATH must contain the directory that has the zookeeper configuration file if zookeeper is NOT managed by HBase).

[[arch.bulk.load.also]]
=== See Also

For more information about the referenced utilities, see <<importtsv>> and  <<completebulkload>>.

See link:http://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/[How-to: Use HBase Bulk Loading, and Why] for an old blog post on loading.

[[arch.bulk.load.adv]]
=== Advanced Usage

Although the `importtsv` tool is useful in many cases, advanced users may want to generate data programmatically, or import data from other formats.
To get started doing so, dig into `ImportTsv.java` and check the JavaDoc for HFileOutputFormat.

The import step of the bulk load can also be done programmatically.
See the `LoadIncrementalHFiles` class for more information.

[[arch.bulk.load.complete.strays]]
==== 'Adopting' Stray Data
Should an HBase cluster lose account of regions or files during an outage or error, you can use
the `completebulkload` tool to add back the dropped data. HBase operator tooling such as
link:https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2[HBCK2] or
the reporting added to the Master's UI under the `HBCK Report` (Since HBase 2.0.6/2.1.6/2.2.1)
can identify such 'orphan' directories.

Before you begin the 'adoption', ensure the `hbase:meta` table is in a healthy state.
Run the `CatalogJanitor` by executing the `catalogjanitor_run` command on the HBase shell.
When finished, check the `HBCK Report` page on the Master UI. Work on fixing any
inconsistencies, holes, or overlaps found before proceeding. The `hbase:meta` table
is the authority on where all data is to be found and must be consistent for
the `completebulkload` tool to work properly.

The `completebulkload` tool takes a directory and a `tablename`.
The directory has subdirectories named for column families of the targeted `tablename`.
In these subdirectories are `hfiles` to load. Given this structure, you can pass
errant region directories (and the table name to which the region directory belongs)
and the tool will bring the data files back into the fold by moving them under the
approprate serving directory. If stray files, then you will need to mock up this
structure before invoking the `completebulkload` tool; you may have to look at the
file content using the <<hfile.tool>> to see what the column family to use is.
When the tool completes its run, you will notice that the
source errant directory has had its storefiles moved/removed. It is now desiccated
since its data has been drained, and the pointed-to directory can be safely
removed. It may still have `.regioninfo` files and other
subdirectories but they are of no relevance now (There may be content still
under the _recovered_edits_ directory; a TODO is tooling to replay the
content of _recovered_edits_ if needed; see
link:https://issues.apache.org/jira/browse/HBASE-22976[Add RecoveredEditsPlayer]).
If you pass `completebulkload` a directory without store files, it will run and
note the directory is storefile-free. Just remove such 'empty' directories.

For example, presuming a directory at the top level in HDFS named
`eb3352fb5c9c9a05feeb2caba101e1cc` has data we need to re-add to the
HBase `TestTable`:

[source,bash]
----
$ ${HBASE_HOME}/bin/hbase --config ~/hbase-conf completebulkload hdfs://server.example.org:9000/eb3352fb5c9c9a05feeb2caba101e1cc TestTable
----

After it successfully completes, any files that were in  `eb3352fb5c9c9a05feeb2caba101e1cc` have been moved
under hbase and the  `eb3352fb5c9c9a05feeb2caba101e1cc` directory can be deleted (Check content
before and after by running `ls -r` on the HDFS directory).

[[arch.bulk.load.replication]]
=== Bulk Loading Replication
HBASE-13153 adds replication support for bulk loaded HFiles, available since HBase 1.3/2.0. This feature is enabled by setting `hbase.replication.bulkload.enabled` to `true` (default is `false`).
You also need to copy the source cluster configuration files to the destination cluster.

Additional configurations are required too:

. `hbase.replication.source.fs.conf.provider`
+
This defines the class which loads the source cluster file system client configuration in the destination cluster. This should be configured for all the RS in the destination cluster. Default is `org.apache.hadoop.hbase.replication.regionserver.DefaultSourceFSConfigurationProvider`.
+
. `hbase.replication.conf.dir`
+
This represents the base directory where the file system client configurations of the source cluster are copied to the destination cluster. This should be configured for all the RS in the destination cluster. Default is `$HBASE_CONF_DIR`.
+
. `hbase.replication.cluster.id`
+
This configuration is required in the cluster where replication for bulk loaded data is enabled. A source cluster is uniquely identified by the destination cluster using this id. This should be configured for all the RS in the source cluster configuration file for all the RS.
+



For example: If source cluster FS client configurations are copied to the destination cluster under directory `/home/user/dc1/`, then `hbase.replication.cluster.id` should be configured as `dc1` and `hbase.replication.conf.dir` as `/home/user`.

NOTE: `DefaultSourceFSConfigurationProvider` supports only `xml` type files. It loads source cluster FS client configuration only once, so if source cluster FS client configuration files are updated, every peer(s) cluster RS must be restarted to reload the configuration.

[[arch.hdfs]]
== HDFS

As HBase runs on HDFS (and each StoreFile is written as a file on HDFS), it is important to have an understanding of the HDFS Architecture especially in terms of how it stores files, handles failovers, and replicates blocks.

See the Hadoop documentation on link:https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html[HDFS Architecture] for more information.

[[arch.hdfs.nn]]
=== NameNode

The NameNode is responsible for maintaining the filesystem metadata.
See the above HDFS Architecture link for more information.

[[arch.hdfs.dn]]
=== DataNode

The DataNodes are responsible for storing HDFS blocks.
See the above HDFS Architecture link for more information.

[[arch.timelineconsistent.reads]]
== Timeline-consistent High Available Reads

[[casestudies.timelineconsistent.intro]]
=== Introduction

HBase, architecturally, always had the strong consistency guarantee from the start.
All reads and writes are routed through a single region server, which guarantees that all writes happen in an order, and all reads are seeing the most recent committed data.


However, because of this single homing of the reads to a single location, if the server becomes unavailable, the regions of the table that were hosted in the region server become unavailable for some time.
There are three phases in the region recovery process - detection, assignment, and recovery.
Of these, the detection is usually the longest and is presently in the order of 20-30 seconds depending on the ZooKeeper session timeout.
During this time and before the recovery is complete, the clients will not be able to read the region data.

However, for some use cases, either the data may be read-only, or doing reads against some stale data is acceptable.
With timeline-consistent high available reads, HBase can be used for these kind of latency-sensitive use cases where the application can expect to have a time bound on the read completion.


For achieving high availability for reads, HBase provides a feature called _region replication_. In this model, for each region of a table, there will be multiple replicas that are opened in different RegionServers.
By default, the region replication is set to 1, so only a single region replica is deployed and there will not be any changes from the original model.
If region replication is set to 2 or more, then the master will assign replicas of the regions of the table.
The Load Balancer ensures that the region replicas are not co-hosted in the same region servers and also in the same rack (if possible).

All of the replicas for a single region will have a unique replica_id, starting from 0.
The region replica having replica_id==0 is called the primary region, and the others _secondary regions_ or secondaries.
Only the primary can accept writes from the client, and the primary will always contain the latest changes.
Since all writes still have to go through the primary region, the writes are not highly-available (meaning they might block for some time if the region becomes unavailable).


=== Timeline Consistency

With this feature, HBase introduces a Consistency definition, which can be provided per read operation (get or scan).
[source,java]
----
public enum Consistency {
    STRONG,
    TIMELINE
}
----
`Consistency.STRONG` is the default consistency model provided by HBase.
In case the table has region replication = 1, or in a table with region replicas but the reads are done with this consistency, the read is always performed by the primary regions, so that there will not be any change from the previous behaviour, and the client always observes the latest data.


In case a read is performed with `Consistency.TIMELINE`, then the read RPC will be sent to the primary region server first.
After a short interval (`hbase.client.primaryCallTimeout.get`, 10ms by default), parallel RPC for secondary region replicas will also be sent if the primary does not respond back.
After this, the result is returned from whichever RPC is finished first.
If the response came back from the primary region replica, we can always know that the data is latest.
For this Result.isStale() API has been added to inspect the staleness.
If the result is from a secondary region, then Result.isStale() will be set to true.
The user can then inspect this field to possibly reason about the data.


In terms of semantics, TIMELINE consistency as implemented by HBase differs from pure eventual consistency in these respects:

* Single homed and ordered updates: Region replication or not, on the write side, there is still only 1 defined replica (primary) which can accept writes.
  This replica is responsible for ordering the edits and preventing conflicts.
  This guarantees that two different writes are not committed at the same time by different replicas and the data diverges.
  With this, there is no need to do read-repair or last-timestamp-wins kind of conflict resolution.
* The secondaries also apply the edits in the order that the primary committed them.
  This way the secondaries will contain a snapshot of the primaries data at any point in time.
  This is similar to RDBMS replications and even HBase's own multi-datacenter replication, however in a single cluster.
* On the read side, the client can detect whether the read is coming from up-to-date data or is stale data.
  Also, the client can issue reads with different consistency requirements on a per-operation basis to ensure its own semantic guarantees.
* The client can still observe edits out-of-order, and can go back in time, if it observes reads from one secondary replica first, then another secondary replica.
  There is no stickiness to region replicas or a transaction-id based guarantee.
  If required, this can be implemented later though.

.Timeline Consistency
image::timeline_consistency.png[Timeline Consistency]

To better understand the TIMELINE semantics, let's look at the above diagram.
Let's say that there are two clients, and the first one writes x=1 at first, then x=2 and x=3 later.
As above, all writes are handled by the primary region replica.
The writes are saved in the write ahead log (WAL), and replicated to the other replicas asynchronously.
In the above diagram, notice that replica_id=1 received 2 updates, and its data shows that x=2, while the replica_id=2 only received a single update, and its data shows that x=1.


If client1 reads with STRONG consistency, it will only talk with the replica_id=0, and thus is guaranteed to observe the latest value of x=3.
In case of a client issuing TIMELINE consistency reads, the RPC will go to all replicas (after primary timeout) and the result from the first response will be returned back.
Thus the client can see either 1, 2 or 3 as the value of x.
Let's say that the primary region has failed and log replication cannot continue for some time.
If the client does multiple reads with TIMELINE consistency, she can observe x=2 first, then x=1, and so on.


=== Tradeoffs

Having secondary regions hosted for read availability comes with some tradeoffs which should be carefully evaluated per use case.
Following are advantages and disadvantages.

.Advantages
* High availability for read-only tables
* High availability for stale reads
* Ability to do very low latency reads with very high percentile (99.9%+) latencies for stale reads

.Disadvantages
* Double / Triple MemStore usage (depending on region replication count) for tables with region replication > 1
* Increased block cache usage
* Extra network traffic for log replication
* Extra backup RPCs for replicas

To serve the region data from multiple replicas, HBase opens the regions in secondary mode in the region servers.
The regions opened in secondary mode will share the same data files with the primary region replica, however each secondary region replica will have its own MemStore to keep the unflushed data (only primary region can do flushes). Also to serve reads from secondary regions, the blocks of data files may be also cached in the block caches for the secondary regions.

=== Where is the code
This feature is delivered in two phases, Phase 1 and 2. The first phase is done in time for HBase-1.0.0 release. Meaning that using HBase-1.0.x, you can use all the features that are marked for Phase 1. Phase 2 is committed in HBase-1.1.0, meaning all HBase versions after 1.1.0 should contain Phase 2 items.

=== Propagating writes to region replicas
As discussed above writes only go to the primary region replica. For propagating the writes from the primary region replica to the secondaries, there are two different mechanisms. For read-only tables, you do not need to use any of the following methods. Disabling and enabling the table should make the data available in all region replicas. For mutable tables, you have to use *only* one of the following mechanisms: storefile refresher, or async wal replication. The latter is recommended.

==== StoreFile Refresher
The first mechanism is store file refresher which is introduced in HBase-1.0+. Store file refresher is a thread per region server, which runs periodically, and does a refresh operation for the store files of the primary region for the secondary region replicas. If enabled, the refresher will ensure that the secondary region replicas see the new flushed, compacted or bulk loaded files from the primary region in a timely manner. However, this means that only flushed data can be read back from the secondary region replicas, and after the refresher is run, making the secondaries lag behind the primary for an a longer time.

For turning this feature on, you should configure `hbase.regionserver.storefile.refresh.period` to a non-zero value. See Configuration section below.

[[async.wal.replication]]
==== Async WAL replication
The second mechanism for propagation of writes to secondaries is done via the
“Async WAL Replication” feature. It is only available in HBase-1.1+. This works
similarly to HBase’s multi-datacenter replication, but instead the data from a
region is replicated to the secondary regions. Each secondary replica always
receives and observes the writes in the same order that the primary region
committed them. In some sense, this design can be thought of as “in-cluster
replication”, where instead of replicating to a different datacenter, the data
goes to secondary regions to keep secondary region’s in-memory state up to date.
The data files are shared between the primary region and the other replicas, so
that there is no extra storage overhead. However, the secondary regions will
have recent non-flushed data in their memstores, which increases the memory
overhead. The primary region writes flush, compaction, and bulk load events
to its WAL as well, which are also replicated through wal replication to
secondaries. When they observe the flush/compaction or bulk load event, the
secondary regions replay the event to pick up the new files and drop the old
ones.

Committing writes in the same order as in primary ensures that the secondaries won’t diverge from the primary regions data, but since the log replication is asynchronous, the data might still be stale in secondary regions. Since this feature works as a replication endpoint, the performance and latency characteristics is expected to be similar to inter-cluster replication.

Async WAL Replication is *disabled* by default. You can enable this feature by
setting `hbase.region.replica.replication.enabled` to `true`.  The Async WAL
Replication feature will add a new replication peer named
`region_replica_replication` as a replication peer when you create a table with
region replication > 1 for the first time. Once enabled, if you want to disable
this feature, you need to do two actions in the following order:
* Set configuration property `hbase.region.replica.replication.enabled` to false in `hbase-site.xml` (see Configuration section below)
* Disable the replication peer named `region_replica_replication` in the cluster using hbase shell or `Admin` class:
[source,bourne]
----
        hbase> disable_peer 'region_replica_replication'
----

Async WAL Replication and the `hbase:meta` table is a little more involved and gets its own section below; see <<async.wal.replication.meta>>

=== Store File TTL
In both of the write propagation approaches mentioned above, store files of the primary will be opened in secondaries independent of the primary region. So for files that the primary compacted away, the secondaries might still be referring to these files for reading. Both features are using HFileLinks to refer to files, but there is no protection (yet) for guaranteeing that the file will not be deleted prematurely. Thus, as a guard, you should set the configuration property `hbase.master.hfilecleaner.ttl` to a larger value, such as 1 hour to guarantee that you will not receive IOExceptions for requests going to replicas.

[[async.wal.replication.meta]]
=== Region replication for META table’s region
Async WAL Replication does not work for the META table’s WAL.
The meta table’s secondary replicas refresh themselves from the persistent store
files every `hbase.regionserver.meta.storefile.refresh.period`, (a non-zero value).
Note how the META replication period is distinct from the user-space
`hbase.regionserver.storefile.refresh.period` value.

==== Async WAL Replication for META table as of hbase-2.4.0+ ====
Async WAL replication for META is added as a new feature in 2.4.0. It is still under
active development. Use with caution. Set
`hbase.region.replica.replication.catalog.enabled` to enable async WAL Replication
for META region replicas. It is off by default.

Regarding META replicas count, up to hbase-2.4.0, you would set the special
property 'hbase.meta.replica.count'. Now you can alter the META table as you
would a user-space table (if `hbase.meta.replica.count` is set, it will take
precedent over what is set for replica count in the META table updating META
replica count to match).

===== Load Balancing META table load =====

hbase-2.4.0 also adds a *new* client-side `LoadBalance` mode. When enabled
client-side, clients will try to read META replicas first before falling back on
the primary. Before this, the replica lookup mode -- now named `HedgedRead` in
hbase-2.4.0 -- had clients read the primary and if no response after a
configurable amount of time had elapsed, it would start up reads against the
replicas.

The new 'LoadBalance' mode helps alleviate hotspotting on the META
table distributing META read load.

To enable the meta replica locator's load balance mode, please set the following
configuration at on the *client-side* (only): set 'hbase.locator.meta.replicas.mode'
to "LoadBalance". Valid options for this configuration are `None`, `HedgedRead`, and
`LoadBalance`. Option parse is case insensitive.  The default mode is `None` (which falls
through to `HedgedRead`, the current default).  Do NOT put this configuration in any
hbase server-side's configuration, Master or RegionServer (Master could make decisions
based off stale state -- to be avoided).

`LoadBalance` also is a new feature. Use with caution.

=== Memory accounting
The secondary region replicas refer to the data files of the primary region replica, but they have their own memstores (in HBase-1.1+) and uses block cache as well. However, one distinction is that the secondary region replicas cannot flush the data when there is memory pressure for their memstores. They can only free up memstore memory when the primary region does a flush and this flush is replicated to the secondary. Since in a region server hosting primary replicas for some regions and secondaries for some others, the secondaries might cause extra flushes to the primary regions in the same host. In extreme situations, there can be no memory left for adding new writes coming from the primary via wal replication. For unblocking this situation (and since secondary cannot flush by itself), the secondary is allowed to do a “store file refresh” by doing a file system list operation to pick up new files from primary, and possibly dropping its memstore. This refresh will only be performed if the memstore size of the biggest secondary region replica is at least `hbase.region.replica.storefile.refresh.memstore.multiplier` (default 4) times bigger than the biggest memstore of a primary replica. One caveat is that if this is performed, the secondary can observe partial row updates across column families (since column families are flushed independently). The default should be good to not do this operation frequently. You can set this value to a large number to disable this feature if desired, but be warned that it might cause the replication to block forever.

=== Secondary replica failover
When a secondary region replica first comes online, or fails over, it may have served some edits from its memstore. Since the recovery is handled differently for secondary replicas, the secondary has to ensure that it does not go back in time before it starts serving requests after assignment. For doing that, the secondary waits until it observes a full flush cycle (start flush, commit flush) or a “region open event” replicated from the primary. Until this happens, the secondary region replica will reject all read requests by throwing an IOException with message “The region's reads are disabled”. However, the other replicas will probably still be available to read, thus not causing any impact for the rpc with TIMELINE consistency. To facilitate faster recovery, the secondary region will trigger a flush request from the primary when it is opened. The configuration property `hbase.region.replica.wait.for.primary.flush` (enabled by default) can be used to disable this feature if needed.




=== Configuration properties

To use highly available reads, you should set the following properties in `hbase-site.xml` file.
There is no specific configuration to enable or disable region replicas.
Instead you can change the number of region replicas per table to increase or decrease at the table creation or with alter table. The following configuration is for using async wal replication and using meta replicas of 3.


==== Server side properties

[source,xml]
----
<property>
    <name>hbase.regionserver.storefile.refresh.period</name>
    <value>0</value>
    <description>
      The period (in milliseconds) for refreshing the store files for the secondary regions. 0 means this feature is disabled. Secondary regions sees new files (from flushes and compactions) from primary once the secondary region refreshes the list of files in the region (there is no notification mechanism). But too frequent refreshes might cause extra Namenode pressure. If the files cannot be refreshed for longer than HFile TTL (hbase.master.hfilecleaner.ttl) the requests are rejected. Configuring HFile TTL to a larger value is also recommended with this setting.
    </description>
</property>

<property>
    <name>hbase.regionserver.meta.storefile.refresh.period</name>
    <value>300000</value>
    <description>
      The period (in milliseconds) for refreshing the store files for the hbase:meta tables secondary regions. 0 means this feature is disabled. Secondary regions sees new files (from flushes and compactions) from primary once the secondary region refreshes the list of files in the region (there is no notification mechanism). But too frequent refreshes might cause extra Namenode pressure. If the files cannot be refreshed for longer than HFile TTL (hbase.master.hfilecleaner.ttl) the requests are rejected. Configuring HFile TTL to a larger value is also recommended with this setting. This should be a non-zero number if meta replicas are enabled.
    </description>
</property>

<property>
    <name>hbase.region.replica.replication.enabled</name>
    <value>true</value>
    <description>
      Whether asynchronous WAL replication to the secondary region replicas is enabled or not. If this is enabled, a replication peer named "region_replica_replication" will be created which will tail the logs and replicate the mutations to region replicas for tables that have region replication > 1. If this is enabled once, disabling this replication also requires disabling the replication peer using shell or Admin java class. Replication to secondary region replicas works over standard inter-cluster replication.
    </description>
</property>
<property>
  <name>hbase.region.replica.replication.memstore.enabled</name>
  <value>true</value>
  <description>
    If you set this to `false`, replicas do not receive memstore updates from
    the primary RegionServer. If you set this to `true`, you can still disable
    memstore replication on a per-table basis, by setting the table's
    `REGION_MEMSTORE_REPLICATION` configuration property to `false`. If
    memstore replication is disabled, the secondaries will only receive
    updates for events like flushes and bulkloads, and will not have access to
    data which the primary has not yet flushed. This preserves the guarantee
    of row-level consistency, even when the read requests `Consistency.TIMELINE`.
  </description>
</property>

<property>
    <name>hbase.master.hfilecleaner.ttl</name>
    <value>3600000</value>
    <description>
      The period (in milliseconds) to keep store files in the archive folder before deleting them from the file system.</description>
</property>

<property>
    <name>hbase.region.replica.storefile.refresh.memstore.multiplier</name>
    <value>4</value>
    <description>
      The multiplier for a “store file refresh” operation for the secondary region replica. If a region server has memory pressure, the secondary region will refresh it’s store files if the memstore size of the biggest secondary replica is bigger this many times than the memstore size of the biggest primary replica. Set this to a very big value to disable this feature (not recommended).
    </description>
</property>

<property>
 <name>hbase.region.replica.wait.for.primary.flush</name>
    <value>true</value>
    <description>
      Whether to wait for observing a full flush cycle from the primary before start serving data in a secondary. Disabling this might cause the secondary region replicas to go back in time for reads between region movements.
    </description>
</property>
----

One thing to keep in mind also is that, region replica placement policy is only enforced by the `StochasticLoadBalancer` which is the default balancer.
If you are using a custom load balancer property in hbase-site.xml (`hbase.master.loadbalancer.class`) replicas of regions might end up being hosted in the same server.

==== Client side properties

Ensure to set the following for all clients (and servers) that will use region replicas.

[source,xml]
----
<property>
    <name>hbase.ipc.client.specificThreadForWriting</name>
    <value>true</value>
    <description>
      Whether to enable interruption of RPC threads at the client side. This is required for region replicas with fallback RPC’s to secondary regions.
    </description>
</property>
<property>
  <name>hbase.client.primaryCallTimeout.get</name>
  <value>10000</value>
  <description>
    The timeout (in microseconds), before secondary fallback RPC’s are submitted for get requests with Consistency.TIMELINE to the secondary replicas of the regions. Defaults to 10ms. Setting this lower will increase the number of RPC’s, but will lower the p99 latencies.
  </description>
</property>
<property>
  <name>hbase.client.primaryCallTimeout.multiget</name>
  <value>10000</value>
  <description>
      The timeout (in microseconds), before secondary fallback RPC’s are submitted for multi-get requests (Table.get(List<Get>)) with Consistency.TIMELINE to the secondary replicas of the regions. Defaults to 10ms. Setting this lower will increase the number of RPC’s, but will lower the p99 latencies.
  </description>
</property>
<property>
  <name>hbase.client.replicaCallTimeout.scan</name>
  <value>1000000</value>
  <description>
    The timeout (in microseconds), before secondary fallback RPC’s are submitted for scan requests with Consistency.TIMELINE to the secondary replicas of the regions. Defaults to 1 sec. Setting this lower will increase the number of RPC’s, but will lower the p99 latencies.
  </description>
</property>
<property>
    <name>hbase.meta.replicas.use</name>
    <value>true</value>
    <description>
      Whether to use meta table replicas or not. Default is false.
    </description>
</property>
----

Note HBase-1.0.x users should use `hbase.ipc.client.allowsInterrupt` rather than `hbase.ipc.client.specificThreadForWriting`.

=== User Interface

In the masters user interface, the region replicas of a table are also shown together with the primary regions.
You can notice that the replicas of a region will share the same start and end keys and the same region name prefix.
The only difference would be the appended replica_id (which is encoded as hex), and the region encoded name will be different.
You can also see the replica ids shown explicitly in the UI.

=== Creating a table with region replication

Region replication is a per-table property.
All tables have `REGION_REPLICATION = 1` by default, which means that there is only one replica per region.
You can set and change the number of replicas per region of a table by supplying the `REGION_REPLICATION` property in the table descriptor.


==== Shell

[source]
----
create 't1', 'f1', {REGION_REPLICATION => 2}

describe 't1'
for i in 1..100
put 't1', "r#{i}", 'f1:c1', i
end
flush 't1'
----

==== Java

[source,java]
----
HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(“test_table”));
htd.setRegionReplication(2);
...
admin.createTable(htd);
----

You can also use `setRegionReplication()` and alter table to increase, decrease the region replication for a table.


=== Read API and Usage

==== Shell

You can do reads in shell using a the Consistency.TIMELINE semantics as follows

[source]
----
hbase(main):001:0> get 't1','r6', {CONSISTENCY => "TIMELINE"}
----

You can simulate a region server pausing or becoming unavailable and do a read from the secondary replica:

[source,bash]
----
$ kill -STOP <pid or primary region server>

hbase(main):001:0> get 't1','r6', {CONSISTENCY => "TIMELINE"}
----

Using scans is also similar

[source]
----
hbase> scan 't1', {CONSISTENCY => 'TIMELINE'}
----

==== Java

You can set the consistency for Gets and Scans and do requests as follows.

[source,java]
----
Get get = new Get(row);
get.setConsistency(Consistency.TIMELINE);
...
Result result = table.get(get);
----

You can also pass multiple gets:

[source,java]
----
Get get1 = new Get(row);
get1.setConsistency(Consistency.TIMELINE);
...
ArrayList<Get> gets = new ArrayList<Get>();
gets.add(get1);
...
Result[] results = table.get(gets);
----

And Scans:

[source,java]
----
Scan scan = new Scan();
scan.setConsistency(Consistency.TIMELINE);
...
ResultScanner scanner = table.getScanner(scan);
----

You can inspect whether the results are coming from primary region or not by calling the `Result.isStale()` method:

[source,java]
----
Result result = table.get(get);
if (result.isStale()) {
  ...
}
----

=== Resources

. More information about the design and implementation can be found at the jira issue: link:https://issues.apache.org/jira/browse/HBASE-10070[HBASE-10070]
. HBaseCon 2014 talk: link:https://hbase.apache.org/www.hbasecon.com/#2014-PresentationsRecordings[HBase Read High Availability Using Timeline-Consistent Region Replicas] also contains some details and link:http://www.slideshare.net/enissoz/hbase-high-availability-for-reads-with-time[slides].

ifdef::backend-docbook[]
[index]
== Index
// Generated automatically by the DocBook toolchain.
endif::backend-docbook[]