1 <?xml version="1.0" encoding="UTF-8"?>
3 Licensed to the Apache Software Foundation (ASF) under one
4 or more contributor license agreements. See the NOTICE file
5 distributed with this work for additional information
6 regarding copyright ownership. The ASF licenses this file
7 to you under the Apache License, Version 2.0 (the
8 "License"); you may not use this file except in compliance
9 with the License. You may obtain a copy of the License at
11 http://www.apache.org/licenses/LICENSE-2.0
13 Unless required by applicable law or agreed to in writing,
14 software distributed under the License is distributed on an
15 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16 KIND, either express or implied. See the License for the
17 specific language governing permissions and limitations
20 <document xmlns="http://maven.apache.org/XDOC/2.0"
21 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22 xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 http://maven.apache.org/xsd/xdoc-2.0.xsd">
24 <title>Powered By Apache HBase™</title>
28 <section name="Powered By Apache HBase™">
29 <p>This page lists some institutions and projects which are using HBase. To
30 have your organization added, file a documentation JIRA or email
31 <a href="mailto:dev@hbase.apache.org">hbase-dev</a> with the relevant
32 information. If you notice out-of-date information, use the same avenues to
35 <p><b>These items are user-submitted and the HBase team assumes no responsibility for their accuracy.</b></p>
37 <dt><a href="http://www.adobe.com">Adobe</a></dt>
38 <dd>We currently have about 30 nodes running HDFS, Hadoop and HBase in clusters
39 ranging from 5 to 14 nodes on both production and development. We plan a
40 deployment on an 80 nodes cluster. We are using HBase in several areas from
41 social services to structured data and processing for internal use. We constantly
42 write data to HBase and run mapreduce jobs to process then store it back to
43 HBase or external systems. Our production cluster has been running since Oct 2008.</dd>
45 <dt><a href="http://spark-packages.org/package/Huawei-Spark/Spark-SQL-on-HBase">Project Astro</a></dt>
47 Astro provides fast Spark SQL/DataFrame capabilities to HBase data,
48 featuring super-efficient access to multi-dimensional HBase rows through
49 native Spark execution in HBase coprocessor plus systematic and accurate
50 partition pruning and predicate pushdown from arbitrarily complex data
51 filtering logic. The batch load is optimized to run on the Spark execution
52 engine. Note that <a href="http://spark-packages.org/package/Huawei-Spark/Spark-SQL-on-HBase">Spark-SQL-on-HBase</a>
53 is the release site. Interested parties are free to make clones and claim
54 to be "latest(and active)", but they are not endorsed by the owner.
57 <dt><a href="http://axibase.com/products/axibase-time-series-database/">Axibase
58 Time Series Database (ATSD)</a></dt>
59 <dd>ATSD runs on top of HBase to collect, analyze and visualize time series
60 data at scale. ATSD capabilities include optimized storage schema, built-in
61 rule engine, forecasting algorithms (Holt-Winters and ARIMA) and next-generation
62 graphics designed for high-frequency data. Primary use cases: IT infrastructure
63 monitoring, data consolidation, operational historian in OPC environments.</dd>
65 <dt><a href="http://www.benipaltechnologies.com">Benipal Technologies</a></dt>
66 <dd>We have a 35 node cluster used for HBase and Mapreduce with Lucene / SOLR
67 and katta integration to create and finetune our search databases. Currently,
68 our HBase installation has over 10 Billion rows with 100s of datapoints per row.
69 We compute over 10<sup>18</sup> calculations daily using MapReduce directly on HBase. We
72 <dt><a href="https://github.com/ermanpattuk/BigSecret">BigSecret</a></dt>
73 <dd>BigSecret is a security framework that is designed to secure Key-Value data,
74 while preserving efficient processing capabilities. It achieves cell-level
75 security, using combinations of different cryptographic techniques, in an
76 efficient and secure manner. It provides a wrapper library around HBase.</dd>
78 <dt><a href="http://caree.rs">Caree.rs</a></dt>
79 <dd>Accelerated hiring platform for HiTech companies. We use HBase and Hadoop
80 for all aspects of our backend - job and company data storage, analytics
81 processing, machine learning algorithms for our hire recommendation engine.
82 Our live production site is directly served from HBase. We use cascading for
83 running offline data processing jobs.</dd>
85 <dt><a href="http://www.celer-tech.com/">Celer Technologies</a></dt>
86 <dd>Celer Technologies is a global financial software company that creates
87 modular-based systems that have the flexibility to meet tomorrow's business
88 environment, today. The Celer framework uses Hadoop/HBase for storing all
89 financial data for trading, risk, clearing in a single data store. With our
90 flexible framework and all the data in Hadoop/HBase, clients can build new
91 features to quickly extract data based on their trading, risk and clearing
92 activities from one single location.</dd>
94 <dt><a href="https://esgyn.com/">EsgynDB</a></dt>
95 <dd>EsgynDB, powered by Apache Trafodion™, provides enterprise SQL on Hadoop.
96 It includes full ACID transactions, online transaction processing and online
97 analytic processing, along with enterprise features such as disaster recovery
98 and full backup/restore. Native tables are stored in HBase, but read and write
99 access to various other file formats such as Apache Parquet and ORC is also supported. </dd>
101 <dt><a href="http://www.explorys.net">Explorys</a></dt>
102 <dd>Explorys uses an HBase cluster containing over a billion anonymized clinical
103 records, to enable subscribers to search and analyze patient populations,
104 treatment protocols, and clinical outcomes.</dd>
106 <dt><a href="http://www.facebook.com/notes/facebook-engineering/the-underlying-technology-of-messages/454991608919">Facebook</a></dt>
107 <dd>Facebook uses HBase to power their Messages infrastructure.</dd>
109 <dt><a href="http://www.filmweb.pl">Filmweb</a></dt>
110 <dd>Filmweb is a film web portal with a large dataset of films, persons and
111 movie-related entities. We have just started a small cluster of 3 HBase nodes
112 to handle our web cache persistency layer. We plan to increase the cluster
113 size, and also to start migrating some of the data from our databases which
114 have some demanding scalability requirements.</dd>
116 <dt><a href="http://www.flurry.com">Flurry</a></dt>
117 <dd>Flurry provides mobile application analytics. We use HBase and Hadoop for
118 all of our analytics processing, and serve all of our live requests directly
119 out of HBase on our 50 node production cluster with tens of billions of rows
120 over several tables.</dd>
122 <dt><a href="http://gumgum.com">GumGum</a></dt>
123 <dd>GumGum is an In-Image Advertising Platform. We use HBase on an 15-node
124 Amazon EC2 High-CPU Extra Large (c1.xlarge) cluster for both real-time data
125 and analytics. Our production cluster has been running since June 2010.</dd>
127 <dt><a href="http://helprace.com/help-desk/">Helprace</a></dt>
128 <dd>Helprace is a customer service platform which uses Hadoop for analytics
129 and internal searching and filtering. Being on HBase we can share our HBase
130 and Hadoop cluster with other Hadoop processes - this particularly helps in
131 keeping community speeds up. We use Hadoop and HBase on small cluster with 4
132 cores and 32 GB RAM each.</dd>
134 <dt><a href="http://hubspot.com">HubSpot</a></dt>
135 <dd>HubSpot is an online marketing platform, providing analytics, email, and
136 segmentation of leads/contacts. HBase is our primary datastore for our customers'
137 customer data, with multiple HBase clusters powering the majority of our
138 product. We have nearly 200 regionservers across the various clusters, and
139 2 hadoop clusters also with nearly 200 tasktrackers. We use c1.xlarge in EC2
140 for both, but are starting to move some of that to baremetal hardware. We've
141 been running HBase for over 2 years.</dd>
143 <dt><a href="http://www.infolinks.com/">Infolinks</a></dt>
144 <dd>Infolinks is an In-Text ad provider. We use HBase to process advertisement
145 selection and user events for our In-Text ad network. The reports generated
146 from HBase are used as feedback for our production system to optimize ad
149 <dt><a href="http://www.kalooga.com">Kalooga</a></dt>
150 <dd>Kalooga is a discovery service for image galleries. We use Hadoop, HBase
151 and Pig on a 20-node cluster for our crawling, analysis and events
154 <dt><a href="http://www.leanxcale.com/">LeanXcale</a></dt>
155 <dd>LeanXcale provides an ultra-scalable transactional & SQL database that
156 stores its data on HBase and it is able to scale to 1000s of nodes. It
157 also provides a standalone full ACID HBase with transactions across
158 arbitrary sets of rows and tables.</dd>
161 <dt><a href="http://www.mahalo.com">Mahalo</a></dt>
162 <dd>Mahalo, "...the world's first human-powered search engine". All the markup
163 that powers the wiki is stored in HBase. It's been in use for a few months now.
164 MediaWiki - the same software that power Wikipedia - has version/revision control.
165 Mahalo's in-house editors produce a lot of revisions per day, which was not
166 working well in a RDBMS. An hbase-based solution for this was built and tested,
167 and the data migrated out of MySQL and into HBase. Right now it's at something
168 like 6 million items in HBase. The upload tool runs every hour from a shell
169 script to back up that data, and on 6 nodes takes about 5-10 minutes to run -
170 and does not slow down production at all.</dd>
172 <dt><a href="http://www.meetup.com">Meetup</a></dt>
173 <dd>Meetup is on a mission to help the world’s people self-organize into local
174 groups. We use Hadoop and HBase to power a site-wide, real-time activity
175 feed system for all of our members and groups. Group activity is written
176 directly to HBase, and indexed per member, with the member's custom feed
177 served directly from HBase for incoming requests. We're running HBase
178 0.20.0 on a 11 node cluster.</dd>
180 <dt><a href="http://www.mendeley.com">Mendeley</a></dt>
181 <dd>Mendeley is creating a platform for researchers to collaborate and share
182 their research online. HBase is helping us to create the world's largest
183 research paper collection and is being used to store all our raw imported data.
184 We use a lot of map reduce jobs to process these papers into pages displayed
185 on the site. We also use HBase with Pig to do analytics and produce the article
186 statistics shown on the web site. You can find out more about how we use HBase
187 in the <a href="http://www.slideshare.net/danharvey/hbase-at-mendeley">HBase
188 At Mendeley</a> slide presentation.</dd>
190 <dt><a href="http://www.ngdata.com">NGDATA</a></dt>
191 <dd>NGDATA delivers <a href="http://www.ngdata.com/site/products/lily.html">Lily</a>,
192 the consumer intelligence solution that delivers a unique combination of Big
193 Data management, machine learning technologies and consumer intelligence
194 applications in one integrated solution to allow better, and more dynamic,
195 consumer insights. Lily allows companies to process and analyze massive structured
196 and unstructured data, scale storage elastically and locate actionable data
197 quickly from large data sources in near real time.</dd>
199 <dt><a href="http://ning.com">Ning</a></dt>
200 <dd>Ning uses HBase to store and serve the results of processing user events
201 and log files, which allows us to provide near-real time analytics and
202 reporting. We use a small cluster of commodity machines with 4 cores and 16GB
203 of RAM per machine to handle all our analytics and reporting needs.</dd>
205 <dt><a href="http://www.worldcat.org">OCLC</a></dt>
206 <dd>OCLC uses HBase as the main data store for WorldCat, a union catalog which
207 aggregates the collections of 72,000 libraries in 112 countries and territories.
208 WorldCat is currently comprised of nearly 1 billion records with nearly 2
209 billion library ownership indications. We're running a 50 Node HBase cluster
210 and a separate offline map-reduce cluster.</dd>
212 <dt><a href="http://olex.openlogic.com">OpenLogic</a></dt>
213 <dd>OpenLogic stores all the world's Open Source packages, versions, files,
214 and lines of code in HBase for both near-real-time access and analytical
215 purposes. The production cluster has well over 100TB of disk spread across
216 nodes with 32GB+ RAM and dual-quad or dual-hex core CPU's.</dd>
218 <dt><a href="http://www.openplaces.org">Openplaces</a></dt>
219 <dd>Openplaces is a search engine for travel that uses HBase to store terabytes
220 of web pages and travel-related entity records (countries, cities, hotels,
221 etc.). We have dozens of MapReduce jobs that crunch data on a daily basis.
222 We use a 20-node cluster for development, a 40-node cluster for offline
223 production processing and an EC2 cluster for the live web site.</dd>
225 <dt><a href="http://www.pnl.gov">Pacific Northwest National Laboratory</a></dt>
226 <dd>Hadoop and HBase (Cloudera distribution) are being used within PNNL's
227 Computational Biology & Bioinformatics Group for a systems biology data
228 warehouse project that integrates high throughput proteomics and transcriptomics
229 data sets coming from instruments in the Environmental Molecular Sciences
230 Laboratory, a US Department of Energy national user facility located at PNNL.
231 The data sets are being merged and annotated with other public genomics
232 information in the data warehouse environment, with Hadoop analysis programs
233 operating on the annotated data in the HBase tables. This work is hosted by
234 <a href="http://www.pnl.gov/news/release.aspx?id=908">olympus</a>, a large PNNL
235 institutional computing cluster, with the HBase tables being stored in olympus's
236 Lustre file system.</dd>
238 <dt><a href="http://www.readpath.com/">ReadPath</a></dt>
239 <dd>|ReadPath uses HBase to store several hundred million RSS items and dictionary
240 for its RSS newsreader. Readpath is currently running on an 8 node cluster.</dd>
242 <dt><a href="http://resu.me/">resu.me</a></dt>
243 <dd>Career network for the net generation. We use HBase and Hadoop for all
244 aspects of our backend - user and resume data storage, analytics processing,
245 machine learning algorithms for our job recommendation engine. Our live
246 production site is directly served from HBase. We use cascading for running
247 offline data processing jobs.</dd>
249 <dt><a href="http://www.runa.com/">Runa Inc.</a></dt>
250 <dd>Runa Inc. offers a SaaS that enables online merchants to offer dynamic
251 per-consumer, per-product promotions embedded in their website. To implement
252 this we collect the click streams of all their visitors to determine along
253 with the rules of the merchant what promotion to offer the visitor at different
254 points of their browsing the Merchant website. So we have lots of data and have
255 to do lots of off-line and real-time analytics. HBase is the core for us.
256 We also use Clojure and our own open sourced distributed processing framework,
257 Swarmiji. The HBase Community has been key to our forward movement with HBase.
258 We're looking for experienced developers to join us to help make things go even
261 <dt><a href="http://www.sematext.com/">Sematext</a></dt>
263 <a href="http://www.sematext.com/search-analytics/index.html">Search Analytics</a>,
264 a service that uses HBase to store search activity and MapReduce to produce
265 reports showing user search behaviour and experience. Sematext runs
266 <a href="http://www.sematext.com/spm/index.html">Scalable Performance Monitoring (SPM)</a>,
267 a service that uses HBase to store performance data over time, crunch it with
268 the help of MapReduce, and display it in a visually rich browser-based UI.
269 Interestingly, SPM features
270 <a href="http://www.sematext.com/spm/hbase-performance-monitoring/index.html">SPM for HBase</a>,
271 which is specifically designed to monitor all HBase performance metrics.</dd>
273 <dt><a href="http://www.socialmedia.com/">SocialMedia</a></dt>
274 <dd>SocialMedia uses HBase to store and process user events which allows us to
275 provide near-realtime user metrics and reporting. HBase forms the heart of
276 our Advertising Network data storage and management system. We use HBase as
277 a data source and sink for both realtime request cycle queries and as a
278 backend for mapreduce analysis.</dd>
280 <dt><a href="http://www.splicemachine.com/">Splice Machine</a></dt>
281 <dd>Splice Machine is built on top of HBase. Splice Machine is a full-featured
282 ANSI SQL database that provides real-time updates, secondary indices, ACID
283 transactions, optimized joins, triggers, and UDFs.</dd>
285 <dt><a href="http://www.streamy.com/">Streamy</a></dt>
286 <dd>Streamy is a recently launched realtime social news site. We use HBase
287 for all of our data storage, query, and analysis needs, replacing an existing
288 SQL-based system. This includes hundreds of millions of documents, sparse
289 matrices, logs, and everything else once done in the relational system. We
290 perform significant in-memory caching of query results similar to a traditional
291 Memcached/SQL setup as well as other external components to perform joining
292 and sorting. We also run thousands of daily MapReduce jobs using HBase tables
293 for log analysis, attention data processing, and feed crawling. HBase has
294 helped us scale and distribute in ways we could not otherwise, and the
295 community has provided consistent and invaluable assistance.</dd>
297 <dt><a href="http://www.stumbleupon.com/">Stumbleupon</a></dt>
298 <dd>Stumbleupon and <a href="http://su.pr">Su.pr</a> use HBase as a real time
299 data storage and analytics platform. Serving directly out of HBase, various site
300 features and statistics are kept up to date in a real time fashion. We also
301 use HBase a map-reduce data source to overcome traditional query speed limits
304 <dt><a href="http://www.tokenizer.org">Shopping Engine at Tokenizer</a></dt>
305 <dd>Shopping Engine at Tokenizer is a web crawler; it uses HBase to store URLs
306 and Outlinks (AnchorText + LinkedURL): more than a billion. It was initially
307 designed as Nutch-Hadoop extension, then (due to very specific 'shopping'
308 scenario) moved to SOLR + MySQL(InnoDB) (ten thousands queries per second),
309 and now - to HBase. HBase is significantly faster due to: no need for huge
310 transaction logs, column-oriented design exactly matches 'lazy' business logic,
311 data compression, !MapReduce support. Number of mutable 'indexes' (term from
312 RDBMS) significantly reduced due to the fact that each 'row::column' structure
313 is physically sorted by 'row'. MySQL InnoDB engine is best DB choice for
314 highly-concurrent updates. However, necessity to flash a block of data to
315 harddrive even if we changed only few bytes is obvious bottleneck. HBase
316 greatly helps: not-so-popular in modern DBMS 'delete-insert', 'mutable primary
317 key', and 'natural primary key' patterns become a big advantage with HBase.</dd>
319 <dt><a href="http://traackr.com/">Traackr</a></dt>
320 <dd>Traackr uses HBase to store and serve online influencer data in real-time.
321 We use MapReduce to frequently re-score our entire data set as we keep updating
322 influencer metrics on a daily basis.</dd>
324 <dt><a href="https://trafodion.apache.org/">Trafodion</a></dt>
325 <dd>Apache Trafodion™ is a webscale SQL-on-Hadoop solution enabling transactional
326 or operational workloads. It uses HBase as its storage engine for SQL tables.</dd>
328 <dt><a href="http://trendmicro.com/">Trend Micro</a></dt>
329 <dd>Trend Micro uses HBase as a foundation for cloud scale storage for a variety
330 of applications. We have been developing with HBase since version 0.1 and
331 production since version 0.20.0.</dd>
333 <dt><a href="http://www.twitter.com">Twitter</a></dt>
334 <dd>Twitter runs HBase across its entire Hadoop cluster. HBase provides a
335 distributed, read/write backup of all mysql tables in Twitter's production
336 backend, allowing engineers to run MapReduce jobs over the data while maintaining
337 the ability to apply periodic row updates (something that is more difficult
338 to do with vanilla HDFS). A number of applications including people search
339 rely on HBase internally for data generation. Additionally, the operations
340 team uses HBase as a timeseries database for cluster-wide monitoring/performance
343 <dt><a href="http://www.udanax.org">Udanax.org</a></dt>
344 <dd>Udanax.org is a URL shortener which use 10 nodes HBase cluster to store URLs,
345 Web Log data and response the real-time request on its Web Server. This
346 application is now used for some twitter clients and a number of web sites.
347 Currently API requests are almost 30 per second and web redirection requests
348 are about 300 per second.</dd>
350 <dt><a href="http://www.veoh.com/">Veoh Networks</a></dt>
351 <dd>Veoh Networks uses HBase to store and process visitor (human) and entity
352 (non-human) profiles which are used for behavioral targeting, demographic
353 detection, and personalization services. Our site reads this data in
354 real-time (heavily cached) and submits updates via various batch map/reduce
355 jobs. With 25 million unique visitors a month storing this data in a traditional
356 RDBMS is not an option. We currently have a 24 node Hadoop/HBase cluster and
357 our profiling system is sharing this cluster with our other Hadoop data
358 pipeline processes.</dd>
360 <dt><a href="http://www.videosurf.com/">VideoSurf</a></dt>
361 <dd>VideoSurf - "The video search engine that has taught computers to see".
362 We're using HBase to persist various large graphs of data and other statistics.
363 HBase was a real win for us because it let us store substantially larger
364 datasets without the need for manually partitioning the data and its
365 column-oriented nature allowed us to create schemas that were substantially
366 more efficient for storing and retrieving data.</dd>
368 <dt><a href="http://www.visibletechnologies.com/">Visible Technologies</a></dt>
369 <dd>Visible Technologies uses Hadoop, HBase, Katta, and more to collect, parse,
370 store, and search hundreds of millions of Social Media content. We get incredibly
371 fast throughput and very low latency on commodity hardware. HBase enables our
372 business to exist.</dd>
374 <dt><a href="http://www.worldlingo.com/">WorldLingo</a></dt>
375 <dd>The WorldLingo Multilingual Archive. We use HBase to store millions of
376 documents that we scan using Map/Reduce jobs to machine translate them into
377 all or selected target languages from our set of available machine translation
378 languages. We currently store 12 million documents but plan to eventually
379 reach the 450 million mark. HBase allows us to scale out as we need to grow
380 our storage capacities. Combined with Hadoop to keep the data replicated and
381 therefore fail-safe we have the backbone our service can rely on now and in
382 the future. !WorldLingo is using HBase since December 2007 and is along with
383 a few others one of the longest running HBase installation. Currently we are
384 running the latest HBase 0.20 and serving directly from it at
385 <a href="http://www.worldlingo.com/ma/enwiki/en/HBase">MultilingualArchive</a>.</dd>
387 <dt><a href="http://www.yahoo.com/">Yahoo!</a></dt>
388 <dd>Yahoo! uses HBase to store document fingerprint for detecting near-duplications.
389 We have a cluster of few nodes that runs HDFS, mapreduce, and HBase. The table
390 contains millions of rows. We use this for querying duplicated documents with
391 realtime traffic.</dd>
393 <dt><a href="http://h50146.www5.hp.com/products/software/security/icewall/eng/">HP IceWall SSO</a></dt>
394 <dd>HP IceWall SSO is a web-based single sign-on solution and uses HBase to store
395 user data to authenticate users. We have supported RDB and LDAP previously but
396 have newly supported HBase with a view to authenticate over tens of millions
397 of users and devices.</dd>
399 <dt><a href="http://www.ymc.ch/en/big-data-analytics-en?utm_source=hadoopwiki&utm_medium=poweredbypage&utm_campaign=ymc.ch">YMC AG</a></dt>
401 <li>operating a Cloudera Hadoop/HBase cluster for media monitoring purpose</li>
402 <li>offering technical and operative consulting for the Hadoop stack + ecosystem</li>
403 <li>editor of <a href="http://www.ymc.ch/en/hbase-split-visualisation-introducing-hannibal?utm_source=hadoopwiki&utm_medium=poweredbypageamp;utm_campaign=ymc.ch">Hannibal</a>, a open-source tool
404 to visualize HBase regions sizes and splits that helps running HBase in production</li>