02分布式NOSQL HBASE - JAVA API 操作HBase

（原文地址：http://www.jb51.cc/cata/500599，转载麻烦带上原文地址。hadoop hive hbasemahout storm spark kafka flume,等连载中，做个爱分享的人）

HBase,可以使用shell界面管理数据，也提供java api 来操作Hbase上的表的数据。所有Column均以二进制格式存储，所以在api中要不断的类型转换。

//1 准备HBase上下文 HBaseAdmin 和user表对象hTable
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir","hdfs://rdc:9000/HRoot");
conf.set("hbase.zookeeper.quorum","p1");
Configuration conf= HBaseConfiguration.create();
HConnection conn= HConnectionManager.createConnection(conf);
HTableInterface userTable = conn.getTable("user");

//2查->Get，HBase只能根据ROW KEY 做随机查。默认是没有二级索引的。相当于只能 SELECT u.name FROM user u where u.id = 2; 不能对非主键的查询 SELECT u.name FROM user u where u.name = "rdc";
String id = "1"; //row key行键
String columnFamily = "info"; //列镞
String colume = "name"; //列名
Get get = new Get(id.getBytes());
final Result result = hTable.get(get);
byte[] valBytes = result.getValue(columnFamily .getBytes(),colume getBytes());
String value = new String(value); // value 相当于sql语句 SELECT u.name FROM user u where u.id = 1，的结果

//2增->Put ，使用Put 一列一列的加。sql： insert into user(name) values("mosi")
//3改->Put ，使用Put 一列一列的改。sql：update user set name "mosi"
Put put = new Put(id.getBytes());
put.add(columnFamily .getBytes(), colume getBytes(),"mosi".getBytes());
hTable.put(put);//和java的map一样。为空新增，不为空覆盖。

//4删 -》sql: delete from user where id = 1
Delete delete = new Delete(id.getBytes());
hTable.delete(delete);
//5 扫描，HBASE还提供scan方式根据rowkey范围扫描,不指定rowkey范围，则为全表扫描 。一次网络连接获取 批量数据缓存在客户端内存中。相对应GET来说。节省网络的连接次数。
// sql: SELECT * FROM user u
Scan scan = new Scan();
final ResultScanner scanner = hTable.getScanner(scan);
for (Result result : scanner) {
final byte[] value = result.getValue(columnFamily .getBytes(),colume.getBytes());
}

03分布式NOSQL HBASE - mapreduce批量读取HBase的数据

• java.lang.Object
• 
  
  • org.apache.hadoop.mapreduce.Mapper<ImmutableBytesWritable,Result,KEYOUT,VALUEOUT>
  • 
    
    • org.apache.hadoop.hbase.mapreduce.TableMapper<KEYOUT,VALUEOUT>
      

• Type Parameters:KEYOUT - The type of the key.VALUEOUT - The type of the value.
  

• java.lang.Object
• 
  
  • org.apache.hadoop.mapreduce.Reducer<KEYIN,org.apache.hadoop.io.Writable>
  • 
    
    • org.apache.hadoop.hbase.mapreduce.TableReducer<KEYIN,KEYOUT>
      

• Type Parameters:KEYIN - The type of the input key.VALUEIN - The type of the input value.KEYOUT - The type of the output key.
  

04分布式NOSQL HBASE - HBase过滤器

（原文地址：http://www.jb51.cc/cata/500599，转载麻烦带上原文地址。hadoop hive hbasemahout storm spark kafka flume,等连载中，做个爱分享的人）

HBase 提供scan 方式来做批量数据扫描。并提供filter机制提供更为精准的数据过滤。并且过滤过程在服务端进行。scan通过setFilter添加过滤器，实现分页、多条件查询等需求。

org.apache.hadoop.hbase.filter.FilterBase 。过滤器的基类，HBase提供了十几个实现类，分别做不同类型的过滤功能。各个过滤器还有各种不一样的过滤条件比较器。过滤功能相当的多。
1、Comparision Filters （比较类型，对行健，列镞等进行比较）
1.1 RowFilter
1.2 FamilyFilter
1.3 QualifierFilter
1.4 ValueFilter
1.5 DependentColumnFilter
2、Dedicated Filters （特殊用途类型，如分页等用途）
2.1 SingleColumnValueFilter （对列的数据过滤）
2.2 SingleColumnValueExcludeFilter
2.3 PrefixFilter
2.4 PageFilter（分页）
2.5 KeyOnlyFilter
2.6 FirstKeyOnlyFilter
2.7 TimestampsFilter
2.8 RandomrowFilter
3、Decorating Filters（Decorating ）
3.1SkipFilter
3.2 WhileMatchFilters

// 过滤器，为scan设置过滤器 。 sql: SELECT * FROM user u where id <> 2；
Scan scan = new Scan();

CompareOp compareOp = CompareOp.NO_OP; // != 比较符；
WritableByteArrayComparable comparable = new BinaryComparator(rowkey.getBytes()) //二进制比较器;
RowFilter rowFilter = new RowFilter(compareOp,comparable); //rowkey 过滤器
scan .setFilter(rowFilter);
final ResultScanner scanner = hTable.getScanner(scan);
for (Result result : scanner) {
final byte[] value = result.getValue(columnFamily .getBytes(),colume.getBytes()); }

05分布式NOSQL HBASE - Coprocessor协调器

（原文地址：http://www.jb51.cc/cata/500599，转载麻烦带上原文地址。hadoop hive hbasemahout storm spark kafka flume,等连载中，做个爱分享的人）

1 那么问题又来了????
1：如果要对HBase的User表，做一些sum，orderby等常用的聚合怎么搞。1：如果javaapi 全部读出来，表太大时显然这种方案不合适，2,mapreduce 分布式的批量读出再算，但是当数据小的时候，mr任务要消耗的资源大，有点牛刀。
为解决这种 sum，orderby等常用的聚合聚合操作的，HBase的org.apache.hadoop.hbase.coprocessor 协处理器机智，提供在服务端region运行代码的机制，的解决方案，如GET 操作，直接在服务端执行。并不需要把结果返回回来。中国山东找蓝翔。

2：假设我们在user表执行DELETE 或者GET等操作之后，要做操作日志
HBase的org.apache.hadoop.hbase.coprocessor 协处理器也能处理这样的需求。

2 协处理器概述
HBase有全表，和单表两种作用域的协处理器，以及 observer和endpoint两种调用方式的协处理器，observer为hbase 的GET,PUT,SCAN,DELETE等操作提供了Hook钩子，类似于onClick do，适合在表的某些操作前后的需求。endpoint是一个服务端的方法，我们要使用时，通过rpc可以直接调用，适合做聚合。

3.1 协处理器observer模式关键API

3.2 协处理器endpoint（RPC）模式关键API
3.2.1服务端类关键接口
//接受到请求后，真正执行的方法。
实现com.google.protobuf.Service里的接口

//初始化获取env对象。
public void start(coprocessorEnvironment env) throws IOException {}
//结束时关闭相关资源
public void stop(coprocessorEnvironment env) throws IOException{}

3.2.2客户端端类关键接口
//发起请求
coprocessorService(java.lang.class,byte[],org.apache.hadoop.hbase.client.coprocessor.Batch.Call)]coprocessorService[/url](Class<T> service,byte[] startKey,byte[] endKey,org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)

//执行rpc请求的时候，真正执行的代码，在服务端执行。
public Long call(TestEndpoint.CountService counter)


4.1 协处理器observer模式程序
有空再补充
4.2 协处理器endpoint（RPC）模式程序 有空再补充

今天关于01分布式NOSQL HBASE HBase 概述和数据模型和分布式数据库hbase的讲解已经结束，谢谢您的阅读，如果想了解更多关于02分布式NOSQL HBASE - JAVA API 操作HBase、03分布式NOSQL HBASE - mapreduce批量读取HBase的数据、04分布式NOSQL HBASE - HBase过滤器、05分布式NOSQL HBASE - Coprocessor协调器的相关知识，请在本站搜索。

以上就是给各位分享NoSQL数据库的类型，其中也会对nosql数据库的类型包括进行解释，同时本文还将给你拓展AWS Around---NoSql数据库的不同类型、NoSQL 数据库的类型、nosql-redis-网络资料学习-05-nosql数据库的四大分类、NoSQL数据库的主主备份等相关知识，如果能碰巧解决你现在面临的问题，别忘了关注本站，现在开始吧！

• NoSQL数据库的类型（nosql数据库的类型包括）
• AWS Around---NoSql数据库的不同类型
• NoSQL 数据库的类型
• nosql-redis-网络资料学习-05-nosql数据库的四大分类
• NoSQL数据库的主主备份

NoSQL数据库的类型（nosql数据库的类型包括）

Nosql纪元

当下已经存在很多的Nosql数据库，比如MongoDB、Redis、Riak、HBase、Cassandra等等。每一个都拥有以下几个特性中的一个：

• 不再使用sql语言，比如MongoDB、Cassandra就有自己的查询语言
• 通常是开源项目
• 为集群运行而生
• 弱结构化——不会严格的限制数据结构类型

AWS Around---NoSql数据库的不同类型

NoSQL 数据库的类型

一、 键值（Key-Value）数据库

键值数据库就像在传统语言中使用的哈希表。你可以通过 key 来添加、查询或者删除数据，鉴于使用主键访问，所以会获得不错的性能及扩展性。

产品：Riak、Redis、Memcached、Amazon’s Dynamo、Project Voldemort

有谁在使用：GitHub （Riak）、BestBuy （Riak）、Twitter （Redis 和 Memcached）、StackOverFlow （Redis）、 Instagram （Redis）、Youtube （Memcached）、Wikipedia（Memcached）

适用的场景

储存用户信息，比如会话、配置文件、参数、购物车等等。这些信息一般都和 ID（键）挂钩，这种情景下键值数据库是个很好的选择。

不适用场景

1. 取代通过键查询，而是通过值来查询。Key-Value 数据库中根本没有通过值查询的途径。

2. 需要储存数据之间的关系。在 Key-Value 数据库中不能通过两个或以上的键来关联数据。

3. 事务的支持。在 Key-Value 数据库中故障产生时不可以进行回滚。

二、 面向文档（Document-Oriented）数据库

面向文档数据库会将数据以文档的形式储存。每个文档都是自包含的数据单元，是一系列数据项的集合。每个数据项都有一个名称与对应的值，值既可以是简单的数据类型，如字符串、数字和日期等；也可以是复杂的类型，如有序列表和关联对象。数据存储的最小单位是文档，同一个表中存储的文档属性可以是不同的，数据可以使用 XML、JSON 或者 JSONB 等多种形式存储。

产品：MongoDB、CouchDB、RavenDB

有谁在使用：SAP （MongoDB）、Codecademy （MongoDB）、Foursquare （MongoDB）、NBC News （RavenDB）

适用的场景

1. 日志。企业环境下，每个应用程序都有不同的日志信息。Document-Oriented 数据库并没有固定的模式，所以我们可以使用它储存不同的信息。

2. 分析。鉴于它的弱模式结构，不改变模式下就可以储存不同的度量方法及添加新的度量。

不适用场景

在不同的文档上添加事务。Document-Oriented 数据库并不支持文档间的事务，如果对这方面有需求则不应该选用这个解决方案。

三、 列存储（Wide Column Store/Column-Family）数据库

列存储数据库将数据储存在列族（column family）中，一个列族存储经常被一起查询的相关数据。举个例子，如果我们有一个 Person 类，我们通常会一起查询他们的姓名和年龄而不是薪资。这种情况下，姓名和年龄就会被放入一个列族中，而薪资则在另一个列族中。

产品：Cassandra、HBase

有谁在使用：Ebay （Cassandra）、Instagram （Cassandra）、NASA （Cassandra）、Twitter （Cassandra and HBase）、Facebook （HBase）、Yahoo!（HBase）

适用的场景

1. 日志。因为我们可以将数据储存在不同的列中，每个应用程序可以将信息写入自己的列族中。

2. 博客平台。我们储存每个信息到不同的列族中。举个例子，标签可以储存在一个，类别可以在一个，而文章则在另一个。

不适用场景

1. 如果我们需要 ACID 事务。Vassandra 就不支持事务。

2. 原型设计。如果我们分析 Cassandra 的数据结构，我们就会发现结构是基于我们期望的数据查询方式而定。在模型设计之初，我们根本不可能去预测它的查询方式，而一旦查询方式改变，我们就必须重新设计列族。

四、 图（Graph-Oriented）数据库

图数据库允许我们将数据以图的方式储存。实体会被作为顶点，而实体之间的关系则会被作为边。比如我们有三个实体，Steve Jobs、Apple 和 Next，则会有两个 “Founded by” 的边将 Apple 和 Next 连接到 Steve Jobs。

产品：Neo4J、Infinite Graph、OrientDB

有谁在使用：Adobe （Neo4J）、Cisco （Neo4J）、T-Mobile （Neo4J）

适用的场景

1. 在一些关系性强的数据中

2. 推荐引擎。如果我们将数据以图的形式表现，那么将会非常有益于推荐的制定

不适用场景

不适合的数据模型。图数据库的适用范围很小，因为很少有操作涉及到整个图

nosql-redis-网络资料学习-05-nosql数据库的四大分类

nosql数据库的四大分类

1、NoSql数据库的四大分类

kv、列值、文档、图

## 1.1 kv 键值 ##
应用举例（16年数据）

• 新浪：BerkeleyDB+redis
• 美团：redis+tair
• 阿里、百度：memcache+redis
• 列表项目

1.2 文档型数据库(bson格式比较多)

例如

• CouchDB
• MongoDB
  MongoDB 是一个基于分布式文件存储的数据库。由 C++ 语言编写。旨在为 WEB 应用提供可扩 展的高性能数据存储解决方案。MongoDB 是一个介于关系数据库和非关系数据库之间的产品，是非关系数据库当中功能最丰富，最像关系数据库的。

1.3 列存储数据库

例如

• Cassandra
• HBase

1.4 图关系型数据库

它不是放图形的，放的是关系比如:朋友圈社交网络、广告推荐系统，社交网络，推荐系统等。专注于构建关系图谱

例如：

• Neo4J
• InfoGrid

NoSQL数据库的主主备份

《Nosql数据库的主主备份》要点：
本文介绍了Nosql数据库的主主备份，希望对您有用。如果有疑问，可以联系我们。

翻译 | 贺雨言

Tarantool DBMS的高性能应该很多人都听说过,包含其丰富的工具套件和某些特定功能.比如,它拥有一个非常强大的on-disk存储引擎Vinyl,并且知道怎样处理JSON文档.然而,大部分文章往往忽略了一个关键点：通常,Tarantool仅仅被视为存储器,而实际上其最大特点是能够在存储器内部写代码,从而高效处理数据.如果你想知道我和igorcoding是怎样在Tarantool内部建立一个系统的,请继续往下看.

如果你用过Mail.Ru电子邮件服务,你应该知道它可以从其他账号收集邮件.如果支持OAuth协议,那么在收集其他账号的邮件时,我们就不必要让用户提供第三方服务凭证了,而是用OAuth令牌来代替.此外,Mail.Ru Group有很多项目要求通过第三方服务授权,并且必要用户的OAuth令牌才能处理某些应用.因此,我们决定建立一个存储和更新令牌的服务.

我猜大家都知道OAuth令牌是什么样的,闭上眼睛回忆一下,OAuth结构由以下3-4个字段组成：

• 拜访令牌(access_token)——允许你执行动作、获取用户数据、下载用户的好友列表等等；

• 更新令牌(refresh_token)——让你重新获取新的access_token,不限次数；

• 过期时间(expires_in)——令牌到期时间戳或任何其他预定义时间,如果你的access_token到期了,你就不能继续拜访所需的资源.

现在我们看一下服务的简单框架.设想有一些前端可以在我们的服务上写入和读出令牌,还有一个独立的更新器,一旦令牌到期,就可以通过更新器从OAuth服务提供商获取新的拜访令牌.

如上图所示,数据库的结构也十分简单,由两个数据库节点(主和从)组成,为了说明两个数据库节点分别位于两个数据中心,二者之间由一条垂直的虚线隔开,其中一个数据中心包含主数据库节点及其前端和更新器,另一个数据中心包含从数据库节点及其前端,以及拜访主数据库节点的更新器.

面临的困难

我们面临的主要问题在于令牌的使用期(一个小时).详细了解这个项目之后,也许有人会问“在一小时内更新1000万条记录,这真的是高负载服务吗?如果我们用一个数除一下,结果大约是3000rps”.然而,如果因为数据库维护或故障,甚至服务器故障(一切皆有可能)导致一部分记录没有得到更新,那事情将会变得比拟麻烦.比如,如果我们的服务(主数据库)因为某些原因持续中断15分钟,就会导致25%的服务中断(四分之一的令牌变成无效,不能再继续使用)；如果服务中断30分钟,将会有一半的数据不能得到更新；如果中断1小时,那么所有的令牌都将失效.假设数据库瘫痪一个小时,我们重启系统,然后整个1000万条令牌都需要进行快速更新.这算不算高负载服务呢?

一开始一切都还进展地比拟顺利,但是两年后,我们进行了逻辑扩展,增加了几个指标,并且开始执行一些辅助逻辑…….总之,Tarantool耗尽了cpu资源.尽管所有资源都是递耗资源,但这样的结果确实让我们大吃一惊.

幸运的是,系统管理员帮我们安装了当时库存中内存最大的cpu,解决了我们随后6个月的cpu需求.但这只是权宜之计,我们必须想出一个解决方法.当时,我们学习了一个新版的Tarantool(我们的系统是用Tarantool 1.5写的,这个版本除了在Mail.Ru Group,其他地方基本没用过).Tarantool 1.6大力提倡主主备份,于是我们想：为什么不在连接主主备份的三个数据中心分别建立一个数据库备份呢?这听起来是个不错的计划.

三个主机、三个数据中心和三个更新器,都分别连接自己的主数据库.即使一个或者两个主机瘫痪了,系统仍然照常运行,对吧?那么这个方案的缺点是什么呢?缺点就是,我们将一个OAuth服务提供商的哀求数量有效地增加到了三倍,也就是说,有多少个副本,我们就要更新几乎相同数量的令牌,这样不行.最直接的解决办法就是,想办法让各个节点自己决定谁是leader,那样就只需要更新存储在leader上的节点了.

选择leader节点

选择leader节点的算法有很多,其中有一个算法叫paxos,相当复杂,不知道怎样简化,于是我们决定用Raft代替.Raft是一个非常通俗易懂的算法,谁能通信就选谁做leader,一旦通信连接失败或者其他因素,就重新选leader.具体实施方法如下：

Tarantool外部既没有Raft也没有paxos,但是我们可以使用net.Box内置模式,让所有节点连接成一个网状网(即每一个节点连接剩下所有节点),然后直接在这些连接上用Raft算法选出leader节点.最后,所有节点要么成为leader节点,要么成为follower节点,或者二者都不是.

如果你觉得Raft算法实施起来有困难,下面的Lua代码可以帮到你：

现在我们给远程服务器发送哀求(其他Tarantool副本)并计算来自每一个节点的票数,如果我们有一个quorum,我们就选定了一个leader,然后发送heartbeats,告诉其他节点我们还活着.如果我们在选举中失败了,我们可以发起另一场选举,一段时间之后,我们又可以投票或被选为leader.

只要我们有一个quorum,选中一个leader,我们就可以将更新器指派给所有节点,但是只准它们为leader服务.

这样我们就规范了流量,由于任务是由单一的节点派出,因此每一个更新器获得大约三分之一的任务,有了这样的设置,我们可以失去任何一台主机,因为如果某台主机出故障了,我们可以发起另一个选举,更新器也可以切换到另一个节点.然而,和其他分布式系统一样,有好几个问题与quorum有关.

“废弃”节点

如果各个数据中心之间失去联系了,那么我们必要有一些适当的机制去维持整个系统正常运转,还必要有一套机制能恢复系统的完整性.Raft成功地做到了这两点：

假设Dataline数据中心掉线了,那么该位置的节点就变成了“废弃”节点,也就是说该节点就看不到其他节点了,集群中的其他节点可以看到这个节点丢失了,于是引发了另一个选举,然后新的集群节点(即上级节点)被选为leader,整个系统仍然坚持运转,因为各个节点之间仍然坚持一致性(大半部分节点仍然互相可见).

那么问题来了,与丢失的数据中心有关的更新器怎么样了呢?Raft说明书没有给这样的节点一个单独的名字,通常,没有quorum的节点和不能与leader联系的节点会被闲置下来.然而,它可以本身建立网络连接然后更新令牌,一般来说,令牌都是在连接模式时更新,但是,也许用一个连接“废弃”节点的更新器也可以更新令牌.一开始我们并不确定这样做有意义,这样不会导致冗余更新吗?

这个问题我们必要在实施系统的过程中搞清楚.我们的第一个想法是不更新：我们有一致性、有quorum,丢失任何一个成员,我们都不应该更新.但是后来我们有了另一个想法,我们看一下Tarantool中的主主备份,假设有两个主节点和一个变量(key)X=1,我们同时在每一个节点上给这个变量赋一个新值,一个赋值为2,另一个赋值为3,然后,两个节点互相交换备份日志(就是X变量的值).在一致性上,这样实施主主备份是很糟糕的(无意冒犯Tarantool开发者).

如果我们必要严格的一致性,这样是行不通的.然而,回忆一下我们的OAuth令牌是由以下两个重要因素组成：

• 更新令牌,本色上永久有效；

• 拜访令牌,有效期为一个小时；

我们的更新器有一个refresh函数,可以从一个更新令牌获取任意数量的拜访令牌,一旦发布,它们都将保持一个小时内有效.

我们考虑一下以下场景：两个follower节点正在和一个leader节点交互,它们更新自己的令牌,接收第一个拜访令牌,这个拜访令牌被复制,于是现在每一个节点都有这个拜访令牌,连接中断了,所以,其中一个follower节点变成了“废弃”节点,它没有quorum,既看不到leader也看不到其他follower,然而,我们允许我们的更新器去更新位于“废弃”节点上的令牌,如果“废弃”节点没有连接网络,那么整个方案都将停止运行.尽管如此,如果发生简单的网络拆分,更新器还是可以维持正常运行.

一旦网络拆分结束,“废弃”节点重新加入集群,就会引发另一场选举或者数据交换.注意,第二和第三个令牌一样,也是“好的”.

原始的集群成员恢复之后,下一次更新将只在一个节点上发生,然后备份.换句话来说,当集群拆分之后,被拆分的各个部分各自独立更新,但是一旦重新整合,数据一致性也因此恢复.通常,需要N/2+1个活动节点(对于一个3节点集群,就是需要2个活动节点)去保持集群正常运转.尽管如此,对我们而言,即使只有1个活动节点也足够了,它会发送尽可能多的外部哀求.

重申一下,我们已经讨论了哀求数量逐渐增加的情况,在网络拆分或节点中断时期,我们能够提供一个单一的活动节点,我们会像平时一样更新这个节点,如果出现绝对拆分(即当一个集群被分成最大数量的节点,每一个节点有一个网络连接),如上所述,OAuth服务提供商的哀求数量将提升至三倍.但是,由于这个事件发生的时间相对短暂,所以情况不是太糟,我们可不希望一直工作在拆分模式.通常情况下,系统处于有quorum和网络连接,并且所有节点都启动运行的状态.

分片

还有一个问题没有解决：我们已经达到了cpu上限,最直接的解决方法就是分片.

假设我们有两个数据库分片,每一个都有备份,有一个这样的函数,给定一些key值,就可以计算出哪一个分片上有所必要的数据.如果我们通过电子邮件分片,一部分地址存储在一个分片上,另一部分地址存储在另一个分片上,我们很清楚我们的数据在哪里.

有两种办法可以分片.一种是客户端分片,我们选择一个返回分片数量的连续的分片函数,比如CRC32、Guava或Sumbur,这个函数在所有客户端的实现方式都一样.这种办法的一个明显优势在于数据库对分片一无所知,你的数据库正常运转,然后分片就发生了.

然而,这种办法也存在一个很严重的缺陷.一开始,客户端非常繁忙.如果你想要一个新的分片,你需要把分片逻辑加进客户端,这里的最大的问题是,可能一些客户端在使用这种模式,而另一些客户端却在使用另一种完全不同的模式,而数据库本身却不知道有两种不同的分片模式.

我们选择另一种办法—数据库内部分片,这种情况下,数据库代码变得更加复杂,但是为了折中我们可以使用简单的客户端,每一个连接数据库的客户端被路由到任意节点,由一个特殊函数计算出哪一个节点应该被连接、哪一个节点应该被控制.前面提到,由于数据库变得更加复杂,因此为了折中,客户端就变得更加简单了,但是这样的话,数据库就要对其数据全权负责.此外,最困难的事就是重新分片,如果你有一大堆客户端无法更新,相比之下,如果数据库负责管理自己的数据,那重新分片就会变得非常简单.

具体怎样实施呢?

六边形代表Tarantool实体,有3个节点组成分片1,另一个3节点集群作为分片2,如果我们将所有节点互相连接,结果会怎样呢?根据Raft,我们可以知道每一个集群的状态,谁是leader服务器谁是follower服务器也一目了然,由于是集群内连接,我们还可以知道其他分片(例如它的leader分片或者follower分片)的状态.总的来说,如果拜访第一个分片的用户发现这并不是他需要的分片,我们很清楚地知道应该指导他往哪里走.

我们来看一些简单的例子：

假设用户向驻留在第一个分片上的key发出哀求,该哀求被第一个分片上的某一个节点接收,这个节点知道谁是leader,于是将哀求重新路由到分片leader,反过来,分片leader对这个key进行读或写,并且将结果反馈给用户.

第二个场景：用户的哀求到达第一个分片中的相同节点,但是被哀求的key却在第二个分片上,这种情况也可以用类似的方法处理,第一个分片知道第二个分片上谁是leader,然后把哀求送到第二个分片的leader进行转发和处理,再将结果返回给用户.

这个方案十分简单,但也存在必定的缺陷,其中最大的问题就是连接数,在二分片的例子中,每一个节点连接到其他剩下的节点,连接数是6*5=30,如果再加一个3节点分片,那么连接数就增加到72,这会不会有点多呢?

我们该如何解决这个问题呢?我们只需要增加一些Tarantool实例,我们叫它代理,而不叫分片或数据库,用代理去解决所有的分片问题：包括计算key值和定位分片领导.另一方面,Raft集群保持自包含,只在分片内部工作.当用户拜访代理时,代理计算出所需要的分片,如果需要的是leader,就对用户作相应的重定向,如果不是leader,就将用户重定向至分片内的任意节点.

由此产生的复杂性是线性的,取决于节点数量.现在一共3个节点,每个节点3个分片,连接数少了几倍.

代理方案的设计考虑到了进一步规模扩展(当分片数量大于2时),当只有2个分片时,连接数不变,但是当分片数量增加时,连接数会剧减.分片列表存储在Lua配置文件中,如果想要获取新列表,我们只必要重载代码就好了.

综上所述,首先,我们进行主主备份,应用Raft算法,然后加入分片和代理,最后我们得到的是一个单块,一个集群,所以说,目前这个方案看上去是比拟简单的.

剩下的就是只读或只写令牌的的前端了,我们有更新器可以更新令牌,获得更新令牌后把它传到OAuth服务提供商,然后写一个新的拜访令牌.

前面说过我们的一些辅助逻辑耗尽了cpu资源,现在我们将这些辅助资源移到另一个集群上.

辅助逻辑主要和地址簿有关,给定一个用户令牌,就会有一个对应的地址簿,地址簿上的数据量和令牌一样,为了不耗尽一台机器上的cpu资源,我们显然需要一个与副原形同的集群,只需要加一堆更新地址簿的更新器就可以了(这个任务比较少见,因此地址簿不会和令牌一起更新).

最后,通过整合这两个集群,我们得到一个相对简单的完整结构：

令牌更新队列

为什么我们本可以使用标准队列却还要用本身的队列呢?这和我们的令牌更新模型有关.令牌一旦发布,有效期就是一个小时,当令牌快要到期时,需要进行更新,而令牌更新必须在某个特定的时间点之前完成.

假设系统中断了,但是我们有一堆已到期的令牌,而在我们更新这些令牌的同时,又有其他令牌陆续到期,虽然我们最后肯定能全部更新完,但是如果我们先更新那些即将到期的(60秒内),再用剩下的资源去更新已经到期的,是不是会更合理一些?(优先级别最低的是还有4-5分钟才到期的令牌)

用第三方软件来实现这个逻辑并不是件容易的事,对于Tarantool来说却不费吹灰之力.看一个简单的方案：在Tarantool中有一个存储数据的元组,这个元组的一些ID设置了基础key值,为了得到我们必要的队列,我们只必要添加两个字段：status(队列令牌状态)和time(到期时间或其他预定义时间).

现在我们考虑一下队列的两个主要功能—put和take.put就是写入新数据.给定一些负载,put时本身设置好status和time,然后写数据,这就是建立一个新的元组.

至于take,是指建立一个基于索引的迭代器,挑出那些等待办理的任务(处于就绪状态的任务),然后核查一下是不是该接收这些任务了,或者这些任务是否已经到期了.如果没有任务,take就切换到wait模式.除了内置Lua,Tarantool还有一些所谓的通道,这些通道本质上是互联光纤同步原语.任何光纤都可以建立一个通道然后说“我在这等着”,剩下的其他光纤可以唤醒这个通道然后给它发送信息.

等待中的函数(等待发布任务、等待指定时间或其他)建立一个通道,给通道贴上适当的标签,将通道放置在某个地方,然后进行监听.如果我们收到一个紧急的更新令牌,put会给通道发出通知,然后take接收更新任务.

Tarantool有一个特殊的功能：如果一个令牌被意外发布,或者一个更新令牌被take接收,或者只是出现接收任务的现象,以上三种情况Tarantool都可以跟踪到客户端中断.我们将每一个连接与指定给该连接的任务联系起来,并将这些映射关系坚持在会话保存中.假设由于网络中断导致更新过程失败,而且我们不知道这个令牌是否会被更新并被写回到数据库.于是,客户端发生中断了,搜索与失败过程相关的所有任务的会话保存,然后自动将它们释放.随后,任意已发布的任务都可以用同一个通道给另一个put发送信息,该put会快速接收和执行任务.

实际上,具体实施方案并不必要太多代码：

Put只是接收用户想要插入队列的所有数据,并将其写入某个空间,如果是一个简单的索引式FIFO队列,设置好状态和当前时间,然后返回该任务.

接下来要和take有点关系了,但仍然比拟简单.我们建立一个迭代器,等待接收新任务.Taken函数只需要将任务标记成“已接收”,但有一点很重要,taken函数还能记住哪个任务是由哪个进程接收的.On_disconnect函数可以发布某个特定连接,或者发布由某个特定用户接收的所有任务.

是否有可选方案

当然有.我们本可以使用任意数据库,不管我们选用什么数据库,我们都要建立一个队列用来处理外部系统、处理更新等等问题.我们不能仅仅按需更新令牌,因为那样会产生不可预估的工作量,不管怎样,我们需要坚持我们的系统充满活力,但是那样,我们就要将延期的任务也插入队列,并且保证数据库和队列之间的一致性,我们还要被迫使用一个quorum的容错队列.此外,如果我们把数据同时放在RAM和一个(考虑到工作量)可能要放入内存的队列中,那么我们就要消耗更多资源.

在我们的方案中,数据库存储令牌,队列逻辑只必要占用7个字节(每个元组只必要7个额外的字节,就可以搞定队列逻辑!),如果使用其他的队列形式,必要占用的空间就多得多了,大概是内存容量的两倍.

总结

首先,我们解决了连接中断的问题,这个问题十分常见,使用上述的系统让我们解脱了这个困扰.

分片赞助我们扩展内存,我们将连接数从二次方减少到了线性,优化了业务任务的队列逻辑：如果发生延期,更新我们所能更新的一切令牌,这些延期并非都是我们的故障引起的,有可能是Google、Microsoft或者其他服务端对OAuth服务提供商进行改造,然后导致我们这边出现大量的未更新的令牌.

去数据库内部运算吧,走近数据,你将拥有方便、高效、可扩展和灵活的运算体验!

《Nosql数据库的主主备份》是否对您有启发，欢迎查看更多与《Nosql数据库的主主备份》相关教程，学精学透。小编PHP学院为您提供精彩教程。

关于NoSQL数据库的类型和nosql数据库的类型包括的介绍现已完结，谢谢您的耐心阅读，如果想了解更多关于AWS Around---NoSql数据库的不同类型、NoSQL 数据库的类型、nosql-redis-网络资料学习-05-nosql数据库的四大分类、NoSQL数据库的主主备份的相关知识，请在本站寻找。

本文将介绍The NoSQL System的详细情况，。我们将通过案例分析、数据研究等多种方式，帮助您更全面地了解这个主题，同时也将涉及一些关于/usr/include/boost/system/error_code.hpp:208：链接时未定义对`boost :: system :: get_system_category（）的引用、Azure 函数的 Cosmos DB 输入绑定无法将 System.String 转换为 System.Guid、c# – 如何将System.Diagnostics.Trace和System.Diagnostics.Debug消息记录到NLog文件？、CentOS7 mysql 连接不上 :[ERROR] InnoDB: The innodb_system data file ''ibdata1'' must be writable的知识。

• The NoSQL System
• /usr/include/boost/system/error_code.hpp:208：链接时未定义对`boost :: system :: get_system_category（）的引用
• Azure 函数的 Cosmos DB 输入绑定无法将 System.String 转换为 System.Guid
• c# – 如何将System.Diagnostics.Trace和System.Diagnostics.Debug消息记录到NLog文件？
• CentOS7 mysql 连接不上 :[ERROR] InnoDB: The innodb_system data file ''ibdata1'' must be writable

The NoSQL System

不同于大多数其他的项目在这本书中，Nosql 不是一种工具，但生态系统组成的几个免费和相互竞争的工具。烙上 Nosql 外号的工具提供了用于存储数据的基于 sql 的关系数据库系统替代方案。若要了解使用 Nosql，我们必须理解可用的工具的空间，看看如何的每一个设计探索的空间数据存储的可能性。

如果你正在考虑使用 Nosql 存储系统，你先要了解的选项，Nosql 系统跨越了广阔的空间。Nosql 系统废除了许多传统的关系数据库系统，舒适，后面一个数据库系统的边界通常封装的业务现在留给应用程序设计人员。这就要求你带上这顶帽子的系统架构师，需要更深入地了解如何构建这样的系统。 13.1.什么是在一个名称？

在界定 Nosql 的空间，让我们先尝试定义名称。字面上看，Nosql 系统给用户不是 sql 的查询接口。Nosql 社区通常需要一个更具包容性的看法，建议使用 Nosql 系统提供替代传统的关系数据库，并允许开发者在设计使用不只一个 sql 接口的项目。在某些情况下，您可能使用 Nosql 的替代方法，替换关系型数据库和在其他你会聘请您在应用程序开发中遇到的不同问题的混合和匹配方法。

在进 Nosql 的世界之前，让我们了解一下 sql 和关系模型适合您的需要，和别人 Nosql 系统可能在哪一个更好的适合的情况。 13.1.1。 sql 和关系模型

sql 是查询数据的声明性语言。声明性语言，是其中一个程序员指定他们想要的系统来做，而不是程序上定义的系统应该怎么做。一些例子包括： 找到的记录为员工 39，投影出只有雇员姓名和电话号码，从他们的整个记录，筛选员工记录那些在会计工作，在每个部门、 员工或参加与经理人表的雇员表中的数据。

到第一近似值，sql 允许你想要问这些问题不加思索的数据在磁盘上，哪个指数，用来访问数据，布局方式或什么算法用来处理数据。大多数关系数据库的一个重要的建筑部分是查询优化器，决定哪些许多的逻辑上等效的查询计划，要执行最快回答查询。这些优化往往比平均的数据库用户，但他们有时没有足够的信息，或者有太简单了系统模型，以产生最有效的执行。

关系数据库，它是在实践中使用的最常见的数据库，请按照关系数据模型。在此模型中，不同的真实世界的实体存储在不同的表中。例如，所有的员工可能会存储在雇员表，和所有部门可能都存储在一个部门表。每个表中的行具有各种属性存储在列中。例如，员工可能有雇员 id、 薪金、 出生日期和第一个/最后一个名字。上述每个属性将存储在列中的雇员表。

关系模型去手中用 sql 语句。简单的 sql 查询，如滤波器、 检索所有记录一些的测试匹配的字段 （例如，雇员 id = 3 或工资 > 美元 20000）。更复杂的构造原因数据库做一些额外的工作，例如将数据从多个表 （例如，什么是哪个员工 3 工程部门的名称吗？） 联接。其他复杂的构造，例如聚合 （例如，什么是我的员工的平均工资?） 可能会导致全表扫描。

关系数据模型定义了与它们之间的严格关系高度结构化的实体。此模型与 sql 的查询将允许复杂数据遍历没有太多自定义的发展。这种建模的复杂性和查询虽然有其局限性，： ? 1 2 3

复杂性会导致不可预测性。sql 的表现力，使每个查询的成本和因而工作量的成本原因所面临的挑战。虽然简单的查询语言可能更为复杂的应用程序逻辑，他们使它易于提供数据存储系统，只有简单的请求作出回应。 有许多方法，模型存在的问题。关系数据模型是严格： 分配给每个表的架构在每行中指定的数据。如果我们存储更少的结构化的数据或行更多的差异，它们存储的列中，关系的模型可能会不必要地限制。同样地，应用程序开发人员可能无法找到关系模型适合每种类型的数据建模。例如，大量的应用程序逻辑写在面向对象的语言，包括高级概念 suc

Unlike most of the other projects in this book,Nosql is not a tool,but an ecosystem composed of several complimentary and competing tools. The tools branded with the Nosql monicker provide an alternative to sql-based relational database systems for storing data. To understand Nosql,we have to understand the space of available tools,and see how the design of each one explores the space of data storage possibilities.

If you are considering using a Nosql storage system,you should first understand the wide space of options that Nosql systems span. Nosql systems do away with many of the Traditional comforts of relational database systems,and operations which were typically encapsulated behind the system boundary of a database are Now left to application designers. This requires you to take on the hat of a systems architect,which requires a more in-depth understanding of how such systems are built. 13.1. What's in a Name?

In defining the space of Nosql,let's first take a stab at defining the name. Taken literally,a Nosql system presents a query interface to the user that is not sql. The Nosql community generally takes a more inclusive view,suggesting that Nosql systems provide alternatives to Traditional relational databases,and allow developers to design projects which use Not Only a sql interface. In some cases,you might replace a relational database with a Nosql alternative,and in others you will employ a mix-and-match approach to different problems you encounter in application development.

Before diving into the world of Nosql,let's explore the cases where sql and the relational model suit your needs,and others where a Nosql system might be a better fit. 13.1.1. sql and the Relational Model

sql is a declarative language for querying data. A declarative language is one in which a programmer specifies what they want the system to do,rather than procedurally defining how the system should do it. A few examples include: find the record for employee 39,project out only the employee name and phone number from their entire record,filter employee records to those that work in accounting,count the employees in each department,or join the data from the employees table with the managers table.

To a first approximation,sql allows you to ask these questions without thinking about how the data is laid out on disk,which indices to use to access the data,or what algorithms to use to process the data. A significant architectural component of most relational databases is a query optimizer,which decides which of the many logically equivalent query plans to execute to most quickly answer a query. These optimizers are often better than the average database user,but sometimes they do not have enough information or have too simple a model of the system in order to generate the most efficient execution.

Relational databases,which are the most common databases used in practice,follow the relational data model. In this model,different real-world entities are stored in different tables. For example,all employees might be stored in an Employees table,and all departments might be stored in a Departments table. Each row of a table has varIoUs properties stored in columns. For example,employees might have an employee id,salary,birth date,and first/last names. Each of these properties will be stored in a column of the Employees table.

The relational model goes hand-in-hand with sql. Simple sql queries,such as filters,retrieve all records whose field matches some test (e.g.,employeeid = 3,or salary > $20000). More complex constructs cause the database to do some extra work,such as joining data from multiple tables (e.g.,what is the name of the department in which employee 3 works?). Other complex constructs such as aggregates (e.g.,what is the average salary of my employees?) can lead to full-table scans.

The relational data model defines highly structured entities with strict relationships between them. Querying this model with sql allows complex data traversals without too much custom development. The complexity of such modeling and querying has its limits,though:

Complexity leads to unpredictability. sql's expressiveness makes it challenging to reason about the cost of each query,and thus the cost of a workload. While simpler query languages might complicate application logic,they make it easier to provision data storage systems,which only respond to simple requests.
There are many ways to model a problem. The relational data model is strict: the schema assigned to each table specifies the data in each row. If we are storing less structured data,or rows with more variance in the columns they store,the relational model may be needlessly restrictive. Similarly,application developers might not find the relational model perfect for modeling every kind of data. For example,a lot of application logic is written in object-oriented languages and includes high-level concepts such as lists,queues,and sets,and some programmers would like their persistence layer to model this.
If the data grows past the capacity of one server,then the tables in the database will have to be partitioned across computers. To avoid JOINs having to cross the network in order to get data in different tables,we will have to denormalize it. Denormalization stores all of the data from different tables that one might want to look up at once in a single place. This makes our database look like a key-lookup storage system,leaving us wondering what other data models might better suit the data.

It's generally not wise to discard many years of design considerations arbitrarily. When you consider storing your data in a database,consider sql and the relational model,which are backed by decades of research and development,offer rich modeling capabilities,and provide easy-to-understand guarantees about complex operations. Nosql is a good option when you have a specific problem,such as large amounts of data,a massive workload,or a difficult data modeling decision for which sql and relational databases might not have been optimized. 13.1.2. Nosql Inspirations

The Nosql movement finds much of its inspiration in papers from the research community. While many papers are at the core of design decisions in Nosql systems,two stand out in particular.

Google's BigTable [CDG+06] presents an interesting data model,which facilitates sorted storage of multi-column historical data. Data is distributed to multiple servers using a hierarchical range-based partitioning scheme,and data is updated with strict consistency (a concept that we will eventually define in Section 13.5).

Amazon's Dynamo [DHJ+07] uses a different key-oriented distributed datastore. Dynamo's data model is simpler,mapping keys to application-specific blobs of data. The partitioning model is more resilient to failure,but accomplishes that goal through a looser data consistency approach called eventual consistency.

We will dig into each of these concepts in more detail,but it is important to understand that many of them can be mixed and matched. Some Nosql systems such as HBase1 sticks closely to the BigTable design. Another Nosql system named Voldemort2 replicates many of Dynamo's features. Still other Nosql projects such as Cassandra3 have taken some features from BigTable (its data model) and others from Dynamo (its partitioning and consistency schemes). 13.1.3. characteristics and Considerations

Nosql systems part ways with the hefty sql standard and offer simpler but piecemeal solutions for architecting storage solutions. These systems were built with the belief that in simplifying how a database operates over data,an architect can better predict the performance of a query. In many Nosql systems,complex query logic is left to the application,resulting in a data store with more predictable query performance because of the lack of variability in queries

Nosql systems part with more than just declarative queries over the relational data. Transactional semantics,consistency,and durability are guarantees that organizations such as banks demand of databases. Transactions provide an all-or-nothing guarantee when combining several potentially complex operations into one,such as deducting money from one account and adding the money to another. Consistency ensures that when a value is updated,subsequent queries will see the updated value. Durability guarantees that once a value is updated,it will be written to stable storage (such as a hard drive) and recoverable if the database crashes.

Nosql systems relax some of these guarantees,a decision which,for many non-banking applications,can provide acceptable and predictable behavior in exchange for improved performance. These relaxations,combined with data model and query language changes,often make it easier to safely partition a database across multiple machines when the data grows beyond a single machine's capability.

Nosql systems are still very much in their infancy. The architectural decisions that go into the systems described in this chapter are a testament to the requirements of varIoUs users. The biggest challenge in summarizing the architectural features of several open source projects is that each one is a moving target. Keep in mind that the details of individual systems will change. When you pick between Nosql systems,you can use this chapter to guide your thought process,but not your feature-by-feature product selection.

As you think about Nosql systems,here is a roadmap of considerations:

Data and query model: Is your data represented as rows,objects,data structures,or documents? Can you ask the database to calculate aggregates over multiple records?
Durability: When you change a value,does it immediately go to stable storage? Does it get stored on multiple machines in case one crashes?
Scalability: Does your data fit on a single server? Do the amount of reads and writes require multiple disks to handle the workload?
Partitioning: For scalability,availability,or durability reasons,does the data need to live on multiple servers? How do you kNow which record is on which server?
Consistency: If you've partitioned and replicated your records across multiple servers,how do the servers coordinate when a record changes?
Transactional semantics: When you run a series of operations,some databases allow you to wrap them in a transaction,which provides some subset of ACID (Atomicity,Consistency,Isolation,and Durability) guarantees on the transaction and all others currently running. Does your business logic require these guarantees,which often come with performance tradeoffs?
Single-server performance: If you want to safely store data on disk,what on-disk data structures are best-geared toward read-heavy or write-heavy workloads? Is writing to disk your bottleneck?
Analytical workloads: We're going to pay a lot of attention to lookup-heavy workloads of the kind you need to run a responsive user-focused web application. In many cases,you will want to build dataset-sized reports,aggregating statistics across multiple users for example. Does your use-case and toolchain require such functionality?

While we will touch on all of these consideration,the last three,while equally important,see the least attention in this chapter. 13.2. Nosql Data and Query Models

The data model of a database specifies how data is logically organized. Its query model dictates how the data can be retrieved and updated. Common data models are the relational model,key-oriented storage model,or varIoUs graph models. Query languages you might have heard of include sql,key lookups,and MapReduce. Nosql systems combine different data and query models,resulting in different architectural considerations. 13.2.1. Key-based Nosql Data Models

Nosql systems often part with the relational model and the full expressivity of sql by restricting lookups on a dataset to a single field. For example,even if an employee has many properties,you might only be able to retrieve an employee by her ID. As a result,most queries in Nosql systems are key lookup-based. The programmer selects a key to identify each data item,and can,for the most part,only retrieve items by performing a lookup for their key in the database.

In key lookup-based systems,complex join operations or multiple-key retrieval of the same data might require creative uses of key names. A programmer wishing to look up an employee by his employee ID and to look up all employees in a department might create two key types. For example,the key employee:30 would point to an employee record for employee ID 30,and employee_departments:20 might contain a list of all employees in department 20. A join operation gets pushed into application logic: to retrieve employees in department 20,an application first retrieves a list of employee IDs from key employee_departments:20,and then loops over key lookups for each employee:ID in the employee list.

The key lookup model is beneficial because it means that the database has a consistent query pattern—the entire workload consists of key lookups whose performance is relatively uniform and predictable. Profiling to find the slow parts of an application is simpler,since all complex operations reside in the application code. On the flip side,the data model logic and business logic are Now more closely intertwined,which muddles abstraction.

Let's quickly touch on the data associated with each key. VarIoUs Nosql systems offer different solutions in this space. Key-Value Stores

The simplest form of Nosql store is a key-value store. Each key is mapped to a value containing arbitrary data. The Nosql store has no kNowledge of the contents of its payload,and simply delivers the data to the application. In our Employee database example,one might map the key employee:30 to a blob containing JSON or a binary format such as Protocol Buffers4,Thrift5,or Avro6 in order to encapsulate the information about employee 30.

If a developer uses structured formats to store complex data for a key,she must operate against the data in application space: a key-value data store generally offers no mechanisms for querying for keys based on some property of their values. Key-value stores shine in the simplicity of their query model,usually consisting of set,get,and delete primitives,but discard the ability to add simple in-database filtering capabilities due to the opacity of their values. Voldemort,which is based on Amazon's Dynamo,provides a distributed key-value store. BDB7 offers a persistence library that has a key-value interface. Key-Data Structure Stores

Key-data structure stores,made popular by Redis8,assign each value a type. In Redis,the available types a value can take on are integer,string,list,set,and sorted set. In addition to set/get/delete,type-specific commands,such as increment/decrement for integers,or push/pop for lists,add functionality to the query model without drastically affecting performance characteristics of requests. By providing simple type-specific functionality while avoiding multi-key operations such as aggregation or joins,Redis balances functionality and performance. Key-Document Stores

Key-document stores,such as CouchDB9,MongoDB10,and Riak11,map a key to some document that contains structured information. These systems store documents in a JSON or JSON-like format. They store lists and dictionaries,which can be embedded recursively inside one-another.

MongoDB separates the keyspace into collections,so that keys for Employees and Department,for example,do not collide. CouchDB and Riak leave type-tracking to the developer. The freedom and complexity of document stores is a double-edged sword: application developers have a lot of freedom in modeling their documents,but application-based query logic can become exceedingly complex. BigTable Column Family Stores

HBase and Cassandra base their data model on the one used by Google's BigTable. In this model,a key identifies a row,which contains data stored in one or more Column Families (CFs). Within a CF,each row can contain multiple columns. The values within each column are timestamped,so that several versions of a row-column mapping can live within a CF.

Conceptually,one can think of Column Families as storing complex keys of the form (row ID,CF,column,timestamp),mapping to values which are sorted by their keys. This design results in data modeling decisions which push a lot of functionality into the keyspace. It is particularly good at modeling historical data with timestamps. The model naturally supports sparse column placement since row IDs that do not have certain columns do not need an explicit NULL value for those columns. On the flip side,columns which have few or no NULL values must still store the column identifier with each row,which leads to greater space consumption.

Each project data model differs from the original BigTable model in varIoUs ways,but Cassandra's changes are most notable. Cassandra introduces the notion of a supercolumn within each CF to allow for another level of mapping,modeling,and indexing. It also does away with a notion of locality groups,which can physically store multiple column families together for performance reasons. 13.2.2. Graph Storage

One class of Nosql stores are graph stores. Not all data is created equal,and the relational and key-oriented data models of storing and querying data are not the best for all data. Graphs are a fundamental data structure in computer science,and systems such as HypergraphDB12 and Neo4J13 are two popular Nosql storage systems for storing graph-structured data. Graph stores differ from the other stores we have discussed thus far in almost every way: data models,data traversal and querying patterns,physical layout of data on disk,distribution to multiple machines,and the transactional semantics of queries. We can not do these stark differences justice given space limitations,but you should be aware that certain classes of data may be better stored and queried as a graph. 13.2.3. Complex Queries

There are notable exceptions to key-only lookups in Nosql systems. MongoDB allows you to index your data based on any number of properties and has a relatively high-level language for specifying which data you want to retrieve. BigTable-based systems support scanners to iterate over a column family and select particular items by a filter on a column. CouchDB allows you to create different views of the data,and to run MapReduce tasks across your table to facilitate more complex lookups and updates. Most of the systems have bindings to Hadoop or another MapReduce framework to perform dataset-scale analytical queries. 13.2.4. Transactions

Nosql systems generally prioritize performance over transactional semantics. Other sql-based systems allow any set of statements—from a simple primary key row retrieval,to a complicated join between several tables which is then subsequently averaged across several fields—to be placed in a transaction.

These sql databases will offer ACID guarantees between transactions. Running multiple operations in a transaction is Atomic (the A in ACID),meaning all or none of the operations happen. Consistency (the C) ensures that the transaction leaves the database in a consistent,uncorrupted state. Isolation (the I) makes sure that if two transactions touch the same record,they will do without stepping on each other's feet. Durability (the D,covered extensively in the next section),ensures that once a transaction is committed,it's stored in a safe place.

ACID-compliant transactions keep developers sane by making it easy to reason about the state of their data. Imagine multiple transactions,each of which has multiple steps (e.g.,first check the value of a bank account,then subtract $60,then update the value). ACID-compliant databases often are limited in how they can interleave these steps while still providing a correct result across all transactions. This push for correctness results in often-unexpected performance characteristics,where a slow transaction might cause an otherwise quick one to wait in line.

Most Nosql systems pick performance over full ACID guarantees,but do provide guarantees at the key level: two operations on the same key will be serialized,avoiding serIoUs corruption to key-value pairs. For many applications,this decision will not pose noticeable correctness issues,and will allow quick operations to execute with more regularity. It does,however,leave more considerations for application design and correctness in the hands of the developer.

Redis is the notable exception to the no-transaction trend. On a single server,it provides a MULTI command to combine multiple operations atomically and consistently,and a WATCH command to allow isolation. Other systems provide lower-level test-and-set functionality which provides some isolation guarantees. 13.2.5. Schema-free Storage

A cross-cutting property of many Nosql systems is the lack of schema enforcement in the database. Even in document stores and column family-oriented stores,properties across similar entities are not required to be the same. This has the benefit of supporting less structured data requirements and requiring less performance expense when modifying schemas on-the-fly. The decision leaves more responsibility to the application developer,who Now has to program more defensively. For example,is the lack of a lastname property on an employee record an error to be rectified,or a schema update which is currently propagating through the system? Data and schema versioning is common in application-level code after a few iterations of a project which relies on sloppy-schema Nosql systems. 13.3. Data Durability

Ideally,all data modifications on a storage system would immediately be safely persisted and replicated to multiple locations to avoid data loss. However,ensuring data safety is in tension with performance,and different Nosql systems make different data durability guarantees in order to improve performance. Failure scenarios are varied and numerous,and not all Nosql systems protect you against these issues.

A simple and common failure scenario is a server restart or power loss. Data durability in this case involves having moved the data from memory to a hard disk,which does not require power to store data. Hard disk failure is handled by copying the data to secondary devices,be they other hard drives in the same machine (RAID mirroring) or other machines on the network. However,a data center might not survive an event which causes correlated failure (a tornado,for example),and some organizations go so far as to copy data to backups in data centers several hurricane widths apart. Writing to hard drives and copying data to multiple servers or data centers is expensive,so different Nosql systems Trade off durability guarantees for performance. 13.3.1. Single-server Durability

The simplest form of durability is a single-server durability,which ensures that any data modification will survive a server restart or power loss. This usually means writing the changed data to disk,which often bottlenecks your workload. Even if you order your operating system to write data to an on-disk file,the operating system may buffer the write,avoiding an immediate modification on disk so that it can group several writes together into a single operation. Only when the fsync system call is issued does the operating system make a best-effort attempt to ensure that buffered updates are persisted to disk.

Typical hard drives can perform 100-200 random accesses (seeks) per second,and are limited to 30-100 MB/sec of sequential writes. Memory can be orders of magnitudes faster in both scenarios. Ensuring efficient single-server durability means limiting the number of random writes your system incurs,and increasing the number of sequential writes per hard drive. Ideally,you want a system to minimize the number of writes between fsync calls,maximizing the number of those writes that are sequential,all the while never telling the user their data has been successfully written to disk until that write has been fsynced. Let's cover a few techniques for improving performance of single-server durability guarantees. Control fsync Frequency

Memcached14 is an example of a system which offers no on-disk durability in exchange for extremely fast in-memory operations. When a server restarts,the data on that server is gone: this makes for a good cache and a poor durable data store.

Redis offers developers several options for when to call fsync. Developers can force an fsync call after every update,which is the slow and safe choice. For better performance,Redis can fsync its writes every N seconds. In a worst-case scenario,the you will lose last N seconds worth of operations,which may be acceptable for certain uses. Finally,for use cases where durability is not important (maintaining coarse-grained statistics,or using Redis as a cache),the developer can turn off fsync calls entirely: the operating system will eventually flush the data to disk,but without guarantees of when this will happen. Increase Sequential Writes by Logging

Several data structures,such as B+Trees,help Nosql systems quickly retrieve data from disk. Updates to those structures result in updates in random locations in the data structures' files,resulting in several random writes per update if you fsync after each update. To reduce random writes,systems such as Cassandra,HBase,Redis,and Riak append update operations to a sequentially-written file called a log. While other data structures used by the system are only periodically fsynced,the log is frequently fsynced. By treating the log as the ground-truth state of the database after a crash,these storage engines are able to turn random updates into sequential ones.

While Nosql systems such as MongoDB perform writes in-place in their data structures,others take logging even further. Cassandra and HBase use a technique borrowed from BigTable of combining their logs and lookup data structures into one log-structured merge tree. Riak provides similar functionality with a log-structured hash table. CouchDB has modified the Traditional B+Tree so that all changes to the data structure are appended to the structure on physical storage. These techniques result in improved write throughput,but require a periodic log compaction to keep the log from growing unbounded. Increase Throughput by Grouping Writes

Cassandra groups multiple concurrent updates within a short window into a single fsync call. This design,called group commit,results in higher latency per update,as users have to wait on several concurrent updates to have their own update be ackNowledged. The latency bump comes at an increase in throughput,as multiple log appends can happen with a single fsync. As of this writing,every HBase update is persisted to the underlying storage provided by the Hadoop distributed File System (HDFS)15,which has recently seen patches to allow support of appends that respect fsync and group commit. 13.3.2. Multi-server Durability

Because hard drives and machines often irreparably fail,copying important data across machines is necessary. Many Nosql systems offer multi-server durability for data.

Redis takes a Traditional master-slave approach to replicating data. All operations executed against a master are communicated in a log-like fashion to slave machines,which replicate the operations on their own hardware. If a master fails,a slave can step in and serve the data from the state of the operation log that it received from the master. This configuration might result in some data loss,as the master does not confirm that the slave has persisted an operation in its log before ackNowledging the operation to the user. CouchDB facilitates a similar form of directional replication,where servers can be configured to replicate changes to documents on other stores.

MongoDB provides the notion of replica sets,where some number of servers are responsible for storing each document. MongoDB gives developers the option of ensuring that all replicas have received updates,or to proceed without ensuring that replicas have the most recent data. Many of the other distributed Nosql storage systems support multi-server replication of data. HBase,which is built on top of HDFS,receives multi-server durability through HDFS. All writes are replicated to two or more HDFS nodes before returning control to the user,ensuring multi-server durability.

Riak,Cassandra,and Voldemort support more configurable forms of replication. With subtle differences,all three systems allow the user to specify N,the number of machines which should ultimately have a copy of the data,and W<N,the number of machines that should confirm the data has been written before returning control to the user.

To handle cases where an entire data center goes out of service,multi-server replication across data centers is required. Cassandra,and Voldemort have rack-aware configurations,which specify the rack or data center in which varIoUs machines are located. In general,blocking the user's request until a Remote Server has ackNowledged an update incurs too much latency. Updates are streamed without confirmation when performed across wide area networks to backup data centers. 13.4. Scaling for Performance

Having just spoken about handling failure,let's imagine a rosier situation: success! If the system you build reaches success,your data store will be one of the components to feel stress under load. A cheap and dirty solution to such problems is to scale up your existing machinery: invest in more RAM and disks to handle the workload on one machine. With more success,pouring money into more expensive hardware will become infeasible. At this point,you will have to replicate data and spread requests across multiple machines to distribute load. This approach is called scale out,and is measured by the horizontal scalability of your system.

The ideal horizontal scalability goal is linear scalability,in which doubling the number of machines in your storage system doubles the query capacity of the system. The key to such scalability is in how the data is spread across machines. Sharding is the act of splitting your read and write workload across multiple machines to scale out your storage system. Sharding is fundamental to the design of many systems,namely Cassandra,Voldemort,and Riak,and more recently MongoDB and Redis. Some projects such as CouchDB focus on single-server performance and do not provide an in-system solution to sharding,but secondary projects provide coordinators to partition the workload across independent installations on multiple machines.

Let's cover a few interchangeable terms you might encounter. We will use the terms sharding and partitioning interchangeably. The terms machine,server,or node refer to some physical computer which stores part of the partitioned data. Finally,a cluster or ring refers to the set of machines which participate in your storage system.

Sharding means that no one machine has to handle the write workload on the entire dataset,but no one machine can answer queries about the entire dataset. Most Nosql systems are key-oriented in both their data and query models,and few queries touch the entire dataset anyway. Because the primary access method for data in these systems is key-based,sharding is typically key-based as well: some function of the key determines the machine on which a key-value pair is stored. We'll cover two methods of defining the key-machine mapping: hash partitioning and range partitioning. 13.4.1. Do Not Shard Until You Have To

Sharding adds system complexity,and where possible,you should avoid it. Let's cover two ways to scale without sharding: read replicas and caching. Read Replicas

Many storage systems see more read requests than write requests. A simple solution in these cases is to make copies of the data on multiple machines. All write requests still go to a master node. Read requests go to machines which replicate the data,and are often slightly stale with respect to the data on the write master.

If you are already replicating your data for multi-server durability in a master-slave configuration,as is common in Redis,CouchDB,or MongoDB,the read slaves can shed some load from the write master. Some queries,such as aggregate summaries of your dataset,which might be expensive and often do not require up-to-the-second freshness,can be executed against the slave replicas. Generally,the less stringent your demands for freshness of content,the more you can lean on read slaves to improve read-only query performance. Caching

Caching the most popular content in your system often works surprisingly well. Memcached dedicates blocks of memory on multiple servers to cache data from your data store. Memcached clients take advantage of several horizontal scalability tricks to distribute load across Memcached installations on different servers. To add memory to the cache pool,just add another Memcached host.

Because Memcached is designed for caching,it does not have as much architectural complexity as the persistent solutions for scaling workloads. Before considering more complicated solutions,think about whether caching can solve your scalability woes. Caching is not solely a temporary band-aid: Facebook has Memcached installations in the range of tens of terabytes of memory!

Read replicas and caching allow you to scale up your read-heavy workloads. When you start to increase the frequency of writes and updates to your data,you will also increase the load on the master server that contains all of your up-to-date data. For the rest of this section,we will cover techniques for sharding your write workload across multiple servers. 13.4.2. Sharding Through Coordinators

The CouchDB project focuses on the single-server experience. Two projects,Lounge and BigCouch,facilitate sharding CouchDB workloads through an external proxy,which acts as a front end to standalone CouchDB instances. In this design,the standalone installations are not aware of each other. The coordinator distributes requests to individual CouchDB instances based on the key of the document being requested.

Twitter has built the notions of sharding and replication into a coordinating framework called Gizzard16. Gizzard takes standalone data stores of any type—you can build wrappers for sql or Nosql storage systems—and arranges them in trees of any depth to partition keys by key range. For fault tolerance,Gizzard can be configured to replicate data to multiple physical machines for the same key range. 13.4.3. Consistent Hash Rings

Good hash functions distribute a set of keys in a uniform manner. This makes them a powerful tool for distributing key-value pairs among multiple servers. The academic literature on a technique called consistent hashing is extensive,and the first applications of the technique to data stores was in systems called distributed hash tables (DHTs). Nosql systems built around the principles of Amazon's Dynamo adopted this distribution technique,and it appears in Cassandra,and Riak. Hash Rings by Example [A distributed Hash Table Ring]

figure 13.1: A distributed Hash Table Ring

Consistent hash rings work as follows. Say we have a hash function H that maps keys to uniformly distributed large integer values. We can form a ring of numbers in the range [1,L] that wraps around itself with these values by taking H(key) mod L for some relatively large integer L. This will map each key into the range [1,L]. A consistent hash ring of servers is formed by taking each server's unique identifier (say its IP address),and applying H to it. You can get an intuition for how this works by looking at the hash ring formed by five servers (A-E) in figure 13.1.

There,we picked L = 1000. Let's say that H(A) mod L = 7,H(B) mod L = 234,H(C) mod L = 447,H(D) mod L = 660,and H(E) mod L = 875. We can Now tell which server a key should live on. To do this,we map all keys to a server by seeing if it falls in the range between that server and the next one in the ring. For example,A is responsible for keys whose hash value falls in the range [7,233],and E is responsible for keys in the range [875,6] (this range wraps around on itself at 1000). So if H('employee30') mod L = 899,it will be stored by server E,and if H('employee31') mod L = 234,it will be stored on server B. Replicating Data

Replication for multi-server durability is achieved by passing the keys and values in one server's assigned range to the servers following it in the ring. For example,with a replication factor of 3,keys mapped to the range [7,233] will be stored on servers A,B,and C. If A were to fail,its neighbors B and C would take over its workload. In some designs,E would replicate and take over A's workload temporarily,since its range would expand to include A's. Achieving Better distribution

While hashing is statistically effective at uniformly distributing a keyspace,it usually requires many servers before it distributes evenly. Unfortunately,we often start with a small number of servers that are not perfectly spaced apart from one-another by the hash function. In our example,A's key range is of length 227,whereas E's range is 132. This leads to uneven load on different servers. It also makes it difficult for servers to take over for one-another when they fail,since a neighbor suddenly has to take control of the entire range of the Failed server.

To solve the problem of uneven large key ranges,many DHTs including Riak create several `virtual' nodes per physical machine. For example,with 4 virtual nodes,server A will act as server A_1,A_2,A_3,and A_4. Each virtual node hashes to a different value,giving it more opportunity to manage keys distributed to different parts of the keyspace. Voldemort takes a similar approach,in which the number of partitions is manually configured and usually larger than the number of servers,resulting in each server receiving a number of smaller partitions.

Cassandra does not assign multiple small partitions to each server,resulting in sometimes uneven key range distributions. For load-balancing,Cassandra has an asynchronous process which adjusts the location of servers on the ring depending on their historic load. 13.4.4. Range Partitioning

In the range partitioning approach to sharding,some machines in your system keep Metadata about which servers contain which key ranges. This Metadata is consulted to route key and range lookups to the appropriate servers. Like the consistent hash ring approach,this range partitioning splits the keyspace into ranges,with each key range being managed by one machine and potentially replicated to others. Unlike the consistent hashing approach,two keys that are next to each other in the key's sort order are likely to appear in the same partition. This reduces the size of the routing Metadata,as large ranges are compressed to [start,end] markers.

In adding active record-keeping of the range-to-server mapping,the range partitioning approach allows for more fine-grained control of load-shedding from heavily loaded servers. If a specific key range sees higher traffic than other ranges,a load manager can reduce the size of the range on that server,or reduce the number of shards that this server serves. The added freedom to actively manage load comes at the expense of extra architectural components which monitor and route shards. The BigTable Way

Google's BigTable paper describes a range-partitioning hierarchical technique for sharding data into tablets. A tablet stores a range of row keys and values within a column family. It maintains all of the necessary logs and data structures to answer queries about the keys in its assigned range. Tablet servers serve multiple tablets depending on the load each tablet is experiencing.

Each tablet is kept at a size of 100-200 MB. As tablets change in size,two small tablets with adjoining key ranges might be combined,or a large tablet might be split in two. A master server analyzes tablet size,load,and tablet server availability. The master adjusts which tablet server serves which tablets at any time. [BigTable-based Range Partitioning]

figure 13.2: BigTable-based Range Partitioning

The master server maintains the tablet assignment in a Metadata table. Because this Metadata can get large,the Metadata table is also sharded into tablets that map key ranges to tablets and tablet servers responsible for those ranges. This results in a three-layer hierarchy traversal for clients to find a key on its hosting tablet server,as depicted in figure 13.2.

Let's look at an example. A client searching for key 900 will query server A,which stores the tablet for Metadata level 0. This tablet identifies the Metadata level 1 tablet on server 6 containing key ranges 500-1500. The client sends a request to server B with this key,which responds that the tablet containing keys 850-950 is found on a tablet on server C. Finally,the client sends the key request to server C,and gets the row data back for its query. Metadata tablets at level 0 and 1 may be cached by the client,which avoids putting undue load on their tablet servers from repeat queries. The BigTable paper explains that this 3-level hierarchy can accommodate 261 bytes worth of storage using 128MB tablets. Handling Failures

The master is a single point of failure in the BigTable design,but can go down temporarily without affecting requests to tablet servers. If a tablet server fails while serving tablet requests,it is up to the master to recognize this and re-assign its tablets while requests temporarily fail.

In order to recognize and handle machine failures,the BigTable paper describes the use of Chubby,a distributed locking system for managing server membership and liveness. ZooKeeper17 is the open source implementation of Chubby,and several Hadoop-based projects utilize it to manage secondary master servers and tablet server reassignment. Range Partitioning-based Nosql Projects

HBase employs BigTable's hierarchical approach to range-partitioning. Underlying tablet data is stored in Hadoop's distributed filesystem (HDFS). HDFS handles data replication and consistency among replicas,leaving tablet servers to handle requests,update storage structures,and initiate tablet splits and compactions.

MongoDB handles range partitioning in a manner similar to that of BigTable. Several configuration nodes store and manage the routing tables that specify which storage node is responsible for which key ranges. These configuration nodes stay in sync through a protocol called two-phase commit,and serve as a hybrid of BigTable's master for specifying ranges and Chubby for highly available configuration management. Separate routing processes,which are stateless,keep track of the most recent routing configuration and route key requests to the appropriate storage nodes. Storage nodes are arranged in replica sets to handle replication.

Cassandra provides an order-preserving partitioner if you wish to allow fast range scans over your data. Cassandra nodes are still arranged in a ring using consistent hashing,but rather than hashing a key-value pair onto the ring to determine the server to which it should be assigned,the key is simply mapped onto the server which controls the range in which the key naturally fits. For example,keys 20 and 21 would both be mapped to server A in our consistent hash ring in figure 13.1,rather than being hashed and randomly distributed in the ring.

Twitter's Gizzard framework for managing partitioned and replicated data across many back ends uses range partitioning to shard data. Routing servers form hierarchies of any depth,assigning ranges of keys to servers below them in the hierarchy. These servers either store data for keys in their assigned range,or route to yet another layer of routing servers. Replication in this model is achieved by sending updates to multiple machines for a key range. Gizzard routing nodes manage Failed writes in different manner than other Nosql systems. Gizzard requires that system designers make all updates idempotent (they can be run twice). When a storage node fails,routing nodes cache and repeatedly send updates to the node until the update is confirmed. 13.4.5. Which Partitioning Scheme to Use

Given the hash- and range-based approaches to sharding,which is preferable? It depends. Range partitioning is the obvIoUs choice to use when you will frequently be performing range scans over the keys of your data. As you read values in order by key,you will not jump to random nodes in the network,which would incur heavy network overhead. But if you do not require range scans,which sharding scheme should you use?

Hash partitioning gives reasonable distribution of data across nodes,and random skew can be reduced with virtual nodes. Routing is simple in the hash partitioning scheme: for the most part,the hash function can be executed by clients to find the appropriate server. With more complicated rebalancing schemes,finding the right node for a key becomes more difficult.

Range partitioning requires the upfront cost of maintaining routing and configuration nodes,which can see heavy load and become central points of failure in the absence of relatively complex fault tolerance schemes. Done well,range-partitioned data can be load-balanced in small chunks which can be reassigned in high-load situations. If a server goes down,its assigned ranges can be distributed to many servers,rather than loading the server's immediate neighbors during downtime. 13.5. Consistency

Having spoken about the virtues of replicating data to multiple machines for durability and spreading load,it's time to let you in on a secret: keeping replicas of your data on multiple machines consistent with one-another is hard. In practice,replicas will crash and get out of sync,replicas will crash and never come back,networks will partition two sets of replicas,and messages between machines will get delayed or lost. There are two major approaches to data consistency in the Nosql ecosystem. The first is strong consistency,where all replicas remain in sync. The second is eventual consistency,where replicas are allowed to get out of sync,but eventually catch up with one-another. Let's first get into why the second option is an appropriate consideration by understanding a fundamental property of distributed computing. After that,we'll jump into the details of each approach. 13.5.1. A Little Bit About CAP

Why are we considering anything short of strong consistency guarantees over our data? It all comes down to a property of distributed systems architected for modern networking equipment. The idea was first proposed by Eric Brewer as the CAP Theorem,and later proved by Gilbert and Lynch [GL02]. The theorem first presents three properties of distributed systems which make up the acronym CAP:

Consistency: do all replicas of a piece of data always logically agree on the same version of that data by the time you read it? (This concept of consistency is different than the C in ACID.)
Availability: Do replicas respond to read and write requests regardless of how many replicas are inaccessible?
Partition tolerance: Can the system continue to operate even if some replicas temporarily lose the ability to communicate with each other over the network?

The theorem then goes on to say that a storage system which operates on multiple computers can only achieve two of these properties at the expense of a third. Also,we are forced to implement partition-tolerant systems. On current networking hardware using current messaging protocols,packets can be lost,switches can fail,and there is no way to kNow whether the network is down or the server you are trying to send a message to is unavailable. All Nosql systems should be partition-tolerant. The remaining choice is between consistency and availability. No Nosql system can provide both at the same time.

Opting for consistency means that your replicated data will not be out of sync across replicas. An easy way to achieve consistency is to require that all replicas ackNowledge updates. If a replica goes down and you can not confirm data updates on it,then you degrade availability on its keys. This means that until all replicas recover and respond,the user can not receive successful ackNowledgment of their update operation. Thus,opting for consistency is opting for a lack of round-the-clock availability for each data item.

Opting for availability means that when a user issues an operation,replicas should act on the data they have,regardless of the state of other replicas. This may lead to diverging consistency of data across replicas,since they weren't required to ackNowledge all updates,and some replicas may have not noted all updates.

The implications of the CAP theorem lead to the strong consistency and eventual consistency approaches to building Nosql data stores. Other approaches exist,such as the relaxed consistency and relaxed availability approach presented in Yahoo!'s PNUTS [CRS+08] system. None of the open source Nosql systems we discuss has adopted this technique yet,so we will not discuss it further. 13.5.2. Strong Consistency

Systems which promote strong consistency ensure that the replicas of a data item will always be able to come to consensus on the value of a key. Some replicas may be out of sync with one-another,but when the user asks for the value of employee30:salary,the machines have a way to consistently agree on the value the user sees. How this works is best explained with numbers.

Say we replicate a key on N machines. Some machine,perhaps one of the N,serves as a coordinator for each user request. The coordinator ensures that a certain number of the N machines has received and ackNowledged each request. When a write or update occurs to a key,the coordinator does not confirm with the user that the write occurred until W replicas confirm that they have received the update. When a user wants to read the value for some key,the coordinator responds when at least R have responded with the same value. We say that the system exemplifies strong consistency if R+W>N.

Putting some numbers to this idea,let's say that we're replicating each key across N=3 machines (call them A,and C). Say that the key employee30:salary is initially set to the value $20,000,but we want to give employee30 a raise to $30,000. Let's require that at least W=2 of A,or C ackNowledge each write request for a key. When A and B confirm the write request for (employee30:salary,$30,000),the coordinator lets the user kNow that employee30:salary is safely updated. Let's assume that machine C never received the write request for employee30:salary,so it still has the value $20,000. When a coordinator gets a read request for key employee30:salary,it will send that request to all 3 machines:

If we set R=1,and machine C responds first with $20,our employee will not be very happy.
However,if we set R=2,the coordinator will see the value from C,wait for a second response from A or B,which will conflict with C's outdated value,and finally receive a response from the third machine,which will confirm that $30,000 is the majority opinion.

So in order to achieve strong consistency in this case,we need to set R=2} so that R+W3}.

What happens when W replicas do not respond to a write request,or R replicas do not respond to a read request with a consistent response? The coordinator can timeout eventually and send the user an error,or wait until the situation corrects itself. Either way,the system is considered unavailable for that request for at least some time.

Your choice of R and W affect how many machines can act strangely before your system becomes unavailable for different actions on a key. If you force all of your replicas to ackNowledge writes,then W=N,and write operations will hang or fail on any replica failure. A common choice is R + W = N + 1,the minimum required for strong consistency while still allowing for temporary disagreement between replicas. Many strong consistency systems opt for W=N and R=1,since they then do not have to design for nodes going out of sync.

HBase bases its replicated storage on HDFS,a distributed storage layer. HDFS provides strong consistency guarantees. In HDFS,a write cannot succeed until it has been replicated to all N (usually 2 or 3) replicas,so W = N. A read will be satisfied by a single replica,so R = 1. To avoid bogging down write-intensive workloads,data is transferred from the user to the replicas asynchronously in parallel. Once all replicas ackNowledge that they have received copies of the data,the final step of swapping the new data in to the system is performed atomically and consistently across all replicas. 13.5.3. Eventual Consistency

Dynamo-based systems,which include Voldemort,allow the user to specify N,R,and W to their needs,even if R + W <= N. This means that the user can achieve either strong or eventual consistency. When a user picks eventual consistency,and even when the programmer opts for strong consistency but W is less than N,there are periods in which replicas might not see eye-to-eye. To provide eventual consistency among replicas,these systems employ varIoUs tools to catch stale replicas up to speed. Let's first cover how varIoUs systems determine that data has gotten out of sync,then discuss how they synchronize replicas,and finally bring in a few dynamo-inspired methods for speeding up the synchronization process. Versioning and Conflicts

Because two replicas might see two different versions of a value for some key,data versioning and conflict detection is important. The dynamo-based systems use a type of versioning called vector clocks. A vector clock is a vector assigned to each key which contains a counter for each replica. For example,if servers A,and C are the three replicas of some key,the vector clock will have three entries,(N_A,N_B,N_C),initialized to (0,0).

Each time a replica modifies a key,it increments its counter in the vector. If B modifies a key that prevIoUsly had version (39,1,5),it will change the vector clock to (39,2,5). When another replica,say C,receives an update from B about the key's data,it will compare the vector clock from B to its own. As long as its own vector clock counters are all less than the ones delivered from B,then it has a stale version and can overwrite its own copy with B's. If B and C have clocks in which some counters are greater than others in both clocks,say (39,5) and (39,6),then the servers recognize that they received different,potentially unreconcilable updates over time,and identify a conflict. Conflict Resolution

Conflict resolution varies across the different systems. The Dynamo paper leaves conflict resolution to the application using the storage system. Two versions of a shopping cart can be merged into one without significant loss of data,but two versions of a collaboratively edited document might require human reviewer to resolve conflict. Voldemort follows this model,returning multiple copies of a key to the requesting client application upon conflict.

Cassandra,which stores a timestamp on each key,uses the most recently timestamped version of a key when two versions are in conflict. This removes the need for a round-trip to the client and simplifies the API. This design makes it difficult to handle situations where conflicted data can be intelligently merged,as in our shopping cart example,or when implementing distributed counters. Riak allows both of the approaches offered by Voldemort and Cassandra. CouchDB provides a hybrid: it identifies a conflict and allows users to query for conflicted keys for manual repair,but deterministically picks a version to return to users until conflicts are repaired. Read Repair

If R replicas return non-conflicting data to a coordinator,the coordinator can safely return the non-conflicting data to the application. The coordinator may still notice that some of the replicas are out of sync. The Dynamo paper suggests,and Cassandra,Riak,and Voldemort implement,a technique called read repair for handling such situations. When a coordinator identifies a conflict on read,even if a consistent value has been returned to the user,the coordinator starts conflict-resolution protocols between conflicted replicas. This proactively fixes conflicts with little additional work. Replicas have already sent their version of the data to the coordinator,and faster conflict resolution will result in less divergence in the system. Hinted Handoff

Cassandra,and Voldemort all employ a technique called hinted handoff to improve write performance for situations where a node temporarily becomes unavailable. If one of the replicas for a key does not respond to a write request,another node is selected to temporarily take over its write workload. Writes for the unavailable node are kept separately,and when the backup node notices the prevIoUsly unavailable node become available,it forwards all of the writes to the newly available replica. The Dynamo paper utilizes a 'sloppy quorum' approach and allows the writes accomplished through hinted handoff to count toward the W required write ackNowledgments. Cassandra and Voldemort will not count a hinted handoff against W,and will fail a write which does not have W confirmations from the originally assigned replicas. Hinted handoff is still useful in these systems,as it speeds up recovery when an unavailable node returns. Anti-Entropy

When a replica is down for an extended period of time,or the machine storing hinted handoffs for an unavailable replica goes down as well,replicas must synchronize from one-another. In this case,Cassandra and Riak implement a Dynamo-inspired process called anti-entropy. In anti-entropy,replicas exchange Merkle Trees to identify parts of their replicated key ranges which are out of sync. A Merkle tree is a hierarchical hash verification: if the hash over the entire keyspace is not the same between two replicas,they will exchange hashes of smaller and smaller portions of the replicated keyspace until the out-of-sync keys are identified. This approach reduces unnecessary data transfer between replicas which contain mostly similar data. Gossip

Finally,as distributed systems grow,it is hard to keep track of how each node in the system is doing. The three Dynamo-based systems employ an age-old high school technique kNown as gossip to keep track of other nodes. Periodically (every second or so),a node will pick a random node it once communicated with to exchange kNowledge of the health of the other nodes in the system. In providing this exchange,nodes learn which other nodes are down,and kNow where to route clients in search of a key. 13.6. A Final Word

The Nosql ecosystem is still in its infancy,and many of the systems we've discussed will change architectures,designs,and interfaces. The important takeaways in this chapter are not what each Nosql system currently does,but rather the design decisions that led to a combination of features that make up these systems. Nosql leaves a lot of design work in the hands of the application designer. Understanding the architectural components of these systems will not only help you build the next great Nosql amalgamation,but also allow you to use current versions responsibly. 13.7. AckNowledgments

I am grateful to Jackie Carter,Mihir Kedia,and the anonymous reviewers for their comments and suggestions to improve the chapter. This chapter would also not be possible without the years of dedicated work of the Nosql community. Keep building! Footnotes

http://hbase.apache.org/
http://project-voldemort.com/
http://cassandra.apache.org/
http://code.google.com/p/protobuf/
http://thrift.apache.org/
http://avro.apache.org/
http://www.oracle.com/technetwork/database/berkeleydb/overview/index.html
http://redis.io/
http://couchdb.apache.org/
http://www.mongodb.org/
http://www.basho.com/products_riak_overview.PHP
http://www.hypergraphdb.org/index
http://neo4j.org/
http://memcached.org/
http://hadoop.apache.org/hdfs/
http://github.com/twitter/gizzard
http://hadoop.apache.org/zookeeper/

/usr/include/boost/system/error_code.hpp:208：链接时未定义对`boost :: system :: get_system_category（）的引用

[FunctionName("AllProducts")]
    public static async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Anonymous,"get",Route = null)] HttpRequest req,[CosmosDB(
            databaseName: "marketplace",collectionName: "products",sqlQuery = "SELECT * FROM product",ConnectionStringSetting = "CosmosDbConnectionString")] IEnumerable<Product> products,ILogger log)
    {
        ...some magic here
    }

public class Product
{
    public Guid Id { get; set; }
    [required]
    public string Name { get; set; }
    [required]
    public string Description { get; set; }
    public bool IsAvailable { get; set; }
    [ForeignKey("User")]
    public string OwnerName { get; set; }
    public User Owner { get; set; }
}

public class Product
{
    [JsonProperty("PropertyID")]
    public Guid Id { get; set; }
}

System.Diagnostics.Debug.WriteLine("This won't end up in the log file");
System.Diagnostics.Trace.WriteLine("This won't end up in the log file");

<system.diagnostics>
        <sources>
            <source name="MyAppNamespace" switchValue="All">
                <listeners>
                    <add name="nlog" />
                </listeners>
            </source>
        </sources>
        <sharedListeners>
            <add name="nlog" type="NLog.NLogTraceListener,NLog" />
        </sharedListeners>
    </system.diagnostics>

NLog.LogManager.GetCurrentClassLogger().Trace("This Works");

<system.diagnostics>
    <trace>
      <listeners>
          <add name="nlog" type="NLog.NLogTraceListener,NLog" />
      </listeners>
    </trace>
</system.diagnostics>

TraceSource source = new TraceSource("MyAppNamespace");
source.TraceEvent(TraceEventType.Warning,2,"File Test not found");

float k = 1.6;
int a = (int)k;
NSLog(@"a = %d",a);

float k = 1.6;
int a = floor(k);
NSLog(@"a = %d",a);

float k = 1.3;
int a = ceil(k);
NSLog(@"a = %d",a);

float k = 1.6;
int a = round(k);
NSLog(@"a = %d",a);

//写入数据的示例伪代码 //data为我们要存储的数据对象 data.title=”title”; data.name=”name”; data.time=”2009-12-01 10:10:01”; data.from=”1”; id=DB.Insert(data);//写入MySQL数据库 NoSQL.Add(id,data);//以写入MySQL产生的自增id为主键写入NoSQL数据库

//写入数据的示例伪代码 //data为我们要存储的数据对象 bool status=false; DB.startTransaction();//开始事务 id=DB.Insert(data);//写入MySQL数据库 if(id>0){     status=NoSQL.Add(id,data);//以写入MySQL产生的自增id为主键写入NoSQL数据库 } if(id>0 && status==true){     DB.commit();//提交事务 }else{     DB.rollback();//不成功，进行回滚 }

//写入数据的示例伪代码 //data为我们要存储的数据对象 data.title=”title”; data.name=”name”; data.time=”2009-12-01 10:10:01”;data.from=”1”;bool status=false; DB.startTransaction();//开始事务 id=DB.Insert(“INSERT INTO table (from) VALUES(data.from)”);//写入MySQL数据库,只写from需要where查询的字段 if(id>0){     status=NoSQL.Add(id,data);//以写入MySQL产生的自增id为主键写入NoSQL数据库 } if(id>0 && status==true){     DB.commit();//提交事务 }else{     DB.rollback();//不成功，进行回滚 }

//从MysqL中查询出评论主键id列表 commentIds=DB.query("SELECT id FROM comments where aid='评论对象id' LIMIT 0,20"); //根据主键id列表，从Nosql取回评论实体数据 CommentsList=Nosql.get(commentIds);