Memory Spills

AWS IAM core concepts explained

2020-12-18T19:00:00+00:00

What is IAM
The core concepts
- Policies
- Users
- Groups
- Roles
Instance profiles
How they play together
Summary

What are policies in IAM?

What are the differences between groups and roles?

What is an instance profile and when shall it be used?

What are the differences between roles and instance profiles?

These are some of the questions I had while working on a side project on AWS. It turns out that users, groups, roles, and policies are core concepts of AWS Identity and Access Management (IAM).

IAM is crucial to the security of applications and infrastructures built on AWS. In this post, I would like to share my understanding of IAM and its core concepts.

TLDR

Use policies to define permissions
To grant long-term permissions to users, attach the policies to groups
To grant short-term permissions to users or services, attach the policies to roles
To pass a role to EC2 instances, use a corresponding instance profile

What is IAM

IAM is an AWS security feature for managing access to AWS services and resources. These services and resources are not only accessed by human users, but also by non-human entities such as other services, instances, or applications. Therefore the term security principal refers to either a human user or such a non-human entity. AWS IAM covers both aspects of access control: authentication and authorization.

Authentication validates principals are who they claim to be, and
Authorization gives the authenticated principals permissions they are supposed to have and denies any other accesses.

The core concepts

For me it was helpful to know for each of the IAM concepts, whether it is relevant to human users or nonhuman entities. This is summarized in the following table:

Concept	Human users	Nonhuman entities
Policies	X	X
Users	X
Groups	X
Roles	X	X

Policies

In IAM, permissions are defined in policies. A policy specifies what kind of accesses are allowed for which resources under what conditions. Policies can be attached to users, groups, and roles¹. Attaching a policy to a user gives the user permissions specified in the policy. The implications of attaching a policy to groups or roles are discussed later in this post.

The example² below is a policy that allows an IAM user to start or stop EC2 instances, but only for instances having an Owner tag with the value matching the user’s username. The DescribeInstances permissions are needed to complete this action on the AWS console.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws:ec2:*:*:instance/*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/Owner": "${aws:username}"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
        }
    ]
}

Users

An IAM user is a person accessing AWS services and resources. Users directly interact with AWS services and have long-term credentials, such as AWS access keys and secret keys, X.509 certificates, SSH keys, passwords for web app logins, or MFA devices.

Groups

An IAM group is a set of IAM users. The policies attached to a group define the permissions shared by its members. Group is a convenience feature for efficiently managing permissions for a set of users. Therefore, although policies can be directly attached to a user, it is a better practice to attach policies to the groups that the user is a member of.

Roles

An IAM role defines a set of permissions via the attached policies. Roles cannot make direct requests to AWS services, they are meant to be assumed by trusted entities, including IAM users and AWS services. The main use case of roles is to delegate accesses with defined permissions to those trusted entities, without having to share long-term credentials with them.

The trust relationship is defined in the role’s trust policy when the role is created, as shown in the screenshot below, where the trusted entity can be either an AWS service, or a user (Another AWS account, Web identity, or SAML 2.0 federation). So a role is a container of polices, which define either permissions or trust relationships.

One can even use roles to delegate access between AWS accounts. To assume a role from a different account, that account must be trusted by the role.

As mentioned earlier, policies can be attached to both roles and groups. Unfortunately, roles can not be attached to groups – that would be a nice feature. A big difference between groups and roles: groups are for users of the same account only, whereas roles can be assumed by users or services, from the same account or other accounts.

Instance profiles

It might be arguable whether instance profile is an IAM core concept – it is even not visible in the AWS console³. Nevertheless, it is a topic that one shall be aware of when working with EC2, which is no doubt a core AWS service.

An instance profile is a container for a single IAM role – it can be mapped to one and only one role. An instance can be launched with or without an instance profile. If it is launched with an instances profile, the corresponding role is associated with the instance, or passed to the instance in AWS terms⁴. This association makes temporary credentials available to the applications on those instances. With the temporary credentials, the applications obtain the permissions defined in the role⁵. Permissions are required, if the applications need to access other AWS services, which is a typical use case.

Without instance profiles, the way of granting permissions to those applications is to deploy the access keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on the EC2 instances. However, it is challenging to do so: the administrator would need to manage those keys, and the DevOps would need to make them available to the applications, e.g., by including them in the application configurations. The access keys are long term credentials owned by users, so groups and polices need to be set up properly. Key rotations and automatically launched instances make it even more complicated.

So the major benefit of using instance profiles is the removal of long-term credentials from EC2 instances. The difference between roles and instance profiles: roles can be assumed by human users, whereas instance profiles can only be attached to instances at the launch time.

However, I still have an open question for myself and the reader: when launching an instance, would it be easier to directly specify the required IAM role, instead of indirectly specifying the same but via an instance profile, which corresponds 1:1 to the role?

How they play together

To grant permissions to a human user, we shall first check if any AWS managed policy can be directly used. If not, a customer managed policy can be created. The policy can be attached to either a group which the user is a member of, or a role which the user can assume.

Let’s assume a set of permissions (for accessing some sample AWS service or resource) are defined in a policy p, which has been attached to a group g and a role r.

For an IAM user to obtain permissions defined in p, there are two options:

being a member of g, or
assuming r.

For an instance to obtain the same set of permissions, there are also two options:

launching it with an instance profile corresponding to role r, or
deploying a pair of access and secret keys (created by a member of g) on it.

Summary

This post documents my understanding of IAM and some of its core concepts, their use cases, and their relationships. It cannot replace any AWS documentation – on the contrary – the official AWS documentation is the source of truth. However, the understanding from an AWS user’s perspective hopefully would be helpful for some other AWS users.

AWS has multiple policy types. In this post, we only discuss the identity-based policies. ↩
The source of the example is here. ↩
When an IAM role is created using the AWS console, an instance profile is created automatically behind the scenes by AWS. ↩
The user launching the instance needs to have the RunInstances and the PassRole permissions. ↩
The applications running on that instance automatically get the (short-term) credentials from that instance’s metadata service and obtain the permissions defined in the role passed via the instance profile. ↩

Null gotchas in Apache Pig

2018-09-01T15:23:07+00:00

Boolean operators and filters
Regular expressions
Null values are viral
Grouping and joining nulls
Summary
References

Apache Pig’s concept of null was not straight forward to me, although it is clearly documented in the Pig book and highlighted in one of its early chapters:

It is important to understand that in Pig the concept of null is the same as in SQL, which is completely different from the concept of null in C, Java, Python, etc. – Gates, A. and Dai, D., 2016, p. 26

I missed to read this and had some fun :wink: investigating a production issue related to nulls. For people like me who do not read (tech) books front to back, it is easier to have all the pieces of information related to a topic together in one place. Therefore the following is a collection of important points on the concept of null from the book, together with some simple experiments.

Boolean operators and filters

For Boolean operators, nulls follow the SQL ternary logic. Thus, x == null results in a value of null, not true (even when x is also null) or false. Filters pass through only those values that are true. – Gates, A. and Dai, D., 2016, pp. 54-55

In Apache Pig, Boolean expressions with nulls can yield three possible values: true, false, and null. This is not always obvious and can cause confusion, because people would expect a Boolean operator to return either true or false, as it does in some other programming languages, e.g., Python or Java.

When I first saw the following code:

A = load 'input.txt' as (k:int, v:int);
B = filter A by k == 1;
C = filter A by k != 1;

I naively believed that the union of B and C is equivalent to A, following the Complement Laws, which says the union of a set $S$ and its absolute complement $S^\complement$ is the universe $U$, i.e.,

\[S \cup S^\complement = U.\]

The problem is that in our example, C is not B’s absolute complement, due to the nulls, i.e.,

\[B^\complement = C \cup \text{\{records whose k field is null\}}\] \[\Longrightarrow\] \[A = B \cup B^\complement = B \cup C \cup \text{\{records whose k field is null\}}.\]

So the Complement Laws still apply. We just need to keep in mind that the absolute complement of filter A by condition is filter A by not condition plus the records whose k field is null. This can be demonstrated by the following experiments.

First, lets prepare an input file, which contains a record with a null field:

cat > input1.txt
1,X
,Y
3,Z
CTRL+D

Then, we can run the following Pig commands and examine the dumped values:

A = load 'input1.txt' using PigStorage(',') as (k:int, v:chararray);

B = filter A by k == 1;
dump B;
-- (1,X)

C = filter A by k != 1;
dump C;
-- (3,Z)

D = filter A by k is null;
dump D;
-- (,Y)

U = union B, C, D;
dump U;
-- (1,X)
-- (,Y)
-- (3,Z)

Note that the record (,Y) passes neither the filter k == 1 nor the filter k != 1. It only qualifies for the filter k is null. Therefore, only the union of B, C, and D can be equivalent to A.

Regular expressions

Likewise, null neither matches nor fails to match any regular expression value. – Gates, A. and Dai, D., 2016, pp. 54-55

Again, lets first prepare an input file, which contains a record with a null field:

cat > input2.txt
a,X
,Y
c,Z
CTRL+D

Then, we can run the following Pig commands and examine the dumped values:

A = load 'input2.txt' using PigStorage(',') as (k:chararray, v:chararray);

B = filter A by k matches 'a';
dump B;
-- (a,X)

C = filter A by not (k matches 'a');
dump C;
-- (c,Z)

D = filter A by k is null;
dump D;
-- (,Y)

U = union B, C, D;
dump U;
-- (a,X)
-- (,Y)
-- (c,Z)

Null values are viral

Null values are viral for all arithmetic operators. That is, x + null == null for all values of x.

Pig also provides a binary condition operator, often referred to as bincond. It begins with a Boolean test, followed by a ?, then the value to return if the test is true, then a :, and finally the value to return if the test is false. If the test returns null, bincond returns null. – Gates, A. and Dai, D., 2016, p. 48

This can be verified with the following tests:

A = load 'input1.txt' using PigStorage(',') as (k:int, v:chararray);
dump A;
-- (1,X)
-- (,Y)
-- (3,Z)

B = foreach A generate k + 1, v;
dump B;
-- (2,X)
-- (,Y)
-- (4,Z)

C = foreach A generate (k == null ? 2 : k), v;
-- (,X)
-- (,Y)
-- (,Z)

D = foreach A generate (k is null ? 2 : k), v;
-- (1,X)
-- (2,Y)
-- (3,Z)

Note the different bincond (ternary operator) expressions used to generate C and D, and k == null always resulted in nulls, whereas k is null delivered the intended results (true or false).

Grouping and joining nulls

Finally, group handles nulls in the same way that SQL handles them: by collecting all records with a null key into the same group. – Gates, A. and Dai, D., 2016, p. 44

A = load 'input3.txt' using PigStorage(',') as (k:int, v:chararray);
dump A;
-- (1,X)
-- (,Y)
-- (3,Z)
-- (,G)

B = group A by k;
dump B;
-- (1,{(1,X)})
-- (3,{(3,Z)})
-- (,{(,G),(,Y)})

As in SQL, null values for keys do not match anything, even null values from the other input. So, for inner joins, all records with null key values are dropped. For outer joins, they will be retained but will not match any records from the other input. – Gates, A. and Dai, D., 2016, p. 47

A = load 'input3.txt' using PigStorage(',') as (k:int, v:chararray);
B = load 'input3.txt' using PigStorage(',') as (k:int, v:chararray);

C = join A by k, B by k;
dump C;
-- (1,X,1,X)
-- (3,Z,3,Z)

D = join A by k full outer, B by k;
dump D;
-- (1,X,1,X)
-- (3,Z,3,Z)
-- (,G,,)
-- (,Y,,)
-- (,,,G)
-- (,,,Y)

Note grouping treats nulls as identical values, whereas joining treats nulls as different (by not matching them).

Summary

In a nut shell, nulls in Apache Pig:

can be produced by Boolean expressions
are viral for all arithmetic operators
yields nulls in equality checks (unless is null is used for the check)
yields nulls in regex match checks
are treated as identical values for grouping, and
are treated as different values for joining

References

Gates, A. and Dai, D., 2016. Programming pig: Dataflow scripting with hadoop. “O’Reilly Media, Inc.”.

YARN的任务调度机制

2018-01-20T14:32:27+00:00

容量调度器
公平调度器
相似和区别
- 两者的相似性
- 两者的区别
总结
参考文档

最近单位的Hadoop集群发生了死锁问题 (deadlock)，于是对Hadoop的调度程序发生了兴趣。YARN支持可插拔的任务调度器（严格来说也许应该叫资源分配器）。官方文档里面讲了两种：容量调度器 (capacity scheduler) 和公平调度器 (fair scheduler)。默认的是容量调度器。下面是啃完文档后的一点总结，浓缩的都是精华 :wink:。

容量调度器

容量调度器旨在允许多组织（或者多用户）共享大型集群，最大限度地提高集群的吞吐量和利用率，同时为每个组织提供容量保证。

层次化队列

容量调度器支持层次化队列结构，以确保在其他队列被允许使用闲置资源之前，在组织的子队列之间共享资源，从而提供更多的控制和可预测性。

下面这个例子里，a，b，和c都是根队列的子队列，a和b还分别定义了自己的子队列：

<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>a,b,c</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.a.queues</name>
  <value>a1,a2</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.b.queues</name>
  <value>b1,b2,b3</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

容量和弹性

容量调度器同时兼顾容量保证和弹性。

容量保证: 队列被分配一定量的可支配的资源。资源可以由集群容量的百分比来表示。
弹性：闲置资源可以分配给任何队列，也就是说队列可以使用超出其容量的资源。

这就确保资源以可预测的和弹性的方式提供给队列，从而防止集群中资源的人为孤岛，以提高利用率。

两个相关的重要参数：

yarn.scheduler.capacity.<queue-path>.capacity 队列容量，用占集群总容量的百分比来表示。所有队列的容量合必须等于100。
yarn.scheduler.capacity.<queue-path>.maximum-capacity 队列最大容量，允许超过队列容量，以便能够使用其它队列的闲置资源。

表达式<queue-path>是队列的路径，比如上面例子里面队列a1的路径为root.a.a1。

上面说到，队列可以使用超出其容量的资源。从运行容量低于保证容量的队列中再请求这些资源时，就要等目前使用这些资源的任务完成，或者不等它们完成就逐步地从资源使用量超过其保证容量的队列中强夺资源。资源的强夺以容器为单位。资源强夺英文叫preemption，是指不等待资源使用者释放资源而强行把资源抢夺过来。

跟资源强夺相关的重要参数：

yarn.resourcemanager.scheduler.monitor.enable 要激活资源强夺，必须将此值设置为true。
yarn.resourcemanager.scheduler.monitor.policies 这里必须是一个跟调度器兼容的策略，默认的策略org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy与容量调度器兼容。
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval 上述策略相邻两次调用之间等待的时间。
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill 发出强夺容器请求和强行终止容器之间的最长等待时间。
yarn.scheduler.capacity.<queue-path>.disable_preemption 可以将此配置设置为true以选择性地禁用从给定队列进行资源强夺。这样可以实现一定程度的队列之间的优先级。

值得一提的是，我从一些测试里观察到，资源强夺并未造成应用程序报错，可能是max_wait_before_kill相对比较长，也可能是Hadoop的容错机制起了作用。

并发度控制

并发度的控制可以通过下面两组参数来配置：

yarn.scheduler.capacity.maximum-applications / yarn.scheduler.capacity.<queue-path>.maximum-applications 应用程序的数量的最大值。这是一个硬性的限制，超过这个限制，提交任何申请时，都会被拒绝（应用程序会出错退出）。
yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent 集群中可用于运行应用程序主控的资源的最大百分比。这个值间接控制并发活动应用程序的数量。这是一个柔性的限制。超过这个限制，提交的申请会被接受并在队列里面等候。

运行时配置

容量调度器支持运行时修改配置。管理员可以在集群运行时以安全的方式更改队列定义和属性（如容量，ACL），以最大限度地减少对用户的干扰。

管理员可以在运行时添加额外的队列，但在运行时不能删除队列。
管理员可以在运行时停止队列，以确保当现有的应用程序运行完成时，不会提交新的应用程序。如果队列处于“已停止”状态，则不能将新应用程序提交给它或它的任何子队列。现有的应用程序将继续完成，因此队列可以正常排空。
管理员也可以启动已停止的队列。

容量调度器的参数主要在capacity-scheduler.xml这个配置文件里。修改这个文件后，可以通过命令来更新队列配置。例如：

$ vi $HADOOP_CONF_DIR/capacity-scheduler.xml
$ $HADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues

另外，一小部分容量调度器相关的参数在yarn-site.xml这个配置文件里。修改这里面的配置可能需要重启资源管理器才能使修改生效。

公平调度器

公平调度器允许YARN应用程序公平地共享大型集群中的资源。旨在让短应用程序在合理的时间内完成，而长应用程序也能得到资源。与容量调度器相似，它也利用分层的队列结构来进行资源分配，这里不再赘述。值得一提的是，公平调度器也允许为队列分配保证的最小份额。而且当队列不需要保证的份额时，多余的份额会在其他正在运行的应用程序之间进行分配，这一点也与容量调度器相似。

调度策略

公平调度器支持可插拔的调度策略，并且支持为每个队列设置不同的调度策略。内置的调度策略有FIFO(FifoPolicy)，FAIR(FairSharePolicy)，和DRF(DominantResourceFairnessPolicy)。默认情况下，公平调度器只基于内存资源使用公平性调度策略 (FAIR)。它可以配置为主导资源公平策略 (DRF) 来调度内存和CPU。关于主导资源，我的理解就是瓶颈资源。

权重和优先级

权重用来确定每个应用程序或队列应该获得的资源的比例。权重体现了应用程序或队列之间的优先级。

两个权重相关的参数：

应用权重：yarn.scheduler.fair.sizebasedweight 是否根据应用程序的大小进行资源分配，而不是向所有应用程序提供相同多的资源而不考虑应用程序的大小。默认为false。
队列权重：weight 队列可以与其他队列不成比例地共享集群。权重默认为1，而权重为2的队列应该接收的资源量约为默认权重的两倍。

并发度控制

公平调度器默认允许所有应用程序运行，但也可以通过配置文件限制每个用户和每个队列运行的应用程序的数量。限制应用程序不会导致任何随后提交的应用程序失败。提交后的应用程序只能在队列中等待，直到某些应用程序完成。这一点跟容器调度器的相关特性不太一样。个人觉得公平调度器的处理更加友好一点。

运行时配置

公平调度器也支持运行时修改配置。公平调度器相关的参数存在于两个配置文件里：yarn-site.xml和分配文件。修改yarn-site.xml里面的配置可能需要重启资源管理器才能使修改生效。而分配文件由yarn.scheduler.fair.allocation.file来指定，默认是fair-scheduler.xml。分配文件每10秒重新加载一次，允许进行更改。

公平调度器支持将正在运行的应用程序移动到不同的队列中。这对于将重要的应用程序移动到更高优先级的队列或将不重要的应用程序移动到更低优先级的队列可能会有用。例如：

yarn application -movetoqueue appID -queue targetQueueName

相似和区别

两者的相似性

都是Hadoop的可插拔调度程序。
都支持分层的队列结构。
都支持队列层面上的最少资源保证。
都支持闲置资源的分配和利用。
都提供了一套限制来防止单个应用程序，用户和队列独占整个队列或集群的资源。
都支持对队列分配策略 (应用程序到队列的映射，一般是基于用户名或组名的一些规则) 进行配置。
都可以通过URLhttps://<ResourceManager URL>/cluster/scheduler查看队列资源的分配和使用情况。
都支持限制每个用户和每个队列运行的应用程序的数量，以防止死锁。
都支持资源的强夺，以防止死锁。

两者的区别

对优先级的支持：容量调度器只是通过给队列分配不同的保证容量以及选择性地禁用资源强夺来部分支持优先级，而公平调度器可以通过应用程序和队列两个层面上的权重来实现优先级。
对并发应用数量的控制：容量调度器是硬性限制（也可以通过控制分配给应用程序主控的资源来间接实现柔性限制），而公平调度器直接是柔性限制。
队列内的资源分配策略：容量调度器是FIFO，而公平调度器支持三种内置的可插拔的策略：FIFO, FAIR, 和DRF。

总结

总体来说，容量调度器和公平调度器的相似多过区别。个人感觉，两者的区别主要是实现上的区别以及参数配置上的区别（运行时的性能和表现肯定也有区别）。公平调度器对优先级以及并发度的控制比容器调度器更直观。另外，本人猜测，容器调度器由HortonWorks主导，而公平调度器由cloudera主导。

最后一点小提示：调度器可以用terasort这样的MapReduce程序来进行简单测试。比如：

# 生成100GB的输入
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen 1073741824 /tmp/teraInput

# 生成20个应用程序，并放入myqueue队列里
for i in $(seq -w 00 19); do
    nohup hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort -Dmapreduce.job.queuename=myqueue /tmp/teraInput /tmp/teraOutput-$i &> $i.log &
    sleep 1;
done

参考文档

容量调度器: capacity scheduler
公平调度器: fair scheduler

How does the GitHub Pages CNAME file work

2018-01-14T08:50:00+00:00

Create a test project on GitHub
Setup
- Set up DNS
- Set up GitHub Pages redirect
Tests
Conclusion

GitHub Pages support custom domains, i.e., sites hosted there can be accessed via DNS names like www.github.com, in addition to the default DNS names like ouyi.github.io or ouyi.github.io/test. I did some tests to figure out how it works. Those tests are described below. To repeat them¹, you need a GitHub account and a test domain.

Create a test project on GitHub

First, lets create a test project on GitHub. This is the test project I created: ouyi.github.io/test. Based on this test project, we can create a minimal working example for GitHub Pages. That requires only a minimalistic HTML index page on the special branch gh-pages. With that in place, we have created a so called Project Pages site, which is accessible at https://ouyi.github.io/test.

Now we can test it with the curl command-line tool².

$ curl -iL "http://ouyi.github.io/test"
HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Content-Type: text/html
Location: https://ouyi.github.io/test
X-GitHub-Request-Id: C902:209A2:BA02A6E:1012C7CB:5A5B61BF
Content-Length: 178
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 14:02:13 GMT
Via: 1.1 varnish
Age: 294
Connection: keep-alive
X-Served-By: cache-hhn1522-HHN
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1515938534.844165,VS0,VE0
Vary: Accept-Encoding
X-Fastly-Request-ID: 9bde18fb5f0118cbb043b30b9aae1850eed3796f

HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Content-Type: text/html
Location: https://ouyi.github.io/test/
X-GitHub-Request-Id: C8FC:4765:17E5632:21B888F:5A5B61BF
Content-Length: 178
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 14:02:13 GMT
Via: 1.1 varnish
Age: 294
Connection: keep-alive
X-Served-By: cache-hhn1545-HHN
X-Cache: HIT
X-Cache-Hits: 2
X-Timer: S1515938534.941957,VS0,VE0
Vary: Accept-Encoding
X-Fastly-Request-ID: 93fba68c26ab60ad62d2ac5a6c584b787532adcf

HTTP/1.1 200 OK
Server: GitHub.com
Content-Type: text/html; charset=utf-8
Last-Modified: Sun, 14 Jan 2018 13:56:49 GMT
Access-Control-Allow-Origin: *
Expires: Sun, 14 Jan 2018 14:07:20 GMT
Cache-Control: max-age=600
X-GitHub-Request-Id: 6990:209A3:5583E4F:760347E:5A5B61BE
Content-Length: 79
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 14:02:13 GMT
Via: 1.1 varnish
Age: 294
Connection: keep-alive
X-Served-By: cache-hhn1545-HHN
X-Cache: HIT
X-Cache-Hits: 2
X-Timer: S1515938534.955052,VS0,VE0
Vary: Accept-Encoding
X-Fastly-Request-ID: 59d0f0fc8487678fe5e489f73b67f96c4de5afc8

<html>
    <head>
    </head>
    <body>
    Hello, World!
    </body>
</html>

Quite a lot was going on to serve our minimal Hello, World! page. The first HTTP response redirects the user agent (curl or a browser) to the HTTPS version of the same location. The second response redirects to the canonical location (which has a trailing slash /) of the site. Finally, the third response returns the HTML page.

Setup

Set up DNS

Now we will set up DNS to point our test domain names to the GitHub Pages IP addresses.

We follow the convention to create an A record for the apex domain builders-it.de to be able to resolve it to an IP address (we picked 192.30.252.154), and a CNAME record for the www subdomain www.builders-it.de as an alias of builders-it.de. How to create those records depends on the DNS provider. I use 1und1.de, which seems to only allow one A record³ for the apex domain (a.k.a. the root domain). But that is sufficient for our tests.

We can verify the DNS set up with the dig command-line tool⁴. After some minutes of waiting, the output shall look like the following:

$ dig www.builders-it.de +nostats +nocomments

; <<>> DiG 9.8.3-P1 <<>> www.builders-it.de +nostats +nocomments
;; global options: +cmd
;www.builders-it.de.		IN	A
www.builders-it.de.	3413	IN	CNAME	builders-it.de.
builders-it.de.		3413	IN	A	192.30.252.154

Set up GitHub Pages redirect

Now we need to tell GitHub Pages our domain name. That can be done by creating a CNAME file containing the www subdomain, again, on the special branch gh-pages. It could also be done using the GitHub Web UI, as shown in the following screen shot:

Tests

Requests using the custom domain name

$ curl -iL "http://www.builders-it.de"
HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 14 Jan 2018 09:58:16 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 79
Vary: Accept-Encoding
Last-Modified: Sun, 14 Jan 2018 08:38:36 GMT
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Expires: Sun, 14 Jan 2018 10:08:16 GMT
Cache-Control: max-age=600
Accept-Ranges: bytes
X-GitHub-Request-Id: FB7C:7582:388994F:4EBCAA3:5A5B29B8

<html>
    <head>
    </head>
    <body>
    Hello, World!
    </body>
</html>

Cool, it works! The following command shows the requests sent to the server.

$ curl -iLv "http://www.builders-it.de" 2>&1 | grep '^> '
> GET / HTTP/1.1
> Host: www.builders-it.de
> User-Agent: curl/7.54.0
> Accept: */*
>

The GitHub Pages IP address (which is most likely a load balancer IP address) in this example is resolved in two steps:

DNS figures out the canonical domain name (which is the root domain name in our setup) for www.builders-it.de using the CNAME record, i.e., www.builders-it.de => builders-it.de.
DNS figures out the IP address for the canonical domain name using the A record, i.e., builders-it.de => 192.30.252.154.

The CNAME could also be www.builders-it.de => ouyi.github.io, as documented in the GitHub official documentation. In that case, the GitHub name servers are responsible for resolving the IP address. The above two steps would be:

DNS figures out the canonical domain name (which is the root domain name in our setup) for www.builders-it.de using the CNAME record, i.e., www.builders-it.de => ouyi.github.io.
GitHub Pages name servers figure out an IP address for the GitHub Pages subdomain, i.e., ouyi.github.io => some ip address.

Requests using the root domain name

Accessing the site using the root domain name involves a 301 redirect to the www subdomain.

$ curl -iL "http://builders-it.de"
HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Date: Sun, 14 Jan 2018 10:29:20 GMT
Content-Type: text/html
Content-Length: 178
Location: http://www.builders-it.de/
X-GitHub-Request-Id: CB47:7585:74DABB1:A2D8546:5A5B3100

HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 14 Jan 2018 10:29:20 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 79
Vary: Accept-Encoding
Last-Modified: Sun, 14 Jan 2018 08:38:36 GMT
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Expires: Sun, 14 Jan 2018 10:39:20 GMT
Cache-Control: max-age=600
Accept-Ranges: bytes
X-GitHub-Request-Id: CB22:7582:389741B:4ED0158:5A5B3100

<html>
    <head>
    </head>
    <body>
    Hello, World!
    </body>
</html>

We can see there were two HTTP requests:

$ curl -iLv "http://builders-it.de" 2>&1 | grep '^> '
> GET / HTTP/1.1
> Host: builders-it.de
> User-Agent: curl/7.54.0
> Accept: */*
>
> GET / HTTP/1.1
> Host: www.builders-it.de
> User-Agent: curl/7.54.0
> Accept: */*
>

The first request goes directly to the GitHub Pages IP address (due to the A record). The GitHub server at that address returns a 301 response, which redirects the user agent to the www subdomain (because of the CNAME file). This kind of redirect seems to be undocumented by GitHub Pages.

Requests using the default DNS name

To see whether the default DNS name for the Project Pages site is still working, we can repeat the very first test we did after we created the test project on GitHub.

$ curl -iL "http://ouyi.github.io/test"
HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Content-Type: text/html
Location: http://www.builders-it.de
X-GitHub-Request-Id: 6402:209A2:B8CB7A7:FF7EA72:5A5B2189
Content-Length: 178
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 09:47:55 GMT
Via: 1.1 varnish
Age: 1473
Connection: keep-alive
X-Served-By: cache-hhn1544-HHN
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1515923275.411885,VS0,VE0
Vary: Accept-Encoding
X-Fastly-Request-ID: dbefaf4a7707a8401e813836d56100c55a5ab412

HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 14 Jan 2018 09:47:55 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 79
Vary: Accept-Encoding
Last-Modified: Sun, 14 Jan 2018 08:38:36 GMT
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
Expires: Sun, 14 Jan 2018 09:57:49 GMT
Cache-Control: max-age=600
Accept-Ranges: bytes
X-GitHub-Request-Id: FBCA:7582:38852CC:4EB67B9:5A5B274B

<html>
    <head>
    </head>
    <body>
    Hello, World!
    </body>
</html>

It works, but differently. There is no redirect to the HTTPS location and there is no redirect to add the trailing slash. The first response redirects the user agent to the www subdomain (due to the CNAME file).

$ curl -iLv "http://ouyi.github.io/test" 2>&1 | grep '^> '
> GET /test HTTP/1.1
> Host: ouyi.github.io
> User-Agent: curl/7.54.0
> Accept: */*
>
> Host: www.builders-it.de
> User-Agent: curl/7.54.0
> Accept: */*
>

Interestingly, the redirect to add the trailing slash still occurs when requesting the HTTPS URL directly, and there is no redirect to the custom domain.

$ curl -iL "https://ouyi.github.io/test"
HTTP/1.1 301 Moved Permanently
Server: GitHub.com
Content-Type: text/html
Location: https://ouyi.github.io/test/
X-GitHub-Request-Id: 6CE4:4767:42B367D:5DF0107:5A5BABD8
Content-Length: 178
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 19:13:28 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-hhn1541-HHN
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1515957208.141728,VS0,VE96
Vary: Accept-Encoding
X-Fastly-Request-ID: 0853013cea20df0ef053f4f85346032fffe9acea

HTTP/1.1 200 OK
Server: GitHub.com
Content-Type: text/html; charset=utf-8
Last-Modified: Sun, 14 Jan 2018 13:56:49 GMT
Access-Control-Allow-Origin: *
Expires: Sun, 14 Jan 2018 14:07:20 GMT
Cache-Control: max-age=600
X-GitHub-Request-Id: 6990:209A3:5583E4F:760347E:5A5B61BE
Content-Length: 79
Accept-Ranges: bytes
Date: Sun, 14 Jan 2018 19:13:28 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-hhn1541-HHN
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1515957208.249646,VS0,VE134
Vary: Accept-Encoding
X-Fastly-Request-ID: b4f06f7769fd14f5924d09fe3729439816761733

<html>
    <head>
    </head>
    <body>
    Hello, World!
    </body>
</html>

Conclusion

Settng up custom domain for GitHub Pages involves two aspects: 1. Set up the domain names to point to the GitHub Pages IP address(es) 2. Add a CNAME file in the project to redirect HTTP requests to the custom domain.

It seems that the DNS name in the CNAME file is used for the HTTP 301 redirect. So, probably a better name for that file could be REDIRECT. Most of the DNS providers nowadays also support HTTP redirects, which can be used, e.g., to redirect HTTP requests to HTTPS locations. But they work the same way as the GitHub redirect does, i.e., they use a Web server.

Footnotes

While doing DNS-related tests, it might be helpful to flush the host DNS cache from time to time, which can be done with various methods, e.g., you can clear host DNS cache with Chrome by opening the URL chrome://net-internals/#dns and click the corresponding button on that page, or if you are on a macbook with macOS Sierra, the host DNS cache can be cleared by the command sudo killall -HUP mDNSResponder. ↩
Tests can also be done with browsers, e.g., Chrome developer tools. ↩
Some DNS providers support multiple A records for the apex domain. ↩
Similar information can also be obtained using the commands host -a www.builders-it.de and host -a builders-it.de. ↩

Rendering table of contents in Jekyll

2017-12-31T13:20:00+00:00

Generating the TOC
Adding the TOC heading

Having a table of contents (TOC) makes an article more readable. Our goal is to automatically generate TOC out of the headings in an article, for a site based on Jekyll and hosted on GitHub Pages, which is the setup of this site.

Generating the TOC

The site generated by Jekyll is basically a set of static HTML pages. It is therefore possible to collect the HTML heading elements (h1, h2, etc.) and generate a TOC using JavaScript, as these libraries do. However, those libraries do not work well, if the heading elements are not only used in the article, but also used in the header or footer of the site, which is the case for this site. For example, the “Memory Spills” site title in the footer is wrapped in a <h2> tag by Jekyll. That means “Memory Spills” would appear in the TOC, which is not nice.

Fortunately, rendering of table of contents (TOC) is supported by kramdown, which is Jekyll 3.0.0’s default Markdown processor, and, as a side note, the only supported Markdown processor on GitHub Pages. Adding the following code snippet directly after the front matter will do the trick:

---
My front matter
---

* TOC
{:toc}
My first paragraph.

Note that no empty lines shall be present between the {:toc} tag and the first paragraph of the article (My first paragraph. in the code example), or else the latter will not be included in the post.excerpt variable, which is used for multiple purposes on this site. First, the post.excerpt variable is used in the home layout to show an excerpt of the post content for each post, in addition to its post title. That way the post content can be previewed on the home page(s), before a reader decides to click the read more button which links to the complete post content. Second, the post.excerpt variable is also used to populate the description meta tags important for SEO.

Adding the TOC heading

The rendered TOC is just a list of article headings. It would be nice if it has the TOC heading “Contents” or “Table of contents”. It turns out this is a bit tricky.

With Markdown

To insert such a TOC heading, one can use the following markdown code:

---
My front matter
---

**Contents**
* TOC
{:toc}
My first paragraph.

This will insert a bold “Contents” line as the heading before the TOC, which is nice, but the word “Contents” becomes a part of the post.excerpt variable, and is consequently shown as a part of the preview, which is not so nice. Therefore, I had to cut off the extra Contents string as follows:

{{ post.excerpt | strip_html | replace_first: 'Contents', '' | lstrip | truncatewords: 40, "" }}<a href="{{ post.url | relative_url }}">&hellip; read more &raquo;</a>

Remember the ‘post.excerpt’ variable is also used to populate the description meta tags. There I found no easy way to get rid of the extra Contents string, which is not nice.

With JavaScript

Therefore, I removed the **Contents** heading from the markdown, and came up with a JavaScript hack.

function domReady() {
    var toc = document.getElementById('markdown-toc');
    if (toc) {
        toc.insertAdjacentHTML('beforebegin', '<p><strong>Table of contents</strong></p>');
    }
}

if ( document.addEventListener ) { // Mozilla, Opera, Webkit
    document.addEventListener( 'DOMContentLoaded', function() {
        document.removeEventListener( 'DOMContentLoaded', arguments.callee, false);
        domReady();
    }, false );
} else if ( document.attachEvent ) { // If IE event model is used
    // ensure firing before onload
    document.attachEvent('onreadystatechange', function() {
        if ( document.readyState === 'complete' ) {
            document.detachEvent( 'onreadystatechange', arguments.callee );
            domReady();
        }
    });
}

The above code can be included ({ % include javascript.html %}) as the last line in _includes/footer.html, so that the <script> tag appears immediately before the closing </body> tag.

This works well without poluting the post.excerpt variable. But is it possible to do this without the JavaScript hack?

With CSS

It turns out arbitrary text can be inserted using CSS, e.g.,

#my-id::before {
  content: "my text before the element with the id my-id";
}

The complete solution is adding the following CSS code to this file:

#markdown-toc::before {
  content: "Contents";
  margin-left: -$spacing-unit;
  line-height: 200%;
  font-weight: bold;
  @include relative-font-size(1.25);
  @include media-query($on-laptop) {
    @include relative-font-size(1.125);
  }
}

This pure CSS solution touches only one file, and, as with the JavaScript solution, it does not polute the post.excerpt variable.

However, I am not sure how to deal with i18n, e.g., for an article written in Chinese, one would prefer to add “目录” instead of “Contents”. That could be easier with the JavaScript solution. If you have a good solution, please leave a comment. Thanks!

Customizing Jekyll theme

2017-12-23T07:56:11+00:00

Override theme defaults
Pagination of the home page
Links to the previous and next pages
Categories and tags
Summary

Jekyll supports a lot of themes, which work quite well out of the box. A theme is a pre-defined set of styles, templates, and template variables. My site is based on the default Jekyll theme: minima. I had to do some customizations to make it work as I wish. Luckily, Jekyll allows customizations of almost everything of the theme. This post shows some of the customizations this site applies on top of minima.

Override theme defaults

The command bundle show minima can be used to find the location where the theme artifacts are installed, e.g.:

[ouyi.github.io]$ tree $(bundle show minima)
/home/ouyi/ouyi.github.io/.bundle/gems/minima-2.1.1
|-- LICENSE.txt
|-- README.md
|-- _includes
|   |-- disqus_comments.html
|   |-- footer.html
|   |-- google-analytics.html
|   |-- head.html
|   |-- header.html
|   |-- icon-github.html
|   |-- icon-github.svg
|   |-- icon-twitter.html
|   `-- icon-twitter.svg
|-- _layouts
|   |-- default.html
|   |-- home.html
|   |-- page.html
|   `-- post.html
|-- _sass
|   |-- minima
|   |   |-- _base.scss
|   |   |-- _layout.scss
|   |   `-- _syntax-highlighting.scss
|   `-- minima.scss
`-- assets
    `-- main.scss

5 directories, 20 files

To override the theme defaults, simply copy the related file from the theme installation location to your project, under the same folder. For example, the following customization adds feed link in the header:

[ouyi.github.io]$ diff .bundle/gems/minima-2.1.1/_includes/header.html _includes/header.html
27a28
>           <a class="page-link" href="/feed.xml"><i class="fa fa-rss" aria-hidden="true"></i></a>

As another example, to include custom styles, make a copy of the main.scss file and add lines to import from any custom style sheets.

[ouyi.github.io]$ diff .bundle/gems/minima-2.1.1/assets/main.scss assets/main.scss
5a6
> @import "custom";
[ouyi.github.io]$ ls _sass/
custom.scss

By default, bundler installs gems in a central location shared by all ruby projects. To change that, set BUNDLE_DISABLE_SHARED_GEMS to true in the bundler config file, e.g.:

cat .bundle/config
---
BUNDLE_DISABLE_SHARED_GEMS: "true"

Pagination of the home page

By default, the minima’s home page shows the complete list of posts, which is not nice. What I would prefer are:

split the home page into multiple pages, if that list become long
control the number of posts displayed per page
link from each page to the previous and next pages

It turns out that Jekyll already has some support for pagination. To enable it, one has to add a line to the _config.yml file, specifying the number of items per page, e.g.: paginator: 8. With pagination enabled, Jekyll populates a paginator liquid object.

The following diff shows my changes to the default home layout, demonstrating the use of the paginator object:

[ouyi.github.io]$ diff .bundle/gems/minima-2.1.1/_layouts/home.html _layouts/home.html
7,8c7,8
<   <h1 class="page-heading">Posts</h1>
<
---
>   <h1 class="post-title">Posts</h1>
>
12c12
<     {% for post in site.posts %}
---
>     {% for post in paginator.posts %}
19a20
>         {{ post.excerpt | strip_html | replace_first: 'Contents', '' | lstrip | truncatewords: 40, "" }}<a href="{{ post.url | relative_url }}">&hellip; read more &raquo;</a>
24,25c25
<   <p class="rss-subscribe">subscribe <a href="{{ "/feed.xml" | relative_url }}">via RSS</a></p>
<
---
>   {% include prev_next.html prev_url=paginator.previous_page_path prev_text='Previous page' next_url=paginator.next_page_path next_text='Next page' %}

Basically, instead of iterating over site.posts, one has to loop over paginator.posts. In addition, pagination only works with the index.html file, which references the home layout. I also had to rename the file index.md (default) to index.html to fix this error:

Pagination: Pagination is enabled, but I couldn't find an index.html page to use as the pagination template. Skipping pagination.

Links to the previous and next pages

The links to the previous and next pages are implemented as a macro in the prev_next.html file. The macro requires four named parameters: prev_url, prev_text, next_url, and next_text. This line includes the macro and passes the required parameters:

{% include prev_next.html prev_url=paginator.previous_page_path prev_text='Previous page' next_url=paginator.next_page_path next_text='Next page' %}

The same macro is reused by the post layout to render the links to the previous and next posts.

As a side note, Jekyll pagination automatically generates the <link> tags with rel="next" and rel="prev" attributes as recommended by Google. Those tags reside in the <head> section of of the HTML page. They are not to be mixed up with the hyperlinks generated by the above macro.

Categories and tags

Categories and tags are two largely overlapping concepts in Jekyll. From a product perspective, I doubt the necessity of having both implemented, which do not provide added value instead of confusing the users. The only difference seems to be that categories become a part of the post URL. That means, a post having the category: hadoop in the front matter would have a URL like https://ouyi.github.io/hadoop/2017/10/08/hbase.html, where hadoop is the category. That also means, posts published on the same date (or in the same month or year) will be put into different folders, if they are of different categories. I do not like that. One could also customize the permalink pattern in Jekyll to remove the categories from the URL. But I chose to set the categories of all posts to post, e.g., the front matter of this post is:

---
layout: post
title:  "Customizing Jekyll theme"
date:   2017-12-23 07:56:11 +0000
last_modified_at: 2017-12-31 14:05:16
category: post
tags: [Blogging, Jekyll]
---

This way all posts go to the post category, which is simple and clear. The tags are used to generate a tags page, where all the posts are indexed by tags. The page generation code can be found here.

Summary

This post showed how to customize a theme in Jekyll, how to paginate the home page, how to generate links to navigate to the neighbouring pages, and discussed a bit about categories and tags. A separate post covers how to automatically generate the table of contents in Jekyll.

Sending emails from Java applications

2017-12-19T22:12:13+00:00

Maven dependencies
The mail sending service implementation
Unit tests
Dependency injection
The REST controller
Calling the REST service using JQuery and Ajax

I have so far worked with two ways of sending emails from Java applications. The first way directly uses the JavaMail API, while the second one utilizes the Spring framework, which makes it a bit easier to work with the JavaMail API. The JavaMail Wikipedia page has sufficient examples for the direct use of the JavaMail API. This post shows code examples of the Spring one. We are implementing a Spring-based REST service for sending emails.

Maven dependencies

The Spring support for emails is provided in the spring-context-support artifact, and we also need the JavaMail artifact. Therefore, the following dependencies are needed in the pom.xml file:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-context-support</artifactId>
    <version>${spring.version}</version>
</dependency>
<dependency>
    <groupId>javax.mail</groupId>
    <artifactId>mail</artifactId>
    <version>1.4.7</version>
</dependency>

The dependency dumbster provides a fake SMTP server, to be used in the unit tests:

<dependency>
    <groupId>com.github.kirviq</groupId>
    <artifactId>dumbster</artifactId>
    <version>1.7.1</version>
    <scope>test</scope>
</dependency>

The mail sending service implementation

We implement the following MailService interface, which can be used by, e.g., a Spring RestController to expose an email-sending REST service.

package io.github.ouyi.mail.service;

import javax.mail.MessagingException;

public interface MailService {
    void sendMail(String to, String subject, String bodyText, String attachmentPath) throws MessagingException;
}

The implementation looks like this:

package io.github.ouyi.mail.service;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.FileSystemResource;
import org.springframework.mail.javamail.JavaMailSender;
import org.springframework.mail.javamail.MimeMessageHelper;

import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;
import java.util.Optional;

public class MailServiceImpl implements MailService {

    private final JavaMailSender mailSender;

    private final String from;

    @Autowired
    public MailServiceImpl(JavaMailSender mailSender, String from) {
        this.mailSender = mailSender;
        this.from = from;
    }

    @Override
    public void sendMail(String to, String subject, String bodyText, String attachmentPath) throws MessagingException {
        send(from, to, subject, bodyText, attachmentPath, !Optional.ofNullable(attachmentPath).orElse("").isEmpty());
    }

    private void send(String from, String to, String subject, String bodyText, String attachmentPath, boolean multipart) throws MessagingException {
        MimeMessage message = mailSender.createMimeMessage();
        MimeMessageHelper helper = new MimeMessageHelper(message, multipart);

        helper.setFrom(from);
        helper.setTo(to);
        helper.setSubject(subject);
        helper.setText(bodyText);

        if (!Optional.ofNullable(attachmentPath).orElse("").isEmpty()) {
            FileSystemResource file = new FileSystemResource(attachmentPath);
            helper.addAttachment(file.getFilename(), file);
        }
        mailSender.send(message);
    }
}

Note a multipart message is required in order to send an email with attachment.

Unit tests

As mentioned earlier, we use dumbster for unit testing:

package io.github.ouyi.mail.service;

import com.dumbster.smtp.SimpleSmtpServer;
import com.dumbster.smtp.SmtpMessage;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.mail.javamail.JavaMailSenderImpl;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Properties;

import static org.junit.Assert.assertEquals;

public class MailServiceImplTest {

    private static final Logger logger = LoggerFactory.getLogger(MailServiceImplTest.class);
    private static final String HEADER_KEY_TO = "To";
    private static final String HEADER_KEY_SUBJECT = "Subject";

    @Test
    public void testSendMail() throws Exception {
        String smtpHost = "localhost";
        int smtpPort = 12345;

        String subject = "MY_SUBJECT";
        String messageBody = "MY_MESSAGE_BODY";
        String fromAddr = "sender@localhost";
        String toAddr = "test@test.com";

        logger.info("Start dumbster fake SMTP server...");
        SimpleSmtpServer server = SimpleSmtpServer.start(12345);

        logger.info("Send mail message");

        JavaMailSenderImpl mailSender = new JavaMailSenderImpl();
        Properties properties = new Properties();
        properties.setProperty("mail.smtp.host", smtpHost);
        properties.setProperty("mail.smtp.port", String.valueOf(smtpPort));
        mailSender.setJavaMailProperties(properties);
        MailServiceImpl mailService = new MailServiceImpl(mailSender, fromAddr);

        mailService.sendMail(toAddr, subject, messageBody, null);
        server.stop();

        assertEquals(1, server.getReceivedEmails().size());

        SmtpMessage email = server.getReceivedEmails().get(0);

        assertEquals(toAddr, email.getHeaderValue(HEADER_KEY_TO));
        assertEquals(subject, email.getHeaderValue(HEADER_KEY_SUBJECT));
        assertEquals(messageBody, email.getBody());
    }

}

Dependency injection

Since we use Spring, we can use dependency injection to create and configure a JavaMailSenderImpl instance, which can be injected into the MailServiceImpl instance:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

    <bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
        <property name="host" value="my.smtp.host"/>
        <property name="port" value="587"/>
        <property name="username" value="user"/>
        <property name="password" value="pass"/>

        <property name="javaMailProperties">
            <props>
                <prop key="mail.smtp.auth">true</prop>
                <prop key="mail.smtp.starttls.enable">true</prop>
            </props>
        </property>
    </bean>
    <bean id="mailService" class="com.hellokoding.account.service.MailServiceImpl">
        <constructor-arg ref="mailSender"></constructor-arg>
        <constructor-arg type="java.lang.String" value="sender@mydomain.com"></constructor-arg>
    </bean>
</beans>

The REST controller

package io.github.ouyi.web;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpSession;
import javax.ws.rs.core.Response;

import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
public class MailController {

    private final Logger logger = LoggerFactory.getLogger(MailController.class);

    @Autowired
    private MailService mailService;

    @Autowired
    private String mailTo;

    private static final String MAIL_SUBJECT = "medshop order";

    private static final String MAIL_CONTENT = "Please find the order in the attachment.";

    @RequestMapping(value = {"/sendmail"}, method = RequestMethod.POST, produces = "application/json")
    public @ResponseBody Response sendmail(HttpServletRequest request, HttpServletResponse response) throws Exception {

        File tempFile = null;
        OutputStream outputStream = null;
        try {
            tempFile = File.createTempFile("mail-attachment-", ".xlsx");
            outputStream = new BufferedOutputStream(new FileOutputStream(tempFile));
            XSSFWorkbook document = fileService.createSpreadsheet();
            document.write(outputStream);
            outputStream.flush();
            mailService.sendMail(mailTo, MAIL_SUBJECT, MAIL_CONTENT, tempFile.getAbsolutePath());
        } finally {
            if (outputStream != null) {
                outputStream.close();
            }
            if (tempFile != null) {
                tempFile.delete();
            }
        }
        return Response.ok().build();
    }
}

The sendmail method of the MailController has to return a Response object. When I changed it to void, I got the following error response from my GlassFish server:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>GlassFish Server Open Source Edition  5.0  - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 405 - Request method &amp;#39&#59;GET&amp;#39&#59; not supported</h1><hr/><p><b>type</b> Status report</p><p><b>message</b>Request method &amp;#39&#59;GET&amp;#39&#59; not supported</p><p><b>description</b>The specified HTTP method is not allowed for the requested resource.</p><hr/><h3>GlassFish Server Open Source Edition  5.0 </h3></body></html>

When it returns a Response object but without the annotation @ResponseBody, I got another error response:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>GlassFish Server Open Source Edition  5.0  - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - Not Found</h1><hr/><p><b>type</b> Status report</p><p><b>message</b>Not Found</p><p><b>description</b>The requested resource is not available.</p><hr/><h3>GlassFish Server Open Source Edition  5.0 </h3></body></html>

Calling the REST service using JQuery and Ajax

Our mail-sending REST service can be consumed by the frontend JavaScript code as follows:

$(document).ready(function(event) {
    $.ajaxPrefilter(function (options, originalOptions, jqXHR) {
      jqXHR.setRequestHeader("${_csrf.headerName}", "${_csrf.token}");
    });

    $('#sendmail-button').click(function() {
        $.ajax({
            type: 'post',
            url: "${contextPath}/sendmail",
            dataType : 'json'
        }).done(function(response) {
            console.log(response);
            alert("Email sent successfully!");
        }).fail(function(response) {
            console.log(response);
            alert("Failed to send email!");
        });
    });
});

Note that ${_csrf.headerName}, ${_csrf.token}, and ${contextPath} are templated on the server-side using jsp. If server-side templating is not used, the CSRF header name and token can be passed to the client as meta tags in the HTML head section, which can then be retrieved by JavaScript code. Details can be found in the Spring security documentation. The context path, which is quite static, can be easily injected into the frontend code at build time.

Understanding Gradle DSL

2017-12-09T19:25:13+00:00

Groovy basics
- Closure
- Method calls
Gradle basics

Most of the time, Gradle works just fine for me, but I had difficulties understanding the build script and the Gradle DSL documentation. This post documents the points which helped me to understand them.

Groovy basics

To understand Gradle, we first need to understand a bit of Groovy, because Gradle is based on Groovy.

Closure

A closure is a function bound to (or executed against) some object instances, which can be one of these three things: this, owner, and delegate.

this: the instance of the enclosing class
owner: same as this, or the enclosing closure if exists
delegate: same as owner, but can be changed

By default, a variable accessed in a Groovy closure is resolved as follows: the closure itself will be checked first, followed by the closure’s this scope, then the closure’s owner, then its delegate.

The following example demostrates changing the delegate of a closure:

class MyOtherClass {
    String myString = "I am in MyOtherClass"
}

class MyClass {
    def myClosure = {
        println myString // myString is undefined (neither on this nor on owner)
    }
}

MyClass myClass = new MyClass()
def closure = new MyClass().myClosure
closure.delegate = new MyOtherClass() // Changing the delegate makes variables in the new delegate available to the closure
closure()   // outputs: "I am in MyOtherClass"

It is important to notice that closure can access member fields and methods defined in another object, which can be changed dynamically at runtime. As a side note: closure and lambda are different concepts. A lambda is simply an anonymous function, without all the this, owner, and delegate stuff.

Method calls

Method calls in Groovy look quite different than in Java. The following are all method call examples in Groovy:

println x // System.out is imported by default, no parenthesis needed if there is at least one parameter

println 1..3 // prints the sequence 1, 2, and 3

printArgs x, a:1, b:2 // named parameters are collected and passed as a map, if the first parameter of the method is a map

doStuff { println it } // closure as the method parameter, `it` is the default closure parameter

list.each { println it } // for each of the list items, execute the closure, with the item as parameter

list.each { i -> println i } // same as above, the closure parameter can also be specified explicitly

The return type distinguishes method definitions from method calls, e.g., this is a method definition:

void repositories() { println "in a closure" }

Gradle basics

With a basic understanding of closures and method calls in Groovy, we can try to make sense of the following method calls:

repositories() { println "in a closure" } // call the repositories() method defined on the project object

repositories { println "in a closure" } // same as above, no parenthesis needed

Note in Gradle DSL, closures are frequently (and idiomatically) used as the last method parameter to configure some object. This is a pattern called configuration closure.

Each closure has a delegate object, which Groovy uses to look up variable and method references which are not local variables or parameters of the closure. Gradle uses this for configuration closures, where the delegate object is set to the object to be configured.

Gradle processes a build script in three phases: initialization, configuration, and execution. The initialization and configuration phases create and configure project objects and construct a DAG of tasks, which are then executed in the execution phase.

Dependencies

With the above being discussed, I understand the following is calling the dependencies() method on the project object. And the documentation indicates the configuration closure is executed against the DependencyHandler object.

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

But where is testCompile defined?

The DependencyHandler documentation says:

To declare a specific dependency for a configuration you can use the following syntax:
dependencies {
    configurationName dependencyNotation1, dependencyNotation2, ...
}

So testCompile is a configurationName, but where is this configurationName defined? It turns out that Gradle plugins can add various stuff to the project. In our case, the testCompile is a dependency configuration added by the java plugin. That is why we need this line in our build script:

apply plugin: 'java' //so that we can use 'compile', 'testCompile' for dependencies

Tasks

Gradle plugins can also add tasks to the project, e.g., the jar task added by the java plugin:

jar {
  manifest {
    attributes(
      'Class-Path': configurations.compile.collect { it.getName() }.join(' '),
      'Main-Class': 'hello.HelloWorld'
    )
  }
}

The documentation says:

Each jar or war object has a manifest property with a separate instance of Manifest.

This explains the manifest {} construct. Following the documentation of Manifest, we can find the method signature Manifest attributes(Map<String,?> attributes), which explains the method call with named parameters attributes(key1:value1, key2:value2).

Custom tasks

The syntax of a custom task is tricky. For example:

task myTask(type:Tar, dependsOn anotherTask) {
    // clousre
}

I could figure out that this was probably calling the method Task task(Map<String,?> args, String name, Closure configureClosure) defined on the project object, but I had no clue how to match the myTask() construct with the name parameter. And I am not alone, similar disussions are here and here.

It turns out that Gradle uses some advanced meta programming features of Groovy (compile-time metaprogramming) to transform the myTask() construct to the name parameter. To be honest, this is the part of Gradle that I do not like, because it seems to be too tricky to implement those syntax sugars (and too much sugar may not be healthy). Afterall, Gradle is just a build tool, which shall be easy to understand, to use, and to extend.

Overall, I find Gradle really cool, because build scripts written in Gradle DSL (or in Groovy) seem to be much cleaner than in XML. With Maven (or Ant), one has to read or write build logic in XML, which is supposed to be processed by machines :wink:.

Migrating from Blogspot to GitHub Pages

2017-10-14T07:56:11+00:00

Travis CI
Hosting images
Redirects and canonical URL tags

Recently I migrated my blog from Blogspot to GitHub Pages. It took a while, but I am glad I did it, because blogging with GitHub Pages is much more enjoyable than with Blogspot, as long as one is comfortable with Git and Markdown. More specifically, I like GitHub Pages because:

Everything of the blog is version controlled, including posts, themes, and settings
Static site generators (I chose Jekyll) allow customizations of almost everything
Markdown produces much cleaner HTML pages than those produced by a WYSIWYG editor

With Blogspot I already had some of my posts in Markdown and in a Git repository. But to publish a post there, I had to convert the post from Markdown to HTML with some tools, and then paste the result page into the Blogspot post editor Web UI.

Now with GitHub Pages and Jekyll, the workflow of blogging is like this:

Writing a post in Markdown with any text editor, and
Add, commit, and push the changes in Git

GitHub Pages will automatically run a jekyll build to generate the HTML pages, which usually become online in a couple of seconds after the push.

The migration process, however, was not always straightforward. I took notes so that it might be helpful for the readers.

Travis CI

The downside of having GitHub Pages building everything automatically behind the scenes is that if there is an issue, e.g., a Markdown syntax error, you would not even notice it. Therefore, I configured it to be built also automatically on Travis CI, which requires just a config file (.travis.yml) in the root folder of the project:

language: ruby
cache: bundler
install:
  - bundle install
script:
  - bundle exec jekyll build --safe
  - bundle exec htmlproofer ./_site --disable-external

This instructs Travis to generate the site (stored in the _site folder) and run htmlproofer on it while ignoring the linked external sites. For all this to work, of course, one has to connect the GitHub project in Travis CI, which is out of scope of this post (and GitHub Pages also works without Travis CI).

When all this has been set up, one got email notifications from Travis on build failures. One got also a small build passing (or failing) badge:

The command htmlproofer is provided by the gem html-proofer. In addition, to make sure the build on Travis has the same dependencies as the build on GitHub, specify gem 'github-pages', group: :jekyll_plugins in the Gemfile. Before the changes are pushed, one can always run bundle exec jekyll serve --port 8080 --host 0.0.0.0 --drafts and open http://localhost:8080 with the browser to proof read the generated site.

Hosting images

I mentioned earlier: “everything of the blog is version controlled”. This does NOT apply to images, because I do not think including images in a Git repository is a great idea. But we need to host the images somewhere, e.g., S3, CDN, or any other hosting services. The solution I chose was found on Statck Overflow. It allows me to host my images on GitHub, based on a secret GitHub feature.

Once I got the URL to the image, I can add the image using standard Markdown syntax (i.e., ![alt text](image_url "title text")) or the HTML img tag (yes, in a Markdown file), e.g.:

<img src="https://user-images.githubusercontent.com/15970333/32409768-84c92cc8-c1b2-11e7-9309-428c99da8cac.png" alt="screen shot 1">

Redirects and canonical URL tags

After a post has been migrated, there are two versions of it on the Web: the old one on Blogspot and the new one on GitHub. Simply removing the old version would mean a bad user experience (they might have bookmarked the old version) and would hurt SEO. Ideally, we would like to:

redirect the browser visiting the old version to the new version, and
tell search engines that the old version is obsolete and the new version is the one to index.

After some searches, I came up with this kind of Blogspot template code snippet:

<head>
    <b:if cond='data:blog.url == &quot;https://ouyi-cs.blogspot.com/2015/12/hadoop-streaming-broken-pipe-issue.html&quot;'>
      <link href='https://ouyi.github.io/hadoop/2015/12/06/hadoop-streaming.html' rel='canonical'/>
      <meta content='0; url=https://ouyi.github.io/hadoop/2015/12/06/hadoop-streaming.html' http-equiv='refresh'/>
    <b:elseif cond='data:blog.url == &quot;http://ouyi-cs.blogspot.com/2015/12/hadoop-streaming-broken-pipe-issue.html&quot;'/>
      <link href='https://ouyi.github.io/hadoop/2015/12/06/hadoop-streaming.html' rel='canonical'/>
      <meta content='0; url=https://ouyi.github.io/hadoop/2015/12/06/hadoop-streaming.html' http-equiv='refresh'/>

    <b:elseif cond='data:blog.url == &quot;https://ouyi-cs.blogspot.com/2015/12/downgrade-rpms-using-ansible.html&quot;'/>
      <link href='https://ouyi.github.io/cicd/2015/12/14/ansible-downgrade-rpm.html' rel='canonical'/>
      <meta content='0; url=https://ouyi.github.io/cicd/2015/12/14/ansible-downgrade-rpm.html' http-equiv='refresh'/>
    <b:elseif cond='data:blog.url == &quot;http://ouyi-cs.blogspot.com/2015/12/downgrade-rpms-using-ansible.html&quot;'/>
      <link href='https://ouyi.github.io/cicd/2015/12/14/ansible-downgrade-rpm.html' rel='canonical'/>
      <meta content='0; url=https://ouyi.github.io/cicd/2015/12/14/ansible-downgrade-rpm.html' http-equiv='refresh'/>

    <!-- one such block for each post migrated ... -->

    <b:else/>
    </b:if>

    <!-- ... -->
</head>

The code snippet can be added to the Blogspot HTML template (in blogger.com Web UI, click “Theme”, then “Edit HTML”), directly after the opening <head> tag. For each migrated post, it will generate two tags (link and meta) in the head section of the Blogspot version. The meta tag with http-equiv='refresh' takes care of redirecting the browser to the GitHub version. The link tag tells search engines that the GitHub version is the one (and the only one) to be indexed. Note also that I had to take care of both the http and https URLs for the Blogspot version.

The syntax seems to be overwhelmingly verbose (working with Jekyll and Liquid, its template system, is way easier). If you find a cleaner or shorter way of doing this, please let me know. More details about the Blogspot template syntax can be found here.

After a few weeks, search engines will remove the Blogspot version from their indexed pages and index the GitHub one. The indexing process is quite out of our control. One has to be patient. From what I observed, Google seems to have picked up the posts on the new site faster than Bing and Yahoo. The following are a couple of articles related to “page URL changes” I found useful:

The blog migration also involves customization of the Jekyll theme I chose, which is described in a separate post.

理解HBase和BigTable

2017-10-08T22:42:27+00:00

都在术语里边
映射表
持久化
分布式
有序的
多维度
稀疏
结束语

本文翻译自Jim R. Wilson的Understanding HBase and BigTable。

学习HBase（Google BigTable的开源实现）最难的部分，就是从概念上理解它到底是什么。

我觉得挺不幸的是，这两个很棒的系统的名字用了表（table) 和库（base）这样的字眼，这很容易把熟悉关系型数据库的那些人（像我自己）搞晕。

这篇文章主要想从概念上描述清楚这两个分布式数据存储系统，读完它以后，当你面对“该用HBase还是该用传统数据库”这样的问题的时候，应该能够做出更加明智的决定。

都在术语里边

幸运的是，Google的BigTable论文清楚地解释了BigTable实际上是什么。这是论文“Data Model”部分的第一句话：

BigTable是一个稀疏的，分布式的，持久化的，多维的，有序的映射表(map)。

注意：读完上面这句话以后，有些读者可能有点晕，现在可以定一下神。

这篇论文接着解释道：

这个map是用row key，column key, 和timestamp来做索引的，每个在map里的值都是未经处理的字节数组。

同样地，HBase架构文档指出：

HBase使用的数据模型与BigTable非常相似。用户将数据行存在有标签的表中。一个数据行有一个可排序的key 和任意多个列。表的存储是稀疏的，所以只要用户喜欢，同一张表的数据行能够有疯狂变化着的不同数量的列。

虽然这一切看起来相当神秘，但是当你把它分成一个个单词的话，就可以理解了。我将按照以下这个顺序来讨论它们：映射表，持久化，分布式，有序，多维度，和稀疏。

与其尝试整个儿来理解一个系统，我发现化整为零更容易理解。

映射表

本质上来说，HBase和BigTable就是一个map。取决于编程语言背景，你可能更熟悉关联数组（PHP），字典（Python），哈希（Ruby），或者对象（JavaScript）。

根据维基百科，map是一个抽象的数据类型，由一个键的集合和一个值的集合组成，每一个键对应一个值。

下边的是个简单的用JSON语法描述的map的例子，它的所有值都是字符串类型：

{
  "zzzzz" : "woot",
  "xyz" : "hello",
  "aaaab" : "world",
  "1" : "x",
  "aaaaa" : "y"
}

持久化

持久化就是说你存进这个特殊的map中的数据“一直存在”，即使创建了或访问了它的程序都已经者结束了。这个概念与其他的持久化存储没什么区别，比如文件系统。接着讲。。。

分布式

HBase和BigTable建立于分布式文件系统之上，以致于底层的文件存储能够分布在一系列的独立的机器上。

HBase是基于Hadoop的分布式文件系统（HDFS）或者亚马逊的简单存储服务（S3）的上层应用，而BigTable则是基于Google文件系统（GFS）。

数据的副本被复制到多个机器节点上的方式和数据在磁盘冗余阵列（RAID）里面分布的方式有点相似。

对于本文的目的来说，我们并不关心具体使用的是哪种分布式文件系统。重要的是知道它是分布的，这样数据在比如集群内的某个节点挂掉时，有一定保障。

有序的

不同于大部分的map的实现，在HBase/BigTable里，键值对严格按字母表顺序排序，就是说键为”aaaaa”的数据行的下一个行键应该是”aaaab”，并且列键为”zzzzz”的数据行比较远。

继续我们的JSON例子，排序后是这样：

{
  "1" : "x",
  "aaaaa" : "y",
  "aaaab" : "world",
  "xyz" : "hello",
  "zzzzz" : "woot"
}

由于这些系统通常很大很分布，这个有序的特点实际上非常重要。相近的键对应的数据行存储在一起，这样当你需要扫描表的时候，你最感兴趣的记录是彼此临近的。

这一点对于选择键的规范是重要的。举个例子，如果一个表的键是域名，将域名反过来写是非常有必要的（用“com.jinbojw.www”而不是“www.jimbojw.com”)，这样子域名的数据行会比较临近父域名的数据行。

继续考虑域名这个例子，键为“mail.jimbojw.com”的数据行将会靠近“com.jimbojw.www”的数据行，而不是临近比如“mail.xyz.com”这种数据行。如果不将域名反过来，就会出现后者这种情况。

注意HBase/BigTable中的有序不是值有序而是键有序。除了键，没有自动索引任何其它东西，这一点就好象普通的map实现一样。

多维度

直到目前，我们还没有提到任何“列”的概念，在概念上我们把“table”当成普通的hash/map，这完全是有意的。“列”这个词像“表”和“库”一样是另一个沉重的单词，承担了多年的关系型数据库经验的感情包袱。

相反，我觉得将它看做一个多维度的map（或者嵌套的map，如果愿意的话）更容易。我们的JSON例子添加一个维度的话，就成了下边这个样子：

{
  "1" : {
    "A" : "x",
    "B" : "z"
  },
  "aaaaa" : {
    "A" : "y",
    "B" : "w"
  },
  "aaaab" : {
    "A" : "world",
    "B" : "ocean"
  },
  "xyz" : {
    "A" : "hello",
    "B" : "there"
  },
  "zzzzz" : {
    "A" : "woot",
    "B" : "1337"
  }
}

在上边的例子中，你会注意到，现在每个键都指向一个正好有两个键“A”“B”的map。从现在开始，我们把最顶层的键值对看做是“行”。而且，按照BigTable和HBase的命名规则，“A”和“B”映射表则被称为“列簇”。

一个表的列簇在表被创建的时候就被指定了。并且以后难以或者说不可能被修改。添加一个列簇也有可能非常昂贵，所以最好一开始就一次性指定好你需要的列簇。

幸运的是，一个列簇可能有任意个数量的列，用“column qualifier”或者“列标签”来标识。下边我们的JSON例子的一部分，这次加上了列标识：

{
  // ...
  "aaaaa" : {
    "A" : {
      "foo" : "y",
      "bar" : "d"
    },
    "B" : {
      "" : "w"
    }
  },
  "aaaab" : {
    "A" : {
      "foo" : "world",
      "bar" : "domination"
    },
    "B" : {
      "" : "ocean"
    }
  },
  // ...
}

注意这两行数据里，“A”列簇有两个列：“foo”和“bar”，“B”列簇仅有一列，并且其列标识为空字符串。

当查询HBase/BigTable的数据时，你必须提供全列名，格式为“:”，举个例子，上边的例子中共有三列：“A:foo”，“A:bar”，和“B:”。

注意，虽然列簇是不变的，但是列是可变的。考虑下面这行扩展了的数据：

{
  // ...
  "zzzzz" : {
    "A" : {
      "catch_phrase" : "woot",
    }
  }
}

这里，数据行“zzzzz”有且仅有一列，“A:catch_phrase”。因为每行可有任意多的不同的列，没有内置的方法来列出所有行中的所有列。要得到这个信息，你得做一个全表扫描。但是你可以查询所有的列簇，因为他们（基本上）是不可变的。

HBase/BigTable里最后的维度是时间。所有的数据都是有版本的，要么是整数的时间戳（Unix时间戳），或者用户选择的其他整数。客户端会在插入数据的时候指定时间戳。

考虑加上了用户指定的整数时间戳的例子：

{
  // ...
  "aaaaa" : {
    "A" : {
      "foo" : {
        15 : "y",
        4 : "m"
      },
      "bar" : {
        15 : "d",
      }
    },
    "B" : {
      "" : {
        6 : "w"
        3 : "o"
        1 : "w"
      }
    }
  },
  // ...
}

每个列簇可以自己决定一个数据单元保留多少个版本（一个rowkey/column唯一指定一个数据单元）。大多数的情况下，应用只会简单的请求给定的数据单元的数据，而不指定时间戳。那样，HBase/BigTable将会返回最新的版本（时间戳最大的版本），因为它是按时间倒序排序的。

如果应用程序要获取一个特定的行并指定时间戳，HBase将会返回时间戳小于或等于给定的时间戳的那个数据单元。

用我们假想的HBase表，查询行／列为“aaaaa”/“A:foo”的数据单元将会返回”y“，而查询行／列／时间戳为“aaaaa”/“A:foo”/10的数据单元将会返回“m”，当查询行／列／时间戳为“aaaaa”/“A:foo”/2 将会返回一个空的结果。

稀疏

最后的关键词是稀疏。像上边提到的，一个给定的数据行的每个列簇能有任意多的列，或者根本没有列。另一种类型的稀疏是基于数据行的空隙，仅指键值之间的空隙。

这只有用本文基于map的术语，而不是用关系型数据库的相关概念来理解HBase/BigTable，才会觉得非常合理。

结束语

希望对你理解HBase的数据模型有所帮助。