What aspect of Chinese characteristics contributed to its huge population throughout history?

Interesting question and answers

Answer by Andy Lee Chaisiri:

Chinese technology was 1,000+ years ahead of everyone else

Like this, but with horses and rice.

Imagine if today's crops suddenly became 30x more productive, that would cause a population boom, right? Agriculture is how human populations exploded in size compared to hunter-gatherer civilizations. So let's talk about some of those tools of agriculture and how population booms were achieved in an era of horse and plow:

Seed Drill: "What if we planted the seeds under the soil?"

Seed drills are tools that bury seeds at a correct depth in a timely manner. Planting seeds at a good depth increases the chances of an individual seed sprouting, without being eaten by birds. The use of seed drills also allows for planting in nice orderly rows with good spacing so the sprouting plants have enough room to draw nutrients from the soil without mutually starving each other. Not every grain will germinate, but using seed drills to plant crops in rows increases the chances of any individual grain germinating. This allows you to eat more grains because you know only a small quantity is needed to replant fields.

Chinese were using metal multi-tubed seed drills as early as 200BC. Seed drills make an appearance in Europe in 1566AD, about 1700 years after their appearance in China. As for how they were planting seeds before that…

Limbourg Brothers for the Duc de Berry (ca. 1415) 'Les Tres Riches Heures

You had a guy with a bag of seeds planting them by hand, then another guy rakes over the earth to cover them. That method leaves a lot of seeds exposed to be eaten by birds, or are planted too shallow to germinate. The crops that do germinate will be competing with other plants that are growing too close to it, and weeding the fields becomes very difficult, if not a waste of time. Out of the grains you wind up harvesting, a larger amount has to be partitioned for future planting, thus less are eaten.

Compared to this hand planting method, using a seed drill to plant crops in rows is 10x-30x more efficient in terms of how much grain you can harvest vs needing to save them for the next planting.

Iron Mouldboard Plough: "Metal cuts better than wood?"

Imagine a plough. You'd probably think of something made out of metal (perhaps with a wedge) right? Well, plows weren't always like that. The earliest ploughs in human history were basically a plank of wood that you knifed into the ground. Around 300BC, Chinese started using plows that were shaped in a way that they simultaneously cut into the earth and turned it too by 100CE, they were made entirely out of iron. Turning the earth is important for getting more nutrients out of your land, and can even turn 'barren' land fertile. 

Around 400AD, a similar mouldboard plow appears in the Roman empire, but widespread adoption is delayed with the fall of the empire. In 1700AD Dutch traders brought Chinese iron mouldboard plows back to Europe, and an agricultural revolution soon followed. Now, what was plowing like without an iron mouldboard plough?

A painting from the 16th century showing a farmer at work, by Pieter Brugel

That is a piece of wood being used to slice into the ground. Because that wooden plough doesn't have a mouldboard the cut soil needs to be tilled through further labor. Iron was expensive and labor intensive to produce, so at best you would have a thin sheet of iron covering the edge of your mostly wooden plough.

So, why did Chinese have all of these iron agricultural tools centuries earlier than Europeans? Because their methods of iron (& steel) production were also centuries ahead.

Blast Furnace: "Like baking a sponge cake made of iron"

The Iron Age is considered to have begun around 1700-1500BC. To extract iron from an ore of iron oxide, the iron has to be separated from oxygen and other impurities in a high temperature process which takes carbon to extract the oxygen out of the ore as carbon dioxide. This is called 'smelting'.

The earliest smelting of iron ore was done at temperatures below the melting point of iron. This left a spongy mass of iron that needed to be shaped by hammering, a very work intensive process.

But some time around 600BC, Chinese developed a furnace that could create a heat intense enough to melt iron, the blast furnace. Once liquified, iron could be poured into casts already in the shape of tools that were needed. The iron casting industry was officially supported by dynastic governments, leading to widespread adoption of iron tools made to a standard.

Now a special note about the difference between iron and steel. Cast iron is very high in carbon content, making it hard but brittle. Steel is iron that has a perfect balance of carbon to retain an edge but also maintain just enough flexibility to avoid brittleness. Around 200BC, Chinese learned that if air was blown over iron as it was being cast the carbon content could be reduced and what you wound up with was steel. Around 600AD steel tools began to widely replace iron ones.

The earliest evidence of blast furnaces in Europe is 1100AD, with widespread adoption occuring in 1400AD. The process of creating steel I described above first appears in the western world in 1855, and there's some contention that the 'inventor' may have actually gotten the idea from Chinese workers in the US.

As another illustration of the difference in iron production, by 1078AD the foundries of northern China could produce 114,000 tons of iron a year. In 1788AD, England produced about 50,000 tons of iron.

Horse Collar: "Over 1,000 years of choking horses"

Imagine a horse pulling a plough. Now, how did you imagine that plough being attached to the horse, with a horse collar, right? Unfortunately for horses, before the collar was invented there was the throat girth harness, which sounds as awful as it is. A plough (or any other load) attached by a throat-girth harness means that a horse is basically pulling with a noose around his trachea. Around 300BC, someone in China thought "What if the horse pulled with its chest instead of its throat?" and so the breast-strap harness was born and horses across China breathed a sigh of relief. This was improved on in 500AD with the horse collar as we know it.

The breast strap harness appears in Russia in 700AD, and shows up further west in Norway around 800AD. The horse collar appears a bit later in 900AD, with widespread adoption by 1200AD.


The difference between China and Europe's population levels throughout history is the difference between their agricultural technology. China had time saving, force multiplying tools (that didn't strangle horses) for centuries, even millenia before adoption in Europe.

What aspect of Chinese characteristics contributed to its huge population throughout history?

国庆

1999年国庆50年,我在广场跳集体舞。
2004年国庆55年,我在广场组织跳集体舞。
2009年国庆60年,我觉得整十年的纪念币很珍贵。
2014年国庆65年,我在加拿大看香港占中的新闻。
2019年我会在哪?

还要不要做大数据

我5月14日发了一条微博,后面的评论和私聊引起了我很多反思。这条微博是这样的:“我从0.16版本开始用Hadoop到现在已经5年了,一直相信大数据会是未来决胜的关键。但是,这个未来看来还有很远。或者说我们遇到的问题不同,是这些发明大数据的老外难以理解的,我们已经超前了很多。”

大数据这个话题近些年越来越热,我其实觉得它热过了头,所以我这里是想泼冷水的。

历史

估计会看到这个内容的大多数是我以前的同事,你们都知道我进这行不晚,2008年在人人网开始搭Hadoop,用0.16.3,那时候还没有什么人谈大数据,第一本Hadoop的书[1]也是2009年6月才出版的。那时候我们也没有概念要怎么用这个东西,唯一的目的就是改变“打点统计Log”模式,一开始就把生产服务搞死了三次。

那时候的服务叫ActiveLog,每一个PV记录一行,格式跟Apache Combined Log很类似,我们把WebServer的日志集中记录在统一的Server上(是的,比Facebook开源Scribe早半年[2])。为了存储空间的问题,引入了Hadoop,分布的存储在几百台服务器上。也就是这个结构,运行MapReduce占交换机带宽过大就会把生产集群挤死。

我记得最早的一个完整24小时Log文件的日期时2008年3月15日。那时候的日志是196GB/天。

当然后来大数据火了,我们有了更多的内部用户,也有了独立服务器甚至独立的机房,千兆直连核心交换机,到我2012年离职时集群已经有700台的规模了。

反思

饮水思源,这些年大数据概念红火带来了项目的红利,受这个影响我自己职业发展也不错。但是,掩盖不了一个一个具体问题的产生,应用范围一直是我最困扰的难题。

这让我回到源头去重新审视自己设计的系统和整个应用体系,然后我才发了最开始的那条微博。

数据量

多大的数据量敢叫大数据呢?Wikipedia里面有一句话:As of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.[3]

2008年人人网的Log数据一个月有6TB,将就着算half dozen吧,偶尔也要算整年的数据。

所以我只敢说我做过Hadoop,实在不敢说成大数据。现在在谈大数据的书和文章,有多少作者是处理过上PB数据的?国内PB级容量的集群又有几个呢?

几百G就用awk吧,几T其实也可以用数据库的。

应用范围

谈到大数据应用就涉及三件事:1) Distributed/Parallel computing. 2) Data mining 3) Business Intelligence

这三个是互相依赖的,直接的需求来自BI,间接的需求来自数据挖掘,实现在Computing上。可是现实的情况给我的切身感受用一句老话来比喻:粗放型经济向集约型经济转型。现在谈集约的下一步绿色经济,还为时尚早。

我们的互联网有几乎取之不尽的用户,打擦边球都能上市的公司,我们真的在乎数据吗?

炒作完大数据概念,真的应用到业务里,产生了利润吗?能挣回成本吗?

我知道国内大多数互联网公司的PM是不用数据做决策的,在谈大数据之前,应该从“小数据”开始。

 

这个切身体会我是到国外工作以后才有的,发那条微博前一周,我转了大概8%的现金到另一家银行开户,第二天,我的客户经理就要约我谈谈“投资需求”。要知道我去招行销金葵花可都没人问原因,销户一个月我的客户经理还打电话跟我说“因为我是金葵花客户,所以邀请办百夫长黑卡”,这是多么大的差距。

但是这还是“小数据”,这些事情还没办法做好,国内的大数据怎么做,做出来给谁看,谁又真的会看?

 

其实我觉得这个问题是无解的,市场决定了这个粗放的大环境,短期内是不会改变的。

现实能做的,不是去贩卖大数据的概念和技术,而是实实在在的让“小数据”先得到应用。
[1] http://wiki.apache.org/hadoop/Books
[2] http://en.wikipedia.org/wiki/Scribe_(log_server)
[3]  http://en.wikipedia.org/wiki/Big_data

Practices of using MySQL and DBPool

给公司写的文档草稿,稍后会写成中文的。

Summary

In large cluster environment, it is always challenge of manage hundreds MySQL databases.

  • Planed down time of database server maintenance
  • Scalability
  • Change of data structures

Ideally, these all handled by proper development process and you have enough software engineers to support.
But in real world, it is more urgent of solving problems.

Here is the 9 best practices to operate thousands of app servers and databases and we don’t have any “user impact” because of database maintenance.

Database design:

1 Global design

We design our database schema like a one-node-cluster, we can run all the service in one box or 1000 boxes.
It is transparent to software engineers.

2 Simple only

Use only basic MySQL features: table, primary key, index, replication.

Application design:

3 Use app server’s CPU.

App servers are scalable, but MySQL is a bottle neck, Use as much as application CPU, which means:
Do not query without index
Do not sort using database
Do not use any query causes temporary tables

4 Use a abstraction layer of tables

We use DBPool for a long time

Ops workflow:

5 Vertical partitioning

When you want to move a few tables to a different master.
a. Setup new master (B) as the slave of the old master (A)
b. Change the DBPool configuration, point master to B.
c. In the middle of the time, (B) will also receive some update requests from old client.
d. Make sure it is no any queries to tables on B
e. Stop the replication and optionally drop old tables on A.

6 Horizontal shading

When you want to distribute data of one table into more physic servers.
a. Estimate how many is needed, find a proper shading key. You should visit only one instance after shading.
b. A good shading number may be 10 or 100. It is human friendly when debug.
c. You don’t need to have 100 physic servers to deploy all tables, DBPool have the ability of route.
d. Use the same method to move tables to new master as showed in (5)

7 Change of data structures

a. We do add column only, no drop column.
b. Application level compatible is required. Make sure new code is working with both old and new data. (if impossible, see (8))
c. Make the changes
d. Update the application to use new column for new feature.

8 Data migration

This situation always involves a big change to the logic, you need to redesign the structure
a. Create a new master (B) of tables using (5)
b. Create new data structures on (B).
c. Create a trigger to update new structure when old data changed on (B).
d. Migrate your old data into the new structure, pay attention to (c) have already moved some recent data.
e. Create a new abstract instance in DBPool, for new structures.
f. Update the application use the new structure for reading.
g. In the same time, old client and new client have the same data for we have (c).
h. Update the application use the new structure for writing.
i. Stop the replication (a), trigger (c) and drop the old tables

9 Planned maintenance

a. DBPool is enough to move MySQL slave servers.
b. Use (5) to make a new master or promote one slave to master.

Free Website Hosting (PaaS)

自从AppFog的免费主机取消了,剩下免费账号也越来越慢。对比了各种云主机,OpenShift是目前最好的选择了。
Update 2015-Mar-06: Appfog的环境里有一个旧版的DBCP包,非常头疼。

OpenShift的免费账号功能亮点很多:

  • 支持3个512M节点
    • 可以建立2个App,1个MySQL;
    • 两个App可以AutoScaleUp
  • 可以ssh登录;
  • 负载均衡采用HAProxy,不计费;
  • 支持自定义域名;
  • 默认域名rhcloud.com支持SSL;
  • 填个信用卡注册了Bronze账号,控制好用量,也可以免费,但是可以支持自定义域名的SSL了。

试用了一下,性能比AppFog高不少,对个人网站来说,是奢侈配置。做开发的功能测试用,也很方便。

OpenSSL and cURL for iOS

昨天升级iOS程序,顺便升级了依赖到的两个库,OpenSSL和cURL。
升级了版本,增加了新的iPhone5s的64位CPU的支持。

I have updated the dependency libraries of my iPhone app, OpenSSL and cURL.
Added support of iPhone5s new 64bit arm64 CPU.
Upgraded to latest version.

Approach

交叉编译这两个库的关键是两个参数:-isysroot和-miphoneos-version-min
cURL是拼上不同的参数实现的,OpenSSL已经内建有iphoneos-cross支持,修改一些参数来支持arm64和模拟器。

The key of cross compiling are two parameters: -isysroot and -miphoneos-version-min
cURL is configured using parameters.
While OpenSSL has a build-in target called iphoneos-cross, I added 64 bit support based on it.

Code

这两个项目的代码提交在了GitHub,目前用iOS SDK7.1在MacOSX 10.8验证通过.
Here are two projects on GitHub, tested on iOS SDK 7.1 and MacOSX 10.8 .
https://github.com/sinofool/build-openssl-ios
https://github.com/sinofool/build-libcurl-ios

Usage

这两个脚本不需要git clone再使用,下载好OpenSSL和cURL的源代码并解压缩,直接运行github上的脚本,就会编译好放在桌面上。
It is not necessary clone the code locally, download the sources from OpenSSL and cURL official website.
Run following scripts, results will be on the desktop.

curl -O http://www.openssl.org/source/openssl-1.0.1f.tar.gz
tar xf openssl-1.0.1f.tar.gz
cd openssl-1.0.1f
curl https://raw.githubusercontent.com/sinofool/build-openssl-ios/master/build_openssl_dist.sh |bash

cURL也是一样:

curl -O http://curl.haxx.se/download/curl-7.35.0.tar.gz
tar xf curl-7.35.0.tar.gz
cd curl-7.35.0
curl https://raw.githubusercontent.com/sinofool/build-libcurl-ios/master/build_libcurl_dist.sh |bash

new Life(location)

有好久没有更新Blog了,在这一年忙着工作,忙着结婚,就在上个月到了加拿大开始全新的生活。

这里的工作文档写的很多,所以有更多可能把工作中可以公开的部分,写成文档放在这里分享。

最近正在修改simplecaptcha(http://simplecaptcha.sourceforge.net/)项目,增加和整理一些功能,修改后的代码发布在GitHub上叫ownCAPTCHA(https://github.com/sinofool/ownCAPTCHA)。
下一个版本完成的时候,会写一些例子在这里。

高可用-高性能-高并发

京ICP备06058813号  京ICP备13006450号