Category Archives: 技术

[Resolved] Codesign error over SSH on macOS 10.12+

Background

I have a shell script running in Jenkins to build my iOS projects. It runs over ssh on a Mac mini.

This stopped working since upgrading to macOS Sierra.

Causes

The root error of this problem is xcodebuild returns permission denied.

After step-by-step try on a hello world program, codesign returns the detailed error:

SecKey API returned: -25308, (null)____FILE_PATH____: unknown error -1=ffffffffffffffff

The explanation of this error is:

$ security error -25308
Error: 0xFFFF9D24 -25308 User interaction is not allowed.

This does not make any sense because the script created a new keychain and unlocked it. It needs no user interaction.

Solution

After three weeks contacting with Apple support, the solution is adding following command before using the newly created keychain:

security set-key-partition-list -S apple: -k <Password> -D <Identity> -t private

This works perfectly, but I cannot find any related document.

The script have been updated.

You-get 用命令行下载离线视频

这个工具很赞,命令行下载离线各大网站的视频和音乐。能自动合并分段的高清视频,并且把flv转换成mp4格式(都是用ffmpeg)。

家里电视连着RaspberryPi跑Kodi,看下载的视频比stream要方便。

上周想看英雄联盟总决赛,但是you-get的腾讯视频功能只能下载最差的质量那个版本。于是花了几个小时写了个patch,这周已经合并到主线了。现在从腾讯视频就可以下载到1080p和720p的版本了,优先选最高的。腾讯视频有限速,但是1MB/s也足够了,1小时的1080p只需要15分钟,限速还不是按IP,可以同时开多个。

网站:https://you-get.org/

源码:https://github.com/soimort/you-get

下一步有空写个浏览器插件,省去复制粘贴,直接post到下载机上就更方便了。

PsiApp: A simple beta app distribution tool

Recently, fir.im and pgyer.com both started to ask personal ID verification for their service.

I use them to install testing build on devices only, It is good to use HockeyApp oversea but clients from China usually have trouble downloading large files.

This result a small distribution tool created to host IPA and APK files easily and privately.

https://github.com/sinofool/PsiApp

  • It is PHP only
  • No database required
  • Local write access is optional
  • Support iOS/Android/UWP/Windows/macOS

 

Resetting the Push Notifications Permissions Alert on iOS

转自Apple官方文档,如何 重新显示 苹果 消息推送 的 权限提示。

Resetting the Push Notifications Permissions Alert on iOS

The first time a push-enabled app registers for push notifications, iOS asks the user if they wish to receive notifications for that app. Once the user has responded to this alert it is not presented again unless the device is restored or the app has been uninstalled for at least a day.

If you want to simulate a first-time run of your app, you can leave the app uninstalled for a day. You can achieve the latter without actually waiting a day by following these steps:

  1. Delete your app from the device.
  2. Turn the device off completely and turn it back on.
  3. Go to Settings > General > Date & Time and set the date ahead a day or more.
  4. Turn the device off completely again and turn it back on.

还要不要做大数据

我5月14日发了一条微博,后面的评论和私聊引起了我很多反思。这条微博是这样的:“我从0.16版本开始用Hadoop到现在已经5年了,一直相信大数据会是未来决胜的关键。但是,这个未来看来还有很远。或者说我们遇到的问题不同,是这些发明大数据的老外难以理解的,我们已经超前了很多。”

大数据这个话题近些年越来越热,我其实觉得它热过了头,所以我这里是想泼冷水的。

历史

估计会看到这个内容的大多数是我以前的同事,你们都知道我进这行不晚,2008年在人人网开始搭Hadoop,用0.16.3,那时候还没有什么人谈大数据,第一本Hadoop的书[1]也是2009年6月才出版的。那时候我们也没有概念要怎么用这个东西,唯一的目的就是改变“打点统计Log”模式,一开始就把生产服务搞死了三次。

那时候的服务叫ActiveLog,每一个PV记录一行,格式跟Apache Combined Log很类似,我们把WebServer的日志集中记录在统一的Server上(是的,比Facebook开源Scribe早半年[2])。为了存储空间的问题,引入了Hadoop,分布的存储在几百台服务器上。也就是这个结构,运行MapReduce占交换机带宽过大就会把生产集群挤死。

我记得最早的一个完整24小时Log文件的日期时2008年3月15日。那时候的日志是196GB/天。

当然后来大数据火了,我们有了更多的内部用户,也有了独立服务器甚至独立的机房,千兆直连核心交换机,到我2012年离职时集群已经有700台的规模了。

反思

饮水思源,这些年大数据概念红火带来了项目的红利,受这个影响我自己职业发展也不错。但是,掩盖不了一个一个具体问题的产生,应用范围一直是我最困扰的难题。

这让我回到源头去重新审视自己设计的系统和整个应用体系,然后我才发了最开始的那条微博。

数据量

多大的数据量敢叫大数据呢?Wikipedia里面有一句话:As of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.[3]

2008年人人网的Log数据一个月有6TB,将就着算half dozen吧,偶尔也要算整年的数据。

所以我只敢说我做过Hadoop,实在不敢说成大数据。现在在谈大数据的书和文章,有多少作者是处理过上PB数据的?国内PB级容量的集群又有几个呢?

几百G就用awk吧,几T其实也可以用数据库的。

应用范围

谈到大数据应用就涉及三件事:1) Distributed/Parallel computing. 2) Data mining 3) Business Intelligence

这三个是互相依赖的,直接的需求来自BI,间接的需求来自数据挖掘,实现在Computing上。可是现实的情况给我的切身感受用一句老话来比喻:粗放型经济向集约型经济转型。现在谈集约的下一步绿色经济,还为时尚早。

我们的互联网有几乎取之不尽的用户,打擦边球都能上市的公司,我们真的在乎数据吗?

炒作完大数据概念,真的应用到业务里,产生了利润吗?能挣回成本吗?

我知道国内大多数互联网公司的PM是不用数据做决策的,在谈大数据之前,应该从“小数据”开始。

 

这个切身体会我是到国外工作以后才有的,发那条微博前一周,我转了大概8%的现金到另一家银行开户,第二天,我的客户经理就要约我谈谈“投资需求”。要知道我去招行销金葵花可都没人问原因,销户一个月我的客户经理还打电话跟我说“因为我是金葵花客户,所以邀请办百夫长黑卡”,这是多么大的差距。

但是这还是“小数据”,这些事情还没办法做好,国内的大数据怎么做,做出来给谁看,谁又真的会看?

 

其实我觉得这个问题是无解的,市场决定了这个粗放的大环境,短期内是不会改变的。

现实能做的,不是去贩卖大数据的概念和技术,而是实实在在的让“小数据”先得到应用。
[1] http://wiki.apache.org/hadoop/Books
[2] http://en.wikipedia.org/wiki/Scribe_(log_server)
[3]  http://en.wikipedia.org/wiki/Big_data

Practices of using MySQL and DBPool

给公司写的文档草稿,稍后会写成中文的。

Summary

In large cluster environment, it is always challenge of manage hundreds MySQL databases.

  • Planed down time of database server maintenance
  • Scalability
  • Change of data structures

Ideally, these all handled by proper development process and you have enough software engineers to support.
But in real world, it is more urgent of solving problems.

Here is the 9 best practices to operate thousands of app servers and databases and we don’t have any “user impact” because of database maintenance.

Database design:

1 Global design

We design our database schema like a one-node-cluster, we can run all the service in one box or 1000 boxes.
It is transparent to software engineers.

2 Simple only

Use only basic MySQL features: table, primary key, index, replication.

Application design:

3 Use app server’s CPU.

App servers are scalable, but MySQL is a bottle neck, Use as much as application CPU, which means:
Do not query without index
Do not sort using database
Do not use any query causes temporary tables

4 Use a abstraction layer of tables

We use DBPool for a long time

Ops workflow:

5 Vertical partitioning

When you want to move a few tables to a different master.
a. Setup new master (B) as the slave of the old master (A)
b. Change the DBPool configuration, point master to B.
c. In the middle of the time, (B) will also receive some update requests from old client.
d. Make sure it is no any queries to tables on B
e. Stop the replication and optionally drop old tables on A.

6 Horizontal shading

When you want to distribute data of one table into more physic servers.
a. Estimate how many is needed, find a proper shading key. You should visit only one instance after shading.
b. A good shading number may be 10 or 100. It is human friendly when debug.
c. You don’t need to have 100 physic servers to deploy all tables, DBPool have the ability of route.
d. Use the same method to move tables to new master as showed in (5)

7 Change of data structures

a. We do add column only, no drop column.
b. Application level compatible is required. Make sure new code is working with both old and new data. (if impossible, see (8))
c. Make the changes
d. Update the application to use new column for new feature.

8 Data migration

This situation always involves a big change to the logic, you need to redesign the structure
a. Create a new master (B) of tables using (5)
b. Create new data structures on (B).
c. Create a trigger to update new structure when old data changed on (B).
d. Migrate your old data into the new structure, pay attention to (c) have already moved some recent data.
e. Create a new abstract instance in DBPool, for new structures.
f. Update the application use the new structure for reading.
g. In the same time, old client and new client have the same data for we have (c).
h. Update the application use the new structure for writing.
i. Stop the replication (a), trigger (c) and drop the old tables

9 Planned maintenance

a. DBPool is enough to move MySQL slave servers.
b. Use (5) to make a new master or promote one slave to master.