elaticsearch 优化

Elasticsearch Optimization Checklist

原文链接

假设

  • hardware 假设
  • index/query rate假设
  • elasticsearch用户运行elasticsearch

优化目标

  • Indexing
  • Searching

Architecture Level

to be continued.

Hardware Level

Elasticsearch Hardware Recommendation

System Level

  • adjust vm.swappiness 1
1
2
3
4
5
6
7
# 这是永久修改
$ echo "vm.swappiness = 1" >> /etc/sysctl.conf

# 这是临时修改,服务器重启后失效
$ sysctl vm.swappiness=1
$ sudo swapoff -a
$ sudo swapon -a

A swappiness of 1 is better than 0, since on some kernel versions a swappiness of 0 can invoke the OOM-killer.

  • Max Open File Descriptors 设置为32k~64k

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # max open file descriptors
    $ cp /etc/security/limits.conf /etc/security/limits.conf.bak

    $ cat /etc/security/limits.conf | grep -v "elasticsearch" > /tmp/system_limits.conf

    $ echo "elasticsearch hard nofile 50000" >> /tmp/system_limits.conf

    $ echo "elasticsearch soft nofile 50000" >> /tmp/system_limits.conf

    $ mv /tmp/system_limits.conf /etc/security/limits.conf
  • configure the maximum map count

    set it permanently by modifying vm.max_map_count setting in your /etc/sysctl.conf.2

1
2
3
4
5
6
7
8
# virtual Memory
$ cp /etc/sysctl.conf /etc/sysctl.conf.bak

$ cat /etc/sysctl.conf | grep -v "vm.max_map_count" > /tmp/system_sysctl.conf

$ echo "vm.max_map_count=262144" >> /tmp/system_sysctl.conf

$ mv /tmp/system_sysctl.conf /etc/sysctl.conf

或者临时修改?[7]

1
sysctl -w vm.max_map_count=262144

查看结果:

1
2
$ sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144

这里有一个使用salt批量做system level调优的脚本,可以直接使用

Application Level

查看结果:

1
2
3
4
$ java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

  • ES_HEAP_SIZE=Xg

    • Ensure that the min (Xms) and max (Xmx) sizes are the same to prevent the heap from resizing at runtime, a very costly process.

    • Give Half Your Memory to Lucene

    • Don’t Cross 32 GB!

  • enable mlockall (elasticsearch.yml)12

Try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out.

Increase RLIMIT_MEMLOCK to prevent failing to lock memory, these can be adjusted by modifying /etc/security/limits.conf, for example:

1
2
3
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

then in elasticsearch.yml:

1
bootstrap.mlockall: true
  • discovery (elasticsearch.yml)
1
2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: master_node_list
  • recovery strategy (elasticsearch.yml)
1
2
3
4
# if you have 10 nodes
gateway.recover_after_nodes: 8
gateway.expected_nodes: 10
gateway.recover_after_time: 10m

ES includes several recovery properties which improve both ElasticSearch cluster recovery and restart times. We have shown some sample values below. The value that will work best for you depends on the hardware you have in use, and the best advice we can give is to test, test, and test again.

1
cluster.routing.allocation.node_concurrent_recoveries:4

This property is how many shards per node are allowed for recovery at any moment in time. Recovering shards is a very IO-intensive operation, so you should set this value with real caution.

1
cluster.routing.allocation.node_initial_primaries_recoveries:18

This controls the number of primary shards initialized concurrently on a single node. The number of parallel stream of data transfer from node to recover shard from peer node is controlled by indices.recovery.concurrent_streams. The value below is setup for the Amazon instance, but if you have your own hardware you might be able to set this value much higher. The property max_bytes_per_sec (as its name suggests) determines how many bytes to transfer per second. This value again need to be configured according to your hardware.

1
2
indices.recovery.concurrent_streams: 4
indices.recovery.max_bytes_per_sec: 40mb

All of the properties described above get used only when the cluster is restarted.[5]

  • Threadpool Properties Prevent Data Loss[5]
    ElasticSearch node has several thread pools in order to improve how threads are managed within a node. At Loggly, we use bulk request extensively, and we have found that setting the right value for bulk thread pool using threadpool.bulk.queue_size property is crucial in order to avoid data loss or _bulk retries
    1
    threadpool.bulk.queue_size: 3000

This property value is for the bulk request. This tells ES the number of requests that can be queued for execution in the node when there is no thread available to execute a bulk request. This value should be set according to your bulk request load. If your bulk request number goes higher than queue size, you will get a RemoteTransportException as shown below.

Note that in ES the bulk requests queue contains one item per shard, so this number needs to be higher than the number of concurrent bulk requests you want to send if those request contain data for many shards. For example, a single bulk request may contain data for 10 shards, so even if you only send one bulk request, you must have a queue size of at least 10. Setting this value “too high” will chew up heap in your JVM, but does let you hand off queuing to ES, which simplifies your clients.

You either need to keep the property value higher than your accepted load or gracefully handle RemoteTransportException in your client code. If you don’t handle the exception, you will end up losing data. We simulated the exception shown below by sending more than 10 bulk requests with a queue size of 10.

1
RemoteTransportException[[<Bantam>][inet[/192.168.76.1:9300]][bulk/shard]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 10) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@13fe9be];

  • Watch Out for delete_all_indices! [5]
    It’s really important to know that the curl API in ES does not have very good authentication built into it. A simple curl API can cause all the indices to delete themselves and lose all data. This is just one example of a command that could cause a mistaken deletion:
    1
    curl -XDELETE ‘http://localhost:9200/*/’

To avoid this type of grief, you can set the following property:

1
action.disable_delete_all_indices: true

This will make sure when above command is given, it will not delete the index and will instead result in an error.

  • cluster settings 优化
    PUT _cluster/settings
    1
    2
    3
    4
    5
    6
    7
    8
    9
    put /_cluster/settings
    {
    "persistent" : {
    "indices.store.throttle.max_bytes_per_sec":"20mb",
    "indices.breaker.fielddata.limit":"60%",
    "indices.breaker.request.limit":"40%",
    "indices.breaker.total.limit":"70%"
    }
    }

上面的都是默认值。如果日志中常出现[your_index_name]... now throttling indexing: numMergesInFlight=6, maxNumMerges=5并且磁盘IO不高 "indices.store.throttle.max_bytes_per_sec"可以更大;如果日志中经常出现java.lang.OutOfMemoryError, 可以减小”indices.breaker.fielddata.limit”,“indices.breaker.request.limit”,“indices.breaker.total.limit”`的值。

如果fielddata需要占用的JVM heap size超过了限定值,请求会被中断(Q:请求中断后,什么返回?),如下日志所示:

1
2
3
4
5
6
[2015-05-27 20:16:44,767][WARN ][indices.breaker          ] [10.19.0.84] [FIELDDATA] New used memory 4848575200 [4.5gb] from field [http_request.raw] would be larger t
han configured breaker: 4831838208 [4.5gb], breaking
[2015-05-27 20:16:44,911][WARN ][indices.breaker ] [10.19.0.84] [FIELDDATA] New used memory 4833426184 [4.5gb] from field [host.raw] would be larger than conf
igured breaker: 4831838208 [4.5gb], breaking
[2015-05-27 20:16:44,914][WARN ][indices.breaker ] [10.19.0.84] [FIELDDATA] New used memory 4833425505 [4.5gb] from field [domain.raw] would be larger than co
nfigured breaker: 4831838208 [4.5gb], breaking

TIP: In Fielddata Size, we spoke about adding a limit to the size of fielddata, to ensure that old unused fielddata can be evicted. The relationship between indices.fielddata.cache.size and indices.breaker.fielddata.limit is an important one. If the circuit-breaker limit is lower than the cache size, no data will ever be evicted. In order for it to work properly, the circuit breaker limit must be higher than the cache size.

  • disk based allocation strategy[6]
    1
    2
    3
    4
    5
    6
    7
    PUT /_cluster/settings -d '{
    "transient" : {
    "cluster.routing.allocation.disk.threshold_enabled" : true,
    "cluster.routing.allocation.disk.watermark.low" : "85%",
    "cluster.routing.allocation.disk.watermark.high" : "90%"
    }
    }'

cluster.routing.allocation.disk.watermark.low controls the low watermark for disk usage. It defaults to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent ES from allocating shards if less than the configured amount of space is available.

cluster.routing.allocation.disk.watermark.high controls the high watermark. It defaults to 90%, meaning ES will attempt to relocate shards to another node if the node disk usage rises above 90%. It can also be set to an absolute byte value (similar to the low watermark) to relocate shards once less than the configured amount of space is available on the node.

  • index template 优化

利用好不同template之间的order关系
默认所有field都是not_analyzed
默认numeric, date, string(not_analyzed), geo_point类型的field都使用doc_values 形式的fielddata
Doc values can be enabled for numeric, date, Boolean, binary, and geo-point fields, and for not_analyzed string fields. They do not currently work with analyzed string fields. Doc values are enabled per field in the field mapping, which means that you can combine in-memory fielddata with doc values.

  • 安装监控工具
1
elastic marvel

验证方法

GET /_nodes/process可以看到

1
2
"max_file_descriptors": 64000, 
"mlockall": true

Index Level

to be continued.


Appendix A Base Template

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
PUT _template/base
{
"order": 0,
"template": "*",
"settings": {
"index.refresh_interval": "120s",
"index.number_of_replicas": "1",
"index.number_of_shards": "10",
"index.routing.allocation.total_shards_per_node": "2",
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.indexing.slowlog.threshold.index.warn": "10s",
"index.indexing.slowlog.threshold.index.info": "5s"
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"integer field": {
"mapping": {
"doc_values": true,
"type": "integer"
},
"match": "*",
"match_mapping_type": "integer"
}
},
{
"date field": {
"mapping": {
"doc_values": true,
"type": "date"
},
"match": "*",
"match_mapping_type": "date"
}
},
{
"long field": {
"mapping": {
"doc_values": true,
"type": "long"
},
"match": "*",
"match_mapping_type": "long"
}
},
{
"float field": {
"mapping": {
"doc_values": true,
"type": "float"
},
"match": "*",
"match_mapping_type": "float"
}
},
{
"double field": {
"mapping": {
"doc_values": true,
"type": "double"
},
"match": "*",
"match_mapping_type": "double"
}
},
{
"byte field": {
"mapping": {
"doc_values": true,
"type": "byte"
},
"match": "*",
"match_mapping_type": "byte"
}
},
{
"short field": {
"mapping": {
"doc_values": true,
"type": "short"
},
"match": "*",
"match_mapping_type": "short"
}
},
{
"binary field": {
"mapping": {
"doc_values": true,
"type": "binary"
},
"match": "*",
"match_mapping_type": "binary"
}
},
{
"geo_point field": {
"mapping": {
"doc_values": true,
"type": "geo_point"
},
"match": "*",
"match_mapping_type": "geo_point"
}
},
{
"string fields": {
"mapping": {
"index": "not_analyzed",
"omit_norms": true,
"doc_values": true,
"type": "string"
},
"match": "*",
"match_mapping_type": "string"
}
}
],
"_all": {
"enabled": false
}
}
}
}

References

  1. Elasticsearch Indexing Under the Hood
    http://www.slideshare.net/GauravKukal/elastic-search-indexing-internals

  2. lucene深入分析博客
    http://www.cnblogs.com/forfuture1978/

  3. Elasticsearch Refresh Interval vs Indexing Performance
    https://sematext.com/blog/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/

  4. Elasticsearch Translog
    https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html

  5. Tuning data ingestion performance for Elasticsearch on Azure
    https://azure.microsoft.com/en-us/documentation/articles/guidance-elasticsearch-tuning-data-ingestion-performance/

  6. Tuning data aggregation and query performance with Elasticsearch on Azure
    https://azure.microsoft.com/en-us/documentation/articles/guidance-elasticsearch-tuning-data-aggregation-and-query-performance/

  7. ElasticSearch Performance Tips
    http://shzhangji.com/blog/2015/04/28/elasticsearch-performance-tips/

  8. Heap: Sizing and Swapping
    https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

  9. Faster bulk indexing in Elasticsearch
    http://www.flax.co.uk/blog/2015/09/28/faster-bulk-indexing-in-elasticsearch/

  10. Performance Considerations for Elasticsearch Indexing
    https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing

  11. Indexing Performance Tips
    https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html

  12. Announcing Rally: Our benchmarking tool for Elasticsearch
    https://www.elastic.co/blog/announcing-rally-benchmarking-for-elasticsearch

  13. Elasticsearch Indexing Performance Cheatsheet
    https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/

  14. Optimizing Elasticsearch: How Many Shards per Index?
    https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index

  15. Performance Considerations for Elasticsearch 2.0 Indexing
    https://www.elastic.co/blog/performance-indexing-2-0

  16. Write Consistency
    https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#bulk-consistency



Written with StackEdit.

ngx_lua常用变量

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
ngx.arg[index]              #ngx指令参数,当这个变量在set_by_lua或者set_by_lua_file内使用的时候是只读的,指的是在配置指令输入的参数.  
ngx.var.varname #读写NGINX变量的值,最好在lua脚本里缓存变量值,避免在当前请求的生命周期内内存的泄漏
ngx.config.ngx_lua_version #当前ngx_lua模块版本号
ngx.config.nginx_version #nginx版本
ngx.worker.exiting #当前worker进程是否正在关闭
ngx.worker.pid #当前worker进程的PID
ngx.config.nginx_configure #编译时的./configure命令选项
ngx.config.prefix #编译时的prefix选项

core constans: #ngx_lua 核心常量
ngx.OK (0)
ngx.ERROR (-1)
ngx.AGAIN (-2)
ngx.DONE (-4)
ngx.DECLINED (-5)
ngx.nil
http method constans: #经常在ngx.location.catpure和ngx.location.capture_multi方法中被调用.
ngx.HTTP_GET
ngx.HTTP_HEAD
ngx.HTTP_PUT
ngx.HTTP_POST
ngx.HTTP_DELETE
ngx.HTTP_OPTIONS
ngx.HTTP_MKCOL
ngx.HTTP_COPY
ngx.HTTP_MOVE
ngx.HTTP_PROPFIND
ngx.HTTP_PROPPATCH
ngx.HTTP_LOCK
ngx.HTTP_UNLOCK
ngx.HTTP_PATCH
ngx.HTTP_TRACE
http status constans: #http请求状态常量
ngx.HTTP_OK (200)
ngx.HTTP_CREATED (201)
ngx.HTTP_SPECIAL_RESPONSE (300)
ngx.HTTP_MOVED_PERMANENTLY (301)
ngx.HTTP_MOVED_TEMPORARILY (302)
ngx.HTTP_SEE_OTHER (303)
ngx.HTTP_NOT_MODIFIED (304)
ngx.HTTP_BAD_REQUEST (400)
ngx.HTTP_UNAUTHORIZED (401)
ngx.HTTP_FORBIDDEN (403)
ngx.HTTP_NOT_FOUND (404)
ngx.HTTP_NOT_ALLOWED (405)
ngx.HTTP_GONE (410)
ngx.HTTP_INTERNAL_SERVER_ERROR (500)
ngx.HTTP_METHOD_NOT_IMPLEMENTED (501)
ngx.HTTP_SERVICE_UNAVAILABLE (503)
ngx.HTTP_GATEWAY_TIMEOUT (504)

Nginx log level constants: #错误日志级别常量 ,这些参数经常在ngx.log方法中被使用.
ngx.STDERR
ngx.EMERG
ngx.ALERT
ngx.CRIT
ngx.ERR
ngx.WARN
ngx.NOTICE
ngx.INFO
ngx.DEBUG

##################
#API中的方法:
##################
print() #与 ngx.print()方法有区别,print() 相当于ngx.log()
ngx.ctx #这是一个lua的table,用于保存ngx上下文的变量,在整个请求的生命周期内都有效,详细参考官方
ngx.location.capture() #发出一个子请求,详细用法参考官方文档。
ngx.location.capture_multi() #发出多个子请求,详细用法参考官方文档。
ngx.status #读或者写当前请求的相应状态. 必须在输出相应头之前被调用.
ngx.header.HEADER #访问或设置http header头信息,详细参考官方文档。
ngx.req.set_uri() #设置当前请求的URI,详细参考官方文档
ngx.set_uri_args(args) #根据args参数重新定义当前请求的URI参数.
ngx.req.get_uri_args() #返回一个LUA TABLE,包含当前请求的全部的URL参数
ngx.req.get_post_args() #返回一个LUA TABLE,包括所有当前请求的POST参数
ngx.req.get_headers() #返回一个包含当前请求头信息的lua table.
ngx.req.set_header() #设置当前请求头header某字段值.当前请求的子请求不会受到影响.
ngx.req.read_body() #在不阻塞ngnix其他事件的情况下同步读取客户端的body信息.[详细]
ngx.req.discard_body() #明确丢弃客户端请求的body
ngx.req.get_body_data() #以字符串的形式获得客户端的请求body内容
ngx.req.get_body_file() #当发送文件请求的时候,获得文件的名字
ngx.req.set_body_data() #设置客户端请求的BODY
ngx.req.set_body_file() #通过filename来指定当前请求的file data。
ngx.req.clear_header() #清求某个请求头
ngx.exec(uri,args) #执行内部跳转,根据uri和请求参数
ngx.redirect(uri, status) #执行301或者302的重定向。
ngx.send_headers() #发送指定的响应头
ngx.headers_sent #判断头部是否发送给客户端ngx.headers_sent=true
ngx.print(str) #发送给客户端的响应页面
ngx.say() #作用类似ngx.print,不过say方法输出后会换行
ngx.log(log.level,...) #写入nginx日志
ngx.flush() #将缓冲区内容输出到页面(刷新响应)
ngx.exit(http-status) #结束请求并输出状态码
ngx.eof() #明确指定关闭结束输出流
ngx.escape_uri() #URI编码(本函数对逗号,不编码,而php的urlencode会编码)
ngx.unescape_uri() #uri解码
ngx.encode_args(table) #将tabel解析成url参数
ngx.decode_args(uri) #将参数字符串编码为一个table
ngx.encode_base64(str) #BASE64编码
ngx.decode_base64(str) #BASE64解码
ngx.crc32_short(str) #字符串的crs32_short哈希
ngx.crc32_long(str) #字符串的crs32_long哈希
ngx.hmac_sha1(str) #字符串的hmac_sha1哈希
ngx.md5(str) #返回16进制MD5
ngx.md5_bin(str) #返回2进制MD5
ngx.today() #返回当前日期yyyy-mm-dd
ngx.time() #返回当前时间戳
ngx.now() #返回当前时间
ngx.update_time() #刷新后返回
ngx.localtime() #返回 yyyy-mm-dd hh:ii:ss
ngx.utctime() #返回yyyy-mm-dd hh:ii:ss格式的utc时间
ngx.cookie_time(sec) #返回用于COOKIE使用的时间
ngx.http_time(sec) #返回可用于http header使用的时间
ngx.parse_http_time(str) #解析HTTP头的时间
ngx.is_subrequest #是否子请求(值为 true or false)
ngx.re.match(subject,regex,options,ctx) #ngx正则表达式匹配,详细参考官网
ngx.re.gmatch(subject,regex,opt) #全局正则匹配
ngx.re.sub(sub,reg,opt) #匹配和替换(未知)
ngx.re.gsub() #未知
ngx.shared.DICT #ngx.shared.DICT是一个table 里面存储了所有的全局内存共享变量
ngx.shared.DICT.get
ngx.shared.DICT.get_stale
ngx.shared.DICT.set
ngx.shared.DICT.safe_set
ngx.shared.DICT.add
ngx.shared.DICT.safe_add
ngx.shared.DICT.replace
ngx.shared.DICT.delete
ngx.shared.DICT.incr
ngx.shared.DICT.flush_all
ngx.shared.DICT.flush_expired
ngx.shared.DICT.get_keys
ndk.set_var.DIRECTIVE #不懂

nginx里request_time和upstream_response_time差别

在根据nginx的accesslog中$request_time进行程序优化时,发现有个接口,直接返回数据,平均的$request_time也比较大。原来$request_time包含了用户数据接收时间,而真正程序的响应时间应该用$upstream_response_time

下面介绍下2者的差别:

1、request_time
官网描述:request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client 。
指的就是从接受用户请求的第一个字节到发送完响应数据的时间,即包括接收请求数据时间、程序响应时间、输出
响应数据时间。

2、upstream_response_time
官网描述:keeps times of responses obtained from upstream servers; times are kept in seconds with a milliseconds resolution. Several response times are separated by commas and colons like addresses in the `$upstream_addr variable

是指从Nginx向后端(php-cgi)建立连接开始到接受完数据然后关闭连接为止的时间。

从上面的描述可以看出,$request_time肯定比$upstream_response_time值大,特别是使用POST方式传递参数时,因为Nginx会把request body缓存住,接受完毕后才会把数据一起发给后端。所以如果用户网络较差,或者传递数据较大时,$request_time会比$upstream_response_time大很多。

所以如果使用nginx的accesslog查看php程序中哪些接口比较慢的话,记得在log_format中加入$upstream_response_time

uwsgi+nginx简洁配置

网上找了一圈的nginx+uwsgi的配置,各种尝试,各种失败。千呼万唤始出来,终于可以正常协同工作了。在此记录下。

uwsgi一定要用pip安装版本的,路径为/usr/bin/uwsgi,使用yum安装版本的可能要进坑

以下是uwsgi的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
## uwsgi.ini ###
[uwsgi]
# the base directory (full path)
chdir = /usr/local/src/alert
module = run ## 启动flask应用的启动文件
enable-threads = true
callable = app ## 启动文件的应用
# master
master = true
# maximum number of worker processes
processes = 4
# the socket (use the full path to be safe
socket = /var/run/api.sock
#socket = 127.0.0.1:2222
# with appropriate permissions
chmod-socket = 664 ##socket文件的权限
# clear environment on exit
vacuum = true
uid = daemon
gid = daemon
# Run in the background as a daemon
daemonize = /var/log/uwsgi/api.log

python应用的启动文件(我这里是一个flask应用)

1
2
3
from alert import app
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0',port=2222)

启动姿势

1
2
3
4
5
6
7
8
9
10

uwsgi的命令行启动:

/usr/bin/uwsgi --socket 0.0.0.0:2222 --protocol=http --module run --callable app —threads 4

daemon启动

/usr/bin/uwsgi —ini uwsgi.ini

nginx的配置

nginx的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
upstream uwsgi {
server unix:///var/run/api.sock;
}

server {
listen 5000;
charset utf-8;
access_log /data/logs/weblog/api_access.log;
error_log /data/logs/weblog/api_error.log;

location / {
uwsgi_pass uwsgi; # 指向uwsgi 所应用的内部地址,所有请求将转发给uwsgi 处理
include uwsgi_params; # the uwsgi_params file you installed
}
}

nginx 限制url的IP访问

针对某些IP开放特苏的URL访问,比如说网站的管理后台。在nginx里我们可以使用geomap的使用,通过变量的判断来实现。当然,最便捷的方式就是通过多域名的方式。

geomap使用的变量是全局有效的,在各server段是共享的

1
2
3
4
5
6
7
8
9
10
11
geo $remote_addr $denied1 {
default 1;
124.127.138.32/27 0;
124.243.227.224/32 0;
106.75.33.149/32 0;
}

map $request_uri $denied2 {
default 0;
~^/admin.php 1;
}

server段里添加判断指令

1
2
3
4
5
6
7
8
9
10
11

if ($denied1) {
set $denied o;
}
if ($denied2) {
set $denied "${denied}o";
}

if ($denied = 'oo') {
return 403;
}

如果后端解析是PHP,也可以通过location指令,不过在location里必须添加完整PHP解析,不然会报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

location ~* ^/admin.php {
allow 1.1.1.1;
allow 12.12.12.0/24;
deny all;
location ~ .*\.php {
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_index index.php;
fastcgi_split_path_info ^(.+\.php)(.*)$;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param PATH_TRANSLATED $document_root$fastcgi_path_info;
include fastcgi_params;
}
}

lua实现验证码验证

配合API提供的验证,实现认证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
location = /code {
proxy_read_timeout 1500;
internal; proxy_pass http://api.renzheng.com:8333/identifycode?code=$arg_code&name=$arg_name;
}

location / {
access_by_lua '
local json = require("cjson")
local whiteiplist = {"1.1.1.1"} --加白
local headers = ngx.req.get_headers()
local request_uri = ngx.var.request_uri
ngx.header["Access-Control-Allow-Headers"]="X-Auth-Token,X-Authorization,Content-type"
if ngx.req.get_method() == "POST" then
if request_uri == "/v2.0/tokens" then
if headers["X-Auth-Token"] == nil then
local code = headers["X-Authorization"]
ngx.req.read_body()
local raw_json = ngx.req.get_body_data()
local success, body = pcall(json.decode,raw_json)
if not success then
return
end
if body.auth ~= nil then
if body.auth.passwordCredentials ~= nil then
if body.auth.passwordCredentials.username ~= nil then
local ip = ngx.req.get_headers()["X-Real-IP"]
if ip == nil then
ip = ngx.var.remote_addr
end
if ip ~= nil then
for _,client in pairs(whiteiplist) do
if client == ip then
return
end
end
end
local name = body.auth.passwordCredentials.username
local res = ngx.location.capture("/code",{args={code=code,name=name}})
if string.len(res.body) == 0 then
ngx.exit(405)
end
end
end
end
end
end
end
ngx.req.clear_header("X-Authorization")
';
try_files $uri @backend;
}

分分钟python入门

Python 是 90 年代初由 Guido Van Rossum 创立的。它是当前最流行的程序语言之一。它那纯净的语法令我一见倾心,它简直就是可以运行的伪码。

请注意:本文以 Python 2.7 为基准,但也应该适用于所有 2.X 版本。还要继续学习最新的 Python 3 哦!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
# Single line comments start with a hash.
# 单行注释由一个井号开头。
""" Multiline strings can be written
using three "'s, and are often used
as comments
三个双引号(或单引号)之间可以写多行字符串,
通常用来写注释。
"""



####################################################
## 1. Primitive Datatypes and Operators
## 1. 基本数据类型和操作符
####################################################


# You have numbers
# 数字就是数字
3 #=> 3


# Math is what you would expect
# 四则运算也是你所期望的那样
1 + 1 #=> 2
8 - 1 #=> 7
10 * 2 #=> 20
35 / 5 #=> 7


# Division is a bit tricky. It is integer division and floors the results
# automatically.
# 除法有一点棘手。
# 对于整数除法来说,计算结果会自动取整。
5 / 2 #=> 2


# To fix division we need to learn about floats.
# 为了修正除法的问题,我们需要先学习浮点数。
2.0 # This is a float
2.0 # 这是一个浮点数
11.0 / 4.0 #=> 2.75 ahhh...much better
11.0 / 4.0 #=> 2.75 啊……这样就好多了


# Enforce precedence with parentheses
# 使用小括号来强制计算的优先顺序
(1 + 3) * 2 #=> 8


# Boolean values are primitives
# 布尔值也是基本数据类型
True
False


# negate with not
# 使用 not 来取反
not True #=> False
not False #=> True


# Equality is ==
# 等式判断用 ==
1 == 1 #=> True
2 == 1 #=> False


# Inequality is !=
# 不等式判断是用 !=
1 != 1 #=> False
2 != 1 #=> True


# More comparisons
# 还有更多的比较运算
1 < 10 #=> True
1 > 10 #=> False
2 <= 2 #=> True
2 >= 2 #=> True


# Comparisons can be chained!
# 居然可以把比较运算串连起来!
1 < 2 < 3 #=> True
2 < 3 < 2 #=> False


# Strings are created with " or '
# 使用 " 或 ' 来创建字符串
"This is a string."
'This is also a string.'


# Strings can be added too!
# 字符串也可以相加!
"Hello " + "world!" #=> "Hello world!"


# A string can be treated like a list of characters
# 一个字符串可以视为一个字符的列表
# (译注:后面会讲到“列表”。)
"This is a string"[0] #=> 'T'


# % can be used to format strings, like this:
# % 可以用来格式化字符串,就像这样:
"%s can be %s" % ("strings", "interpolated")


# A newer way to format strings is the format method.
# This method is the preferred way
# 后来又有一种格式化字符串的新方法:format 方法。
# 我们推荐使用这个方法。
"{0} can be {1}".format("strings", "formatted")


# You can use keywords if you don't want to count.
# 如果你不喜欢数数的话,可以使用关键字(变量)。
"{name} wants to eat {food}".format(name="Bob", food="lasagna")


# None is an object
# None 是一个对象
None #=> None


# Don't use the equality `==` symbol to compare objects to None
# Use `is` instead
# 不要使用相等符号 `==` 来把对象和 None 进行比较,
# 而要用 `is`。
"etc" is None #=> False
None is None #=> True


# The 'is' operator tests for object identity. This isn't
# very useful when dealing with primitive values, but is
# very useful when dealing with objects.
# 这个 `is` 操作符用于比较两个对象的标识。
# (译注:对象一旦建立,其标识就不会改变,可以认为它就是对象的内存地址。)
# 在处理基本数据类型时基本用不上,
# 但它在处理对象时很有用。


# None, 0, and empty strings/lists all evaluate to False.
# All other values are True
# None、0 以及空字符串和空列表都等于 False,
# 除此以外的所有值都等于 True。
0 == False #=> True
"" == False #=> True




####################################################
## 2. Variables and Collections
## 2. 变量和集合
####################################################


# Printing is pretty easy
# 打印输出很简单
print "I'm Python. Nice to meet you!"




# No need to declare variables before assigning to them.
# 在赋值给变量之前不需要声明
some_var = 5 # Convention is to use lower_case_with_underscores
# 变量名的约定是使用下划线分隔的小写单词
some_var #=> 5


# Accessing a previously unassigned variable is an exception.
# See Control Flow to learn more about exception handling.
# 访问一个未赋值的变量会产生一个异常。
# 进一步了解异常处理,可参见下一节《控制流》。
some_other_var # Raises a name error
# 会抛出一个名称错误


# if can be used as an expression
# if 可以作为表达式来使用
"yahoo!" if 3 > 2 else 2 #=> "yahoo!"


# Lists store sequences
# 列表用于存储序列
li = []
# You can start with a prefilled list
# 我们先尝试一个预先填充好的列表
other_li = [4, 5, 6]


# Add stuff to the end of a list with append
# 使用 append 方法把元素添加到列表的尾部
li.append(1) #li is now [1]
#li 现在是 [1]
li.append(2) #li is now [1, 2]
#li 现在是 [1, 2]
li.append(4) #li is now [1, 2, 4]
#li 现在是 [1, 2, 4]
li.append(3) #li is now [1, 2, 4, 3]
#li 现在是 [1, 2, 4, 3]
# Remove from the end with pop
# 使用 pop 来移除最后一个元素
li.pop() #=> 3 and li is now [1, 2, 4]
#=> 3,然后 li 现在是 [1, 2, 4]
# Let's put it back
# 我们再把它放回去
li.append(3) # li is now [1, 2, 4, 3] again.
# li 现在又是 [1, 2, 4, 3] 了


# Access a list like you would any array
# 像访问其它语言的数组那样访问列表
li[0] #=> 1
# Look at the last element
# 查询最后一个元素
li[-1] #=> 3


# Looking out of bounds is an IndexError
# 越界查询会产生一个索引错误
li[4] # Raises an IndexError
# 抛出一个索引错误


# You can look at ranges with slice syntax.
# (It's a closed/open range for you mathy types.)
# 你可以使用切片语法来查询列表的一个范围。
# (这个范围相当于数学中的左闭右开区间。)
li[1:3] #=> [2, 4]
# Omit the beginning
# 省略开头
li[2:] #=> [4, 3]
# Omit the end
# 省略结尾
li[:3] #=> [1, 2, 4]


# Remove arbitrary elements from a list with del
# 使用 del 来删除列表中的任意元素
del li[2] # li is now [1, 2, 3]
# li 现在是 [1, 2, 3]


# You can add lists
# 可以把列表相加
li + other_li #=> [1, 2, 3, 4, 5, 6] - Note: li and other_li is left alone
#=> [1, 2, 3, 4, 5, 6] - 请留意 li 和 other_li 并不会被修改


# Concatenate lists with extend
# 使用 extend 来合并列表
li.extend(other_li) # Now li is [1, 2, 3, 4, 5, 6]
# 现在 li 是 [1, 2, 3, 4, 5, 6]


# Check for existence in a list with in
# 用 in 来检查是否存在于某个列表中
1 in li #=> True


# Examine the length with len
# 用 len 来检测列表的长度
len(li) #=> 6




# Tuples are like lists but are immutable.
# 元组很像列表,但它是“不可变”的。
tup = (1, 2, 3)
tup[0] #=> 1
tup[0] = 3 # Raises a TypeError
# 抛出一个类型错误


# You can do all those list thingies on tuples too
# 操作列表的方式通常也能用在元组身上
len(tup) #=> 3
tup + (4, 5, 6) #=> (1, 2, 3, 4, 5, 6)
tup[:2] #=> (1, 2)
2 in tup #=> True


# You can unpack tuples (or lists) into variables
# 你可以把元组(或列表)中的元素解包赋值给多个变量
a, b, c = (1, 2, 3) # a is now 1, b is now 2 and c is now 3
# 现在 a 是 1,b 是 2,c 是 3
# Tuples are created by default if you leave out the parentheses
# 如果你省去了小括号,那么元组会被自动创建
d, e, f = 4, 5, 6
# Now look how easy it is to swap two values
# 再来看看交换两个值是多么简单。
e, d = d, e # d is now 5 and e is now 4
# 现在 d 是 5 而 e 是 4




# Dictionaries store mappings
# 字典用于存储映射关系
empty_dict = {}
# Here is a prefilled dictionary
# 这是一个预先填充的字典
filled_dict = {"one": 1, "two": 2, "three": 3}


# Look up values with []
# 使用 [] 来查询键值
filled_dict["one"] #=> 1


# Get all keys as a list
# 将字典的所有键名获取为一个列表
filled_dict.keys() #=> ["three", "two", "one"]
# Note - Dictionary key ordering is not guaranteed.
# Your results might not match this exactly.
# 请注意:无法保证字典键名的顺序如何排列。
# 你得到的结果可能跟上面的示例不一致。


# Get all values as a list
# 将字典的所有键值获取为一个列表
filled_dict.values() #=> [3, 2, 1]
# Note - Same as above regarding key ordering.
# 请注意:顺序的问题和上面一样。


# Check for existence of keys in a dictionary with in
# 使用 in 来检查一个字典是否包含某个键名
"one" in filled_dict #=> True
1 in filled_dict #=> False


# Looking up a non-existing key is a KeyError
# 查询一个不存在的键名会产生一个键名错误
filled_dict["four"] # KeyError
# 键名错误


# Use get method to avoid the KeyError
# 所以要使用 get 方法来避免键名错误
filled_dict.get("one") #=> 1
filled_dict.get("four") #=> None
# The get method supports a default argument when the value is missing
# get 方法支持传入一个默认值参数,将在取不到值时返回。
filled_dict.get("one", 4) #=> 1
filled_dict.get("four", 4) #=> 4


# Setdefault method is a safe way to add new key-value pair into dictionary
# Setdefault 方法可以安全地把新的名值对添加到字典里
filled_dict.setdefault("five", 5) #filled_dict["five"] is set to 5
#filled_dict["five"] 被设置为 5
filled_dict.setdefault("five", 6) #filled_dict["five"] is still 5
#filled_dict["five"] 仍然为 5




# Sets store ... well sets
# set 用于保存集合
empty_set = set()
# Initialize a set with a bunch of values
# 使用一堆值来初始化一个集合
some_set = set([1,2,2,3,4]) # some_set is now set([1, 2, 3, 4])
# some_set 现在是 set([1, 2, 3, 4])


# Since Python 2.7, {} can be used to declare a set
# 从 Python 2.7 开始,{} 可以用来声明一个集合
filled_set = {1, 2, 2, 3, 4} # => {1, 2, 3, 4}
# (译注:集合是种无序不重复的元素集,因此重复的 2 被滤除了。)
# (译注:{} 不会创建一个空集合,只会创建一个空字典。)


# Add more items to a set
# 把更多的元素添加进一个集合
filled_set.add(5) # filled_set is now {1, 2, 3, 4, 5}
# filled_set 现在是 {1, 2, 3, 4, 5}


# Do set intersection with &
# 使用 & 来获取交集
other_set = {3, 4, 5, 6}
filled_set & other_set #=> {3, 4, 5}


# Do set union with |
# 使用 | 来获取并集
filled_set | other_set #=> {1, 2, 3, 4, 5, 6}


# Do set difference with -
# 使用 - 来获取补集
{1,2,3,4} - {2,3,5} #=> {1, 4}


# Check for existence in a set with in
# 使用 in 来检查是否存在于某个集合中
2 in filled_set #=> True
10 in filled_set #=> False




####################################################
## 3. Control Flow
## 3. 控制流
####################################################


# Let's just make a variable
# 我们先创建一个变量
some_var = 5


# Here is an if statement. Indentation is significant in python!
# prints "some_var is smaller than 10"
# 这里有一个条件语句。缩进在 Python 中可是很重要的哦!
# 程序会打印出 "some_var is smaller than 10"
# (译注:意为“some_var 比 10 小”。)
if some_var > 10:
print "some_var is totally bigger than 10."
# (译注:意为“some_var 完全比 10 大”。)
elif some_var < 10: # This elif clause is optional.
# 这里的 elif 子句是可选的
print "some_var is smaller than 10."
# (译注:意为“some_var 比 10 小”。)
else: # This is optional too.
# 这一句也是可选的
print "some_var is indeed 10."
# (译注:意为“some_var 就是 10”。)




"""
For loops iterate over lists
for 循环可以遍历列表
prints:
如果要打印出:
dog is a mammal
cat is a mammal
mouse is a mammal
"""

for animal in ["dog", "cat", "mouse"]:
# You can use % to interpolate formatted strings
# 别忘了你可以使用 % 来格式化字符串
print "%s is a mammal" % animal
# (译注:意为“%s 是哺乳动物”。)


"""
`range(number)` returns a list of numbers
from zero to the given number
`range(数字)` 会返回一个数字列表,
这个列表将包含从零到给定的数字。
prints:
如果要打印出:
0
1
2
3
"""

for i in range(4):
print i


"""
While loops go until a condition is no longer met.
while 循环会一直继续,直到条件不再满足。
prints:
如果要打印出:
0
1
2
3
"""

x = 0
while x < 4:
print x
x += 1 # Shorthand for x = x + 1
# 这是 x = x + 1 的简写方式


# Handle exceptions with a try/except block
# 使用 try/except 代码块来处理异常


# Works on Python 2.6 and up:
# 适用于 Python 2.6 及以上版本:
try:
# Use raise to raise an error
# 使用 raise 来抛出一个错误
raise IndexError("This is an index error")
# 抛出一个索引错误:“这是一个索引错误”。
except IndexError as e:
pass # Pass is just a no-op. Usually you would do recovery here.
# pass 只是一个空操作。通常你应该在这里做一些恢复工作。




####################################################
## 4. Functions
## 4. 函数
####################################################


# Use def to create new functions
# 使用 def 来创建新函数
def add(x, y):
print "x is %s and y is %s" % (x, y)
# (译注:意为“x 是 %s 而且 y 是 %s”。)
return x + y # Return values with a return statement
# 使用 return 语句来返回值


# Calling functions with parameters
# 调用函数并传入参数
add(5, 6) #=> prints out "x is 5 and y is 6" and returns 11
# (译注:意为“x 是 5 而且 y 是 6”,并返回 11)


# Another way to call functions is with keyword arguments
# 调用函数的另一种方式是传入关键字参数
add(y=6, x=5) # Keyword arguments can arrive in any order.
# 关键字参数可以以任意顺序传入


# You can define functions that take a variable number of
# positional arguments
# 你可以定义一个函数,并让它接受可变数量的定位参数。
def varargs(*args):
return args


varargs(1, 2, 3) #=> (1,2,3)




# You can define functions that take a variable number of
# keyword arguments, as well
# 你也可以定义一个函数,并让它接受可变数量的关键字参数。
def keyword_args(**kwargs):
return kwargs


# Let's call it to see what happens
# 我们试着调用它,看看会发生什么:
keyword_args(big="foot", loch="ness") #=> {"big": "foot", "loch": "ness"}


# You can do both at once, if you like
# 你还可以同时使用这两类参数,只要你愿意:
def all_the_args(*args, **kwargs):
print args
print kwargs
"""
all_the_args(1, 2, a=3, b=4) prints:
(1, 2)
{"a": 3, "b": 4}
"""



# When calling functions, you can do the opposite of varargs/kwargs!
# Use * to expand tuples and use ** to expand kwargs.
# 在调用函数时,定位参数和关键字参数还可以反过来用。
# 使用 * 来展开元组,使用 ** 来展开关键字参数。
args = (1, 2, 3, 4)
kwargs = {"a": 3, "b": 4}
all_the_args(*args) # equivalent to foo(1, 2, 3, 4)
# 相当于 all_the_args(1, 2, 3, 4)
all_the_args(**kwargs) # equivalent to foo(a=3, b=4)
# 相当于 all_the_args(a=3, b=4)
all_the_args(*args, **kwargs) # equivalent to foo(1, 2, 3, 4, a=3, b=4)
# 相当于 all_the_args(1, 2, 3, 4, a=3, b=4)


# Python has first class functions
# 函数在 Python 中是一等公民
def create_adder(x):
def adder(y):
return x + y
return adder


add_10 = create_adder(10)
add_10(3) #=> 13


# There are also anonymous functions
# 还有匿名函数
(lambda x: x > 2)(3) #=> True


# There are built-in higher order functions
# 还有一些内建的高阶函数
map(add_10, [1,2,3]) #=> [11, 12, 13]
filter(lambda x: x > 5, [3, 4, 5, 6, 7]) #=> [6, 7]


# We can use list comprehensions for nice maps and filters
# 我们可以使用列表推导式来模拟 map 和 filter
[add_10(i) for i in [1, 2, 3]] #=> [11, 12, 13]
[x for x in [3, 4, 5, 6, 7] if x > 5] #=> [6, 7]


####################################################
## 5. Classes
## 5. 类
####################################################


# We subclass from object to get a class.
# 我们可以从对象中继承,来得到一个类。
class Human(object):


# A class attribute. It is shared by all instances of this class
# 下面是一个类属性。它将被这个类的所有实例共享。
species = "H. sapiens"


# Basic initializer
# 基本的初始化函数(构造函数)
def __init__(self, name):
# Assign the argument to the instance's name attribute
# 把参数赋值为实例的 name 属性
self.name = name


# An instance method. All methods take self as the first argument
# 下面是一个实例方法。所有方法都以 self 作为第一个参数。
def say(self, msg):
return "%s: %s" % (self.name, msg)


# A class method is shared among all instances
# They are called with the calling class as the first argument
# 类方法会被所有实例共享。
# 类方法在调用时,会将类本身作为第一个函数传入。
@classmethod
def get_species(cls):
return cls.species


# A static method is called without a class or instance reference
# 静态方法在调用时,不会传入类或实例的引用。
@staticmethod
def grunt():
return "*grunt*"




# Instantiate a class
# 实例化一个类
i = Human(name="Ian")
print i.say("hi") # prints out "Ian: hi"
# 打印出 "Ian: hi"


j = Human("Joel")
print j.say("hello") # prints out "Joel: hello"
# 打印出 "Joel: hello"


# Call our class method
# 调用我们的类方法
i.get_species() #=> "H. sapiens"


# Change the shared attribute
# 修改共享属性
Human.species = "H. neanderthalensis"
i.get_species() #=> "H. neanderthalensis"
j.get_species() #=> "H. neanderthalensis"


# Call the static method
# 调用静态方法
Human.grunt() #=> "*grunt*"




####################################################
## 6. Modules
## 6. 模块
####################################################


# You can import modules
# 你可以导入模块
import math
print math.sqrt(16) #=> 4


# You can get specific functions from a module
# 也可以从一个模块中获取指定的函数
from math import ceil, floor
print ceil(3.7) #=> 4.0
print floor(3.7) #=> 3.0


# You can import all functions from a module.
# Warning: this is not recommended
# 你可以从一个模块中导入所有函数
# 警告:不建议使用这种方式
from math import *


# You can shorten module names
# 你可以缩短模块的名称
import math as m
math.sqrt(16) == m.sqrt(16) #=> True


# Python modules are just ordinary python files. You
# can write your own, and import them. The name of the
# module is the same as the name of the file.
# Python 模块就是普通的 Python 文件。
# 你可以编写你自己的模块,然后导入它们。
# 模块的名称与文件名相同。


# You can find out which functions and attributes
# defines a module.
# 你可以查出一个模块里有哪些函数和属性
import math
dir(math)

普通人扯ceph架构

今天在来扯点啥呢?平时由于吊丝运维的工作关系,杂七杂八地也搞了不少东西。偶然间邂逅了ceph,各种高大上的概念与理念就像是妹子身上的幽香(此处运用通感手法),吸引着让我继续探索(读者切毋遐想)。
什么是ceph呢?引述下官文结出的定义: Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. 鄙人英文水平有限,通过有道翻译,把几个关键字拎出来。

1
2
unified   //统一的
distributed //分布的

官文就是给力,寥寥几句就能给人醍醐灌顶的赶脚。所谓的“统一的”就是指ceph同时可能向外提供对象存储、块存储、文件存储这三类存储功能。“分布式”是说ceph没有中心结构,可以平滑地水平扩展。
说得好高大啊(言外之意就是逼格好高),这几类存储是啥鸟玩样儿呢?鄙人再来小扯下。

Read More

Cobbler的简单使用

一、 cobbler介绍

Cobbler是一个快速网络安装linux的服务,而且在经过调整也可以支持网络安装windows。该工具使用python开发,小巧轻便(才15k行python代码),使用简单的命令即可完成PXE网络安装环境的配置,同时还可以管理DHCPDNS、以及yum仓库、构造系统ISO镜像。
Cobbler支持命令行管理,web界面管理,还提供了API接口,可以方便二次开发使用。
Cobbler客户端 Koan 支持虚拟机安装和操作系统重新安装,使重装系统更便捷。
Cobbler提供以下服务集成:

  • PXE服务支持
  • DHCP服务管理
  • DNS服务管理
  • 电源管理
  • Kickstart服务支持
  • yum仓库管理

Read More