外卖平台订单爬取之美团外卖商家版

Qifei
网络爬虫

发布于：2020年10月17日
次浏览

1：root手机

昨天，我利用mitmproxy成功拦截了饿了么服务器返回的订单信息。今天，我准备故技重施，获取美团订单。残酷的现实给我炽热地心浇了一盆冷水。无论我如何调整mitmproxy和美团外卖商家版，都无法获取到美团服务器返回的数据。更确切地说，美团服务器似乎发现，有这么一个小偷，准备偷取它返回给客户端地数据。因此拒绝给客户端传递数据。

我在网上找了一些解答，在安卓7.0之后，安卓系统允许安卓应用不信任用户上传地CA证书，众所周知，在中间人攻击中，将代理服务器的CA证书添加到安卓用户证书里，这样代理服务器在客户端和https服务端之间做个信使，他们的来回信件，信使就能拆开阅读，再封好发出去。现在安卓系统允许安卓应用设置是否信任用户证书，很明显，美团外卖商家版，不支持用户自定义证书。

好在，安卓系统有个要求，就是系统证书都是被认可的。而添加系统证书，需要修改手机system文件夹，需要将手机root。

待root手机：三星S8+

工具：Odin线刷工具，第三方recovery TWRP，Tomato卡刷包（卡刷包含root）。

2：添加CA证书为系统证书

mitmproxy会在其运行目录生成一个.mitmproxy隐藏目录，目录中的mitmproxy-ca-cert.pem是可以直接放到系统证书目录下的证书。首先需要将它更名为统一的格式。

1
2
3
4
#openssl版本在1.0以上的版本的执行这一句
openssl x509 -inform PEM -subject_hash_old -in mitmproxy-ca-cert.pem
#openssl版本在1.0以下的版本的执行这一句
openssl x509 -inform PEM -subject_hash -in mitmproxy-ca-cert.pem

生成347bacb5证书的hash值，然后将mitmproxy-ca-cert.pem更名为347bacb5.0

将更名后的证书放到安卓系统证书目录，root后的安卓手机，可以直接打开RE管理器进入系统目录

1
/system/etc/security/cacerts

重启手机。修改wifi代理，并连接mitmproxy。

打开美团外卖商家版，不再报没有网络连接的错误。此时即可以从代理中截获解码后的信息了。

3：截获订单请求数据

开启mitmproxy，开始收集请求。在美团上下单，可以截获下单后，美团服务器发送到客户端的订单信息。

进行中订单请求地址，部分内容已打码：

1
https://eapi.waimai.meituan.com/api/retail/order/supplement/increment/orders?region_id=10***0100&region_version=153***1&__skck=8f5973b0***&__skts=16***8&__skua=d41d8c***8427e&__skno=94899011-***

预订单请求地址：

1
https://eapi.waimai.meituan.com/api/retail/order/pre/orders?region_id=1000510100&region_version=1534166411&__skck=8f5973b085446090f224af74e30e0181&__skts=1603035614&__skua=d41d8cd98f00b204e9800998ecf8427e&__skno=ed5fa567-5e33-4186-806a-e1ee884b5d5c&__skcy=hNEhcBUBRBinXjXwhZ%2BoMtvkMu8%3D

进行中响应内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
{
	"msg": "Success",
	"code": 0,
	"data": {
		"serverTime": 1603013219,
		"news": [{
			"agreeBtn": 0,
			"reject_reason": "",
			"rejectBtn": 0,
			"apply_refund_type": 0,
			"appealReason": "",
			"pay_status": 3,
			"payExpireTime": 0,
			"orderDescription": "",
			"pay_status_desc": "已付款",
			"apply_reason": "",
			"applyRefundTime": 0,
			"rejectRefundTime": "",
			"appealRefundTime": 0,
			"refundFinishTime": "",
			"isPartRefundApply": 0,
			"isAllowPartRefund": 0,
			"isAllowAllRefund": false,
			"hasMigratedToRetail": true,
			"returnSubStatus": 0,
			"logistics": {
				"id": -1,
				"type": 101,
				"icon": "",
				"name": "",
				"status": -1,
				"statusDesc": "",
				"sendTime": -1,
				"confirmTime": -1,
				"arrivePoiTime": -1,
				"selfDeliveryTime": -1,
				"fetchTime": -1,
				"completedTime": -1,
				"cancelTime": -1,
				"collectTime": 0,
				"departureTime": 0,
				"reachTime": 0,
				"takeTime": 0,
				"timeOut": -1,
				"code": "",
				"valid": 1,
				"shippingFee": 0.0,
				"tipFee": 0.0,
				"shippingTips": "",
				"couponAmount": 0.0,
				"activityName": "",
				"activityAmount": 0.0,
				"extraFee": 0.0,
				"payAmount": 0.0,
				"shippingFeeBase": 0.0,
				"payType": "",
				"zbType": 0,
				"orderDistance": 0.0,
				"ifSelfLogistics": 0,
				"isCanChangeSelfLogistics": 0,
				"dispatcher": {
					"name": "",
					"mobile": "",
					"icon": "",
					"role": "配送员",
					"type": -1,
					"typeName": ""
				},
				"dispatchMaster": {
					"name": "站点",
					"mobile": "",
					"icon": "",
					"role": "站长"
				},
				"dispatchAssistant": {
					"name": "站点备用",
					"mobile": "",
					"icon": "",
					"role": "站长助理"
				},
				"unionDispatcher": {
					"name": "",
					"mobile": ""
				},
				"logisticsSubCode": "",
				"dispatchOrderId": -1,
				"delayPushSecond": 0,
				"delayPushMin": 0,
				"assignTime": 0,
				"thirdLogisticsCode": ""
			},
			"printRelayBarCode": false,
			"utime": 1603013218,
			"refundType": -1,
			"isCanChangeSelfLogistics": 0,
			"refundProcessEndTips": "",
			"ifSelfLogistics": 0,
			"cancelButtonShow": 0,
			"foodDoneInfo": {
				"isShowFoodDoneBlock": 0,
				"isfoodDone": 0,
				"foodDoneStatus": "拣货完成",
				"foodDoneTime": 0,
				"foodSendOutTimeDeadline": 0,
				"diningMinutes": 0,
				"foodSendDeviceInfo": ""
			},
			"logisticsAbnormalInfo": {
				"isLogisticsAbnormal": 0,
				"logisticsAbnormalDesc": "",
				"logisticsAbnormalType": 0,
				"isShowLogisticsReport": 0,
				"logisticsReportDesc": ""
			},
			"refundOtherInfos": {
				"returnStatusDesc": "",
				"returnSubStatusDesc": "",
				"returnMaxPayAcount": 100.0,
				"returnSubStatus": 0,
				"countDownDesc": "",
				"refundViewId": 0,
				"serviceType": 0,
				"refundRuleUrl": "http://s3plus.sankuai.com/v1/mss_03d0d9cf21144ba0b7747ba1dc1acf6e/order/refund_rules.html"
			},
			"refundInfoList": [],
			"refundGoodsInfoList": [],
			"isAppealApply": 0,
			"compensationInfo": {
				"id": 0,
				"title": "",
				"time": 0,
				"moneyDesc": "",
				"reason": "",
				"pictures": [],
				"appealInfo": ""
			},
			"orderImValid": 1,
			"imGuideContent": "",
			"imReplyDefaultContent": "",
			"isAllowPriceDiffRefund": false,
			"confirmDeliveryStatus": 0,
			"id": 56367**02186,
			"wm_poi_id": 5636780,
			"wm_poi_order_dayseq": 1,
			"wm_order_id_view": 563678***186,
			"num": 1,
			"poi_name": "**鲜花店",
			"order_source": 1,
			"order_source_desc": "手机APP下单",
			"shipping_fee": 0.0,
			"recipient_name": "谢先生",
			"recipient_phone": "178**858,4109",
			"recipient_address": "",
			"total_after": 10.0,
			"total_before": 10.0,
			"boxpriceTotal": 0.0,
			"remark": "[贺卡内容]贺卡内容 [其他备注]备注内容",
			"environmentalFriendlyOrder": 0,
			"status": 4,
			"order_time": 1603013207,
			"order_time_fmt": "10-18 17:26",
			"confirmOrderTime": 1603013218,
			"details": [{
				"id": 1945297344,
				"food_name": "8枝红色非洲菊 ",
				"food_price": 10.0,
				"originalFoodPrice": 10.0,
				"unit": "份",
				"count": 1,
				"box_num": 1.0,
				"box_price": 0.0,
				"cartId": 0,
				"cartName": "1号口袋",
				"upcCode": "",
				"skuCode": "",
				"locatorCode": "",
				"discountPoint": "",
				"discountRemark": "",
				"picUrl": "http://p0.meituan.net/wmproduct/d852b****0fa2d7f3d1adf7d95423.jpg",
				"specification": ""
			}],
			"discounts": [],
			"tips": [],
			"canBeCancelled": 1,
			"cancel_reason": "",
			"is_pre_order": 0,
			"delivery_btime": 0,
			"pre_order_delivery_info": "",
			"pay_utime": 1603013218,
			"pay_utime_fmt": "2020-10-18 17:26:58",
			"online_paid_info": "",
			"pre_order_remind": 0,
			"pre_order_tip": "预订单：期望送达时间",
			"overtime_paid_info": "",
			"invoice_title": "",
			"wm_order_pay_type": 2,
			"actual_pay_type": 2,
			"dispatch_code": "",
			"longitude": 104251302,
			"latitude": 30880900,
			"inArea": 1,
			"orderDistance": 2120.0,
			"addressLongitude": 104251302,
			"addressLatitude": 30880900,
			"riderPayment": "",
			"zbShippingFee": "",
			"zbShippingTips": "",
			"tipsVos": [],
			"status_desc": "已接单",
			"delivery_btime_fmt": "",
			"logistics_code": "",
			"logisticsService": 1,
			"logistics_status": -1,
			"hasFrozen": 0,
			"estimateArrivalTime": 1603020418,
			"poi_push_day": 20201018,
			"bindedPhone": "178**858,4109",
			"isLargeAmountOrder": 0,
			"userOrderCount": 2,
			"pickType": 0,
			"isUsePrivacyPhone": 1,
			"privacyPhone": "1788**4858,4109",
			"phoneShow": "178**8转4109",
			"privacyPhoneStatus": 0,
			"recipientPhoneShow": "手机尾号2605",
			"packageBagMoney": 0.0,
			"wmUserId": 371311890,
			"confirmTime": 1603013218,
			"cansunStatus": 0,
			"cansunInfo": [],
			"wmChargeBusiness": {
				"wmOrderViewId": 56367802727202186,
				"activityAmount": "0.0",
				"activityDetails": [],
				"commisionAmount": "-1.0",
				"settleAmount": "8.95",
				"offlineOrderSkPayAmount": "0.0",
				"shippingType": "0000",
				"commisionDetails": [{
					"chargeAmount": "-1.0",
					"chargeDesc": "服务费"
				}],
				"userOnlinePayAfter": "",
				"discounts": [],
				"isDowngrade": false,
				"shippingFee": "0.0",
				"riderPayment": "",
				"isRiderPay": false,
				"giftDetails": [],
				"userPayAmount": "0.0",
				"foodAmount": "10.0",
				"userPayTotalAmount": "10.0",
				"donateAmount": "-0.05",
				"agreementAmount": 0.0,
				"chargeMode": 1
			},
			"userTips": [{
				"category": 17,
				"type": "下单2次",
				"info": "该用户过去90天完成订单数"
			}],
			"bindedPrivacyPhoneList": [{
				"privacyPhoneShow": "130**773转5108",
				"privacyPhone": "13**4773,5108"
			}],
			"backupField": [],
			"onTimeInsurance": {
				"effect": 0,
				"desc": "",
				"status": 0
			},
			"foodSafeInsurance": {
				"effect": 0,
				"desc": "",
				"status": 0
			},
			"thirdLogistics": {
				"courierName": "",
				"courierPhone": "",
				"logisticsPlatformName": "",
				"logisticsPlatformCode": "",
				"deliverTime": "",
				"allowAlterType": true
			},
			"preMakerValid": 1,
			"retailOrderPreMakerInfo": {
				"preMakerPrivacyPhoneShow": "1788**37转9232",
				"preMakerPhoneShow": "178**7转9232",
				"preMakerPhone": "178**137,9232",
				"preMakerTag": "购花使者",
				"preMakerPrivacyPhoneShowAfter": "1******8137转9232",
				"preMakerBackupPrivacyPhone": [{
					"privacyPhoneShow": "185**029转1199",
					"privacyPhone": "18**4029,1199"
				}]
			}
		}],
		"changes": [],
		"orderSynCtime": 1603013219,
		"orderSynUtime": 1603013219
	}
}

3.1 确认送达时间：

1
"estimateArrivalTime": 1603020418,

从信息中提取到送达时间的时间戳，时间戳是指，格林威治时间从1970-01-01 00:00:00到某个时间点所经历的秒数，可以表达绝对时间领域的一个时刻。其转化为我们所能够阅读的时间，是根据计算机系统时间来计算的。不同时区的计算机得到的结果是不一样的。上述时间戳，转为北京时间后是2020-10-18 19:26:58。在转换时，默认时根据计算机所在时区转换。所以，最好指定转换时区。

1
2
3
4
5
6
import pytz
ts = 1566395745
tz = pytz.timezone(' Asia/Shanghai')
dt = pytz.datetime.datetime.fromtimestamp(ts, tz)
dt.strftime('%Y-%m-%d %H:%M:%S')
12345

结果为：2019-08-21 09:55:45

3.2 送达地点

在信息中，是没有配送地点的内容的，如果，通过appium来点击查看地址在爬取地址信息，就过于麻烦。再次研究订单信息JSON数据后，我发现，在订单中，有一行数据为

1
2
"addressLongitude": 104251302,
"addressLatitude": 30880900,

这是地址编码，对应的是经纬度信息。在高德地图API接口手册中，我找到了适合地址解码的api

1
https://restapi.amap.com/v3/geocode/regeo?output=xml&location=116.310003,39.991957&key=<用户的key>&radius=1000&extensions=all

测试后，从api返回的地址正是收货地址。

可以看到，Longitude经度，Latitude维度。这个参数是以六位小数结尾的数据。。而订单中，是没有把小数位表现出来的，所以需要单独修改。

为了让饿了么和美团的订单，具有相同的格式，我们需要修改之前确定的饿了么订单对象。把送达时间统一更改为时间戳，把地址信息统一转为地理编码后的经纬度。

4：Python编程

下面是包含了饿了么和美团外卖的完整程序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
# -*- encoding: utf-8 -*-
'''
@file_name    :order.py
@description  :
@time         :2020/10/16 21:25:01
@author       :Qifei
@version      :1.0
'''

import mitmproxy.http
import json

#顾客类
class Customer(object):
    def __init__(self,name,tel):
        self.name = name 
        self.tel = tel
     

#商品类
class Good(object):
    def __init__(self,name,number,img):
        self.name = name
        self.number = number
        self.img = img

#订单类
class Order(Customer):
    def __init__(self,platform,name,tel,addr_long,addr_latu,note,arri_time,order_id,goods):
        Customer.__init__(self,name,tel)
        self.addr_long = addr_long
        self.addr_latu = addr_latu
        self.note = note
        self.arri_time = arri_time
        self.order_id = order_id
        self.goods = goods
        self.platform = platform

#截获饿了么订单
class ElemeOrders():
    def __init__(self):
        self.orders = []

    def response(self, flow: mitmproxy.http.flow):
        if "bwm_newretail.recrm_order_j/queryorderapp/queryOrderInfoList?" in flow.request.url:
            data = json.loads(flow.response.text)["data"]
            for total_order_list in data['total_order_list']:
                for order_list in total_order_list['order_list']:
                    goods = []
                    for good in order_list['order_goods']['goods_list']:
                        good_buffer = Good(good['name'],good['number'],good['url'])
                        goods.append(good_buffer)
                    order = Order('eleme',order_list['order_basic']['user_real_name']+order_list['order_basic']['sex'],order_list['order_basic']['user_phone'],str(order_list['order_basic']['user_address_lng']),str(order_list['order_basic']['user_address_lat']),order_list['order_basic']['user_note'],order_list['order_basic']['takeout_average_time'],order_list['order_basic']['order_id'],goods)
                    self.orders.append(order)
            print(self.orders)

 #截获美团订单
class MeituanOrders():
    def __init__(self):
        self.orders = []

    def response(self, flow: mitmproxy.http.flow):
        if "eapi.waimai.meituan.com/api/retail/order/supplement/increment/orders?" in flow.request.url:
            data = json.loads(flow.response.text)["data"]
            for order_list in data['news']:
                goods = []
                for good in order_list['details']:
                    good_buffer = Good(good['food_name'],good['count'],good['picUrl'])
                    goods.append(good_buffer)
                order = Order('meituan',order_list['recipient_name'],order_list['recipient_phone'],self.geo_format(order_list['longitude']),self.geo_format(order_list['latitude']),order_list['remark'],order_list['estimateArrivalTime'],order_list['id'],goods)
                self.orders.append(order)
            print(self.orders[0].name)

    #将经纬度162787612转为162.787612
    def geo_format(self,int_geo):
        str_geo = str(int_geo)  #162787612
        list_geo = list(str_geo)    #['1','6','2','7','8','7','6','1','2']
        list_geo.insert(-6,'.')
        new_str = ''.join(list_geo)
        return new_str

#mitmproxy插件
addons = [
    ElemeOrders(),
    MeituanOrders()
]

5：参考资料

《给Android7及以上的手机安装系统级证书，实现Fiddler或者其他程序的HTTPS的抓包》

《python将时间戳转换为指定时区时间》

相关文章
爬取Youtube搜索结果爬虫基础如何利用Selenium实现更加高效的爬虫外卖平台订单爬取之饿了么零售商家版

博客内容遵循 署名-非商业性使用-相同方式共享 4.0 国际 (CC BY-NC-SA 4.0) 协议

本文永久链接是：https://blog.sci.ci/2020/10/17/%E5%A4%96%E5%8D%96%E5%B9%B3%E5%8F%B0%E8%AE%A2%E5%8D%95%E7%88%AC%E5%8F%96%E4%B9%8B%E7%BE%8E%E5%9B%A2%E5%A4%96%E5%8D%96%E5%95%86%E5%AE%B6%E7%89%88/

更新于：2024年3月26日

爬虫

评论