1:简单说两句

这是我一个准备做的完整项目中的一个小部分。从很早以前,我就在思考,如何将美团和饿了么以及我自己做的小程序订单合并在一起,这样来实现统一的管理。。之所以要统一管理,是因为我母亲从事鲜花零售,在节假日里,会有大量的订单,而鲜花的包扎是个时间活。制作和配送,以及订单的处理,繁杂的程序会让人晕头转向。为了减轻我母亲的节假日压力,我想到了一些方法,来帮助她管理订单。

完整的项目是,汇总小程序、美团、饿了么的订单。将订单返回的信息,通过处理,按配送时间排序,经过处理后通过企业微信api推送给微信。因为,花店的配送人员都是我母亲找的,他们受教育程度低,有的因为不识字,连导航都不会用。所以,需要把订单的位置信息,通过百度地图api转为url发送给配送员。另外,还可以制作一个程序,在程序上实时更新,新订单。。每当一束花被做好,我母亲就可从程序中将该订单从等待制作的列表中移除。。每当,有一个订单被送出,也可在程序上控制美团等客户端发送对应消息。

这一切都是非常美好的,对零售商家的体验会非常友好。但是,因为我无法取得美团和饿了么的开发者资格,也就拿不到他们的接口授权。。于是,我想到了,利用利用mitmproxy获取美团饿了么的订单信息。通过appium自动化控制手机。无论是mitmrpoxy还是appium对python程序都是非常友好的。

那么现在开始我的第一步,利用mitmproxy爬取饿了么零售商家版订单信息。

2:mitmproxy安装与使用

安装

1
pip install mitmproxy

开启

1
2
mitmweb
#或者mitmproxy mitmdump

与python配合

1
mitmdump -s eleme.py

手机使用mitmproxy开启的代理

更改wifi高级设置里的代理模式为手动,主机填写mitmproxy运行的电脑局域网ip。可通过ifconfig查看。

端口为8080

此时打开baidu.com会提示不安全,而且mitmproxy应该开始记录了。

手机打开网页mitm.it下载对应系统的证书文件。安卓下载后直接安装。。

再上网就不会提示不安全了。

3:响应内容

在开启mitmweb后,从打开的网页上可以观察到手机的整个数据流,在手机上刷新订单页,就能截获饿了么发送回来的订单信息。

将响应内容提取出来,并把JSON数据格式化,部分关键信息已打码。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
{
"data": {
"curr_page": 1,
"order_count": 1,
"page_count": 1,
"page_size": 20,
"total_order_list": [{
"mills": 1602518400000,
"order_list": [{
"cold_chain_cost": {
"price": "0.00",
"title": "冷链服务费"
},
"compensationInfo": {
"enableCompensable": false,
"timeRemaining": 0
},
"extract_commission": {
"brief_desc": "请以结算账单为准",
"commission_total": "15.00",
"title": "抽取佣金"
},
"goods_discount": {
"discount_account": "-¥3.00",
"discount_list": [{
"desc": "",
"hasLineBelow": 1,
"list": [{
"price": "¥3.00",
"title": "商家配送费补贴"
}, {
"price": "¥3.00",
"title": "小计"
}],
"title": "商户补贴金额"
}, {
"desc": "",
"hasLineBelow": null,
"list": [{
"title": "配送费满减"
}, {
"title": "(商家补贴:¥3.00元)"
}],
"title": "活动明细"
}],
"title": "顾客优惠合计",
"user_discount_account": "-¥3.00"
},
"grayInfo": {
"hummingBirdGray": 0
},
"orderSettle": {
"baseLogisticsAmt": "0.00",
"baseLogisticsDesc": "本单收取固定物流费",
"commissionAmt": "15.00",
"commissionDesc": "请以结算账单为准",
"shopInAmount": "0.00"
},
"order_basic": {
"apply_cancel_feed": {
"feed_list": [{
"cancel_content": "",
"cancel_extra": "",
"cancel_time": "10月13日 12:55",
"cancel_title": "订单已完成(商户)",
"end_time": "",
"left_time": 0,
"left_time_show": "",
"refund_price": "",
"user_apply_refund_pic": []
}, {
"cancel_content": "",
"cancel_extra": "",
"cancel_time": "10月13日 09:10",
"cancel_title": "商家已接单",
"end_time": "",
"left_time": 0,
"left_time_show": "",
"refund_price": "",
"user_apply_refund_pic": []
}, {
"cancel_content": "",
"cancel_extra": "",
"cancel_time": "10月13日 09:10",
"cancel_title": "订单已支付",
"end_time": "",
"left_time": 0,
"left_time_show": "",
"refund_price": "",
"user_apply_refund_pic": []
}, {
"cancel_content": "",
"cancel_extra": "",
"cancel_time": "10月13日 09:09",
"cancel_title": "创建订单",
"end_time": "",
"left_time": 0,
"left_time_show": "",
"refund_price": "",
"user_apply_refund_pic": []
}]
},
"apply_cancel_status": 886,
"auto_confirm_order": 0,
"batch": 2000000000000000001,
"business_type": 0,
"call_delivery_delay": "",
"cancel_reason": "",
"cancel_reason_explain": "",
"cancel_reason_status": "0",
"check_out_button": 0,
"city_name": "成都市",
"confirmButton": 1,
"confirm_time": "1602551413",
"create_time": "1602551397",
"delivery_delay_time": "",
"delivery_exception_code": 0,
"delivery_exception_state": "",
"delivery_name": "",
"delivery_party": "self",
"delivery_staff_phone": "",
"delivery_state": "0",
"delivery_state_desc": "",
"delivery_tip_amount": 0.0,
"distance": "<1.8千米",
"due_deliver_time_by_ofc": "",
"ele_fengniao_info": {
"is_fengniao_call_again": 0,
"is_fengniao_switch_self": 0,
"no_delivery": 0
},
"ele_zhongbao_info": {
"delivery_fee": 0,
"is_show": false,
"op_status": 0
},
"eleme_finish_refund": 0,
"eleme_order_id": "5009809679236497411",
"evoip_order": 2,
"express_company": "",
"express_id": "0",
"feed_to_show": "order_log",
"finished_time": "1602564946",
"gift_greeting": "",
"gift_phone": "13***502",
"hbirdRunErrandsInfo": {
"addTipsButton": 0,
"callButton": 0,
"cancelCallButton": 0,
"payType": null
},
"idempotentId": "20201017025815254005000100027411822742",
"invoice_price": "100.00",
"invoice_title": "",
"is_alipay_order": 0,
"is_baidu_logistics": 0,
"is_can_auto_confirm": 0,
"is_city_delivery": 0,
"is_cold_order": 0,
"is_compensation_show": 0,
"is_cook_overtime": 0,
"is_from_tp_new_retail": true,
"is_print": 1,
"is_privacy_hide": 1,
"is_ranger_show": 1,
"is_show_express": 0,
"ivr_tip": "",
"meal_num": "",
"need_invoice": 0,
"orderUserPhone": "136****3502",
"orderUserPrivacyPhone": "",
"order_from": "2",
"order_id": "50098***97411",
"order_index": "1",
"order_list_type": {
"is_cancel": 0,
"is_exception": 0,
"is_new": 0,
"is_remind": 0,
"is_reserve": 1,
"is_user_part_refund": 0
},
"order_time_line": [{
"code": "101004",
"time": "10/13 12:55",
"title": "订单已完成(商户)"
}, {
"code": "100004",
"time": "10/13 09:10",
"title": "商家已接单"
}, {
"code": "100001",
"time": "10/13 09:10",
"title": "订单已支付"
}, {
"code": "100000",
"time": "10/13 09:09",
"title": "创建订单"
}],
"order_type": 4,
"part_refund_support": 0,
"pay_display": 1,
"pay_status": "已支付",
"pc_order_label": [],
"pick_info": {
"expect_finish_time": "02:58:15",
"left_time": "0",
"pick_button": "0",
"show": "0",
"timestamp": ""
},
"privacy_phone": "",
"range_status": 0,
"real_status": 5,
"real_status_desc": "商家已经确认订单",
"refund_attribute": 0,
"refund_logs": [],
"refund_req_info": [],
"refuseButton": 1,
"remind_undeal": 0,
"remind_wait_time": "",
"responsible_party": "暂无",
"self_order_completed_button": 0,
"send_time": "请10月13日 13:00送达",
"send_time_print": "13日 13:00送达",
"service_phone": "10105757",
"set_express": 0,
"sex": "女士",
"shopPicked": 1,
"shop_info": {
"location_shop": {
"latitude": "35**3**7",
"longitude": "***4431E7"
},
"shop_address": null
},
"shop_name": "**鲜花店",
"shop_phone": "13***65",
"shop_user_distance": "",
"show_add_tip": 0,
"show_cancel_button": false,
"show_know_button": 0,
"show_shop_pack": 0,
"status": 9,
"status_desc": " 已完结",
"supplier_id": "0",
"supplier_name": "",
"takeout_average_time": 1602565200,
"takeout_phone": "136****065",
"taxer_id": "",
"thirty_min_service": 0,
"userTags": [2],
"user_address": "为了保护顾客隐私,隐藏地址信息",
"user_address_lat": "30.890345",
"user_address_lng": "104.261772",
"user_id": "576192986",
"user_note": "送给我好朋友的生日花,写几句祝福语,谢谢",
"user_order_num_str": "超百次下单",
"user_phone": "177****7520",
"user_phone_call": "",
"user_phone_star": "",
"user_real_name": "珊",
"vip_status": 0,
"waimai_release_id": "159766693",
"wxIconPath": "",
"wxNickname": "",
"zone_tips_required": 0
},
"order_goods": {
"cart_num": 1,
"goods_list": [{
"customer_price": "98.00",
"ext": {
"attr": "",
"cart_id": "1",
"cold_chain": null,
"combo_attr": [],
"customer_total_discount": "",
"discount_desc": "",
"ext_code": "",
"is_combo": 0,
"process_label": [],
"product_index": null,
"property_label": [],
"shop_total_discount": "",
"store_attr": {
"bar_code": "wm367**65",
"category_name": "爱情表白",
"shelf_position": "",
"sku_id": "1597871782222488"
},
"unique_id": "13***678"
},
"fix_weight": 0,
"gift_info": null,
"gmids": null,
"hasgift": 0,
"isNoReasonToReturn": false,
"isfreegift": 0,
"name": "【七夕专属】19朵红玫瑰礼盒+银叶",
"number": 1,
"orig_price": "98.00",
"orig_unit_price": "98.00",
"printing_price": "98.00",
"shop_price": "98.00",
"shop_unit_price": "98.00",
"total_weight": 999,
"unique_id": "130047***9119678",
"url": "https://img.alic***i2/111483146/O1CN01wVILGS1Z6udgswHe4_!!111483146-0-***.jpg",
"weight": 999,
"weight_can_update": false,
"weight_flag": 2
}],
"goods_total": 1
},
"order_meal_fee": {
"price": "0.00",
"title": "包装费"
},
"order_operations": [],
"order_sub_total": {
"discount": 0,
"formula_desc": "",
"number": 1,
"price": "98.00",
"title": "商品小计"
},
"order_total": {
"customer_price": "100.00",
"customer_title": "本单顾客实际支付",
"number": 1,
"pay_type": 1,
"shop_price": "85.00",
"shop_price_final": "85.00",
"title": "预计收入",
"title_after": null,
"total_desc": "本单顾客实际支付100.00元"
},
"other_total_cost": {
"price": "5.00",
"title": "其他费用合计"
},
"prescriptionDrugsInfo": {
"auditButton": 0,
"prescriptionOrder": false,
"processMsg": "系统正在处理中"
},
"prescription_info": {
"is_prescription_order": 0
},
"shop_other_discount": {
"customer_discount_list": [],
"customer_discount_total": 0,
"price": "-0.00",
"shop_discount_list": [],
"shop_discount_total": 0,
"sub_total_desc": "以上明细为商户承担的顾客优惠费用,最终订单收入请以对账单为准。",
"title": "商户其他优惠"
},
"takeout_cost": {
"price": "5.00",
"title": "配送费"
}
}],
"title": "10月13日"
}]
},
"errmsg": "",
"errno": 0,
"req_params": [{
"asap_type": null,
"end_timestamp": "2020-10-16",
"is_asap": null,
"keyword": "",
"new_system": 1,
"order_status": 9,
"page": 1,
"pageSize": 20,
"shop_id": null,
"start_timestamp": "2020-10-10"
}, {
"appVersion": "2.2.3",
"authedSupplierIds": [],
"brand": "samsung",
"channel": "mobile",
"cuid": "12B9828F9D0D3E389BA181D5A286A514|fw+G+2QPhBBAD8Yubtg6qGDX",
"from": "android",
"isBoss": false,
"isManager": false,
"isMobile": false,
"isShop": true,
"isSupplier": false,
"manager": {
"bossUserId": null
},
"pcVersion": "",
"roleInfo": {
"shopRoleId": "159766693"
},
"shopInfo": {
"eleId": 159766693,
"wid": "159766693"
},
"supplierInfo": {
"supplierId": "0"
},
"userInfo": {
"loginRole": 1,
"loginSubUserType": 0,
"loginUserId": "1597***50001",
"phone": "13***65",
"shopUserId": "15973**001",
"userName": "x***65"
}
}]
}

4:Python程序

对上述JSON数据进行分析,在Python中用json.load把JSON数据转化为字典和列表格式数据,方便我们提取信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import mitmproxy.http
import json

#顾客类
class Customer(object):
def __init__(self,name,tel):
self.name = name
self.tel = tel


#商品类
class Good(object):
def __init__(self,name,number,img):
self.name = name
self.number = number
self.img = img

#订单类
class Order(Customer):
def __init__(self,platform,name,tel,addr_long,addr_latu,note,arri_time,order_id,goods):
Customer.__init__(self,name,tel)
self.addr_long = addr_long
self.addr_latu = addr_latu
self.note = note
self.arri_time = arri_time
self.order_id = order_id
self.goods = goods
self.platform = platform


class ElemeOrders():
def __init__(self):
self.orders = []

def response(self, flow: mitmproxy.http.flow):
if "bwm_newretail.recrm_order_j/queryorderapp/queryOrderInfoList?" in flow.request.url:
data = json.loads(flow.response.text)["data"]
for total_order_list in data['total_order_list']:
for order_list in total_order_list['order_list']:
goods = []
for good in order_list['order_goods']['goods_list']:
good_buffer = Good(good['name'],good['number'],good['url'])
goods.append(good_buffer)
order = Order('eleme',order_list['order_basic']['user_real_name']+order_list['order_basic']['sex'],order_list['order_basic']['user_phone'],str(order_list['order_basic']['user_address_lng']),str(order_list['order_basic']['user_address_lat']),order_list['order_basic']['user_note'],order_list['order_basic']['takeout_average_time'],order_list['order_basic']['order_id'],goods)
self.orders.append(order)
print(self.orders)

经过简单的测试,程序能够拿到完整的订单信息。一次性搞定,没有bug。

在这个程序中,主要需要关注的是,Mitm类中的self.orders这个变量,可以看到,这个变量是一个列表,它包含了大于等于1个订单。每个订单,都是一个Order类的实例化,Order类中最后一个变量goods是一个列表,这个列表由多个实例化的Good类组成。因为一个订单可能包含多种商品。

具体如何利用这些信息,还需要再Mitm()类中做修改。而提取单个订单类,比如第一个订单,直接使用self.orders[0],要提取第一个订单中的第一件商品self.orders[0].goods[0]

其他的,比如第一个订单收货人:self.orders[0].name

第一个订单的第一件商品的名称:self.orders[0].goods[0].name

后续我可能会将它保存到数据库中,方便下次提取和修改信息。

5:参考资料

使用 mitmproxy + python 做拦截代理

APP爬虫入门,Appium+Mitmproxy强势组合实现抖音的数据爬取

评论