ag试玩

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

身邊有很多運維工程師,做了幾年的運維自動化,但依然不能確定選擇哪個工自動化工作?還有,怎樣更優雅的實施運維自動化和避免事實當中的坑?
馬哥教育大咖講堂-175期特別邀請了一線資深工程師張強為我們帶來了一場《saltstack在自動化運維實踐》的分享,嘗試帶著他對工作問題的思考以及部署自動化工具的經驗給大家一些幫助和啟發。
————————————
作者介紹:
張強,馬哥教育Linux大咖講堂金牌講師,現就職于伙伴智慧運維工程師,負責主線業務平臺,3年Linux一線經驗,擅長shell腳本、自動化_發布、web應用等,現在關注自動化運維、分布式數據庫,虛擬化技術。


自動化工具比較

puppetyexushisikuangongjuzhongzuishenrurenxinde。jiukeyongcaozuo、mokuaiheyonghujiemianeryan,tashizuiquanmiande。puppetchengxianleshujuzhongxinxiediaodequanmao,jihuhangaimeiyigeyunxingxitong,weigedacaozuoxitongtigongleshenrudegongju。chushishezhibijiaojiandan,zhixuyaozaixuyaojiayiguanlidemeigexitongshanganzhuangzhufuwuqihekehuduandailiruanjian。minglingxingjiekou(cli)jiandanzhiguan,yunxutongguopuppetminglingxiazaiheanzhuangmokuai。ranhou,xuyaoduipeizhiwenjianjinxinggenggai,haorangmokuaishihesuoxuderenwu;yingjiedaozhilingdekehuduanyuzhufuwuqilianxishi,huigenggaipeizhiwenjian,huozhekehuduantongguolijichufagenggaipeizhiwenjiandetuisong(push)laijinxinggenggai。

ag试玩ansibleguanzhudezhongdianshiliqiujingjianhekuaisu,erqiebuxuyaozaijiedianshanganzhuangdailiruanjian。yinci,ansibletongguosshzhixingsuoyougongneng。xuyaoguanlidejiedianbeitianjiadaoansiblepeizhihuanjing,sshshouquanmiyaobeifujiadaomeigejiedianshang,zheyuyunxingansibledeyonghuyouguan。yidanwanchenglezhebu,ansiblezhufuwuqikeyitongguosshyujiedianjinxingtongxin,zhixingsuoyoubiyaoderenwu。ansiblekeyishiyongparamiko(jiyussh2xieyidepythonshixian)huobiaozhunsshyongyutongxin,buguohuanyouyizhongjiasumoshi,yunxugengkuaisu、gengdaguimodetongxin。

saltleisiansible,yinweitayeshijiyuclidegongju,caiyongletuisongfangfashixiankehuduantongxin。takeyitongguogithuotongguochengxubaoguanlixitonganzhuangdaozhufuwuqihekehuduanshang。kehuduanhuixiangzhufuwuqitichuqingqiu,qingqiuzaizhufuwuqishangdedaojieshouhou,jiukeyikongzhigaikehuduanle。saltkeyitongguoputongdesshyukehuduanjinxingtongxin,danruguoshiyongmingweiminiondekehuduandailiruanjian,keyidadazengqiangkekuozhanxing。ciwai,salthanyouyigeyibuwenjianfuwuqi,keyiweikehuduanjiakuaiwenjianfuwusudu,zhewanquanshisaltzhuzhonggaokuozhanxingdeyigetixian。yuansibleyiyang,nikeyizhijietongguocli,xiangkehuduanfachumingling,biruqidongfuwuhuoanzhuangchengxubao;niyekeyishiyongmingweistatedeyamlpeizhiwenjian,chulibijiaofuzaderenwu。huanyou“pillar”,zhexieshifangzaijizhongdifangdeshujuji,yamlpeizhiwenjiankeyizaiyunxingqijianfangwentamen。

ag试玩zongjie:gerenguandianpuppetzuidaquedianjiushimorenqingkuangxiaagentmeige30fenzhongxiangmastertongbuzhuangtai,masterzhudongtuisonggongnengbijiaoboruo(2.7banben),ansiblejiyusshfuwuzhixing,ruguofuwuqiguoduobujianyishiyong,tashishiyonglunxundefangshi。saltjiyuxiaoxiduilie。xingnengxiangdanghao,shihedaliangshengchanhuanjing。

SaltStack簡介與特性

saltstack shiyizhongjiyu c/s jiagoudefuwuqijichujiagoujizhonghuaguanlipingtai,guanliduanchengwei master,kehuduanchengwei minion。saltstack jubeipeizhiguanli、yuanchengzhixing、jiankongdenggongneng,yibankeyilijieweishijianhuabande puppet hejiaqiangbande func。saltstack benshenshijiyu python yuyankaifashixian,jieheleqingliangjidexiaoxiduilieruanjian zeromq yu python disanfangmokuai(pyzmq、pycrypto、pyjinjia2、python-msgpack he pyyaml deng)goujian。

ag试玩tongguobushu saltstack huanjing,yunweirenyuankeyizaichengqianshangwantaifuwuqishangzuodaopiliangzhixingmingling,genjubutongdeyewutexingjinxingpeizhijizhonghuaguanli、fenfawenjian、caijixitongshujujiruanjianbaodeanzhuangyuguanlideng。

SaltStack 具有以下特性:

1、bushujiandan、fangbian;

2、zhichidabufenunix/linuxjiwindowshuanjing;

3、zhucongjizhonghuaguanli;

4、peizhijiandan、gongnengqiangda、kuozhanxingqiang;

5、zhukongduan(master)hebeikongduan(minion)jiyuzhengshurenzheng,anquankekao。

ag试玩6、zhichiapijizidingyimokuai,ketongguopythonqingsongkuozhan。

SaltStack 的工作原理

saltstack caiyong c/s jiegoulaiduiyunhuanjingneidefuwuqicaozuoguanlijipeizhiguanli。weilegenghaodelijietadegongzuofangshijiguanlimoxing,jiangtongguotuxingfangshiduiqiyuanlijinxingchanshu。

ag试玩saltstack kehuduan(minion)zaiqidongshi,huizidongshengchengyitaomiyao,baohansiyaohegongyao。zhihoujianggongyaofasonggeifuwuqiduan,fuwuqiduanyanzhengbingjieshougongyao,yicilaijianlikekaoqiejiamidetongxinlianjie。tongshitongguoxiaoxiduilie zeromq zaikehuduanyufuwuduanzhijianjianlixiaoxifabulianjie。jutitongxinyuanlitu,rutu 1 suoshi,minglingzhixingrutu 2 suoshi:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

zhuanyeshuyushuoming:

ag试玩minion shi saltstack xuyaoguanlidekehuduananzhuangzujian,huizhudongqulianjie master duan,bingcong master duandedaoziyuanzhuangtaixinxi,tongbuziyuanguanlixinxi。

ag试玩master zuoweikongzhizhongxinyunxingzaizhujifuwuqishang,fuze salt minglingyunxingheziyuanzhuangtaideguanli。

ag试玩zeromq shiyikuankaiyuandexiaoxiduilieruanjian,yongyuzai minion duanyu master duanjianlixitongtongxinqiaoliang。

ag试玩daemon shiyunxingyumeiyigechengyuanneideshouhujincheng,chengdanzhufabuxiaoxijitongxinduankoujiantingdegongneng。

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

yuanlitushuoming:

minion shi saltstack xuyaoguanlidekehuduananzhuangzujian,huizhudongqulianjie master duan,bingcong master duandedaoziyuanzhuangtaixinxi,tongbuziyuanguanlixinxi。

master zuoweikongzhizhongxinyunxingzaizhujifuwuqishang,fuze salt minglingyunxingheziyuanzhuangtaideguanli。

ag试玩master shangzhixingmoutiaozhilingtongguoduiliexiafadaogege minions quzhixing,bingfanhuijieguo。

SaltStack 的架構設計

weilerangdajiagenghaodelijie saltstack jizhonghuaguanlifangmiandeyoushi,yinci,genjuxiangmudeshijiqingkuanghuizhilebushujiagoutu,bingzaiwenzhongduijiagoutujinxinglexiangxishuoming。rutu 3 suoshi:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

shuoming:

ag试玩saltstack desuoyoubeiguanlikehuduanjiedian(rutu 3 suoshi db he web),doushitongguomiyaojinxingjiamitongxin,shiyongduankouwei 4506。kehuduanyufuwuqiduandeneirongchuanshu,shitongguoxiaoxiduiliewancheng,shiyongduankouwei 4505。master keyifasongrenhezhilingrang minion zhixing,salt youhenduokezhixingmokuai,birushuo cmd mokuai,zaianzhuang minion deshihouyijingzidaile,tamentongchangweiyunide python kuzhong,locate salt | grep /usr/?keyikandao salt zidaidesuoyoudongxi。

ag试玩weilegenghaodelijiejiagouyongyi,yixiajiangzhanshizhuyaodeminglingfabuguocheng:

saltstack de master yu minion zhijiantongguo zeromq jinxingxiaoxichuandi,shiyongle zeromq defabudingyuemoshi,lianjiefangshibaokuo tcp he ipc。

ag试玩salt mingling,jiang cmd.run ls minglingcong salt.client.localclient.cmd_cli fabudao master,huoquyige jodid,genju jobid huoquminglingzhixingjieguo。

ag试玩master jieshoudaominglinghou,jiangyaozhixingdeminglingfasonggeikehuduan minion。

minion congxiaoxizongxianshangjieshoudaoyaochulidemingling,jiaogei minion._handle_aes chuli。

ag试玩minion._handle_aes faqiyigebendixianchengdiaoyong cmdmod zhixing ls mingling。xianchengzhixingwan ls hou,diaoyong minion._return_pub fangfa,jiangzhixingjieguotongguoxiaoxizongxianfanhuigei master。

master jieshoudaokehuduanfanhuidejieguo,diaoyong master.handle_aes fangfajiangjieguoxiedewenjianzhong。

ag试玩salt.client.localclient.cmd_cli tongguolunxunhuoqu job zhixingjieguo,jiangjieguoshuchudaozhongduan。

SaltStack 的安裝與配置

dui saltstack youleyigechubudelejiezhihou,tongguoshijianlicaozuojinyibulejie saltstack。

yi、anzhuangsalt

ag试玩saltxuyaoepelyuanzhichi,suoyouanzhuangqianxuyaoxiananzhuangepelyuanbao。

1、salt-master

ag试玩# yum -y install salt-master

2、salt-minion

# yum -y install salt-minion

er、peizhisalt

ag试玩1、master(/etc/salt/master)

ag试玩# saltyunxingdeyonghu,yingxiangdaosaltdezhixingquanxian

user: root

ag试玩#saltdeyunxingxiancheng,kaidexianchengyueduoyibanchulidesuduyuekuai,danyibanbuyaochaoguocpudegeshu

worker_threads: 10

# masterdeguanliduankou

publish_port : 4505

ag试玩# mastergenminiondetongxunduankou,yongyuwenjianfuwu,renzheng,jieshoufanhuijieguodeng

ag试玩ret_port : 4506

# ruguozhegemasteryunxingdesalt-syndiclianjiedaoleyigegenggaocengjidemaster,neimezhegecanshuxuyaopeizhichenglianjiedaodezhegegaocengjimasterdejiantingduankou

syndic_master_port : 4506

# zhidingpidwenjianweizhi

ag试玩pidfile: /var/run/salt-master.pid

# saltstack keyikongzhidewenjianxitongdekaishiweizhi

root_dir: /

# rizhiwenjiandizhi

ag试玩log_file: /var/log/salt_master.log

# fenzushezhi

nodegroups:

ag试玩group_all: ‘*’

# salt statezhixingshihoudegenmulu

file_roots:

base:

ag试玩– /etc/salt/

# shezhipillar degenmulu

ag试玩pillar_roots:

base:

ag试玩– /etc/pillar

2、peizhiminion(/etc/salt/minion)

master: mail? #zhekuaidemailzhideshizai/etc/hostswenjianzhongsuodingyidezhujiming

id: node1

3、qidongsalt

ag试玩service salt-master start

ag试玩service salt-minion start

ag试玩# saltstack shishiyongpython2deyuyanbianxie,duipython3dejianrongxingbuhao,qingshiyongpython2dehuanjing

4、renzhengminglingjieshao

salt-key #zhengshuguanli

ag试玩# salt-key –l ? ? ? #chakansuoyouminion-key

# salt-key –a ? ? ?#jieshoumougeminion-key

# salt-key –a ? ? ?#jieshousuoyouminion-key

# salt-key –d ? ? ? #shanchumougeminion-key

# salt-key –d ? ? ? #shanchusuoyouminion-key

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

5、saltminglingjieshao

minglinggeshi:salt [options] [arguments]

ag试玩li:salt \* cmd.run ‘uptime’

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

SaltStack minion匹配方式

ag试玩1、 glob(saltmorendetargetleixing,shiyongshelldetongpeifulaizhidingyigehuoduogeminion id)

# salt \* test.ping huo salt ‘*’ test.ping

2、pcrejianrongzhengzebiaodashi

# salt –e ‘^[m|m]in.[e|o|u]n$’ test.ping

3、subnet(tongguozhidingyigeipv4dizhihuoyigecidrdeipv4ziwang)

ag试玩# salt –s 192.168.0.42 test.ping

ag试玩# salt –s 192.168.0.0/16 test.ping

4、grains(saltkeyitongguocaozuoxitong、cpujiagoujizidingyixinxidengjiqitezhengjinxingtarget minion)

# salt –g ‘os:ubuntu’ test.ping

ag试玩# salt –g ‘os_family:debian’ test.ping

5、pillar(saltzhichitongguopillarshujujinxingpipei)

# salt –i ‘my_val:my_val’ test.ping

ag试玩6、hunhe(compound)

# salt –c ‘web* or g@os:arch’ test.ping

7、jiedianzu(nodegroup)

jiedianzuxuyaoshixiandingyi,peizhifangfaruxia:

# vim /etc/salt/master

nodegroups:

node: ‘l@node1,node2’

# salt -n node test.ping

SaltStack常用模塊

ag试玩1、statusmokuai(chakanxitongxinxi)

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

# salt “*” status.diskstats? ? #chakancipanxinxi

# salt “*” status.meminfo? ? ? #chakanneicunxinxi

# salt “*” status.w? ? ? ? ? ? #wminglingfanhuixinxi

2、chakansuoyoumoduleliebiao

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

ag试玩3、chakanzhidingmoduledesuoyoufunctionfangfa

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

ag试玩4、chakanzhidingmoduleyongfa

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

5、jutimokuaideshiyong(lizi)

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

tongshidaozhidingjiqichakan

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

cmd.runmokuaideshiyong

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

Grains

Static bits of information that a minion collects about the system when the minion first starts.

the grains interface is made available to salt modules and components so that the right salt minion commands are automatically available on the right systems.

以上是官方的解釋,大致意思是說grains是minion第一次啟動的時候采集的靜態數據,可以用在salt的模塊和其他組件中。例如,當os_family的Grain數據為Centos時,則會使用yum工具組件來進行軟件包管理。Grains會在Minion進程啟動時加載,并緩存在內存中。這樣salt-minion進程就無須每次操作都重新檢索系統來獲取Grain,極大的提高了Minion的性能。

1、womenzhelijiandanzuoyigeshuchuceshi,keyikandaominionjiediandeyixiexinxiruxia:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

chakanjutimeiyixiangxinxi

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

2、yingyongchangjing:

grainsdetexing–meiciqidonghuibao、jingtaijuedingletameiyoupillarlinghuo,yaozhidaopillarshisuishikebiande,zhiyaozaimasterduanxiugaileneiyibandouhuilikeshengxiaode。suoyigrainsgengshihezuoyixiejingtaideshuxingzhidecaiji,lirushebeidejiaose(role),cipangeshu(disk_num),caozuoxitongbanbendengzhurucileifeichanggudingdeshuxing。jiandanzongjieqilaigrainsdeyongturuxia:

ag试玩(1),grainskeyizaistatexitongyingyongzhong,yonghupeizhiguanlimokuai。

ag试玩(2),grainskeyizaitargetzhongshiyong,yonglaipipeiminion,biruos,yong-g。

ag试玩(3),grainskeyiyongyuxinxichaxun,grainsbaocunzhushoujidaodekehuduandexinxi。

neimewomenjiukeyidedaoyigedazhidepanduan,ruguonixiangdingyideshuxingzhishijingchangbianhuade,neiqingcaiyongpillar,ruguoshihenguding、buyibiandeneiqingyonggrains。

3、grainsyouxianji

ag试玩grainskeyibaochizaiminionduan、tongguomasterduanxiafadengduogefangshilaifenfa。danbutongdefangfayoubutongdeyouxianjide(youdidaogao):

(1). /etc/salt/grains

ag试玩(2) /etc/salt/minion

(3)./srv/salt/_grains/? masterduan_grainsmuluxia

youxianjishunxuyiciweicunzaizaiminionduan/etc/salt/minionpeizhiwenjianzhongdetongminggrainshuifugai/etc/salt/grainswenjianzhongdezhi,ertongguomasterduan_grainsmuluxiagrainswenjianxiafadezhikeyihuifugaiminionduandesuoyoutongmingzhi。bijiaoaokou,zongzhijide,tongguomasterxiafadegrainsyouxianjishizuigaodeke,/etc/salt/minioncizhi,/etc/salt/grainszuidi(core grainsbudadong,jiubutaolunle,zhegebi/etc/salt/grainshuandi)。

4、grainsdexiafa

ag试玩grainsdexiafadazhikeyifenweilianggesilu:

ag试玩(1)zidingyide(_grains)keyitongguostate.highstate、saltutil.sync_grains、saltutil.sync_all dengfangfapiliangxiafa,qiejisuoyouzai_grainsmuluxiadesuoyouzidingyigrainszhidouhuixiafadaominion,zheshixuedejiaoxun。

ag试玩(2)gudingcunfangzaiminionduanpeizhiwenjianzhong,rugrains、minionwenjianzhong,keyitongguofile managerdefangfaqupiliangxiafa/etc/salt/grainsdengpeizhiwenjianshixiangrainsdepiliangxiafa,dangranleyetongguobiedefangshibazhegewenjianpiliangxiafaxiaqu,doushiokde。

duibi:

ag试玩(1)tongguostate.highstate xiafadegrainshaochushiwuxuzhongqiminionjikeshengxiao,dantongguoxiafa/etc/salt/grainswenjianxiafadegrainszhizebixuzhongqiminionduanfuwucaikeyishengxiao。

ag试玩(2)zidingyide_grainsmeicizaihighstatediaoyongdeshihoujiuhuizidongxiafa、shuaxin,er/etc/salt/grainswenjiandezebuhui。

Pillar

ag试玩zaidaduoshuchangjingzhong,pillardebiaoxianxingweihegrainyizhi,danyougehendadequbieshi:pillarzaimastershangjinxingdingyi,cunzaiyuyigejizhonghuadelujing。pillarshujushiyutedingminionguanliande,yejiushishuomeiyigeminiondouzhinengkandaozijideshuju,suoyipillarkeyiyonglaichuandiminganshuju(zaisaltdeshejizhong,pillarshiyongdulidejiamisession,yeshiweilebaozhengminganshujudeanquanxing)。

pillarkeyiyongzaineixiedifang:

1、minganshuju

lirussh key,jiamizhengshudeng,youyupillarshiyongdulidejiamisession,keyiquebaozhexieminganshujububeiqitaminionkandao。

2、bianliang

keyizaipillarzhongchulipingtaichayixing,biruzhenduibutongdecaozuoxitongshezhiruanjianbaodemingzi,ranhouzaistatezhongyinyong。

3、qitarenheshuju

ag试玩keyizaipillarzhongtianjiarenhexuyaoyongdaodeshuju。birudingyiyonghuheuiddeduiyingguanxi,mniondejiaosedeng。

ag试玩4、yongzaitargettingzhong

pillarkeyiyonglaixuanzeminion,shiyong-ixuanxiang。

dingyipillar:

ag试玩masterpeizhiwenjianzhongdingyi:

ag试玩morenqingkuangxia,masterpeizhiwenjianzhongdesuoyoushujudoutianjiadaopillarzhong,qieduisuoyouminionkeyong。ruguoyaojinyongzheyimorenzhi,keyizaimasterpeizhiwenjianzhongtianjiaruxiashuju,zhongqifuwuhoushengxiao:

ag试玩pillar_opts: false

shiyongslswenjiandingyipillar

pillarshiyongyustatexiangsideslswenjian。pillarwenjianfangzaimasterpeizhiwenjianzhongpillar_rootsdingyidemuluxia。shiliruxia:

ag试玩pillar_roots:

base:

– /srv/pillar

xiamianzheduandaimadingyilebasehuanjingxiadepillarwenjianbaocunzai/srv/pillar/muluxia。yustatexiangsi,pillaryeyoutop file,yeshiyongxiangtongdepipeifangshijiangshujuyingyongdaominionshang。shiliruxia:

# cat /srv/pillar/top.sls:

base:

‘*’:

ag试玩– packages

# cat /srv/pillar/packages.sls:

{% if grains[‘os’] == ‘redhat’ %}

apache: httpd

git: git

{% elif grains[‘os’] == ‘debian’ %}

apache: apache2

git: git-core

{% endif %}

basehuanjingzhongsuoyoudeminiondoujuyoupackageszhongdingyideshuju。pillarcaiyongyufile serverxiangtongdewenjianyingshefangshi,zaibenlizhong,packagesyingshedaowenjian/srv/pillar/packages.sls。zhuyikeyyuvalueyaoyongmaohaojiakonggefenge,meiyoukonggedehuajiangjiexishibai。

ruhezhidaominionyongyouneixiepillarshuju?

zaimastershangxiugaipillarwenjianhou,xuyaoyongyixiaminglingshuaxinminionshangdeshuju:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

ag试玩shiyongpillarhuoquzidingyishuju:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

State

ag试玩jianshu:sls(daibiaosalt statewenjian)shisalt statexitongdehexin。slsmiaoshulexitongdemubiaozhuangtai,yougeshijiandandeshujugoucheng。zhejingchangbeichengzuopeizhiguanli

top.slsshipeizhiguanliderukouwenjian,yiqiedoushicongzhelikaishi,zaimaster zhujishang,morencunfangzai/srv/salt/mulu.

ag试玩top.sls morencong base biaoqiankaishijiexizhixing,xiayijishicaozuodemubiao,keyitongguozhengze,grainmokuai,huofenzuming,laijinxingpipei,zaixiayijishiyaozhixingdestatewenjian,bubaohuankuozhanming。

ag试玩chuangjian/srv/salt/top.sls

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

stateshizhan

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

————廣告時間————

《馬哥Linux云計算及架構師》課程,由知名Linux布道師馬哥創立,經歷了8年的發展,聯合阿里巴巴、唯品會、大眾點評、騰訊、陸金所等大型互聯網一線公司的馬哥課程團隊的工程師進行深度定制開發,課程采用 Centos7.2系統教學,加入了大量實戰案例,授課案例均來自于一線的技術案例。

開課時間:11月06號

相關新聞

AG真人 ag试玩 AG捕鱼下载