用python获取yarn资源fair-scheduler.xml里的队列信息

yarn资源队列信息一般都保存在一个xml文件里。具体路径可以在yarn-site.xml中查找。
具体格式如下：

<property><name>yarn.scheduler.fair.allocation.filename><value>/etc/hadoop/conf/fair-scheduler.xmlvalue>property>

fair-scheduler.xml格式如下：

<allocations><queue name="root"><aclSubmitApps>aclSubmitApps><aclAdministerApps>aclAdministerApps><defaultFairSharePreemptionTimeout>600defaultFairSharePreemptionTimeout><defaultFairSharePreemptionThreshold>0.8defaultFairSharePreemptionThreshold>><queue name="production"><minResources>8192mb,8vcoresminResources><maxResources>419840mb,125vcoresmaxResources><maxRunningApps>60maxRunningApps><schedulingMode>fairschedulingMode><weight>7.5weight><aclSubmitApps>*aclSubmitApps><aclAdministerApps>productionaclAdministerApps>queue><queue name="spark"><minResources>8192mb,8vcoresminResources><maxResources>376480mb,110vcoresmaxResources><maxRunningApps>50maxRunningApps><schedulingMode>fairschedulingMode><weight>1weight><aclSubmitApps>*aclSubmitApps><aclAdministerApps>sparkaclAdministerApps>queue><queue name="default"><minResources>8192mb,8vcoresminResources><maxResources>202400mb,20vcoresmaxResources><maxRunningApps>20maxRunningApps><schedulingMode>FIFOschedulingMode><weight>0.5weight><aclSubmitApps>*aclSubmitApps><aclAdministerApps>*aclAdministerApps>queue><queue name="hbase"><minResources>8192mb,8vcoresminResources><maxResources>69120mb,16vcoresmaxResources><maxRunningApps>20maxRunningApps><schedulingMode>fairschedulingMode><aclSubmitApps>*aclSubmitApps><weight>1weight><aclAdministerApps>hbaseaclAdministerApps>queue>queue>

python解析xml文件可以直接导入模块xml.sax

import xml.saxclass YarnQueue(xml.sax.ContenHandler):def __init__(self):self.CurrentData = ""self.minResources = ""self.maxResources =""self.aclSubmitApps =""self.aclAdminsterApps = ""#获取每个队列的队列名    def get_queuename(self, tag, attributes):self.CurrentData = tagif tag == "queue":name = attributes["name"]#区分每个队列print("[yarn_{}]".format(name))print("name = root.{}".format(name))def get_queuevlu(self, tag):if self.CurrentData == "minResources":if len(self.minResources.split()) == 4:min_cores = self.minResources.split()[2]min_memory = int(self.minResources.split()[0])/1024print("min_cores = {}".format(min_cores))#int()是为了不带小数点print("min_memory = {}g".format(int(min_memory)))elif self.CurrentData == "maxResources":if len(self.maxResources.split()) == 4:max_cores = self.maxResources.split()[2]max_memory = int(self.maxResources.split()[0])/1024print("max_cores = {}".format(max_cores))#int()是为了不带小数点print("max_memory = {}g".format(int(max_memory)))elif self.CurrentData == "aclSubmitApps":print("aclSubmitApps = {}".format(aclSubmitApps))elif self.CurrentData == "aclAdminsterApps":print("aclAdminsterApps = {}".format(aclAdminsterApps))self.CurrentData = ""#处理内容def queue_res(self, content):if self.CurrentData == "minResources":self.minResources = contentelif self.CurrentData == "maxResources":self.maxResources = contentelif self.CurrentData == "aclSubmitApps":self.aclSubmitApps = contentelif self.CurrentData == "aclAdminsterApps":self.aclAdminsterApps = content
if (__name__ == "__main__"):#创建一个xml readerparser = xml.sax.make_parser()parser.setFeature(xml.sax.handler.feature_namespaces, 0)#重写ContenHandlerHandler = YarnQueue()parser.setContentHandler(Handler)parser.parse("fair-scheduler.xml")

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！