dtomlinson/dtomlinson-cv
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
Files
a285fad1ee237c79ae50a405a6193127ad908add
dtomlinson-cv/vrops-api/readme.md
dtomlinson a285fad1ee adding projects to repo
2019-11-04 14:38:31 +00:00

18 KiB
Raw Blame History

VROPS API Module

Author: Daniel Tomlinson

Team: Capacity & Monitoring

Date: October 2019

See example.py for an example of the following methods.

Requires: python version 3.6+

For any bugs create a gitlab issue and give a description of the error and sample code where necessary.

To jump right in, clone the repo, install the dependencies and run approach1.py and approach2.py for a quick and easy demonstration.

If you need VROPS api attributes (rollUpTypes, statKeys etc.) follow the links in Get stats for VMs.



Requirements

See requirements.txt and install to a virtualenv

python3 -m venv vrops-api
source vrops-api/bin/activate
pip install -r requirements.txt

Modules

The current working directory has to be added to the system path for python to import a custom module.

import sys
import os
sys.path.append(os.getcwd())
from vropsAPI import vropsAPI

Workflow

The following documents how you can use the module to do common tasks with the VROPs api. We cover pulling VM metrics for each VM in a cluster.

Get stats for every VM in a cluster

To pull back data from VMs you need two things:

  1. A dictionary containing the name:id pairs for each VM.
  2. A list of these id's to pass to the api call.

Step 1 is optional, but without it you won't have the name of the VM when you query the API.

The following approaches document how to use this module to accomplish this.

There are two approaches to do this using this module.

  • The first approach is if you don't have a dictionary containing the name:id pairs for all VMs in the cluster. Do this if you haven't run the script before.
  • The second approach can skip a lot of the steps and jump straight to querying the api using this dictonary of the name:id pairs from an earlier run. Do this if you've saved the dictionary from a previous run.

Approach 1: If you do not have a dictionary of name:id pairs for all VMs in a cluster.

see approach1.py for an example of these steps

  1. Authenticate a session with VROPS api. - Authenticate
  2. Get all cluster names and choose the one you require. - Get all clusters
  3. Get all hosts in this cluster + store these names in a list. - Get all hosts
  4. Get all VMs under these hosts and store the names and id's in a dictionary. - Get VMs
  5. (Optional) Dump this dictionary in a json file to be reused for future runs. - Save VM dictionary as json for import
  6. Get data for these VMs by passing in a list of VM id's. - Get stats for VMs
  7. Save output to json. - Save results to json file on disk

Approach 2: If you have a dictionary of name:id pairs for all VMs from a previous run.

see approach2.py for an example of these steps

  1. Authenticate a session with VROPS api. - Authenticate
  2. Load this dictionary from a json file, and update the module's dictionary. - Load saved dictionary and import
  3. Get data for these VMs by passing in a list of VM id's. - Get stats for VMs
  4. Save output to json. - Save results to json file on disk

Usage

Authenticate

Authenticates a session with VROPs and generates a token.

Definition:

@classmethod
def authenticate(
    cls,
    vropsURL: str,
    username: str,
    authSource: str,
    password: str,
    verify: bool = True,
):
  ...
  return cls(vropsURL, authToken, verify)

Example:

vrops = vropsAPI.authenticate(
    'https://sc1prapvro01/',
    'svc_splunkVROPS@Group.WilliamHill.PLC',
    'whgroup',
    base64.b64decode(b'UmFjaW5nMjEyMg==').decode(),
    verify=False,
)

Get all clusters

Pull back clusters

To get all clusters you need to do the following two methods: vropsAPI.getClusters() and vropsAPI.getClusterIdentifiers()

You can then use the vropsAPI.getKeysList() method to get a list of names without the ids.

Definition:

def getClusters(self):
  ...
  return(self)

def getClusterIdentifiers(self):
  ...
  return(self)

Example:

vropsAPI.getClusters()
vropsAPI.getClusterIdentifiers()

print(vropsAPI.allClusters)

Attributes created:

  • vropsAPI.allClusters - A dictionary containing the name:id pairs of all clusters.

Output:

{
   "SC1PRINNOGEUXCCL01": "00276144-2eda-4f2b-9660-fa9f9cf3cd82",
   "BRSDRGEUX01": "037f8d5e-01d5-411b-afdc-c25868f04a27",
   "BRSDRCTXWN01": "05c2ba9a-d6d4-47aa-8fe5-855b40364625",
   "LD6-BRS-Hosts": "0612327c-d637-4e95-8782-97c97d1e99ed",
   "BRSPRPCMGEUXCCL01": "08b7d0fb-92ee-4cd9-ba7d-96f21965d7a4",
   "LD6DRGEUX01": "84f052a8-8aed-4efb-8c39-8bce0f4a3c54"
}
Convert to list

Optionally you can convert this to a list of names with

allClustersList = vropsAPI.getKeysList(vropsAPI.allClusters)
print(allClustersList)

Output:

[
   "SC1PRINNOGEUXCCL01",
   "BRSDRGEUX01",
   "BRSDRCTXWN01",
   "LD6-BRS-Hosts",
   "BRSPRPCMGEUXCCL01",
   "LD6DRGEUX01"
]

Get all hosts

Pull back hosts

To get all hosts for cluster SC1PRCONTXWHCUXCCL01 you need to use the following two methods getHostsFromCluster() and getHostIdentifiers():

Definition:

def getHostsFromCluster(self, cluster='SC1PRCONTXWHCUXCCL01'):
  ...
  return(self)

def getHostIdentifiers(self):
  ...
  return(self)

Attributes created:

  • vropsAPI.allHosts. - A dictionary containing the name:ID pairs for all hosts.

Example:

vropsAPI.getHostsFromCluster(cluster='SC1PRCONTXWHCUXCCL01')
vropsAPI.getHostIdentifiers()

print(vropsAPI.allHosts)

Output:

{
   "sc1hsesx156.prod.williamhill.plc": "15b3ea0c-9f62-4fc2-93b8-d4281196043e",
   "sc1hsesx187.prod.williamhill.plc": "18ecb9d5-842f-4a4b-b43b-f8bbcdd54775",
   "sc1hsesx148.prod.williamhill.plc": "1bdf892a-121e-461e-8ef7-8dd174c4c01a",
   "sc1hsesx155.prod.williamhill.plc": "1ef01a7b-9e61-40b7-8d72-78363352fbfc"
}
Convert to list

You can convert this to a list of names with:

allHostsList = vropsAPI.getKeysList(vropsAPI.allHosts)
print(allHostsList)

Output:

[
   "sc1hsesx156.prod.williamhill.plc",
   "sc1hsesx187.prod.williamhill.plc",
   "sc1hsesx148.prod.williamhill.plc",
   "sc1hsesx155.prod.williamhill.plc",
   "sc1hsesx093.prod.williamhill.plc"
]

Get VMs

Pull back VMs

To get all VMs for a single host or a list of hosts (all hosts from a cluster) you need to use the following two methods getVMSFromHost() and getVMSIdentifiers():

Defintion:

    def getVMSFromHost(self, host: Union(str, list)):
      ...
      return self

    def getVMSIdentifiers(self):
      ...
      return self

Attributes created:

  • vropsAPI.allVMS - A dictionary containing the name:ID pairs for all VMs.
  • vropsAPI.totalVMS - An integer showing the total number of VMs.

Example (pulling back all VMs for all hosts in a cluster:

vropsAPI.getVMSFromHost(allHostsList)
vropsAPI.getVMSIdentifiers()

Print total number of VMs

print(vropsAPI.totalVMS)

Print VM IDs and names

print(vropsAPI.allVMS)

Output:

OrderedDict(
    [
        ('prdxinf21box001', '0981bfe1-b6ba-4a62-bfcc-39892fd038c6'),
        ('prdxcmr21ssb001', '3017e298-67e8-4a5f-bf10-f8c903e3d75f'),
        ('prdxtms14sln014', '35c5a694-2361-44dd-9e5e-893cea16119d'),
        ('prdxtms10sln014', '3b76b4b1-76b3-4fa7-a032-6448a60eded5'),
        ('prdxria01trd003', '458c11be-88ba-4301-aa32-3b748c92a47b')
    ]
)
Convert to list

You will need a list of IDs, not names, to pull back data from the API

You can convert this to a list of names with

allVMSList = vropsAPI.getKeysList(vropsAPI.allVMS)

You can convert this to a list of IDs with

allVMSIdList = vropsAPI.getValuesList(vropsAPI.allVMS)
Save VM dictionary as json for import

You can save this dictionary of VM names:ids for import in future runs.

vropsAPI.saveToDisk(vropsAPI.allVMS, indent=4, filePrefix='vm-export')
Load saved dictionary and import

You can do the following if you already have a list of VMs for the cluster/hosts in question. This means you can import, and go straight to importing metrics, rather than querying the API for clusters/hosts each time.

Assuming file on disk is:

vm-export-21-10-2019_09-12-53.json

Load VM information

Load this .json into python as a dictionary

import json
with open('vm-export-21-10-2019_09-12-53.json', 'r') as file:
    myFile = json.load(file)
Set the VM dictionary
vropsAPI.allVMS = myFile

You should see

Successfully imported the dictionary.

You now have the results from a previous run added to your instance.


Get stats for VMs

Uses post to pull back data from VROPs for a list of VMs

VROPs needs epoch time in ms format without the decimal (e.g 1571607237000)

You can use the following to pull data back for VMs:

Definition:

def getStatsFromVMS(
    self,
    begin: int,
    end: int,
    intervalType: str,
    intervalQuantifier: str,
    rollUpType: str,
    resourceId: list,
    statKey: Union[str, list],
):
  ...
  return self

Attributes:

  • Attributes:
    • begin: epoch time for start
    • end: epoch time for end
    • intervalType: <ns3:intervalType> see documentation
    • intervalQuantifier: int
    • rollUpType: <ns3:rollUpType> see documentation
    • resourceId: list of vrops resourceIds (not Names!)
    • statKey: list containing vrops StatKey metrics see documentation
Get results

Give 5 min avg for CPU usage % and number of cores assigned for past 10 minutes.

for relative time go back 11 minutes to ensure we have a complete 10 minute interval.

You need to pass in a list for resourceId. If you have a single VM this should be a list with one entry only.

Example, passing in allVMSIdList from before:

vropsAPI.getStatsFromVMS(
    begin=vropsAPI.epochRelativeTime(vropsAPI.epochNow, minutes=-11),
    end=vropsAPI.epochNow,
    intervalType='MINUTES',
    intervalQuantifier='5',
    rollUpType='AVG',
    resourceId=list(allVMSIdList),
    statKey=['cpu|usage_average', 'config|hardware|num_Cpu'],
)

Attributes returned:

  • vropsAPI.allVMS - A string containing the raw output from the VROPS api. You should use the method vropsAPI.exportVMData() for a more workable format.

Output:

prdxmes21zoo004 has 2 vCPUs and an average CPU utilisation of 1.2% for the past 10 minutes.

{
    "values": [
        {
            "name": "prdxmes21zoo004",
            "resourceId": "528eb4db-813b-45a1-a456-ce5b44751da6",
            "stat-list": {
                "stat": [
                    {
                        "timestamps": [
                            1571608439999,
                            1571608739999
                        ],
                        "statKey": {
                            "key": "config|hardware|num_Cpu"
                        },
                        "rollUpType": "AVG",
                        "intervalUnit": {
                            "quantifier": 5,
                            "intervalType": "MINUTES"
                        },
                        "data": [
                            2.0,
                            2.0
                        ]
                    },
                    {
                        "timestamps": [
                            1571608439999,
                            1571608739999
                        ],
                        "statKey": {
                            "key": "cpu|usage_average"
                        },
                        "rollUpType": "AVG",
                        "intervalUnit": {
                            "quantifier": 5,
                            "intervalType": "MINUTES"
                        },
                        "data": [
                            1.218666672706604,
                            1.2406666278839111
                        ]
                    }
                ]
            }
        }
    ]
}
Save results to json file on disk

To save results to disk you should use vropsAPI.exportVMData() then use vropsAPI.saveToDisk().

The method vropsAPI.exportVMData() will format the raw output from VROPS and create the attribute vropsAPI.export which is a list that contains a json formatted string for each datapoint you requested. You can pass this list to vropsAPI.saveToDisk(breakLine=True) (you should include breakLine=True) which will save each item as a json formatted dictionary.

Definition:

def exportVMData(self):
  ...
  return self

Attributes returned:

  • vropsAPI.export - A python list where each entry is a json string containing the dictionary representation for each datapoint requested.

Example:

vropsAPI.exportVMData()
vropsAPI.saveToDisk(
  vropsAPI.export, indent=4, filePrefix='SC1PRCONTXWHCUXCCL01', breakLine = True
)

Output:

{
    "name": "prdxmes14sln010",
    "timestamp": 1571841899999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}
{
    "name": "prdxmes14sln010",
    "timestamp": 1571842199999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}
{
    "name": "prdxmes14sln010",
    "timestamp": 1571842499999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}

Get epoch time

The following method is built in to provide easy generation of epoch times. This functions similar to Splunks relative_time and time snapping.

Get epoch time now
  • vropsAPI.epochNow - The following attribute is available instance wide to get the current time in Epoch correctly formatted for the API.
Get epoch time relative to another tie

Get epoch time relative to another time:

  1. Can go back N hours/minutes etc.
  2. Can set the hour/minute etc. to a specified value (snapping)

Definition:

def epochRelativeTime(
    epochTime: int,
    year: int = datetime.now().year,
    month: int = datetime.now().month,
    day: int = datetime.now().day,
    hour: int = datetime.now().hour,
    minute: int = datetime.now().minute,
    second: int = datetime.now().second,
    years: int = 0,
    months: int = 0,
    days: int = 0,
    hours: int = 0,
    minutes: int = 0,
    seconds: int = 0,
) -> int:
  ...
  return relativeTime

Attributes:

**kwargs:
    epochTime: epoch time for start

    year: int = datetime.now().year    # set year
    month: int = datetime.now().month    # set month
    day: int = datetime.now().day    # set day
    hour: int = datetime.now().hour    # set hour
    minute: int = datetime.now().minute    # set minute
    second: int = datetime.now().second    # set second

    years: int = 0    # go back/forward N years
    months: int = 0    # go back/forward N months
    days: int = 0    # go back/forward N days
    hours: int = 0    # go back/forward N hours
    minutes: int = 0    # go back/forward N minutes
    seconds: int = 0    # go back/forward N seconds

Examples:

Get epoch 5 minutes ago.

vropsAPI.epochRelativeTime(vropsAPI.epochNow, minutes=-5)

Get epoch at start of current hour.

vropsAPI.epochRelativeTime(
    vropsAPI.epochNow,
    hour=0,
    minute=0,
    second=0,
)

Get epoch 1 week ago at start of day.

vropsAPI.epochRelativeTime(
    vropsAPI.epochNow,
    days=-7
    hour=0,
    minute=0,
    second=0,
)

Reference in New Issue View Git Blame Copy Permalink