# VROPS API Module

_Author: Daniel Tomlinson_

_Team: Capacity & Monitoring_

_Date: October 2019_


See example.py for an example of the following methods.

**Requires**: python version 3.6+

For any bugs create a gitlab issue and give a description of the error and sample code where necessary.

**To jump right in, clone the repo, install the dependencies and run `approach1.py` and `approach2.py` for a quick and easy demonstration.**

If you need VROPS api attributes (rollUpTypes, statKeys etc.) follow the links in [Get stats for VMs](#get-stats-for-vms).


<br/>

<!-- MarkdownTOC  -->

- [Requirements](#requirements)
- [Modules](#modules)
- [Workflow](#workflow)
  * [Get stats for every VM in a cluster](#get-stats-for-every-vm-in-a-cluster)
    + [Approach 1: If you **do not** have a dictionary of name:id pairs for all VMs in a cluster.](#approach-1-if-you-do-not-have-a-dictionary-of-nameid-pairs-for-all-vms-in-a-cluster)
    + [Approach 2: If you have a dictionary of name:id pairs for all VMs from a previous run.](#approach-2-if-you-have-a-dictionary-of-nameid-pairs-for-all-vms-from-a-previous-run)
  * [Usage](#usage)
    + [Authenticate](#authenticate)
    + [Get all clusters](#get-all-clusters)
      - [Pull back clusters](#pull-back-clusters)
      - [Convert to list](#convert-to-list)
    + [Get all hosts](#get-all-hosts)
      - [Pull back hosts](#pull-back-hosts)
      - [Convert to list](#convert-to-list-1)
    + [Get VMs](#get-vms)
      - [Pull back VMs](#pull-back-vms)
      - [Convert to list](#convert-to-list-2)
      - [Save VM dictionary as json for import](#save-vm-dictionary-as-json-for-import)
      - [Load saved dictionary and import](#load-saved-dictionary-and-import)
      - [Load VM information](#load-vm-information)
      - [Set the VM dictionary](#set-the-vm-dictionary)
    + [Get stats for VMs](#get-stats-for-vms)
      - [Get results](#get-results)
      - [Save results to json file on disk](#save-results-to-json-file-on-disk)
    + [Get epoch time](#get-epoch-time)
      - [Get epoch time now](#get-epoch-time-now)
      - [Get epoch time relative to another tie](#get-epoch-time-relative-to-another-tie)

<!-- /MarkdownTOC -->

<br/>

## Requirements

See requirements.txt and install to a virtualenv

```bash
python3 -m venv vrops-api
source vrops-api/bin/activate
pip install -r requirements.txt
```

## Modules

The current working directory has to be added to the system path for python to import a custom module.

```python
import sys
import os
sys.path.append(os.getcwd())
from vropsAPI import vropsAPI
```

## Workflow

The following documents how you can use the module to do common tasks with the VROPs api. We cover pulling VM metrics for each VM in a **cluster.**


### Get stats for every VM in a cluster

To pull back data from VMs you need two things:

1. A dictionary containing the name:id pairs for each VM.
2. A list of these id's to pass to the api call.

Step 1 is optional, but without it you won't have the name of the VM when you query the API.

The following approaches document how to use this module to accomplish this.

There are two approaches to do this using this module.

- The first approach is if you don't have a dictionary containing the name:id pairs for all VMs in the cluster. **Do this if you haven't run the script before.**
- The second approach can skip a lot of the steps and jump straight to querying the api using this dictonary of the name:id pairs from an earlier run. **Do this if you've saved the dictionary from a previous run.**

#### Approach 1: If you **do not** have a dictionary of name:id pairs for all VMs in a cluster.

*see ```approach1.py``` for an example of these steps*

1. Authenticate a session with VROPS api. - [Authenticate](#authenticate)
2. Get all cluster names and choose the one you require. - [Get all clusters](#get-all-clusters)
3. Get all hosts in this cluster + store these **names in a list**. - [Get all hosts](#get-all-hosts)
4. Get all VMs under these hosts and store the **names and id's in a dictionary.** - [Get VMs](#get-vms)
5. *(Optional) Dump this dictionary in a json file to be reused for future runs.* - [Save VM dictionary as json for import](#save-vm-dictionary-as-json-for-import)
6. Get data for these VMs by passing in a list of VM **id's.** - [Get stats for VMs](#get-stats-for-vms)
7. Save output to json. - [Save results to json file on disk](#save-results-to-json-file-on-disk)

#### Approach 2: If you have a dictionary of name:id pairs for all VMs from a previous run.

*see ```approach2.py``` for an example of these steps*

1. Authenticate a session with VROPS api. - [Authenticate](#authenticate)
2. Load this dictionary from a json file, and update the module's dictionary. - [Load saved dictionary and import](#load-saved-dictionary-and-import)
3. Get data for these VMs by passing in a list of VM **id's.** - [Get stats for VMs](#get-stats-for-vms)
4. Save output to json. - [Save results to json file on disk](#save-results-to-json-file-on-disk)

<br>

### Usage

#### Authenticate

Authenticates a session with VROPs and generates a token.

Definition:

```python
@classmethod
def authenticate(
    cls,
    vropsURL: str,
    username: str,
    authSource: str,
    password: str,
    verify: bool = True,
):
  ...
  return cls(vropsURL, authToken, verify)
```

Example:

```python
vrops = vropsAPI.authenticate(
    'https://sc1prapvro01/',
    'svc_splunkVROPS@Group.WilliamHill.PLC',
    'whgroup',
    base64.b64decode(b'UmFjaW5nMjEyMg==').decode(),
    verify=False,
)
```

#### Get all clusters

##### Pull back clusters

To get all clusters you need to do the following two methods: ```vropsAPI.getClusters()``` and ```vropsAPI.getClusterIdentifiers()```

You can then use the ```vropsAPI.getKeysList()``` method to get a list of names without the ids.

Definition:

```python
def getClusters(self):
  ...
  return(self)
```
```python
def getClusterIdentifiers(self):
  ...
  return(self)
```

Example:

```python
vropsAPI.getClusters()
vropsAPI.getClusterIdentifiers()
```

```python
print(vropsAPI.allClusters)
```

Attributes created:
- `vropsAPI.allClusters` - A dictionary containing the name:id pairs of all clusters.

Output:

```json
{
   "SC1PRINNOGEUXCCL01": "00276144-2eda-4f2b-9660-fa9f9cf3cd82",
   "BRSDRGEUX01": "037f8d5e-01d5-411b-afdc-c25868f04a27",
   "BRSDRCTXWN01": "05c2ba9a-d6d4-47aa-8fe5-855b40364625",
   "LD6-BRS-Hosts": "0612327c-d637-4e95-8782-97c97d1e99ed",
   "BRSPRPCMGEUXCCL01": "08b7d0fb-92ee-4cd9-ba7d-96f21965d7a4",
   "LD6DRGEUX01": "84f052a8-8aed-4efb-8c39-8bce0f4a3c54"
}
```

##### Convert to list

Optionally you can convert this to a list of names with

```python
allClustersList = vropsAPI.getKeysList(vropsAPI.allClusters)
print(allClustersList)
```

Output:

```python
[
   "SC1PRINNOGEUXCCL01",
   "BRSDRGEUX01",
   "BRSDRCTXWN01",
   "LD6-BRS-Hosts",
   "BRSPRPCMGEUXCCL01",
   "LD6DRGEUX01"
]
```

<br>

#### Get all hosts

##### Pull back hosts

To get all hosts for cluster ```SC1PRCONTXWHCUXCCL01``` you need to use the following two methods `getHostsFromCluster()` and `getHostIdentifiers()`:

Definition:

```python
def getHostsFromCluster(self, cluster='SC1PRCONTXWHCUXCCL01'):
  ...
  return(self)
```

```python
def getHostIdentifiers(self):
  ...
  return(self)
```

Attributes created:
- `vropsAPI.allHosts`. - A dictionary containing the name:ID pairs for all hosts.

Example:

```python
vropsAPI.getHostsFromCluster(cluster='SC1PRCONTXWHCUXCCL01')
vropsAPI.getHostIdentifiers()
```

```python
print(vropsAPI.allHosts)
```

Output:

```
{
   "sc1hsesx156.prod.williamhill.plc": "15b3ea0c-9f62-4fc2-93b8-d4281196043e",
   "sc1hsesx187.prod.williamhill.plc": "18ecb9d5-842f-4a4b-b43b-f8bbcdd54775",
   "sc1hsesx148.prod.williamhill.plc": "1bdf892a-121e-461e-8ef7-8dd174c4c01a",
   "sc1hsesx155.prod.williamhill.plc": "1ef01a7b-9e61-40b7-8d72-78363352fbfc"
}
```

##### Convert to list

You can convert this to a list of names with:

```python
allHostsList = vropsAPI.getKeysList(vropsAPI.allHosts)
print(allHostsList)
```

Output:

```python
[
   "sc1hsesx156.prod.williamhill.plc",
   "sc1hsesx187.prod.williamhill.plc",
   "sc1hsesx148.prod.williamhill.plc",
   "sc1hsesx155.prod.williamhill.plc",
   "sc1hsesx093.prod.williamhill.plc"
]
```

<br>

#### Get VMs

##### Pull back VMs

To get all VMs for a single host or a list of hosts (all hosts from a cluster) you need to use the following two methods `getVMSFromHost()` and `getVMSIdentifiers()`:

Defintion:

```python
    def getVMSFromHost(self, host: Union(str, list)):
      ...
      return self
```

```python
    def getVMSIdentifiers(self):
      ...
      return self
```

Attributes created:
- `vropsAPI.allVMS` - A dictionary containing the name:ID pairs for all VMs.
- `vropsAPI.totalVMS` - An integer showing the total number of VMs.

Example (pulling back all VMs for all hosts in a cluster:

```python
vropsAPI.getVMSFromHost(allHostsList)
vropsAPI.getVMSIdentifiers()

```

Print total number of VMs

```python
print(vropsAPI.totalVMS)
```

Print VM IDs and names

```python
print(vropsAPI.allVMS)
```

Output:

```python
OrderedDict(
    [
        ('prdxinf21box001', '0981bfe1-b6ba-4a62-bfcc-39892fd038c6'),
        ('prdxcmr21ssb001', '3017e298-67e8-4a5f-bf10-f8c903e3d75f'),
        ('prdxtms14sln014', '35c5a694-2361-44dd-9e5e-893cea16119d'),
        ('prdxtms10sln014', '3b76b4b1-76b3-4fa7-a032-6448a60eded5'),
        ('prdxria01trd003', '458c11be-88ba-4301-aa32-3b748c92a47b')
    ]
)
```

##### Convert to list

**You will need a list of IDs, not names, to pull back data from the API**

You can convert this to a list of names with

```python
allVMSList = vropsAPI.getKeysList(vropsAPI.allVMS)
```

You can convert this to a list of IDs with

```python
allVMSIdList = vropsAPI.getValuesList(vropsAPI.allVMS)
```

##### Save VM dictionary as json for import

You can save this dictionary of VM names:ids for import in future runs.

```python
vropsAPI.saveToDisk(vropsAPI.allVMS, indent=4, filePrefix='vm-export')
```

##### Load saved dictionary and import

You can do the following if you already have a list of VMs for the cluster/hosts in question. This means you can import, and go straight to importing metrics, rather than querying the API for clusters/hosts each time.


Assuming file on disk is:

`vm-export-21-10-2019_09-12-53.json`

##### Load VM information

Load this .json into python as a dictionary

```python
import json
with open('vm-export-21-10-2019_09-12-53.json', 'r') as file:
    myFile = json.load(file)
```

##### Set the VM dictionary

```python
vropsAPI.allVMS = myFile
```

You should see

```
Successfully imported the dictionary.
```

You now have the results from a previous run added to your instance.


<br>

#### Get stats for VMs

Uses post to pull back data from VROPs for a list of VMs

VROPs needs epoch time in ms format without the decimal (e.g 1571607237000)

You can use the following to pull data back for VMs:

Definition:

```python
def getStatsFromVMS(
    self,
    begin: int,
    end: int,
    intervalType: str,
    intervalQuantifier: str,
    rollUpType: str,
    resourceId: list,
    statKey: Union[str, list],
):
  ...
  return self
```

Attributes:

- Attributes:
    - begin: epoch time for start
    - end: epoch time for end
    - intervalType: `<ns3:intervalType>` [see documentation](https://sc1prapvro01/suite-api/docs/rest/models.html#repr-1190589417)
    - intervalQuantifier: int
    - rollUpType: `<ns3:rollUpType>` [see documentation](https://sc1prapvro01/suite-api/-docs/rest/models.html#repr-1735704374)
    - resourceId: list of vrops resourceIds (**not Names!**)
    - statKey: list containing vrops StatKey metrics [see documentation](https://docs.vmware.com/en/vRealize-Operations-Manager/6.7/com.vmware.vcom.metrics.doc/GUID-1322F5A4-DA1D-481F-BBEA-99B228E96AF2.html)

##### Get results

Give 5 min avg for CPU usage % and number of cores assigned for past 10 minutes.

*for relative time go back 11 minutes to ensure we have a complete 10 minute interval.*

**You need to pass in a list for resourceId. If you have a single VM this should be a list with one entry only.**

Example, passing in allVMSIdList from before:

```python
vropsAPI.getStatsFromVMS(
    begin=vropsAPI.epochRelativeTime(vropsAPI.epochNow, minutes=-11),
    end=vropsAPI.epochNow,
    intervalType='MINUTES',
    intervalQuantifier='5',
    rollUpType='AVG',
    resourceId=list(allVMSIdList),
    statKey=['cpu|usage_average', 'config|hardware|num_Cpu'],
)
```

Attributes returned:

- `vropsAPI.allVMS` - A string containing the raw output from the VROPS api. You should use the method `vropsAPI.exportVMData()` for a more workable format.

Output:

prdxmes21zoo004 has 2 vCPUs and an average CPU utilisation of 1.2% for the past 10 minutes.

```json
{
    "values": [
        {
            "name": "prdxmes21zoo004",
            "resourceId": "528eb4db-813b-45a1-a456-ce5b44751da6",
            "stat-list": {
                "stat": [
                    {
                        "timestamps": [
                            1571608439999,
                            1571608739999
                        ],
                        "statKey": {
                            "key": "config|hardware|num_Cpu"
                        },
                        "rollUpType": "AVG",
                        "intervalUnit": {
                            "quantifier": 5,
                            "intervalType": "MINUTES"
                        },
                        "data": [
                            2.0,
                            2.0
                        ]
                    },
                    {
                        "timestamps": [
                            1571608439999,
                            1571608739999
                        ],
                        "statKey": {
                            "key": "cpu|usage_average"
                        },
                        "rollUpType": "AVG",
                        "intervalUnit": {
                            "quantifier": 5,
                            "intervalType": "MINUTES"
                        },
                        "data": [
                            1.218666672706604,
                            1.2406666278839111
                        ]
                    }
                ]
            }
        }
    ]
}
```

##### Save results to json file on disk

To save results to disk you should use `vropsAPI.exportVMData()` then use `vropsAPI.saveToDisk()`.

The method `vropsAPI.exportVMData()` will format the raw output from VROPS and create the attribute `vropsAPI.export` which is a list that contains a json formatted string for each datapoint you requested. You can pass this list to ```vropsAPI.saveToDisk(breakLine=True)``` **(you should include `breakLine=True`)** which will save each item as a json formatted dictionary.

Definition:

```python
def exportVMData(self):
  ...
  return self
```

Attributes returned:

- `vropsAPI.export` - A python list where each entry is a json string containing the dictionary representation for each datapoint requested.


Example:

```python
vropsAPI.exportVMData()
vropsAPI.saveToDisk(
  vropsAPI.export, indent=4, filePrefix='SC1PRCONTXWHCUXCCL01', breakLine = True
)
```

Output:

```json
{
    "name": "prdxmes14sln010",
    "timestamp": 1571841899999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}
{
    "name": "prdxmes14sln010",
    "timestamp": 1571842199999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}
{
    "name": "prdxmes14sln010",
    "timestamp": 1571842499999,
    "value": 4.0,
    "statKey": "config|hardware|num_Cpu",
    "rollUpType": "AVG",
    "intervalUnit": 5
}
```

<br>

#### Get epoch time

The following method is built in to provide easy generation of epoch times. This functions similar to Splunks relative_time and time snapping.

##### Get epoch time now

- `vropsAPI.epochNow` - The following attribute is available instance wide to get the current time in Epoch correctly formatted for the API.



##### Get epoch time relative to another tie

Get epoch time relative to another time:
1. Can go back N hours/minutes etc.
2. Can set the hour/minute etc. to a specified value (snapping)

Definition:

```python
def epochRelativeTime(
    epochTime: int,
    year: int = datetime.now().year,
    month: int = datetime.now().month,
    day: int = datetime.now().day,
    hour: int = datetime.now().hour,
    minute: int = datetime.now().minute,
    second: int = datetime.now().second,
    years: int = 0,
    months: int = 0,
    days: int = 0,
    hours: int = 0,
    minutes: int = 0,
    seconds: int = 0,
) -> int:
  ...
  return relativeTime
```

Attributes:

```
**kwargs:
    epochTime: epoch time for start

    year: int = datetime.now().year    # set year
    month: int = datetime.now().month    # set month
    day: int = datetime.now().day    # set day
    hour: int = datetime.now().hour    # set hour
    minute: int = datetime.now().minute    # set minute
    second: int = datetime.now().second    # set second

    years: int = 0    # go back/forward N years
    months: int = 0    # go back/forward N months
    days: int = 0    # go back/forward N days
    hours: int = 0    # go back/forward N hours
    minutes: int = 0    # go back/forward N minutes
    seconds: int = 0    # go back/forward N seconds
```

Examples:

Get epoch 5 minutes ago.

```python
vropsAPI.epochRelativeTime(vropsAPI.epochNow, minutes=-5)
```

Get epoch at start of current hour.
```python
vropsAPI.epochRelativeTime(
    vropsAPI.epochNow,
    hour=0,
    minute=0,
    second=0,
)
```

Get epoch 1 week ago at start of day.
```python
vropsAPI.epochRelativeTime(
    vropsAPI.epochNow,
    days=-7
    hour=0,
    minute=0,
    second=0,
)
```
<br>