How to extend NABox with Capacity information without having OCUM - my first python

Monitoring a Netapp is important, the "unofficial" Monitoring NAbox is installed like a slice of cake.

If you never heard about it - check it out here .. https://nabox.org/

But it does not collect the "capacity" information of his objects (volumes and aggregates) without the help of a OCUM (or Active IQ Unified Manager) . 

Instead of installing another Appliance you can collect the needed data from Netapp Systems running Ontap 9.6+ by NAbox itself per REST-API.

First create a authentification certificate (which is usable for Powershell too, see my next post) : 

Log into the NABox with SSH - (root:NetappGrafanaVA are the default login credentials). 

mkdir /opt/myontapcol/

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /opt/myontapcol/ontapkey.key -out /opt/myontapcol/ontap.pem -subj "/C=DE/ST=WONDERLAND/L=EMERALDCITY/O=IT/CN=nabox"

BE AWARE OF "CN=nabox" this will be the user which we configure in  the next step.

cat /opt/myontapcol/ontap.pem

connect with a 2nd SSH Shell to the Netapp and run this command, take the output from above into clipboard, you need to paste it.

security certificate install -type client-ca -vserver mycluster

it sometimes needs 3 "enter".

security ssl modify -client-enabled true -vserver mycluster

security login create -user-or-group-name nabox -application http -authentication-method cert -role readonly -vserver mycluster

your netapp answers now REST API requests,

You can test it on the nabox shell with

curl -s --key /opt/myontapcol/ontap.key --cert /opt/myontapcol/ontap.pem -k https://ontap-a.acme.corp/api/storage/aggregates

Now lets a script feed the NABox with data, except the python extension jsonpickle the nabox have all neccesary things allready installed.

pip install -U jsonpickle

I uploaded the script here .. naboxcapacol unzip the file open up the naboxcapacol.py file with notepad.

[UPDATE - Newer NA Boxes do not use the port 2004 and use 2013 for pickles counter - find the config file with ps -au|grep carbon and grep -A 5 ^.pick /etc/go-carbon/go-carbon.conf  ]

Copy the script into the file and save it.

nano /opt/myontapcol/ontapcapacol.py

#make it executable

chmod +x /opt/myontapcol/ontapcapacol.py

#and let it run every 5min

crontab -e

*/5 * * * * /opt/myontapcol/ontapcapacol.py >> ~/cron.log 2>&1

# after 1h take a look for some of the data 

https://nabox/graphite/?width=800&lineMode=connected&showTarget=mgtechhead.*.*.aggregates.*.volumes.*.used&height=600&target=mgtechhead.*.*.aggregates.*.volumes.*.used

You can now import my "simple" capacity dashboard ontapcapacol.json from the zip or just create your own. I created this for a customer who is not using so much qtrees.

The dashboards which came with harvest are not connecting volume and aggregate together, if you want to have the data here you have to alter the script in a way that it pumps the data to this paths.

netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_total
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/snapshot_reserve_total
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/snapshot_used_percent
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_avail
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_daily_growth_rate
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/overwrite_reserve_total
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/snapshot_reserve_avail
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_used_percent
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/overwrite_reserve_used
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/overwrite_reserve_avail
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/actual_volume_size
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_used_per_day
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/snapshot_reserve_used
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/total
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/afs_used
netapp/capacity/$Group/$Cluster/svm/$SVM/vol/$Volume/quota_committed_space
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/compression_space_savings
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/dedupe_space_savings
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/snapshot_reserve_total
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/size_used
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/size_used_per_day
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/size_total
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/size_available
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/daily_growth_rate
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/snapshot_reserve_avail
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/compression_space_savings_percent
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/dedupe_space_savings_percent
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/space_total_committed
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/size_used_percent
netapp/capacity/$Group/$Cluster/node/$Node/aggr/aggr_$Aggregate/snapshot_reserve_used

Known issue - if you move a volume you have to delete the counters 

rm /opt/graphite/storage/whisper/mgtechhead/*/*/aggregates.[former_aggregate].volumes.[volume_name]/*
rmdir /opt/graphite/storage/whisper/mgtechhead/*/*/aggregates.[former_aggregate].volumes.[volume_name]

Add a Netapp LUN as VM Datastore per Powershell

You can speak Powershell to Netapp and VMware, so adding a LUN as Datastore should not a problem, despite the fact that Netapp Powershell command get-nclun does not deliver the worldwidename of the lun per default.

But according this KB from Netapp a LUN NAA is just a prefix + serial ascii to hex converted.

https://kb.netapp.com/app/answers/answer_view/a_id/1033595/~/how-to-match-a-luns-naa-number-to-its-serial-number-

This is "stupid" adding all LUNs as Datastore using the filename of the LUN as Suffix.

Get-NcLun|select Node,Vserver,Path,
 @{N='NAA';E={"naa.600a0980$(-join ($_.SerialNumber.ToCharArray()|%{'{0:x}' -f $([byte][char]$_)}))"}}|%{
  get-vmhost esxserver|New-Datastore -Vmfs -Name "VMFS_$($_.Path.Split('/')[2])" -Path $_.NAA
  }

Maybe better declaring a function to make Code better readable ...

function Convert-ASCIItoHEX_STRING ([string]$inputstring){
return $(-join ($inputstring.ToCharArray()|%{'{0:x}' -f $([byte][char]$_)}))
}

Get-NcLun|select Node,Vserver,Path,
 @{N='NAA';E={"naa.600a0980$(Convert-ASCIItoHEX_STRING $_.SerialNumber)"}}|%{
  get-vmhost esxserver|New-Datastore -Vmfs -Name "VMFS_$($_.Path.Split('/')[2])" -Path $_.NAA
  }

I hope this comes handy sometimes ..

How to recover access to a Brocade SAN switch after password dementia.

There are already too much blog posts around ? Most of them just talk about going onsite - using serial cable or "back door" user which are mostly disabled.

When the default password 'fibranne' for the root user did not help you, but you have still access to another switch in the fabric, this ist the way to push the password database from one switch you can access to the one you locked yourself out.

distribute -p PWD -d [#SwitchID]

#SwitchID is the decimal number in the first row in fabricshow output.

After this you can logon with the same password you know from switch A.

This is also a fine way to prevent to setup every password for admin, user, factory etc. by hand. Just setup one switch in the fabric properly and "push" the settings to each - or to configure fresh passwords FAST in a environment.

Get Fibrechannel WWPNs and alot other HW Info from Fujitsu iRMC4 per Powershell

Pssst .. wanna get ALL the cool hardware info you see on an iRMC4 WebGui in a Powershell object ?
If you need even the fibrechannel stuff, you won't get it by REST,CIM or WSMAN - i deep-dived some days into all kind of documentations, wasted alot of time in enumerating and finally gave up, the only way seems to be get it from the WebGui - shame on Fujitsu - DELL do it better, see my next post.

This quick-n-dirty shit worked fine for me for PRIMERGY RX2540 M1, PRIMERGY RX2540 M2, PRIMERGY RX4770 M2 and PRIMERGY RX300 S8 - nice for report all MAC, WWPNs, Serials, Firmware etc ... if you use it let me know.

function Get-iRMC_HWInfo($irmcname,$irmcuser,$irmcpw)
{
    $rsacred=New-Object System.Management.Automation.PsCredential($irmcuser,$(ConvertTo-SecureString -String $irmcpw -AsPlainText -force)) 
    $body = @{APPLY = 99;P99='Login'}
    $irmcwebreq=Invoke-WebRequest "http://$irmcname/login" -SessionVariable irmcsession -Credential $rsacred -Method Post -Body $body
    $hwinfo=$($irmcwebreq.AllElements|?{$_.TagName -eq 'TR'}|?{$_.innerText -match '^System Type|^Serial|^System GUID|^System Name|^System\ O/S'}).innerText    
    $networkinventorylink=$($irmcwebreq.Links|?{$_.innerHTML -eq 'Network Inventory'}).href
    $irmcwebreq2=Invoke-WebRequest "http://$irmcname/$networkinventorylink" -WebSession $irmcsession 
    $tabellen=$irmcwebreq2.ParsedHtml.getElementsByTagName("TABLE")
    $nictabelle=$tabellen|?{$_.summary -match 'Ethernet'}
    $fctabelle=$tabellen|?{$_.summary -match 'Fibre'}
    $nicports=@()
    foreach($datarow in $nictabelle.rows){
        if($datarow.cells[0].tagName -eq "TD"){
            $nicports+=@{Enabled=$datarow.cells[0].innerHTML -match 'ok.gif';
                SlotID=$datarow.cells[1].innerText;
                FunctionID=$datarow.cells[2].innerText;
                PortID=$datarow.cells[3].innerText;
                Firmware=$datarow.cells[5].innerText;
                OpROM=$datarow.cells[6].innerText;
                Interface=$datarow.cells[7].innerText;
                VenID=$datarow.cells[9].innerText;
                DevID=$datarow.cells[10].innerText;
                SubVenID=$datarow.cells[11].innerText;
                SubDevID=$datarow.cells[12].innerText;
                MAC=$datarow.cells[14].innerText
            }
        }
    }

    $fcports=@()
    foreach($datarow in $fctabelle.rows){
        if($datarow.cells[0].tagName -eq "TD"){
            $fcports+=@{Enabled=$datarow.cells[0].innerHTML -match 'ok.gif';
                SlotID=$datarow.cells[1].innerText;
                FunctionID=$datarow.cells[2].innerText;
                PortID=$datarow.cells[3].innerText;
                Firmware=$datarow.cells[5].innerText;
                OpROM=$datarow.cells[6].innerText;
                Interface=$datarow.cells[7].innerText;
                VenID=$datarow.cells[9].innerText;
                DevID=$datarow.cells[10].innerText;
                SubVenID=$datarow.cells[11].innerText;
                SubDevID=$datarow.cells[12].innerText;
                WWNN=$datarow.cells[14].innerText;
                WWPN=$datarow.cells[15].innerText}
            }
        }
    return [pscustomobject]@{
        SysType=$hwinfo[0].Split(':')[1];
        SysSerial=$hwinfo[1].Split(':')[1];
        SysGUID=$hwinfo[2].Split(':')[1];
        SysName=$hwinfo[3].Split(':')[1];
        SysOS=$hwinfo[4].Split(':')[1];
        nicports=$nicports;
        fcports=$fcports
 }

}

Yeah thats all folks - ugly quick-n-dirty.

Nasty behavior of ESXi - VMs arent able to mount ISO from certain Datastore

I observed a strange behavior in VMWare ESXi, exactly 6.0U3 Build 5572656 but i am sure many other versions are affected too.

We planned to ONE certain LUN for Scratch-Location, Kerneldumps, VMware Tools and our ISO files, but after some "configuration" changes the ESX servers did not allowed to mount any ISO from this certain datastore anymore.


2017-09-25T08:49:47.872Z| vmx| A100: ConfigDB: Unsetting "ide1:0.clientDevice"
2017-09-25T08:49:47.872Z| vmx| W115: ConfigDB_Set: Cannot make config edit ide1:0.fileName="/vmfs/volumes/58e6316b-7473644d-acaf-2c600c945e0e/DEPOT/ISOS/SCCM_LIVE_X64.iso"
2017-09-25T08:49:47.872Z| vmx| I125: Msg_Post: Warning 2017-09-25T08:49:47.872Z| vmx| I125: [msg.configrules.validate.failed.reject] Invalid value "/vmfs/volumes/58e6316b-7473644d-acaf-2c600c945e0e/DEPOT/ISOS/SCCM_LIVE_X64.iso" for configuration key "ide1:0.fileName".  The value was rejected by rule "No System Files".

I found that that ESXi has a config file with rules and wonder why this datastore is there.

/etc/vmware/configrules

  # /etc/init.d/hostd will sync the lines between the below markers on start.
  # SPECIAL_PATHS_START_TAG
  reject regex_case "^/vmfs/volumes/5971d493-023fb4b0-b2b2-246e965a4370/"
  reject regex_case "^/vmfs/volumes/1b49f9ee-f9cac734-9d40-577e80580578/"
  reject regex_case "^/vmfs/volumes/51c0fee1-3f13fc40-7a38-680ea30b2816/"
  reject regex_case "^/vmfs/volumes/58e6316b-7473644d-acaf-2c600c945e0e/" <-- why is this line here ??
  # SPECIAL_PATHS_END_TAG

Browsing the datastore showed that misconfiguration caused a var and a log directory in the root partition of the datastore, as soon we deleted them the hostd put the full pathname of the scratch location into the configrules file. It seems that the agent look into scratch config and starts searching for a "var" folder on the datastore - and put the folder where it finds "any" var folder into this file.

Ontap Clustermode does memorize beside SIDS also the SamAccountName and seems never to forget it

Update : Issue is gone with Ontap 9.6+

This some nasty behavior I observed on Ontap, if you rename Users or Groups in your active directory c-Dot Ontap will keep them in the CIFS ACL, even after deleting all the caches.

Everything will work but it just bugs when you troubleshoot CIFS access. As it memorize the SID fine you can use the function from my last blog post to set remove and add them, this happens so fast that i did not see any disruption.

$admcdot = New-Object System.Management.Automation.PsCredential('admin',$(ConvertTo-SecureString -String 'huehuehue' -AsPlainText -force)) 

Connect-NcController MyNetappCdotStorage -Credential $admcdot 


function Get-SID_NAME(
    [Parameter(Position=2)][string]$domain=$env:userdomain,
    [Parameter(Mandatory=$True,Position=1)][string]$search,
    [switch]$Local)
{
 if($search -match '\\'){
    $domain=$search.Split('\')[0]
    $search=$search.Split('\')[1]
    }
 if($search -match '^S-1-5-21-'){
    $objSID = New-Object System.Security.Principal.SecurityIdentifier($search)
    $objUser = $objSID.Translate( [System.Security.Principal.NTAccount])
    return $objUser.Value
    }else{
        if($Local){
            $objUser = New-Object System.Security.Principal.NTAccount($search)
            $strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
            return $strSID.Value
        } else {
            $objUser = New-Object System.Security.Principal.NTAccount($domain, $search)
            $strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
            return $strSID.Value
        }
    }
} 


Get-NcCifsShareAcl|?{($_.UserOrGroup -match '^MYDOMAIN') -and ($_.UserOrGroup -notmatch 'admins$') -and ($_.Vserver -notmatch 'mc$') }|%{
     if(-not $(Get-SID_NAME $_.UserOrGroup)){
           Remove-NcCifsShareAcl -Share $_.Share -User $_.UserOrGroup  -VserverContext $_.Vserver
          Add-NcCifsShareAcl -Share $_.Share -UserOrGroup $(Get-SID_Name $_.Winsid) -Permission $_.Permission -VserverContext $_.Vserver
    }
} 

Access a Huawei OceanStor Storage over REST-API per Powershell

So a device which cannot be automated is in my opinion not for big buisness, so here my first steps in dealing with the REST-API which involved a lot of Try&Error. You will need to read the API documentation anyway to know which kind of objects you can put into $HWMGTRessource parameter. If you ask why i put this Pipeline Switch there - it is for later use to "Pipe".

REST-Api Documentation can be found here

V300R003C20: http://support.huawei.com/enterprise/en/doc/DOC1000126989

V300R003C10: http://support.huawei.com/enterprise/en/doc/DOC1000111390


function Start-Huawei_RestSession {
  [Cmdletbinding()]
Param(
[Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=0,Mandatory=$true)]
[String]$HWMGTHostName,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=1,Mandatory=$true,ParameterSetName="HWMGTUser")]
[String]$HWMGTUser,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=2,Mandatory=$false)]
[String]$HWMGTPassword,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=1,Mandatory=$true,ParameterSetName="HWMGTCred")]
[System.Management.Automation.PSCredential]$HWMGTCred,
    [Switch]$ADUser
)
    
    switch ($PsCmdlet.ParameterSetName)
    {
        "HWMGTUser" {
            if($HWMGTPassword){
                $HWMGTCred=New-Object System.Management.Automation.PsCredential($HWMGTUser,$(ConvertTo-SecureString -String $HWMGTPassword -AsPlainText -force))
            } else {
                throw "You musst provide a Password for User $HWMGTUser"
            }
        }
        "HWMGTCred" {
            $HWMGTUser=$HWMGTCred.GetNetworkCredential().UserName
            $HWMGTPassword=$HWMGTCred.GetNetworkCredential().Password
        }
    }


    $body = @{username = $HWMGTUser;
             password = $HWMGTPassword;
             scope = if($ADUser){1}else {0}}

    $logonsession=Invoke-RestMethod -Method Post -Uri "https://$($HWMGTHostName):8088/deviceManager/rest/xxxxx/sessions" -Body (ConvertTo-Json $body) -SessionVariable WebSession
    
    $CredentialsBytes = [System.Text.Encoding]::UTF8.GetBytes(-join("{0}:{1}" -f $HWMGTUser,$HWMGTPassword))
    $EncodedCredentials = [Convert]::ToBase64String($CredentialsBytes)
    
    $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add("Authorization", "Basic $EncodedCredentials")
    $headers.Add("iBaseToken", $logonsession.data.iBaseToken)

    return [pscustomobject]@{
HWMGTHostName = $HWMGTHostName
HWMGTCred = $HWMGTCred
DeviceId=$logonsession.data.deviceid
                    WebSession=$WebSession
                    Headers=$headers
                    iBaseToken=$logonsession.data.iBaseToken
                    error=$logonsession.error
    }
}

function Invoke-Huawei_RestMethod {
  [Cmdletbinding()]
Param(
[Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=0,Mandatory=$true)]
[pscustomobject]$HWMGTSession,
    [Parameter(Position=1,Mandatory=$true)]
    [ValidateSet('GET','POST','PUT','DELETE')]
    [string]$Method,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=2,Mandatory=$true)]
[String]$HWMGTRessource,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Position=3,Mandatory=$false)]
[int]$HWMGTRessourceID,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Mandatory=$False)]
[ValidateScript({-not $($_ -inotmatch '^filter=\w*(:{1,2})\w*$|^range=\[\d{1,5}-\d{1,5}\]$')})]
    [string[]]$HWMGTFilters,
  [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Mandatory=$false)]
    [System.Collections.Hashtable]$HWMGTRequestBody,
    [Parameter(ValueFromPipeline=$True,ValueFromPipelineByPropertyName=$True,Mandatory=$false)]
    [Switch]$PipeLine
)

    $URI="https://$($HWMGTSession.HWMGTHostName):8088/deviceManager/rest/$($HWMGTSession.DeviceId)/$HWMGTRessource"
    
    if($HWMGTRessourceID){
        $URI+="/$HWMGTRessourceID"
    } elseif($HWMGTFilters) {
        $URI+="?$($HWMGTFilters -join('&'))"
    }

    if($HWMGTRequestBody){
        $result=Invoke-RestMethod -Method $Method -uri $uri -Headers $HWMGTSession.Headers -WebSession $HWMGTSession.WebSession -ContentType "application/json" -Credential $HWMGTSession.HWMGTCred -Body $(ConvertTo-Json $HWMGTRequestBody)
    }else{
        $result=Invoke-RestMethod -Method $Method -uri $uri -Headers $HWMGTSession.Headers -WebSession $HWMGTSession.WebSession -ContentType "application/json" -Credential $HWMGTSession.HWMGTCred
    }
    if ($PipeLine){
        return [pscustomobject]@{
            Result=$result
HWMGTSession=$HWMGTSession
HWMGTRessource = $HWMGTRessource
HWMGTFilters=$HWMGTFilters
            URI=$URI}
        } else {return $result}
}

So here a example how to report all LUNs .. First you have to start a session

$huasession=Start-Huawei_RestSession -HWMGTHostName "MyChineseStorage" -HWMGTCred $(Get-Credential)

Invoke the REST Command to the Storage

$huarequest=Invoke-Huawei_RestMethod $huasession GET 'lun'

Get what you want from the Result

$LUNTable=$huarequest|Select -ExpandProperty data data|Select NAME,ParentName,WWN,OWNINGCONTROLLER,ALLOCCAPACITY,ID

Delete the Session properly .. will happen itself after 20min

Invoke-Huawei_RestMethod $huasession DELETE sessions|Out-Null