Skip to content

Instantly share code, notes, and snippets.

@nmattam
Last active April 4, 2022 12:09
Show Gist options
  • Select an option

  • Save nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f to your computer and use it in GitHub Desktop.

Select an option

Save nmattam/bcfbc8a4ebd9a520c2ac50ab0137e58f to your computer and use it in GitHub Desktop.
[Splunk] Automate "Apply Changes" for Monitoring Console
# Create DMC groups
# This is basically updating the system/local/distsearch.conf
# I got all the IPs used below from /servicesNS/admin/-/search/distributed/peers. I used DNS names for search head (I had only one SH) and cluster master. So it was easy to separate the indexer IPs.
# 8089 is my API PORT
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_search_head -d member=<search_head_member1:8089> -d member=<search_head_member2:8089> -d default=false
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_cluster_master -d member=<cluster_master:8089> -d default=false
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_group_indexer -d member=<indexer1:8089> -d member=<indexer2:8089> -d default=true # not idea why default=true for indexers
curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups -d name=dmc_indexerclustergroup_<CLUSTER_LABEL> -d member=<peer1:8089> -d member=<peer2:8089> -d default=false # Configure all peers in the cluster.
# Second time you run the above commands, you will run into an ERROR that says that the group already exists.
# So if the group exists, hit the "edit" endpoint to update the list of peers in a group.
# curl -k -u <user>:<password> <Monitoring Console>/services/search/distributed/groups/<dmc_group_name>/edit -d member=<member1:8089> -d member=<member2:8089> -d default=<true/false> # true for indexers, false for everything else
# Now configure Monitoring Console.
# Update the splunk_monitoring_console_assets.conf
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/splunk_monitoring_console/configs/conf-splunk_monitoring_console_assets/settings -d configuredPeers=<comma separated lists of all the peers> -d output_mode=json -d eai:appName=splunk_monitoring_console -d eai:userName=nobody
# Kicks off the search to build the assets.
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC+Asset+-+Build+Full/dispatch -d output_mode=json -d trigger_actions=true -d dispatch.auto_cancel=30 -d dispatch.buckets=300 -d dispatch.enablePreview=true
# Finally update the app.
curl -k -u <user>:<password> <Monitoring Console>/servicesNS/nobody/system/apps/local/splunk_monitoring_console -d output_mode=json -d author=Splunk -d check_for_updates=1 -d configured=1 -d description=The+Splunk+Monitoring+Console+application+gives+you+insight+into+your+Splunk+deployment. -d label=Monitoring+Console -d version=<Splunk Version> -d visible=1
# There was another endpoint that was hit to update the nav/default.xml to reveal new distributed dashboards. The request body
# had the entire xml, hence I landed it via Chef using the cerner_splunk cookbook apps hash (https://github.com/cerner/cerner_splunk/blob/stable/docs/databags.md#apps-hash). But this was the content
<nav color='#65A637'>
<view name='monitoringconsole_overview' default='true' />
<view name='monitoringconsole_check' />
<view name='monitoringconsole_instances' />
<collection label='Indexing'>
<collection label='Performance'>
<view name='indexing_performance_instance' />
<view name='indexing_performance_deployment' />
</collection>
<collection label='Indexer Clustering'>
<!--<a href='Clustering'>Indexer Clustering:Status</a>-->
<view name='indexer_clustering_status' />
<view name='indexer_clustering_service_activity' />
</collection>
<collection label='Indexes and Volumes'>
<view name='indexes_and_volumes_instance' />
<view name='indexes_and_volumes_deployment' />
<view name='index_detail_instance' />
<view name='index_detail_deployment' />
<view name='volume_detail_instance' />
<view name='volume_detail_deployment' />
</collection>
<collection label='Inputs'>
<view name='http_event_collector_instance' />
<view name='http_event_collector_deployment' />
<view name='splunk_tcpin_performance_instance' />
<view name='splunk_tcpin_performance_deployment' />
<view name='data_quality' />
</collection>
<collection label='License Usage'>
<view name='license_usage_today' />
<view name='license_usage_30days' />
</collection>
</collection>
<collection label='Search'>
<collection label='Activity'>
<view name='search_activity_instance' />
<view name='search_activity_deployment' />
<view name='search_usage_statistics_instance' />
<view name='search_usage_statistics_deployment' />
</collection>
<collection label='Distributed Search'>
<view name='distributed_search_instance' />
<view name='distributed_search_deployment' />
</collection>
<collection label='Search Head Clustering'>
<view name='shc_status_and_conf' />
<view name='shc_conf_rep' />
<view name='shc_artifact_replication' />
<view name='shc_scheduler_delegation_statistics' />
<view name='shc_app_deployment' />
</collection>
<collection label='Scheduler Activity'>
<view name='scheduler_activity_instance' />
<view name='scheduler_activity_deployment' />
</collection>
<collection label='KV Store'>
<view name='kv_store_instance' />
<view name='kv_store_deployment' />
</collection>
</collection>
<collection label='Resource Usage'>
<view name='resource_usage_instance' />
<view name='resource_usage_machine' />
<view name='resource_usage_deployment' />
</collection>
<collection label='Forwarders'>
<view name='forwarder_instance' />
<view name='forwarder_deployment' />
</collection>
<collection label='Settings'>
<view name='monitoringconsole_configure' />
<view name='monitoringconsole_forwarder_setup' />
<view name='monitoringconsole_alerts_setup' />
<view name='monitoringconsole_overview_preferences'/>
<view name='monitoringconsole_check_list' />
</collection>
<a href='search'>Run a Search</a>
</nav>
@Rohlik

Rohlik commented Jun 28, 2021

Copy link
Copy Markdown

It doesn't work anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment