Configuring Cloud Data Sources

1. Configuring AWS CloudWatch

Property	Value	Description
access_key_id	`string`	AWS access key ID for authentication
secret_access_key	`string`	AWS secret access key for authentication
region	`string`	AWS region to query (e.g., us-east-1)
assume_role_arn	`string`	ARN of the IAM role to assume (optional)
auth_type	`string`	Authentication type: `keys`, `credentials`, or `default`
custom_metrics	`boolean`	Enable custom metrics (default: `false`)

Example: Adding AWS CloudWatch Data Source

{
  "name": "CloudWatch-Prod",
  "type": "cloudwatch",
  "jsonData": {
    "region": "us-west-2",
    "authType": "keys",
    "customMetrics": true
  },
  "secureJsonData": {
    "accessKey": "AKIAIOSFODNN7EXAMPLE",
    "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  }
}

Note: Ensure that the IAM role or credentials provided have appropriate permissions to read CloudWatch metrics and logs.

2. Setting CloudWatch Authentication

Property	Value	Description
authType	`accessKey`	Authentication method to use for CloudWatch. Options: `accessKey`, `credentialsFile`, `ecs`, `ec2`
accessKey	`string`	Required when `authType` is `accessKey`. AWS Access Key ID for authentication.
secretKey	`string`	Required when `authType` is `accessKey`. AWS Secret Access Key for authentication.
region	`string`	Required. AWS region to query CloudWatch data from (e.g., `us-east-1`, `eu-west-1`).
credentialsFile	`string`	Required when `authType` is `credentialsFile`. Path to the AWS credentials file (e.g., `/home/user/.aws/credentials`).
profile	`string`	Optional. AWS profile name to use from the credentials file (e.g., `default`, `myprofile`).

Example: Using Access Key Authentication

{
  "authType": "accessKey",
  "accessKey": "AKIAIOSFODNN7EXAMPLE",
  "secretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "region": "us-west-2"
}

Example: Using Credentials File Authentication

{
  "authType": "credentialsFile",
  "credentialsFile": "/home/user/.aws/credentials",
  "profile": "default",
  "region": "us-east-1"
}

Note: Ensure that the IAM user or role used for authentication has appropriate permissions to read CloudWatch metrics and logs. Required policies include cloudwatch:GetMetricData, cloudwatch:DescribeAlarms, and logs:FilterLogEvents.

3. Configuring Azure Monitor

Field	Description	Value
cloudProvider	Specifies the cloud provider	`azure`
tenantId	GUID of the Azure AD tenant	`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
clientId	Application (client) ID from Azure AD app registration	`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
clientSecret	Client secret value from Azure AD app registration	`secret-value`
subscriptionId	GUID of the Azure subscription	`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`
resourceGroup	Name of the Azure resource group	`my-resource-group`
workspaceId	Workspace ID for Log Analytics or Application Insights	`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`

Example: Azure Monitor Data Source Configuration

{
  "name": "Azure Monitor",
  "type": "grafana-azure-monitor-datasource",
  "access": "proxy",
  "jsonData": {
    "cloudProvider": "azure",
    "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  },
  "secureJsonData": {
    "clientSecret": "secret-value"
  }
}

Note: Ensure the Azure AD application has appropriate permissions to access the selected resources.

Warning: The client secret will be stored in plain text. Use managed identity or Azure Key Vault for enhanced security.

4. Setting Azure Log Analytics

Property	Value	Description
cloudProvider	`azure`	Specifies the cloud provider for the data source
azureLogAnalytics	`true`	Enables Azure Log Analytics integration
workspaceId	`string`	The unique identifier for the Azure Log Analytics workspace
tenantId	`string`	The Azure Active Directory tenant ID
clientId	`string`	The client ID of the service principal used for authentication
clientSecret	`string`	The client secret for the service principal
subscriptionId	`string`	The Azure subscription ID containing the Log Analytics workspace

Example: Azure Log Analytics Configuration

{
  "name": "Azure Log Analytics",
  "type": "azure-log-analytics-datasource",
  "access": "proxy",
  "jsonData": {
    "cloudProvider": "azure",
    "workspaceId": "12345678-1234-1234-1234-123456789012",
    "tenantId": "abcd1234-ef56-7890-abcd-ef1234567890",
    "clientId": "efgh5678-1234-5678-90ab-cdef12345678",
    "subscriptionId": "ijkl9012-3456-7890-abcd-ef1234567890"
  },
  "secureJsonData": {
    "clientSecret": "secret-key-here"
  }
}

Note: Ensure the service principal has appropriate permissions to query Azure Log Analytics data.

Warning: The client secret should be stored securely and not exposed in configuration files.

5. Configuring Google Cloud Monitoring

Field	Value	Description
name	`Google Cloud Monitoring`	Name of the data source in Grafana
type	`stackdriver`	Stackdriver is the legacy name for Google Cloud Monitoring
access	`proxy`	Access mode; proxy through Grafana server
jsonData	`{}`	JSON object containing configuration options

Example: Adding Google Cloud Monitoring Data Source

{
  "name": "Google Cloud Monitoring",
  "type": "stackdriver",
  "access": "proxy",
  "jsonData": {
    "gceBackend": true,
    "gkeBackend": false
  }
}

Note: Ensure that the Grafana server has appropriate permissions to access Google Cloud Monitoring API. This typically involves setting up a service account with the necessary roles (e.g., monitoring.viewer).

Warning: The stackdriver data source type is deprecated in favor of the newer prometheus or google-cloud-monitoring types. Consider migrating to the newer type for better long-term support.

Configuration Option	Default Value	Description
gceBackend	`false`	Enable GCE backend for monitoring
gkeBackend	`false`	Enable GKE backend for monitoring
customMetrics	`false`	Allow custom metrics from GCP

Summary of Steps to Configure Google Cloud Monitoring

Create a service account with appropriate GCP permissions
Download the JSON key file for the service account
In Grafana, navigate to Configuration > Data Sources
Add a new data source of type stackdriver
Enter the service account key in the appropriate field
Save and test the connection

6. Using GCP Service Account

Property	Value	Description
type	`gcp`	Specifies the data source type as GCP
project	`my-gcp-project`	GCP project ID to use for queries
credentials	`service-account.json`	Path to the GCP service account key file
auth_type	`service_account`	Authentication method to use (e.g., service account)

Example: Configuring GCP Data Source with Service Account

{
  "name": "GCP-Data-Source",
  "type": "gcp",
  "access": "proxy",
  "jsonData": {
    "project": "my-gcp-project",
    "authType": "service_account"
  },
  "secureJsonData": {
    "credentials": "{...}"
  }
}

Note: The credentials field should contain the base64-encoded JSON key file of the GCP service account.

7. Configuring Cloud Data Source Variables

Variable Type	Description	Usage	Notes
Text Box	Allows users to input custom values	`${varName}`	Useful for dynamic queries
Query	Uses a data source query to generate values	`${varName:value}`	Supports multi-value selection
Constant	Fixed value that does not change	`${varName:value}`	Good for environment-specific values
Interval	Generates time intervals for queries	`${varName:duration}`	Useful for time-based filtering

Example: Query Variable


query = "SELECT DISTINCT region FROM metrics"

Example: Text Box Variable


var region = {
  name: "region",
  label: "Region",
  description: "Select a region to filter metrics",
  type: "textbox",
  value: "us-west-1"
}

Note: Variables can be referenced in dashboard panels using ${variableName} syntax.

Warning: Query variables may impact performance if not optimized properly.

8. Setting Cross-Account Access

Configuring cross-account access in Grafana allows users to view and manage data sources across multiple AWS accounts. This is particularly useful in environments where data is distributed across various accounts for security or organizational reasons.

Configuration	Description	Value
datasource.access	Defines the access mode for data sources	`cross-account`
datasource.accountId	The AWS account ID to which access is granted	`123456789012`
datasource.roleArn	ARN of the IAM role used for cross-account access	`arn:aws:iam::123456789012:role/GrafanaCrossAccountRole`

Example: Configuring Cross-Account Access in Grafana

{
  "name": "CrossAccountDataSource",
  "type": "prometheus",
  "access": "cross-account",
  "account": {
    "id": "123456789012",
    "roleArn": "arn:aws:iam::123456789012:role/GrafanaCrossAccountRole"
  }
}

Note: Ensure that the IAM role specified has appropriate permissions to access the target data sources in the cross-account environment.

Warning: Misconfiguring cross-account access can lead to unauthorized data access. Always validate role permissions and account IDs.

9. Configuring Cloud Query Timeouts

Property	Values	Description
queryTimeout	`30s`	Maximum time a query can take before being cancelled
maxConcurrentQueries	`10`	Maximum number of queries that can run concurrently
timeoutBehavior	`cancel`	Action to take when a query exceeds timeout: `cancel` or `fail`

Example: Setting Query Timeout in Grafana Configuration

datasources:
  - name: "Prometheus"
    type: "prometheus"
    url: "http://localhost:9090"
    timeout: 60s

Note: Timeout settings can be overridden per data source in the Grafana UI or via configuration files.

10. Using Cloud Annotations

Property	Description	Values	Notes
datasource	Cloud data source to use for annotations	`prometheus`, `loki`, `elasticsearch`	Must be a valid data source configured in Grafana
query	Query to fetch annotations from the cloud data source	`query` field with appropriate syntax	Depends on the selected data source type
tagKeys	Comma-separated list of tag keys to include in annotations	`tagKeys` field with comma-separated values	Optional, defaults to all tags if not specified
titleFormat	Template for annotation title	`{{__interval}} - {{__value}}` or custom template	Supports templating with Grafana variables

Example: Prometheus Annotation Query

rate(http_requests_total[5m]) > 0

Example: Loki Annotation Query

{job="api-server"} |= "error"

Note: Cloud annotations are fetched from the configured cloud data source and displayed on dashboards as events or markers.

Warning: Ensure that the cloud data source has sufficient permissions to query annotations.