Cloud and virtualization

Three routes bring cloud data in. Push receivers on the gateway need no agent; the agent modules poll APIs with credentials that live only on the agent host.

Push receivers (gateway)

Both use an ingestion key (wti_...). The tenant is the key's; any wt.* attribute in the payload is deleted before stamping. Delivery is at-least-once: both services retry on any non-2xx answer.

AWS Firehose: POST /aws/firehose

Create a Kinesis Data Firehose delivery stream with an HTTP endpoint destination:

• URL https://<gateway>/aws/firehose, access key = your wti_ key (sent as X-Amz-Firehose-Access-Key).

• Metrics: a CloudWatch Metric Stream with output format OpenTelemetry 1.0 into that stream. Needs scope ingest:metrics.

• Logs: a CloudWatch Logs subscription filter into that stream (gzip JSON). Needs scope ingest:logs.

A bad or missing key answers 401, a key without the scope for the records sent answers 403, both in the Firehose reply format (requestId, timestamp, errorMessage). The body limits of the OTLP endpoints apply to the request and, together, to the decompressed size of all log records.

GCP Pub/Sub: POST /gcp/pubsub?key=wti_...

Create a log sink to a topic, and a push subscription to https://<gateway>/gcp/pubsub?key=<key> (Pub/Sub push cannot set headers). The key needs scope ingest:logs. The gateway never logs the query, and error text never repeats it. Treat the subscription URL as a secret and rotate the key if it leaks. A message that is not a LogEntry answers 400 and Pub/Sub redelivers it until it expires or goes to your dead-letter topic.

Agent modules

Remote config can only switch these on or off and change the interval. Credentials come from the local file or the environment, never from the server, and are never exported or logged.

modules:
  aws:                      # standard AWS credential chain; no keys in this file
    region: eu-west-1
    namespaces: ["AWS/Lambda", "AWS/EC2", "AWS/RDS"]
    max_metrics: 100         # per namespace (default 100)
    period_s: 300            # 300 (default) or 60
    stats: ["Average"]       # default; each extra stat doubles the cost
    inventory: true          # Resource Groups Tagging API -> cloud entities
    # role_arn: arn:aws:iam::123456789012:role/watchtower-read   # cross-account; assumed on top of the chain
    # external_id: from-the-role-trust-policy
    # namespaces: ["auto"]   # every AWS/* namespace with metrics in the last 3 h (at most 40, re-listed every 15 min)
  azure_eventhub:
    connection_string_env: AZURE_EVENTHUB_CONNECTION_STRING
    topics: ["insights-logs-auditevent", "insights-metrics-pt1m"]
  vsphere:
    targets: [{name: vc1, url: "https://vcenter/sdk", username: wt, password_env: VC_PASSWORD}]
  proxmox:
    targets: [{name: pve, url: "https://pve:8006", token_id: "wt@pve!agent", token_secret_env: PVE_TOKEN}]

• AWS metrics are named amazonaws.com/<Namespace>/<Metric> as gauges with the attributes aws.cloudwatch.stat and aws.dim.<Dimension>. The metric stream sends Summary points with a different attribute shape, so the two do not share series; use one route per namespace. Each resource in the tagging inventory becomes an entity of kind cloud (cloud.resource_id is the ARN, tags are cloud.tag.*); the walk is capped at 5,000 resources.

• Azure: point a diagnostic setting at an Event Hub, then list its names under topics. Records with metricName become azure.monitor.<metric> gauges, others become logs. Delivery is at-least-once: offsets are committed only after the records were exported, and fetching pauses while the export buffer is full. A new consumer group starts at the latest record, not the oldest (set start_at_earliest: true to read the retained history, which can be large).

• vSphere: host and VM CPU, memory, disk and network from the PerformanceManager (real-time 20 s samples), plus inventory. A VM is an entity of kind vm (keyed by the vCenter instance UUID) whose parent is the ESXi host. Each poll logs in and out of vCenter, so vCenter records session events every interval (60 s by default).

• Proxmox: nodes, qemu and lxc status and counters; guests are vm entities under their node.

When the agent runs on the hypervisor it polls (the usual Proxmox install), the host is described by the agent's own resource and no second host resource is sent, so the host entity does not change identity between sources.

CloudWatch cost

CloudWatch bills GetMetricData at about $0.01 per 1,000 metrics requested. Cost per namespace and month = max_metrics x stats x (minutes per month / poll interval in minutes) / 1,000 x $0.01. With the defaults (100 metrics, one stat, every 5 minutes) that is 100 x 8,640 = 864,000 metrics, about $8.64 per namespace per month. At period_s: 60 (polled every minute) the same 100 metrics cost about $43 per namespace per month, and each additional stat adds the same again. The agent lists a namespace's metrics at most every 15 minutes and reports the metrics it requested as wt_agent_cloudwatch_metric_requests. The newest two minutes are never read, because CloudWatch is still filling them.

Every network call has a timeout and each target or namespace is polled on its own, so one slow endpoint does not delay the rest.

Managed database discovery

A cloud pull with provider: databases lists AWS RDS and Aurora (DescribeDBInstances, DescribeDBClusters), Azure SQL Database and Flexible Server for PostgreSQL and MySQL (ARM list calls), and GCP Cloud SQL (sqladmin instances list) every 10 minutes (interval, floor 1m). Each database becomes a cloud entity with db.system, cloud.provider, cloud.region, cloud.resource_id, db.engine, db.version and cloud.tag.*. A failed refresh keeps the last good list of that provider. Credentials stay in the local (sealed) agent config. For AWS metrics add AWS/RDS to the aws module namespaces (CPUUtilization, DatabaseConnections, FreeStorageSpace, ReadIOPS, WriteIOPS).

{"pulls":[{"provider":"databases","config":{
  "aws":{"region":"eu-west-1","access_key_id":"...","secret_access_key":"..."},
  "azure":{"tenant_id":"...","client_id":"...","client_secret":"...","subscription_id":"..."},
  "gcp":{"project":"p","service_account":{"client_email":"...","private_key":"..."}}}}]}

Least privilege: AWS {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["rds:DescribeDBInstances","rds:DescribeDBClusters","cloudwatch:GetMetricData","cloudwatch:ListMetrics"],"Resource":"*"}]}; Azure role Reader on the subscription; GCP role roles/cloudsql.viewer.

Known gap G-21: GCP and Azure management-API pollers are not built; use the push routes above.

More: cloud pulls (GCP, Azure, Cloudflare), Vercel drains, Cloudflare Logpush.