谈谈CloudWatch

Cloudwatch是云上的眼睛,用来记录服务的指标和日志.

指标

先说说指标,每个服务都自带会有相关的metics,比如EC2的CPU占用率和网络进出的流量,但是像实例内部的指标比如内存和磁盘占用率就检测不到.这个时候就需要自定义指标, 其实也就是调用了PutMetricData的API,比如亚马逊云自家的CloudWatch Agent.(如果自定义的话,接受过去两周和未来两小时的数据)

然后就可以根据这个指标的数值进行设置警报和对应用程序的扩缩.甚至还可以使用机器学习方法对警报进行智能检测

维度(时间,聚合数值)
• Standard: 1 minute (60 seconds)
• High Resolution: 1/5/10/30 second(s) – Higher cost

日志

云山所有的日子都可以传输到CloudWatch Log中,比如各种agent输入的日志,各种应用程序的日志,甚至无服务的日志(Lambda,ECS,EKS,APIGATEWAY),甚至包括VPC Flow log,过滤之后的cloudtrail.和R53的DNS日志查询.

日志组 -> 日志流

可以定义过期时间(或者甚至设置永不过期),当然日志也可以被发送到其他目的地做进一步的持久化存储

• S3(CreateExportTask API不是实时或者近实时,最多12小时)
• kinesis data stream and Firehose
• Lambda
• OpenSearch

日志默认是KMS加密的,当然可以修改为自己制定的KMS密钥进行加密.使用Logsight可以对关键字进行查询

cloudwatch agent

现在正在使用的叫做 CloudWatch Unified Agent, 由于旧版本的CloudWatch Logs Agent只有采集日志的功能,新版本增加了收集系统内部指标的功能,常见指标如下: (装了agent是不是就可以获得更细粒度的监控?)

• CPU (active, guest, idle, system, user, steal)
• Disk metrics (free, used, total), Disk IO (writes, reads, bytes, iops) • RAM (free, inactive, used, total, cached)
• Netstat (number of TCP and UDP connections, net packets, bytes) • Processes (total, dead, bloqued, idle, running, sleep)
• Swap Space (free, used, used %)

除此之外还可以把参数存在SSM Parameter Store中,然后使用参数进行引用
https://aws.amazon.com/cn/blogs/china/cloudwatch-agent-in-cloudwatch-ec2-rom/

关于警报,可以设置当某个指标的统计维度到达一定程度并且满足一定时间的时候触发警报,
可以设置单个警报,也可以设置多个警报,用AND或者OR进行逻辑的计算.用多个警报负责的逻辑计算有助于防止误报.

警报的目标如下:

• EC2: Stop, Terminate, Reboot, or Recover an EC2 Instance
• Auto Scaling
• Amazon SNS(可以触发Lambda,几乎可以做任何事)

log 跨账户写入Kinesis
Logs Insights 查询多个账户的多个日子组(非实时)
可以基于Metric Filter同步指标到cloudwatch和设置警报.

centos 安装cloudwatch-agent的步骤:

安装：
wget https://s3.amazonaws.com/amazoncloudwatch-agent/centos/amd64/latest/amazon-cloudwatch-agent 
t.rpm

sudo rpm -U ./amazon-cloudwatch-agent.rpm

生成配置文件：

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

On which OS are you planning to use the agent?
1. linux
2. windows
3. darwin
default choice: [1]:
1
Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:
1
Which user are you planning to run the agent?
1. root
2. cwagent
3. others
default choice: [1]:
1
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor metrics from CollectD? WARNING: CollectD must be installed or the Agent will fail to start
1. yes
2. no
default choice: [1]:

2
Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:
1
Do you want to monitor cpu metrics per core?
1. yes
2. no
default choice: [1]:
2
Do you want to add ec2 dimensions (ImageId, InstanceId, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
1. yes
2. no
default choice: [1]:
2
Do you want to aggregate ec2 dimensions (InstanceId)?
1. yes
2. no
default choice: [1]:
2
Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
1. 1s
2. 10s
3. 30s
4. 60s
default choice: [4]:
4
Which default metrics config do you want?
1. Basic
2. Standard
3. Advanced
4. None
default choice: [1]:
2
Current config as follows:
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
"metrics": {
"metrics_collected": {
"cpu": {
"measurement": [
"cpu_usage_idle",
"cpu_usage_iowait",
"cpu_usage_user",
"cpu_usage_system"
],
"metrics_collection_interval": 60,
"totalcpu": false
},
"disk": {
"measurement": [
"used_percent",
"inodes_free"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"diskio": {
"measurement": [
"io_time"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
},
"swap": {
"measurement": [
"swap_used_percent"
],
"metrics_collection_interval": 60
}
}
}
}
Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional items.
1. yes
2. no
default choice: [1]:
1
Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html ) configuration file to import for migration?
1. yes
2. no
default choice: [2]:
2
Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:
2
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
"metrics": {
"metrics_collected": {
"cpu": {
"measurement": [
"cpu_usage_idle",
"cpu_usage_iowait",
"cpu_usage_user",
"cpu_usage_system"
],
"metrics_collection_interval": 60,
"totalcpu": false
},
"disk": {
"measurement": [
"used_percent",
"inodes_free"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"diskio": {
"measurement": [
"io_time"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
},
"swap": {
"measurement": [
"swap_used_percent"
],
"metrics_collection_interval": 60
}
}
}
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.

使用命令行启动 CloudWatch 代理
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json