Public Ops & support industry page within Technology & IT with peer questions, pay and benefits, career topics, workflow details, and real workplace experience for professionals in the United States.
tech-ops-support
有次值班遇到一个很迷惑的问题：应用报警说无法写入日志，但 df -h 看磁盘还有 30% 空间，服务重启也没用。开发那边以为是权限问题，运维这边看目录权限也正常。后来我查 df -i 才发现 inode 用完了，原因是某个临时目录堆了几十万个小文件，logrotate 也没覆盖到这类文件。 处理过程比较朴素：先用 find 按目录统计文件数量，确认问题目录；再停掉生成临时文件的 job，批量清理过期文件；最后把 systemd tim…
tech-ops-support
Our dashboards said backups were green for months, but the first real restore drill failed on a staging server. The dump was present and the storage job had a success status. The failure came later: the target database…
tech-ops-support
我以前参与过一次发布，灰度只看了接口能不能访问，没看新旧版本数据是否兼容。前10%的流量没出大问题，放到全量后才发现老版本写入的字段新版本读不了，回滚也没用，因为数据已经变了。后面我做灰度会先确认三件事：配置能不能独立开关，数据库和缓存是否前后兼容，回滚后老版本还能不能处理新版本留下的数据。灰度指标也不能只看CPU和错误率，要看关键业务动作，比如登录、下单、支付回调、消息发送这些链路是否正常。发布前我会先在预发环境用生产相近的数据跑一…
tech-ops-support
遇到的问题： 上周值班碰到一个挺容易误判的故障。客服同事说几个远程员工能打开系统首页，但登录后马上被踢回登录页，偶尔还提示 MFA code expired。前端看起来像会话丢失，Nginx access log 里是 200 后跟一串 401，应用日志提示 JWT iat is in the future。我一开始也怀疑 session cookie、Redis session 或负载均衡粘性没配好。 解决过程： 我先把影响面收住…
tech-ops-support
I once had a nightly export job that showed success in cron but delivered an empty folder to the vendor SFTP. The service account had no error email, so support only heard about it when the vendor asked why the report…
tech-ops-support
有台内部服务重启要两三分钟，业务同事只看到页面打不开，值班同学一直重启 systemctl restart，结果每次都慢。服务本身启动不慢，慢在 systemd 等一个网络挂载。 我排查时先看 systemctl status 和 journalctl -u service -b，确认卡在哪个时间点；再用 systemd-analyze blame 和 critical-chain 看启动链路。最后发现 unit 文件里 After=…
tech-ops-support
DNS cutovers always sound simpler in the planning meeting than they feel during the actual window. Change the record, wait for propagation, watch traffic move. In practice, one cached resolver, one forgotten subdomain…
tech-ops-support
有次线上服务写日志失败，df -h 看磁盘空间还剩不少，大家一开始以为是权限问题。后来查到是 inode 被小文件耗尽，磁盘没满但新文件已经创建不了。 我的处理顺序是先跑 df -i 看 inode 使用率，再用 find 按目录统计小文件数量，重点看临时目录、上传缓存和日志切割目录。最后发现一个失败任务每分钟生成空的 retry 文件，logrotate 也管不到那一层。我们先暂停任务，归档并删除旧 retry 文件，再给任务加失败…
tech-ops-support
I dealt with this recently in an internal help desk queue where remote staff used Windows laptops, a corporate VPN, and split DNS. The visible problem was users could reconnect to VPN after sleep, but internal…
tech-ops-support
I dealt with a Windows laptop that looked healthy from the user's side. Wi-Fi worked, Outlook worked, Teams worked, but Intune policy updates would not arrive. The user needed a compliance profile before accessing a…
tech-ops-support
A small ops issue got bigger than it should have because our shared laptops had messy names. Some were named after users, some after rooms, and a few still had the vendor image name. When tickets came in saying "the…
tech-ops-support
Comparto una experiencia de campo en Ops & support. Venia trabajando con soporte interno para usuarios hibridos que saltan entre oficina, casa y VPN cuando aparecio un laptop entraba a la VPN, pero al desconectar ya no…
tech-ops-support
Alert fatigue does not start when there are too many alerts. It starts when the team stops trusting them. I have seen a pretty dashboard still fail the on-call person because every warning had the same urgency. Disk at…
tech-ops-support
值班最怕一上来就被群里催，CPU、磁盘、网络、应用日志全都红一点。我的习惯是先看影响面，再看最近变更，不急着重启服务。很多事故其实是证书、DNS、配置发布这种小地方拖出来的。你们排障时第一眼会先看监控、日志，还是发布记录？
tech-ops-support
SSL certificate renewal sounds routine until the first time a quiet renewal fails and customers see browser warnings before the team notices. I used to treat certs as a calendar reminder. Now I treat them like a small…
tech-ops-support
The migration looked small on paper: add a few columns, backfill old records, then switch the application to read the new shape. The risk was that the table was hot all day and the old worker code would still be…
tech-ops-support
有一次在一个有秒杀活动的电商后台值班，问题一开始看着不大：Redis memory usage 突然冲到 92%，接口还没挂，但缓存 miss 和响应时间已经开始抖。现场几个人的说法都合理，可放到一起就是对不上。 我先让大家暂停继续扩大影响，再逐项核对证据。我先确认不是连接数暴涨，再看 keyspace 和 big key，临时调低部分非核心缓存 TTL，同时把商品详情热 key 做了分片和预热。这件事让我感觉，内存告警不是等到 OO…
tech-ops-support
有一次在一个有秒杀活动的电商后台值班，问题一开始看着不大：Redis memory usage 突然冲到 92%，接口还没挂，但缓存 miss 和响应时间已经开始抖。现场几个人的说法都合理，可放到一起就是对不上。 我先让大家暂停继续扩大影响，再逐项核对证据。我先确认不是连接数暴涨，再看 keyspace 和 big key，临时调低部分非核心缓存 TTL，同时把商品详情热 key 做了分片和预热。这件事让我感觉，内存告警不是等到 OO…
Pay in Ops & support depends on city, experience, role type, company size, skills, and certifications. Compare base pay together with overtime, bonus, benefits, stability, and long-term growth.
To move into Ops & support, start by learning entry-level roles, common skills, certification requirements, and real day-to-day work. Then build credible experience through projects, part-time work, junior roles, or concrete examples.
The outlook for Ops & support depends on local demand, technology changes, market cycles, and skill depth. Real job postings, peer experience, and transferable skills are usually better signals than broad trends alone.
Ops & support often requires role-specific technical skills, communication, collaboration, and safety or compliance judgment. Some jobs also require licenses, certifications, equipment experience, or project examples.
Beginners can start with basic terms, typical workflows, common problems, and entry-level tools. Real cases, peer Q&A, and small projects help turn basic knowledge into practical judgment.