DevOps Troubleshooting: Linux Server Best Practices

The DevOps approach to system administration describes a world where Linux developers and sysadmins work far more closely than in traditional environments. DevOps is optimized to support today's confluence of Linux trends, including cloud migrations, the rise of startups using hosted services, greater requirements to analyze big data, and the transition to NoSQL databases. One of DevOps' most powerful benefits is its support for better, faster troubleshooting. In this book, pioneering Linux sysadmin Kyle Rankin teaches DevOps' standardized, repeatable troubleshooting techniques, showing administrators and developers how to work together to reduce costs and improve effectiveness. Rankin helps both developers and admins fill the gaps in their respective troubleshooting skillsets, so they can stop pointing fingers - and start getting results. Writing clearly and simply, he walks through using DevOps techniques to identify and resolve these and other common problems in Linux environments: * Slow servers (including CPU, RAM, and disk I/O bottlenecks) * System boot failures * Full or corrupt disks * Down servers and websites * Failures in DNS server hostname resolution * Email delivery problems * Slow databases * Hardware faults

Preface         xiii

Acknowledgments         xix

About the Author         xxi

 

Chapter 1: Troubleshooting Best Practices         1

Divide the Problem Space 3

Practice Good Communication When Collaborating 4

Favor Quick, Simple Tests over Slow, Complex Tests 8

Favor Past Solutions 9

Document Your Problems and Solutions 10

Know What Changed 12

Understand How Systems Work 13

Use the Internet, but Carefully 14

Resist Rebooting 15

 

Chapter 2: Why Is the Server So Slow? Running Out of CPU, RAM, and Disk I/O         17

System Load 18

Diagnose Load Problems with top 20

Troubleshoot High Load after the Fact 29

 

Chapter 3: Why Won’t the System Boot? Solving Boot Problems         35

The Linux Boot Process 36

BIOS Boot Order 45

Fix GRUB 47

Disable Splash Screens 51

Can’t Mount the Root File System 51

Can’t Mount Secondary File Systems 55

 

Chapter 4: Why Can’t I Write to the Disk? Solving Full or Corrupt Disk Issues         57

When the Disk Is Full 58

Out of Inodes 61

The File System Is Read-Only 62

Repair Corrupted File Systems 63

Repair Software RAID 64

 

Chapter 5: Is the Server Down? Tracking Down the Source of Network Problems         67

Server A Can’t Talk to Server B 68

Troubleshoot Slow Networks 78

Packet Captures 83

 

Chapter 6: Why Won’t the Hostnames Resolve? Solving DNS Server Issues         93

DNS Client Troubleshooting 95

DNS Server Troubleshooting 98

 

Chapter 7: Why Didn’t My Email Go Through? Tracing Email Problems         119

Trace an Email Request 120

Understand Email Headers 123

Problems Sending Email 125

Problems Receiving Email 135

 

Chapter 8: Is the Website Down? Tracking Down Web Server Problems         141

Is the Server Running? 143

Test a Web Server from the Command Line 146

HTTP Status Codes 149

Parse Web Server Logs 154

Get Web Server Statistics 158

Solve Common Web Server Problems 163

 

Chapter 9: Why Is the Database Slow? Tracking Down Database Problems          171

Search Database Logs 172

Is the Database Running? 174

Get Database Metrics 177

Identify Slow Queries 182

 

Chapter 10: It’s the Hardware’s Fault! Diagnosing Common Hardware Problems         185

The Hard Drive Is Dying 186

Test RAM for Errors 190

Network Card Failures 191

The Server Is Too Hot 192

Power Supply Failures 194

 

Index         197