DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD

The first guide to DTrace: the breakthrough debugging tool for Mac OS X, Unix, Solaris, and OpenSolaris operating systems and applications

 

Complete coverage: architecture, implementation, components, usage, and much more

Covers integrating DTrace into open source code, and integrating probes into application software

Includes full chapter of advanced tips and techniques

For users of DTrace on all platforms

Foreword by Bryan Cantril, creator of DTrace

 

 DTrace represents a revolution in debugging. Using it, administrators, developers, and service personnel can dynamically instrument operating systems and applications to quickly ask and answer virtually any question about how their operating systems or user programs are behaving. Now available for Solaris 10 and OpenSolaris, Mac OS X, and FreeBSD, thousands of professionals are discovering DTrace - but, until now, there's been no comprehensive, authoritative guide to using it. This book fills that gap. Written by four key contributors to the DTrace community, it's the first single source reference to this powerful new technology. The authors cover everything technical professionals need to know to succeed with DTrace, regardless of the operating system or application they want to instrument. The book also includes a full chapter of advanced tips and techniques.

Foreword xxi

Preface xxv

Acknowledgments xxxi

About the Authors xxxv

 

Part I: Introduction

Chapter 1: Introduction to DTrace 1

What Is DTrace? 1

Why Do You Need It? 1

Capabilities 2

Dynamic and Static Probes 4

DTrace Features 4

A First Look 6

Overview 8

Architecture 16

Summary 17

 

Chapter 2: D Language 19

D Language Components 20

Probes 23

Variables 26

Aggregations 33

Actions 37

Options 43

Example Programs 44

Summary 49

 

Part II: Using DTrace

Chapter 3: System View 51

Start at the Beginning 52

Observing CPUs 56

Observing Memory 95

Observing Disk and Network I/O 125

Summary 148

 

Chapter 4: Disk I/O 151

Capabilities 152

Disk I/O Strategy 154

Checklist 155

Providers 156

Scripts 172

Case Studies 269

Summary 290

 

Chapter 5: File Systems 291

Capabilities 292

Strategy 295

Checklist 296

Providers 297

Scripts 313

Case Study 387

Summary 397

 

Chapter 6: Network Lower-Level Protocols 399

Capabilities 400

Strategy 402

Checklist 403

Providers 404

Scripts 445

Common Mistakes 548

Summary 555

 

Chapter 7: Application-Level Protocols 557

Capabilities 558

Strategy 558

Checklist 559

Providers 560

Scripts 574

Summary 668

 

Chapter 8: Languages 669

Capabilities 671

Strategy 672

Checklist 674

Providers 675

C 679

C++ 689

Java 691

JavaScript 705

Perl 719

PHP 731

Python 740

Ruby 751

Shell 764

Tcl 774

Summary 782

 

Chapter 9: Applications 783

Capabilities 784

Strategy 784

Checklist 786

Providers 787

Scripts 804

Case Studies 817

Summary 832

 

Chapter 10: Databases 833

Capabilities 834

Strategy 835

Providers 836

MySQL 837

PostgreSQL 851

Oracle 858

Summary 865

 

Part III: Additional User Topics

Chapter 11: Security 867

Privileges, Detection, and Debugging 867

Scripts 875

Summary 892

 

Chapter 12: Kernel 893

Capabilities 894

Strategy 896

Checklist 897

Providers 897

Scripts 932

Summary 945

 

Chapter 13: Tools 947

The DTraceToolkit 948

Chime 962

DTrace GUI Plug-in for NetBeans and Sun Studio 966

DLight, Oracle Solaris Studio 12.2 966

Mac OS X Instruments 971

Analytics 973

Summary 985

 

Chapter 14: Tips and Tricks 987

Tip 1: Known Workloads 987

Tip 2: Write Target Software 989

Tip 3: Use grep to Search for Probes 991

Tip 4: Frequency Count 991

Tip 5: Time Stamp Column, Postsort 992

Tip 6: Use Perl to Postprocess 993

Tip 7: Learn Syscalls 994

Tip 8: timestamp vs. vtimestamp 995

Tip 9: profile:::profile-997 and Profiling 996

Tip 10: Variable Scope and Use 997

Tip 11: strlen() and strcmp() 999

Tip 12: Check Assumptions 1000

Tip 13: Keep It Simple 1001

Tip 14: Consider Performance Impact 1001

Tip 15: drops and dynvardrops 1003

Tip 16: Tail-Call Optimization 1003

Further Reading 1003

 

Appendix A: DTrace Tunable Variables 1005

 

Appendix B: D Language Reference 1011

 

Appendix C: Provider Arguments Reference 1025

Providers 1025

Arguments 1038

 

Appendix D: DTrace on FreeBSD 1045

Enabling DTrace on FreeBSD 7.1 and 8.0 1045

DTrace for FreeBSD: John Birrell 1047

 

Appendix E: USDT Example 1051

USDT Bourne Shell Provider 1052

Case Study: Implementing a Bourne Shell Provider 1057

 

Appendix F: DTrace Error Messages 1063

Privileges 1063

Drops 1064

Aggregation Drops 1065

Dynamic Variable Drops 1066

Invalid Address 1066

Maximum Program Size 1067

Not Enough Space 1068

 

Appendix G: DTrace Cheat Sheet 1069

Synopsis 1069

Finding Probes 1069

Finding Probe Arguments 1070

Probes 1070

Vars 1070

Actions 1071

Switches 1071

Pragmas 1071

One-Liners 1072

 

Bibliography 1073

Suggested Reading 1073

Vendor Manuals 1075

 

Index 1089