structured, leveled logging

[jba]

This discussion has led to a proposal and is now finished. Please comment on the proposal.

We would like to add structured logging with levels to the standard library. Structured logging is the ability to output logs with machine-readable structure, typically key-value pairs, in addition to a human-readable message. Structured logs can be parsed, filtered, searched and analyzed faster and more reliably than logs designed only for people to read. For many programs that aren't run directly by person, like servers, logging is the main way for developers to observe the detailed behavior of the system, and often the first place they go to debug it. Logs therefore tend to be voluminous, and the ability to search and filter them quickly is essential.

In theory, one can produce structured logs with any logging package:

log.Printf(`{"message": %q, "count": %d}`, msg, count)

In practice, this is too tedious and error-prone, so structured logging packages provide an API for expressing key-value pairs. This draft proposal contains such an API.

We also propose generalizing the logging "backend." The log package provides control only over the io.Writer that logs are written to. In the new package (tentative name: log/slog), every logger has a handler that can process a log event however it wishes. Although it is possible to have a structured logger with a fixed backend (for instance, zerolog outputs only JSON), having a flexible backend provides several benefits: programs can display the logs in a variety of formats, convert them to an RPC message for a network logging service, store them for later processing, and add to or modify the data.

Lastly, we include levels in our design, in a way that accommodates both traditional named levels and logr-style verbosities.

Our goals are:

• Ease of use. A survey of the existing logging packages shows that programmers want an API that is light on the page and easy to understand. This proposal adopts the most popular way to express key-value pairs, alternating keys and values.

• High performance. The API has been designed to minimize allocation and locking. It provides an alternative to alternating keys and values that is more cumbersome but faster (similar to Zap's Fields).

• Integration with runtime tracing. The Go team is developing an improved runtime tracing system. Logs from this package will be incorporated seamlessly into those traces, giving developers the ability to correlate their program's actions with the behavior of the runtime.

What Does Success Look Like?

Go has many popular structured logging packages, all good at what they do. We do not expect developers to rewrite their existing third-party structured logging code en masse to use this new package. We expect existing logging packages to coexist with this one for the foreseeable future.

We have tried to provide an API that is pleasant enough to prefer to existing packages in new code, if only to avoid a dependency. (Some developers may find the runtime tracing integration compelling.) We also expect newcomers to Go to become familiar with this package before learning third-party packages, so they will naturally prefer it.

But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share.

Prior Work

The existing log package has been in the standard library since the release of Go 1 in March 2012. It provides formatted logging, but not structured logging or levels.

Logrus, one of the first structured logging packages, showed how an API could add structure while preserving the formatted printing of the log package. It uses maps to hold key-value pairs, which is relatively inefficient.

Zap grew out of Uber's frustration with the slow log times of their high-performance servers. It showed how a logger that avoided allocations could be very fast.

zerolog reduced allocations even further, but at the cost of reducing the flexibility of the logging backend.

All the above loggers include named levels along with key-value pairs. Logr and Google's own glog use integer verbosities instead of named levels, providing a more fine-grained approach to filtering high-detail logs.

Other popular logging packages are Go-kit's log, HashiCorp's hclog, and klog.

Overview of the Design

Here is a short program that uses some of the new API:

import "log/slog"

func main() {
    slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr)))
    slog.Info("hello", "name", "Al")
    slog.Error("oops", net.ErrClosed, "status", 500)
    slog.LogAttrs(slog.ErrorLevel, "oops",
        slog.Int("status", 500), slog.Any("err", net.ErrClosed))
}

It begins by setting the default logger to one that writes log records in an easy-to-read format similar to logfmt . (There is also a built-in handler for JSON.)

The program then outputs three log messages augmented with key-value pairs. The first logs at the Info level, passing a single key-value pair along with the message. The second logs at the Error level, passing an error and a key-value pair.

The third produces the same output as the second, but more efficiently. Functions like Any and Int construct slog.Attr values, which are key-value pairs that avoid memory allocation for some values. slog.Attr is modeled on zap.Field.

The Design

Interaction Between Existing and New Behavior

The slog package works to ensure consistent output with the log package. Writing to slog's default logger without setting a handler will write structured text to log's default logger. Once a handler is set, as in the example above, the default log logger will send its text output to the structured handler.

Handlers

A slog.Handler describes the logging backend. It is defined as:

type Handler interface {
    // Enabled reports whether this handler is accepting records.
    Enabled(Level) bool

    // Handle processes the Record.
    Handle(Record) error

    // With returns a new Handler whose attributes consist of                                                                                                                        
    // the receiver's attributes concatenated with the arguments.                                                                                                                              
    With(attrs []Attr) Handler
}

The main method is Handle. It accepts a slog.Record with the timestamp, message, level, caller source position, and key-value pairs of the log event. Each call to a Logger output method, like Info, Error or LogAttrs, creates a Record and invokes the Handle method.

The Enabled method is an optimization that can save effort if the log event should be discarded. Enabled is called early, before any arguments are processed.

The With method is called by Logger.With, discussed below.

The slog package provides two handlers, one for simple textual output and one for JSON. They are described in more detail below.

The Record Type

The Record passed to a handler exports Time, Message and Level methods, as well as four methods for accessing the sequence of Attrs:

• Attrs() []Attr returns a copy of the Attrs as a slice.
• NumAttrs() int returns the number of Attrs.
• Attr(int) Attr returns the i'th Attr.
• SetAttrs([]Attr) replaces the sequence of Attrs with the given slice.

This API allows an efficient implementation of the Attr sequence that avoids copying and minimizes allocation. SetAttrs supports "middleware" handlers that want to alter the Attrs, say by removing those that contain sensitive data.

The Attr Type

The Attr type efficiently represents a key-value pair. The key is a string. The value can be any type, but Attr improves on any by storing common types without allocating memory. In particular, integer types and strings, which account for the vast majority of values in log messages, do not require allocation. The default version of Attr uses package unsafe to store any value in three machine words. The version without unsafe requires five.

There are convenience functions for constructing Attrs with various value types:

• Int(k string, v int) Attr
• Int64(k string, v int64) Attr
• Uint64(k string, v uint64) Attr
• Float64(k string, v float64) Attr
• String(k, v string) Attr
• Bool(k string, v bool) Attr
• Duration(k string, v time.Duration) Attr
• Time(k string, v time.Time) Attr
• Any(k string, v any) Attr

The last of these dispatches on the type of v, using a more efficient representation if Attr supports it and falling back to an any field in Attr if not.

The Attr.Key method returns the key. Extracting values from an Attr is reminiscent of reflect.Value: there is a Kind method that returns an enum, and a variety of methods like Int64() int64 and Bool() bool that return the value or panic if it is the wrong kind.

Attr also has an Equal method, and an AppendValue method that efficiently appends a string representation of the value to a []byte, in the manner of the strconv.AppendX functions.

Loggers

A Logger consists of a handler and a list of Attrs. There is a default logger with no attributes whose handler writes to the default log.Logger, as explained above. Create a Logger with New:

func New(h Handler) *Logger

To add attributes to a Logger, use With:

l2 := l1.With("url", "http://example.com/")

The arguments are interpreted as alternating string keys and and arbitrary values, which are converted to Attrs. Attrs can also be passed directly. Loggers are immutable, so this actually creates a new Logger with the additional attributes. To allow handlers to preprocess attributes, the new Logger’s handler is obtained by calling Handler.With on the old one.

You can obtain a logger's handler with Logger.Handler.

The basic logging methods are

func (*Logger) Log(level Level, message string, kvs ...any)

which logs a message at the given level with a list of attributes that are interpreted just as in Logger.With, and the more efficient

func (Logger) LogAttrs(level Level, message string, attrs ...Attr)

These functions first call Handler.Enabled(level) to see if they should proceed. If so, they create a Record with the current time, the given level and message, and a list of attributes that consists of the receiver's attributes followed by the argument attributes. They then pass the Record to Handler.Handle.

Each of these methods has an alternative form that takes a call depth, so other functions can wrap them and adjust the source line information.

There are four convenience methods for common levels:

func (*Logger) Info(message string, kvs ...any)
func (*Logger) Warn(message string, kvs ...any)
func (*Logger) Debug(message string, kvs ...any)
func (*Logger) Error(message string, err error, kvs ...any)

They all call Log with the appropriate level. Error first appends Any("err", err) to the attributes.

There are no convenience methods for LogAttrs. We expect that most programmers will use the more convenient API; those few who need the extra speed will have to type more, or provide wrapper functions.

All the methods described in this section are also names of top-level functions that call the corresponding method on the default logger.

Context Support

Passing a logger in a context.Context is a common practice and a good way to include dynamically scoped information in log messages. For instance, you could construct a Logger with information from an http.Request and pass it through the code that handles the request by adding it to r.Context().

The slog package has two functions to support this pattern. One adds a Logger to a context:

func NewContext(ctx context.Context, l *Logger) context.Context

As an example, an HTTP server might want to create a new Logger for each request. The logger would contain request-wide attributes and be stored in the context for the request:

func handle(w http.ResponseWriter, r *http.Request) {
    rlogger := slog.With(
        "method", r.Method,
        "url", r.URL,
        "traceID", getTraceID(r))
    ctx := slog.NewContext(r.Context(), rlogger)
    // ... use ctx ...
}

To retrieve a Logger from a context, call FromContext:

slog.FromContext(ctx).Info(...)

FromContext returns the default logger if it can't find one in the context.

Levels

A level is a positive integer, where lower numbers designate more severe or important log events. The slog package provides names for common levels, with gaps between the assigned numbers to accommodate other level schemes. (For example, Google Cloud Platform supports a Notice level between Info and Warn.)

Some logging packages like glog and Logr use verbosities instead, where a verbosity of 0 corresponds to the Info level and higher values represent less important messages. To use a verbosity of v with this design, pass slog.InfoLevel + v to Log or LogAttrs.

Provided Handlers

The slog package includes two handlers, which behave similarly except for their output format. TextHandler emits attributes as KEY=VALUE, and JSONHandler writes line-delimited JSON objects. Both can be configured with the same options:

• The boolean AddSource option controls whether the file and line of the log call. It is false by default, because there is a small cost to extracting this information.

• The LevelRef option, of type LevelRef, provides control over the maximum level that the handler will output. For example, setting a handler's LevelRef to Info will suppress output at Debug and higher levels. A LevelRef is a safely mutable pointer to a level, which makes it easy to dynamically and atomically change the logging level for an entire program.

• To provide fine control over output, the ReplaceAttr option is a function that both accepts and returns an Attr. If present, it is called for every attribute in the log record, including the four built-in ones for time, message, level and (if AddSource is true) the source position. ReplaceAttr can be used to change the default keys of the built-in attributes, convert types (for example, to replace a time.Time with the integer seconds since the Unix epoch), sanitize personal information, or remove attributes from the output.

Interoperating with Other Log Packages

As stated earlier, we expect that this package will interoperate with other log packages.

One way that could happen is for another package's frontend to send slog.Records to a slog.Handler. For instance, a logr.LogSink implementation could construct a Record from a message and list of keys and values, and pass it to a Handler. To facilitate that, slog provides a way to construct Records directly and add attributes to it:

func NewRecord(t time.Time, level Level, msg string, calldepth int) Record

func (*Record) AddAttr(Attr)

Another way for two log packages to work together is for the other package to wrap its backend as a slog.Handler, so users could write code with the slog package's API but connect the results to an existing logr.LogSink, for example. This involves writing a slog.Handler that wraps the other logger's backend. Doing so doesn't seem to require any additional support from this package.

Acknowledgements

Ian Cottrell's ideas about high-performance observability, captured in the golang.org/x/exp/event package, informed a great deal of the design and implementation of this proposal.

Seth Vargo’s ideas on logging were a source of motivation and inspiration. His comments on an earlier draft helped improve the proposal.

Michael Knyszek explained how logging could work with runtime tracing.

Tim Hockin helped us understand logr's design choices, which led to significant improvements.

Abhinav Gupta helped me understand Zap in depth, which informed the design.

Russ Cox provided valuable feedback and helped shape the final design.

Appendix: API

package slog // import "golang.org/x/exp/slog"


FUNCTIONS

func Debug(msg string, args ...any)
    Debug calls Logger.Debug on the default logger.

func Error(msg string, err error, args ...any)
    Error calls Logger.Error on the default logger.

func Info(msg string, args ...any)
    Info calls Logger.Info on the default logger.

func Log(level Level, msg string, args ...any)
    Log calls Logger.Log on the default logger.

func LogAttrs(level Level, msg string, attrs ...Attr)
    LogAttrs calls Logger.LogAttrs on the default logger.

func NewContext(ctx context.Context, l *Logger) context.Context
    NewContext returns a context that contains the given Logger. Use FromContext
    to retrieve the Logger.

func SetDefault(l *Logger)
    SetDefault makes l the default Logger. After this call, output from the
    log package's default Logger (as with log.Print, etc.) will be logged at
    InfoLevel using l's Handler.

func Warn(msg string, args ...any)
    Warn calls Logger.Warn on the default logger.


TYPES

type Attr struct {
	// Has unexported fields.
}
    An Attr is a key-value pair. It can represent some small values without an
    allocation. The zero Attr has a key of "" and a value of nil.

func Any(key string, value any) Attr
    Any returns an Attr for the supplied value.

    Any does not preserve the exact type of integral values. All signed integers
    are converted to int64 and all unsigned integers to uint64. Similarly,
    float32s are converted to float64.

    However, named types are preserved. So given

        type Int int

    the expression

        log.Any("k", Int(1)).Value()

    will return Int(1).

func Bool(key string, value bool) Attr
    Bool returns an Attr for a bool.

func Duration(key string, value time.Duration) Attr
    Duration returns an Attr for a time.Duration.

func Float64(key string, value float64) Attr
    Float64 returns an Attr for a floating-point number.

func Int(key string, value int) Attr
    Int converts an int to an int64 and returns an Attr with that value.

func Int64(key string, value int64) Attr
    Int64 returns an Attr for an int64.

func String(key, value string) Attr
    String returns a new Attr for a string.

func Time(key string, value time.Time) Attr
    Time returns an Attr for a time.Time.

func Uint64(key string, value uint64) Attr
    Uint64 returns an Attr for a uint64.

func (a Attr) AppendValue(dst []byte) []byte
    AppendValue appends a text representation of the Attr's value to dst.
    The value is formatted as with fmt.Sprint.

func (a Attr) Bool() bool
    Bool returns the Attr's value as a bool. It panics if the value is not a
    bool.

func (a Attr) Duration() time.Duration
    Duration returns the Attr's value as a time.Duration. It panics if the value
    is not a time.Duration.

func (a1 Attr) Equal(a2 Attr) bool
    Equal reports whether two Attrs have equal keys and values.

func (a Attr) Float64() float64
    Float64 returns the Attr's value as a float64. It panics if the value is not
    a float64.

func (a Attr) Format(s fmt.State, verb rune)
    Format implements fmt.Formatter. It formats a Attr as "KEY=VALUE".

func (a Attr) HasValue() bool
    HasValue returns true if the Attr has a value.

func (a Attr) Int64() int64
    Int64 returns the Attr's value as an int64. It panics if the value is not a
    signed integer.

func (a Attr) Key() string
    Key returns the Attr's key.

func (a Attr) Kind() Kind
    Kind returns the Attr's Kind.

func (a Attr) String() string
    String returns Attr's value as a string, formatted like fmt.Sprint.
    Unlike the methods Int64, Float64, and so on, which panic if the Attr is of
    the wrong kind, String never panics.

func (a Attr) Time() time.Time
    Time returns the Attr's value as a time.Time. It panics if the value is not
    a time.Duration.

func (a Attr) Uint64() uint64
    Uint64 returns the Attr's value as a uint64. It panics if the value is not
    an unsigned integer.

func (a Attr) Value() any
    Value returns the Attr's value as an any. If the Attr does not have a value,
    it returns nil.

func (a Attr) WithKey(key string) Attr
    WithKey returns an attr with the given key and the receiver's value.

type Handler interface {
	// Enabled reports whether this handler is accepting records
	// at the given level.
	Enabled(Level) bool

	// Handle processes the Record.
	// Handle methods that produce output should observe the following rules:
	//   - If r.Time() is the zero time, do not output it.
	//   - If r.Level() is Level(0), do not output it.
	Handle(Record) error

	// With returns a new Handler whose attributes consist of
	// the receiver's attributes concatenated with the arguments.
	With(attrs []Attr) Handler
}
    A Handler processes log records produced by Logger output. Any of the
    Handler's methods may be called concurrently with itself or with other
    methods. It is the responsibility of the Handler to manage this concurrency.

type HandlerOptions struct {
	// Add a "source" attributes to the output whose value is of the form
	// "file:line".
	AddSource bool

	// Ignore records with levels above LevelRef.Level.
	// If nil, accept all levels.
	LevelRef *LevelRef

	// If set, ReplaceAttr is called on each attribute of the message,
	// and the returned value is used instead of the original. If the returned
	// key is empty, the attribute is omitted from the output.
	//
	// The built-in attributes with keys "time", "level", "source", and "msg"
	// are passed to this function first, except that time and level are omitted
	// if zero, and source is omitted if AddSource is false.
	ReplaceAttr func(a Attr) Attr
}
    HandlerOptions are options for a TextHandler or JSONHandler. A zero
    HandlerOptions consists entirely of default values.

func (opts HandlerOptions) NewJSONHandler(w io.Writer) *JSONHandler
    NewJSONHandler creates a JSONHandler with the given options that writes to
    w.

func (opts HandlerOptions) NewTextHandler(w io.Writer) *TextHandler
    NewTextHandler creates a TextHandler with the given options that writes to
    w.

type JSONHandler struct {
	// Has unexported fields.
}
    JSONHandler is a Handler that writes Records to an io.Writer as
    line-delimited JSON objects.

func NewJSONHandler(w io.Writer) *JSONHandler
    NewJSONHandler creates a JSONHandler that writes to w, using the default
    options.

func (h JSONHandler) Enabled(l Level) bool
    Enabled reports whether l is less than or equal to the maximum level.

func (h *JSONHandler) Handle(r Record) error
    Handle formats its argument Record as a JSON object on a single line.

    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
    and the value is output in RFC3339 format with millisecond precision.

    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
    and the value of Level.String is output.

    If the AddSource option is set and source information is available,
    the key is "source" and the value is output as "FILE:LINE".

    The message's key is "msg".

    To modify these or other attributes, or remove them from the output,
    use [HandlerOptions.ReplaceAttr].

    Values are formatted as with encoding/json.Marshal.

    Each call to Handle results in a single, mutex-protected call to
    io.Writer.Write.

func (h *JSONHandler) With(attrs []Attr) Handler
    With returns a new JSONHandler whose attributes consists of h's attributes
    followed by attrs.

type Kind int
    Kind is the kind of an Attr's value.

const (
	AnyKind Kind = iota
	BoolKind
	DurationKind
	Float64Kind
	Int64Kind
	StringKind
	TimeKind
	Uint64Kind
)
func (k Kind) String() string

type Level int
    A Level is the importance or severity of a log event. The higher the level,
    the less important or severe the event.

const (
	ErrorLevel Level = 10
	WarnLevel  Level = 20
	InfoLevel  Level = 30
	DebugLevel Level = 31
)
    Names for common levels.

func (l Level) String() string
    String returns a name for the level. If the level has a name, then that
    name in uppercase is returned. If the level is between named values, then an
    integer is appended to the uppercased name. Examples:

        WarnLevel.String() => "WARN"
        (WarnLevel-2).String() => "WARN-2"

type LevelRef struct {
	// Has unexported fields.
}
    A LevelRef is a reference to a level. LevelRefs are safe for use by multiple
    goroutines. Use NewLevelRef to create a LevelRef.

    If all the Handlers of a program use the same LevelRef, then a single Set on
    that LevelRef will change the level for all of them.

func NewLevelRef(l Level) *LevelRef
    NewLevelRef creates a LevelRef initialized to the given Level.

func (r *LevelRef) Level() Level
    Level returns the LevelRef's level. If LevelRef is nil, it returns the
    maximum level.

func (r *LevelRef) Set(l Level)
    Set sets the LevelRef's level to l.

type Logger struct {
	// Has unexported fields.
}
    A Logger generates Records and passes them to a Handler.

    Loggers are immutable; to create a new one, call New or Logger.With.

func Default() *Logger
    Default returns the default Logger.

func FromContext(ctx context.Context) *Logger
    FromContext returns the Logger stored in ctx by NewContext, or the default
    Logger if there is none.

func New(h Handler) *Logger
    New creates a new Logger with the given Handler.

func With(attrs ...any) *Logger
    With calls Logger.With on the default logger.

func (l *Logger) Debug(msg string, args ...any)
    Debug logs at DebugLevel.

func (l *Logger) Enabled(level Level) bool
    Enabled reports whether l emits log records at level.

func (l *Logger) Error(msg string, err error, args ...any)
    Error logs at ErrorLevel. If err is non-nil, Error appends Any("err",
    err) to the list of attributes.

func (l *Logger) Handler() Handler
    Handler returns l's Handler.

func (l *Logger) Info(msg string, args ...any)
    Info logs at InfoLevel.

func (l *Logger) Log(level Level, msg string, args ...any)
    Log emits a log record with the current time and the given level and
    message. The Record's Attrs consist of the Logger's attributes followed by
    the Attrs specified by args.

    The attribute arguments are processed as follows:
      - If an argument is an Attr, it is used as is.
      - If an argument is a string and this is not the last argument, the
        following argument is treated as the value and the two are combined into
        an Attr.
      - Otherwise, the argument is treated as a value with key "!BADKEY".

func (l *Logger) LogAttrs(level Level, msg string, attrs ...Attr)
    LogAttrs is a more efficient version of Logger.Log that accepts only Attrs.

func (l *Logger) LogAttrsDepth(calldepth int, level Level, msg string, attrs ...Attr)
    LogAttrsDepth is like Logger.LogAttrs, but accepts a call depth argument
    which it interprets like Logger.LogDepth.

func (l *Logger) LogDepth(calldepth int, level Level, msg string, args ...any)
    LogDepth is like Logger.Log, but accepts a call depth to adjust the file
    and line number in the log record. 0 refers to the caller of LogDepth;
    1 refers to the caller's caller; and so on.

func (l *Logger) Warn(msg string, args ...any)
    Warn logs at WarnLevel.

func (l *Logger) With(attrs ...any) *Logger
    With returns a new Logger whose handler's attributes are a concatenation of
    l's attributes and the given arguments, converted to Attrs as in Logger.Log.

type Record struct {
	// Has unexported fields.
}
    A Record holds information about a log event.

func NewRecord(t time.Time, level Level, msg string, calldepth int) Record
    NewRecord creates a new Record from the given arguments. Use Record.AddAttr
    to add attributes to the Record. If calldepth is greater than zero,
    Record.SourceLine will return the file and line number at that depth.

    NewRecord is intended for logging APIs that want to support a Handler as a
    backend. Most users won't need it.

func (r *Record) AddAttr(a Attr)
    AddAttr appends a to the list of r's attributes. It does not check for
    duplicate keys.

func (r *Record) Attr(i int) Attr
    Attr returns the i'th Attr in r.

func (r *Record) Attrs() []Attr
    Attrs returns a copy of the sequence of Attrs in r.

func (r *Record) Level() Level
    Level returns the level of the log event.

func (r *Record) Message() string
    Message returns the log message.

func (r *Record) NumAttrs() int
    NumAttrs returns the number of Attrs in r.

func (r *Record) SourceLine() (file string, line int)
    SourceLine returns the file and line of the log event. If the Record
    was created without the necessary information, or if the location is
    unavailable, it returns ("", 0).

func (r *Record) Time() time.Time
    Time returns the time of the log event.

type TextHandler struct {
	// Has unexported fields.
}
    TextHandler is a Handler that writes Records to an io.Writer as a sequence
    of key=value pairs separated by spaces and followed by a newline.

func NewTextHandler(w io.Writer) *TextHandler
    NewTextHandler creates a TextHandler that writes to w, using the default
    options.

func (h TextHandler) Enabled(l Level) bool
    Enabled reports whether l is less than or equal to the maximum level.

func (h *TextHandler) Handle(r Record) error
    Handle formats its argument Record as a single line of space-separated
    key=value items.

    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
    and the value is output in RFC3339 format with millisecond precision.

    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
    and the value of Level.String is output.

    If the AddSource option is set and source information is available,
    the key is "source" and the value is output as FILE:LINE.

    The message's key "msg".

    To modify these or other attributes, or remove them from the output,
    use [HandlerOptions.ReplaceAttr].

    Keys are written as unquoted strings. Values are written according to their
    type:
      - Strings are quoted if they contain Unicode space characters or are over
        80 bytes long.
      - If a value implements [encoding.TextMarshaler], the result of
        MarshalText is used.
      - Otherwise, the result of fmt.Sprint is used.

    Each call to Handle results in a single, mutex-protected call to
    io.Writer.Write.

func (h *TextHandler) With(attrs []Attr) Handler
    With returns a new TextHandler whose attributes consists of h's attributes
    followed by attrs.

[evanphx]

👋🏻 hclog developer here! Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into. slog.Logger could be something an implementation of that interface.

Were that route taken, I'd propose the surface area that looks something like this:

type Logger interface {
   Log(level int, msg string, args ...any)
   Debug(msg string, args ...any)
   Info(msg string, args ...any)
   Warn(msg string, args ...any)
   Error(msg string, args ...any)
   With(args ...any) Logger
}

var DefaultLogger Logger

func Debug(msg string, args ...any) {
   DefaultLogger.Debug(msg, args...)
}
// Toplevel for Info, Warn, Error, and With

func InContext(ctx context.Context, log Logger) context.Context { ... }
func FromContext(ctx context.Context) Logger { ... } // returns DefaultLogger if none available

[akshayjshah]

Curious about seeing if the stdlib could mostly contain the interface surface area for different implementations to step into.

Is this suggesting that the slog package export the Logger interface you outline, which third-party packages may then implement ("step into")?

If so, the interface you propose ends up being fairly slow because passing ints, strings, and other values as any often allocates. These allocations are problematic because they're nearly impossible to avoid. Even if debug-level logging is disabled,

logger.Debug("some high-volume thing happened", "key1", someString, "key2", someOtherString)

will make multiple allocations just to call the logger implementation, which then realizes that debug-level logs are disabled and does nothing. These unavoidable allocations make logging difficult on code paths that need to be fast.

This is often just fine - most code doesn't need to be particularly high-performance. If we're going to put structured logging into the stdlib, though, the logger interface should probably expose enough internals to be useful to performance-sensitive applications.

[evanphx]

That's a fair concern! Perhaps adding In(level int) bool to allow a user to guard the statements in high throughput areas is prudent here. It's been a little while since I benchmarked the allocations of string->any and int->any conversions, which are the 2 vast majority of argument cases.

With the more complex APIs, the trade is carefully crafting them to not allocate and having them sometimes suffer ergonomically and decrease the developer usage.

[evanphx]

Looking at the benchmarking, the only allocation today is is creating the slice it appears, no individual allocs for the arguments.

[vearutop]

I have a use case where I would like to control logging level based on context (e.g. enabling Debug for requests with magical header). Seems I can do this by creating Logger instance using debug-enabled Handler and then making NewContext with it.

However, this context instrumentation needs to happen before any other (for example context may already have a Logger.With(...)) to avoid loss of state.
Maybe it would make sense to allow replacing Handler using WithHandler(h Handler) *Logger to make such manipulations more flexible.

[jba]

The question is, is this common enough to warrant built-in support?

[jochumdev]

@jba that would mean the "root" handler (in this case "logger1") needs to have the lowest supported Level in subloggers, right?

[jba]

First of all, the comparison was wrong in this code because we changed the direction of levels. I made an edit to fix that.

Now to answer your question, I don't think it matters what logger1's level is. The "level" of a logger is just a way of talking about how its handler's Enabled method is written, and in this code the outermost handler has complete control—it does not delegate to the handler it wraps.

[jochumdev]

@jba ohh I see, so Enabled will not be called on the wrapped Handler, that's nice.

[Davincible]

The question is, is this common enough to warrant built-in support?

@jba if possible it would be very nice to have built-in support for this. Having to create a wrapper like that for a sub-logger with a different level feels a bit clunky. I feel like enabling different log levels for different components of your code could generally be quite useful.

[vearutop]

What would happen if kvs ...any arguments are not well-formed (odd number or key is not a string)?

[jba]

Is the intention of this library to be abstracted by other libraries

Intention is a strong word, but as I've written elsewhere in this discussion, I think it's inevitable.

alternatives such as zerolog exist, which "box" you into not making mistakes

Zerolog has a lovely API. I experimented with something like .String("k1", "s").Int("k2", 1)... at one point. There are a few downsides, though.

First, you have to remember to write Msg at the end of the line.

Second, it's a bit redundant to have to write Warn().....Msg(...). But you need the level at the beginning to filter, and something at the end to say you're done, so there's no alternative.

Third, it's hard to get good performance from the chaining API if you want a general backend that remembers the attributes. Either you allocate for each attribute, or you copy something that can hold several attributes, which gets expensive. zerolog doesn't have that problem, because it converts each attribute to bytes immediately and appends to a []byte. If that buffer is pool-allocated with a large enough size, you won't see any allocations (hence the name).

[meling]

It has probably been proposed by others in another context and rejected, but it would seem like a nice addition to the language to be able to specify variadic pairs. Something like ...{string, any}.

[edgmnt]

@meling, a struct containing a string and an any should be enough to prevent unbalanced key-value pairs (although other improvements may be considered):

type KV struct {
        Key   string
        Value any
}

func (log *Logger) Error(err error, msg string, kvs ...KV)

// Usage:
log.Error(err, "message",
        slog.KV{"handle", handle},
        slog.KV{"user", user})

[jba]

specify variadic pairs

I think this would fall under the category of eliding type names in literals. For example, #21496.

[jba]

type KV struct {
        Key   string
        Value any
}

Our Attr type is essentially this, but will not allocate memory for many values that this would.

[willfaught]

Looks promising. Some thoughts:

But more important than any traction gained by the "frontend" is the promise of a common "backend." An application with many dependencies may find that it has linked in many logging packages. If all of the packages support the handler interface we propose, then the application can create a single handler and install it once for each logging library to get consistent logging across all its dependencies. Since this happens in the application's main function, the benefits of a unified backend can be obtained with minimal code churn. We hope that this proposal's handlers will be implemented for all popular logging formats and network protocols, and that every common logging framework will provide a shim from their own backend to a handler. Then the Go logging community can work together to build high-quality backends that all can share.

Why is this being proposed now? Why not 5–8 years ago?

This seems to tout handlers as an innovation over the state of the art. Do none of the existing solutions have a comparable handler design?

Will this effort to establish a common interface for the community extend to other problem domains, like audio?

The program then outputs three log messages augmented with key-value pairs.

In my opinion, the loose ...any key/value pairs parameter design, like in go-kit, feels a little too "loose". I understand going with it, because I can't think of a better way to do it either, but it always felt like there was a better way out there somewhere. It's not worth getting hung up on, but I'd be very interested in any viable alternatives.

What happens if a value is missing?

The second logs at the Error level, passing an error and a key-value pair.

What if there isn't a Go error value when logging an error condition? What value should be used?

FromContext returns the default logger if it can't find one in the context.

Could this hide configuration or setup mistakes, where there should have been a logger, but there wasn't?

InfoLevel Level = 30
DebugLevel Level = 31

Why Is DebugLevel 31, when the other levels increment by 10?

TextHandler emits attributes as KEY=VALUE

What is the exact intended initial format, whether or not you want to document it?

What is the order of the fields? Are the "built-in" fields first?

Are the pairs delimited by a space?

Are strings containing whitespace quoted?

JSONHandler writes line-delimited JSON objects

Is the JSON minimized?

What is the order of the fields? Are the "built-in" fields first?

ReplaceAttr can be used to change the default keys of the built-in attributes

What are the default keys?

Functions like Any and Int construct slog.Attr values
slog.LogAttrs

Nit: Consider using the names Field/Fields, like zap. "Attr" is a little more abstract than "field," and "WithFields" is a little clearer and reads a little better than "WithAttrs," in my humble opinion.

Any(k string, v any) Attr

Why not include variants for all built-in types, like Int8, Rune or Complex64? Would generics help here?

The boolean AddSource option controls whether the file and line of the log call.

Have you considered the potential demand for a stack trace option (as opposed to just the current file name and line)?

func (l *Logger) With(attrs ...any) *Logger

I don't see a Logger.WithAttrs variant. Is that because it wouldn't help avoid allocations? If so, why is that?

[willfaught]

A common, popular logger design that has an error parameter

That would be logr

I don't think that design counts as common, as I showed with the design survey; only logr has this design.

I don't think that design counts as popular, either, but I acknowledge that logr is one of the more notable log packages, albeit in the lowest "tier" of notable (i.e. less than logrus and zap; at the same level as go-kit).

No, the main point is that it's more convenient. That is just a side benefit.

No, I assumed that none of them were. That is what I meant when I said the percentages alone would be evidence for an error argument: because even if none were mistakes, programmers would have an easier time overall.

OK, so this isn't about foot-guns, or about user-perceived convenience that we've measured, it's about observing that supplying error values correlates quite a bit with a particular log level, specializing part of the design around that particular use case, and assuming that users will find that more preferable. The correlation threshold for doing this is ~2/3 of the time; ~4/10 of the time is too low.

Let's consider your zap numbers again. Your analysis takes into account the convenience that the error parameter would have had ~2/3 of the time when there was an error value, but it doesn't take into account the inconvenience it would have had ~1/3 of the time when there wasn't an error value. It also doesn't take into account the design downsides of having a different method signature from the other levels. While weighing those trade-offs does involve some subjectivity, I think the consensus of the community is against the design of logr, and against this design, in this particular respect.

[smlx]

It also doesn't take into account the design downsides of having a different method signature from the other levels.

I also find this choice of inconsistency confusing. In my experience you often want to log errors at any log level, not just Error. This design also implies that if you have an error then you should be logging at Error, which is not correct.

[jba]

It's new and unfamiliar.
There's evidence that it is a net benefit. (It makes some things worse, but more things better.)
Let's try it. There will be plenty of time to kick the tires.

[hherman1]

Putting a vote in for the convenience of the error argument. Not clear to me why the slight inconsistency with other log levels matters.

[thockin]

As one of the logr authors, I can see both sides of this. My own argument AGAINST the logr style error arg is the need to give it a hard-coded "magic" key ("err").

I can buy the argument that fewer magic keys would be better. The counter argument is that there is value in a consistent name across all call-sites.

logr was informed by kubernetes, which really wanted the consistency and really does have an error in almost all Error() calls

[akshayjshah]

👋🏽 I'm glad to see this discussion reignited! I'm one of the original zap authors, and may have a little additional context to share. Points below notwithstanding, this proposal seems carefully thought-through and researched. Many thanks to the authors for all the work they've put in 😍

Previous art

Peter Bourgon and Chris Hines made a proposal with similar goals in 2017. The overall situation hasn't changed much since then - if anything, the proliferation of log-like APIs in distributed tracing packages has made it even worse. If the authors of this proposal haven't seen the previous doc, it's worth reviewing. I'm not sure if Chris and Peter are still interested in logging, but their perspectives would be valuable here.

I particularly liked the very small interface they proposed and the Valuer interface, even if both come at some performance cost.

Namespacing

I notice that this proposal doesn't include support for namespacing key-value pairs. For example, we might want a gRPC interceptor to produce JSON records like this:

{ ... standard fields ..., "grpc": {"method": "some.package/Method", ... other gRPC stuff...}}

Zap exposes this via Namespace. At least at Uber, this was critical for practical use: like many storage backends, ElasticSearch stops indexing if a field changes type. Implicitly, this forces each binary to have a schema for its log output. It's very hard to do this if dozens of loosely-coordinated packages are writing to a shared, flat keyspace. Namespacing cuts the problem into manageable pieces: each package uses its own name as a namespace, within which it can attempt to adhere to some implicit schema. This problem is significant enough that people have attempted to write static analysis to mitigate it.

Of course, not all storage engines have this limitation - but many do. Food for thought.

First-class error support

The error interface is infamously common in Go code, and logging errors is also common. It seems worth building in special support, rather than forcing users to call err.Error() eagerly. This also opens the door to a richer structured representation of errors. For example, the proposed multi-error support in the stdlib will be nearly unreadable if users shove err.Error() into JSON. If the old x/errors proposal for automatically capturing stacktraces re-emerges, we'd also want a sane way to include them in structured log output.

Zap spends a fair bit of effort to special-case go.uber.org/multierr and github.com/pkg/errors to paper over some of these complexities.

Edit: on a second reading, I see the error argument to slog.Error. I still think that it may be valuable to have a func Error(err error) Attr to use with other log levels (particularly debug).

Delegating serialization & complex objects

It's convenient for end users to let types implement their own serialization logic (a la json.Marshaler or zap.ObjectMarshaler). I don't see an equivalent here. How do you imagine types customizing their representation in logs? Implementing encoding-specific interfaces like json.Marshaler is difficult, since there's no telling what encoding the owner of main prefers. Should packages expose a function that allocates a []log.Attr for each loggable type? Is this wholly out of scope? Do users log each field of interest by hand?

Generics

My experience with zap is that the mere existence of a faster API, even it's painfully verbose, puts a lot of pressure on developers to use it. Making the lower-allocation API as ergonomic as possible has a disproportionate effect on developer experience.

One of my enduring regrets about zap is its fussy API for producing fields. Nowadays, this seems like the sort of problem we ought to be able to solve nicely with a

type Attr[T any] struct {
  key string
  value T
}

Minimizing allocations would require a fast path through marshaling that doesn't go through any, which I think would require support for type switches on parametric types. Personally, I'd love to see a sketch of what might be possible with #45380.

vet support

It'd be nice to also have go vet check for mistakes in the friendlier, alternating key-value API. People don't make many mistakes in practice, but it's nice to have a little extra assurance that you haven't left off an argument or mixed up types. Perhaps there's space for function names as unique as printf and friends, so that the same vet rule could apply to any similarly-named functions?

It's great though!

Even with the concerns above, I want to reiterate how encouraging this is. Large sections of the Go community seem to really want structured logs, and the current fragmentation is painful. Working around the limitations of a single stdlib implementation will be much better than working around the quirks of the many structured loggers today.

Edit: I realize that this proposal isn't a referendum on structured logging in general. Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. 🤷🏽‍♂️ I seem to be out of step with the rest of the community on this, though, and I do see this proposal as a step forward for structured logging.

[rabbbit]

I'm a biased zap user but would like to +1 @akshayjshah ideas on:

• namespacing: it is quite frustrating when you import a new library, it starts logging, and now either all my dashboards are garbage, or just disappear when ELK invisibly stops ingesting. This is painful when you don't control either of the libraries.
• specialized func Error(err error) Attr. It's surprisingly pleasant not to even have to deal with naming the error (slog.Info("e", err.Error()) || slog.Info("err", err.Error()) || slog.Info("error", err.Error()) vs just slog.Info(slog.Err, err)

[AndrewHarrisSPU]

Nevertheless, as one of the perpetrators of zap, I do feel compelled to say that I'm personally still not convinced that the fussiness of structured logging is worth it. On balance, I prefer leaving this degree of structure to distributed tracing and Prometheus-style tagged metrics and keeping logs as simple printfs. 🤷🏽‍♂️

This made me grin. FWIW, my own experience with zap was that I did end up finding structured logging useful; I enjoyed it a lot more when I switched from zap's sugared logger to DIY, local sugar.

[thockin]

Regarding a specific method for types to marshal log-data -- logr defines such a method, too. It seemed "obviously useful", so +1 on that

[riking]

Specific method for types to marshal log data - isn't this just MarshalJSON, MarshalText, and friends in other encoding schemes? Or are we talking about something producing a bunch of Attr?

[thockin]

In logr (which does not use an Attr equivalent), this method returns an any, which is then rendered in the normal way (whatever the given backend would do).

This means you can return a string, but you can also return an int, a map, a slice, a struct, etc. The original key for this value still holds.

So, for example, if I am logging a struct, I can use this method to wipe sensitive fields or convert to a simpler structure entirely.

[seankhliao]

ReplaceAttr func(a Attr) Attr

This allows changing or omitting an Attr, but not splitting it out into multiple.
At our company, logging calls take a context.Context,
and TraceID/SpanID (and other metadata) is extracted from there and injected into log records.
We do it this was as it's easier to remember, and span ids aren't yet visible at higher levels (eg wrapping handlers).

type Level int

I'd like to see it implement encoding.TextMarshaler / encoding.TextUnmarshaler for use in flags

type Handler interface

I'd like to see a standard handler that forwards to testing.TB.Logf

func SetDefault

zerolog provides a global logger as a separate package,
I've found it quite useful in catching if you use the wrong logger (global vs context-derived).

---

The other recent (big?) project that I'm aware of in logging standardization is OpenTelemetry's Log Data Model, which has recently been stabilized.

There they chose a different mapping of Levels / Severity to numbers,
I wonder if it would be worth using the same numbering as theirs?
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-severitynumber

---

I'll echo the desire for namespaced key-values,
but I think having it on the handler level isn't quite right,
as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries.
For JSON it looks halfway possible with Any,
maybe there could be another one for Attrs(key string, attrs...Attr) Attr?

[willfaught]

Perhaps there should be (*Logger).WithNamespace(string) *Logger and NewLoggerNamespace(string) *Logger that prepend/wrap namespaces around keys.

[jba]

I'll echo the desire for namespaced key-values,
but I think having it on the handler level isn't quite right,
as there are cases when you want to write into different namespaces partitioned by business logic rather than hard package boundaries.

Can you give an example of how this would look?

[seankhliao]

I've seen 2 patterns used:

• log with dotted key strings and rely on the backend to merge them:
  in middleware log = log.With("workflow.request_tyoe","...", "request.source", "...")
  and later log.Info("done", "workflow.result")
  but this can be somewhat difficult to process properly
• update a context object that's passed along
  and leave it to the final caller or logger processing hooks to expand it into the appropriate fields,
  the above log would be {"msg":"done", "request":{"source":"..."},"workflow":{"request_type":"...","result":"..."}}

would be really nice if somehow interoperated with opentelemetry's attribute package, our developers are already exposed to that for annotating traces/spans.

[hherman1]

What if there was just a new Attr type that was a namespace attr, and supported passing in a collection of sub-attrs? Seems performant + tiny extension to the existing api

[hherman1]

E.g ‘sllog.Namespace(ns string, attrs Attr…)

[sagikazarmark]

Thanks for starting this discussion. The sheer amount of logging libraries out there seems to be an indicator that there should be a solution for this in the standard library.

A couple thoughts about the proposal:

OpenTelemetry

I think it would make sense to look at the work the OpenTelemetry working group does...at least to make sure this proposal is not incompatible with what they are working on.

I know they don't focus on libraries itself at the moment, but a wire format for logs.

Sugared logger

Personally, I'm quite fond of Zap's default logger enforcing attributes to be Field instances. Strong type safety sounds like a good default. For people who are willing to sacrifice that can always use the Sugared logger.

Maybe reflecting that in this proposal would also make sense by creating separate implementations.

Attr

I was wondering if it would make sense to make Attr an interface:

type Attr interface {
    Key() string
    Value() string
}

That way converting a value to string could be handed off to a custom Attr implementation. I don't know if that would affect allocations though. Maybe they would, but it's good enough for Zap. This would also allow to create separate types instead of using a single Attr with an any value.

I was wondering what the purpose of Kind is. Where would it be useful?

Context

Personally, I prefer extracting information from the context in the logger, not the other way around.

Let's say a context travels through a set of layers, each of them adding information to the context that you would like to log.
The best place to extract all those information is at the place you actually want to log something.

Therefore I'd want to pass the context to the logger somehow, not the other way around. For example, I could implement a handler that extracts all the relevant information from the context. The only question is: how do we pass a context to the logger?

[pohly]

How does that work if two different, unrelated packages both want to get additional values logged? If two functions add such an interface to the context, the handler will only retrieve the more recent one. Each function which adds that interface would need to check whether there is already one and support chaining in its interface implementation. This all feels rather clunky and would still depend on adding multiple things to a context (values plus interface for extracting them).

Or do you suggest that the main function adds the interface only once? That only works if it knows about all additional values that packages may want to log. If that's a long list, then the handler will have to check every single one on each log call, including the ones that were not added in the current call chain.

It also only solves "what to log" problem. It doesn't address "how to log". A handler could be attached to a context instead of a logger, but IMHO attaching a logger and encapsulating additional values in that logger is simpler overall.

[edgmnt]

A helper or set of helpers is all you need to log using a context, while the core API just takes a logger. E.g.

func A(log slog.Logger) {
        log.Info(...)
}

func B(ctx context.Context) {
        slog.FromContext(ctx).Info(...)
        // ... or ...
        slog.Info(ctx, ...)
}

func C(ctx context.Context) {
        A(slog.FromContext(ctx).WithValues(...))
        B(slog.WithValues(ctx, ...))
}

You could find a way to track attributes separately from the logger, perhaps lumping them up in a single context-like object, but I don't see the point.

I also think some comments may be missing the subtle fact that this doesn't have to be a complete, heavy logger that's expensive to create, pass around or swap out, it's simply a harness that does something with the logged parameters to allow decoupling. It's a sort of callback/closure with a bit of extra structure.

[edgmnt]

I think it's more worthwhile exploring whether you can collect and stuff attributes into normal error values. Because then you'll also be able to return errors as usual and log them higher in the call chain, where they may ve recoverable. All while avoiding logging the same error multiple times, which is fairly problematic when packages decide to both log and return the same base error.

[mgood]

Why couldn't you do exactly that (and pass the context down as a Logger attribute)?

I'll try to provide a more concrete example of a similar concern I have about the context and tracing. Here's a simple adaptation of one of the examples from https://pkg.go.dev/go.opentelemetry.io/otel/trace to add a logging line. It uses FromContext to look up a logger, but that logger won't have info about the current span:

var tracer trace.Tracer

func init() {
	tracer = otel.Tracer("instrumentation/package/name")
}

func operation(ctx context.Context) {
	var span trace.Span
	ctx, span = tracer.Start(ctx, "operation")
	defer span.End()
	
	slog.FromContext(ctx).Info("msg")
}

So, IIUC the way you would have to do that is wrap each tracer.Start call to create a new logger with the span info, and return slog.NewContext to include the attributes.

That would work, however there may be cases where the connection between tracing and logging is more decoupled. E.g. let's assume these functions are split across packages:

func TracedOp(ctx context.Context) {
	var span trace.Span
	ctx, span = tracer.Start(ctx, "operation")
	defer span.End()
	foo.LoggedOp(ctx)
}

package foo
func LoggedOp(ctx context.Context) {
	slog.FromContext(ctx).Info("msg")
}

Here, we've lost the context info about the span again, but now we have to make one of these places aware of either logging or tracing. If these are 3rd party packages that's even more difficult.

So, we have the necessary info available on the context at the point of the logging call, but to actually get to that information we would need to add additional coupling between logging and tracing throughout the application.

Providing a way to pass the context through would enable you to configure a Context-aware logger at some top-level entry point instead. Then each logging call would extract the info needed from the Context provided at that point.

[svzhl]

I agree with the original comment about working with context and believe that extracting logger from a context is a design smell.
Lets look at the following code:

func (f *Foo) Bar(ctx context.Contex) {
	...
	f.logger.Info(ctx, "Bar processed")
	...
}

If you read this line you know that:

1. A customized logger configured for Foo struct is used. You can easily find out of logger configuration if required.
2. All logging attributes added to context will be logged.

The semantics is pretty strait forward and clear.

Compare to the proposal:

func (f *Foo) Bar(ctx context.Contex) {
	logger := slog.FromContext(ctx)
	...
	logger.Info("Bar processed")
	...
}

Looking at Info method call it is unclear which logger is used. Is it taken from context? Is it a brand new instance of logging? Does the logger contain all necessary baggage attributes? Things are getting worse if a function works with 2 instances of context. Every log method call requires careful investigation.

If we use another way:

func (f *Foo) Bar(ctx context.Contex) {
	...
	slog.FromContext(ctx).Info(ctx, "Bar processed")
	...
}

It is clear that a logger from context is used. But it would be difficult to find all the places where the instance of logger is modified and added to context for a particular call stack. So we really don't know here how logger is configured. A third party library could override the logger in a wrong way and add it to context which is not safe. Fixing a bug would be not an easy journey.

Although most of popular existing log libraries don't accept context as a parameter it is possible to wrap the logger. This is what we do usually. The proposed slog package also has such opportunity. But I believe a modern log package should enforce accepting context.

[rogpeppe]

I think this is promising. A bunch of thoughts:

• a few times in the docs, it talks about an Attr "having" a value. I'm not sure that's very helpful (and it wasn't very obvious what it meant to me). How about just calling it "nil" as that's presumably what it is under the hood. Then HasValue becomes redundant - it's just a.Kind() == NilValue
• why not make the zero value of LevelRef usable?
• I'd like to see more justification for the arbitrary-seeming level constants. Why 10 apart and not 1, 100 or 1000? Why no gap between Info and Debug? It seems to me like the arbitrary numbers make it easier to get things wrong and are less efficient (no trivial mapping to an array)
• some mention could be made of how NaN and Infinity are handled when marshaling to JSON.
• "Strings are quoted if they contain Unicode space characters" - they should surely be quoted if they contain a quote character too? The quote format should also be defined (presumably Go string syntax, but this could be made clear)
• It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments. If there are any other handlers defined externally, that's the pattern they'll need to use, and as methods, there's a distinct possibility that they might be pulled into a binary even when a json dependency isn't desired.

[jba]

a few times in the docs, it talks about an Attr "having" a value

Agreed. We can remove Attr.HasValue and any talk of an Attr "having" a value.

It seems a bit odd to me that the handler constructors are methods on the options, rather than just regular functions that take the options as arguments.

This pattern avoids a nil argument to the constructor. It has some precedent in the standard library:

net: Listen, ListenConfig.Listen
go/printer: Fprint, Config.Fprint
go/types: Config.Check (although there is no top-level Check)

If there are any other handlers defined externally, that's the pattern they'll need to use

Why?

[pohly]

they should surely be quoted if they contain a quote character too

We got complaints from Kubernetes developers after switching to structured logging and doing exactly that in the klog text output for structured log calls: the problem was that in a few places, the value was a long, multi-line string with plenty of quotation marks. The output then became completely unreadable.

The solution was a different quoting scheme for strings with line breaks - see kubernetes/klog#273

[firelizzard18]

I'm working with a framework that catches and logs panics (not a behavior I like but I don't really have a choice). It quotes the stack trace and replaces newlines and tabs with \n and \t. This makes the long output entirely unusable until I copy it into an editor and unescape those characters. I'd love something like what kubernetes/klog#273 describes.

[rogpeppe]

Using generics

In one of the comments above, there was mention of the possibility of using generics to mitigate some of the API blowup from having many explicitly different Attr constructors and methods on Attr to retrieve values.

I'll expand a bit on my reply above. I think we can use generics to mitigate allocations and reduce the API, even in the absence of #45380.

Here's how the API might look. We'd remove the Duration, Time, etc constructors and methods and replace them with this:

// Mk returns an Attr for the supplied value.
// Mk does not preserve the exact type of integral values. All signed integers
// are converted to int64 and all unsigned integers to uint64. Similarly,
// float32s are converted to float64.
//
// It special-cases the following types to avoid allocations:
//
//	bool
//	int, int8, int16, etc
//	uint, uintptr, uint8, uint16, etc
//	time.Duration
//	string
// 	time.Time
//	float32, float64
//
//   However, named types are preserved. So given
// 
//       type Int int
// 
//   the expression
// 
//       log.Any("k", Int(1)).Value()
// 
//   will return Int(1).
func Mk[T any](key string, value T) Attr

// Value returns the value part of a. It panics if T is not [any]
// and doesn't match the kind reported by a.Kind().
//
// The above condition implies that T must be one of the
// following types:
//	any, int, uint, time.Duration, time.Time, string, bool, float64
func Value[T any](a Attr) T

Another possibility to make the performance implications of creating an Attr a bit clearer might be to define a constraint that allows all the known concrete types, and then keep Any but define the above Mk in terms of that:

type Types interface {
    time.Duration | time.Time | int | uint | ...
}

func Mk[T Types](key string, value T) Attr

Another possible spelling for Mk might be A (or if Attr was renamed to Field, F).

As I mentioned in the original comment, it's possible to implement this API without incurring allocations, although there is some performance overhead.

[jba]

Thanks for finding this trick. (I had gotten as far as switching on *T, but never considered that any(val) wouldn't allocate.)

One possibility is to replace Any with Any[T]. Any[any] would behave like the current Any. Most or all of the constructors could go away.

I'm less convinced of Value[T](Attr). It's much more clumsy to write slog.Value[string](a) than a.String(), and the constraints on T make it no more powerful than the handful of methods we have now. It might be worth it if

slog.Value[MyInt](slog.Any("k", 1))

would work, where MyInt's underlying type is int. But that would require switching on ~int.

[AndrewHarrisSPU]

But that would require switching on ~int.

FWIW, there's an unsafe trick that bypasses Kind() or switching, and could be used on ~int:

func value[T any](rhs any) T {
	var t T
	lhs := any(t)
	lhsPtr := (*uint64)(unsafe.Pointer(&lhs))
	rhsPtr := (*uint64)(unsafe.Pointer(&rhs))
	if *lhsPtr != *rhsPtr {
		panic("unsafe assertion")
	}

	return rhs.(T)
}

This must take an any because it relies on the first word of an interface being associated with dynamic type/index into runtime itab table. Any constraint on what an Attr can hold is exogenous to this call. Works on godbolt.org for a lot of things. I don't know if it's a great idea.

edit: this is a dumb example, just a type assertion in practice - the trick is really that the unsafe pointer value for rhs can be grabbed and stored ahead of time

[jba]

You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you. The implementation of reflect.TypeOf is not much more than what you have.

Anyway, this is an exact check, and to make slog.Value[MyInt](slog.Any("k", 1)) work we need to ask if MyInt's underlying type is int.

[AndrewHarrisSPU]

You might see if reflect.TypeOf(lhs) == reflect.TypeOf(rhs) works for you

Pragmatically it's fine. Mostly I've been interested/distracted in exploring what's possible where the FGG/FG paper suggests covariant method receiver types. I feel like the typing and type checking of Attrs is a case where exhaustive pattern matching on a type switch would be reasonable way to structure code, it's just not quite in the language.

Another fuzzy idea, in slog this would be a bit nasty:

type TypicalInt[T constraints.Signed ] int64

func (TypicalInt[T]) Typical() {}

Attrs:

type Attr struct {
	k string
	n int64
	some interface{ Typical() }
}

func Any[T interface{ Typical() }]( key string, value T ) Attr {
	if v, ok := any(value).( interface{ UnwrapInt() int64 }); ok {
		return Attr{key, v.UnwrapInt(), value } // or nil for the last field is possible
	}

	return Attr{key, 0, value}
}

func (attr Attr) Int() int64 {
	if _, ok := attr.some.(interface{ UnwrapInt() int64 }); ok {
		return attr.n
	}
	panic("oops")
}

But, downstream, in myPkg importing slog, this is possible, and any TypicalInt is known to be convertible to int64

// package myPkg
type myInt slog.TypicalInt[myInt]

func(Int) Typical() {}
func (n Int) UnwrapInt() int64 {
	return int64(n)
}

Dunno if this would actually help to coordinate backends / loggers or just pushes the problem elsewhere. It's messy, and it's type erasure and coupling.

edit: fixed exporting

[rogpeppe]

The docs should document what should happen when With is used to attach the same attribute key twice. Does it override the original attribute, get ignored, or is the behaviour undefined?

[jba]

No duplicate checking is done anywhere. Too expensive. The Record we deliver to your Handler will have a raw sequence of Attrs, exactly as you added them. A Handler can of course dedup if it wishes, but the ones we provide do not.

I will make this clear in the next version of the doc.

[willfaught]

"attrs" in

func (l *Logger) With(attrs ...any) *Logger

should be "args" like in

func (l *Logger) Warn(msg string, args ...any)

to not suggest that attrs must be Attrs.

[jba]

Will do.

[mvrhov]

I hate the passing the attributes as key, values e.g kvs ...any from the bottom of the heart. This is going to be a bottomless pit of problems, Where one will mess up the key value pairs and the the hole log line will be messed up.
I've messed up the string replacers key value pairs numerous times and
Another thing is when logging a lot of data long lines are completely unreadable and one really needs to take a time to see on what goes with what.
I really like what zerolog does it with "fluent" calls. and a function for each type.

[vearutop]

I never had such a problem and I think that was due to using one line per pair convention.

slog.Info("message",
  "foo", 1,
  "bar", 2,
)

[jba]

@mvrhov, I think the vet check will help a lot.
It seems that experience with other, similar log packages has shown that this can work.

[sagikazarmark]

Whenever I have to use an API like this, a part of me dies as well. This is where IMO Go takes simplicity too far, instead of making a type system/standard library that's able to represent key-value pairs efficiently. The result is that we are losing build time safety. But that's just my opinion.

[sebfan2]

I agree with this, instead of typed arguments we are relaying a custom and unique formatting style (1 argument the on first line then 2 augment pairs on the following lines) to make the code readable and vetting to ensure it's correct. We also miss out on formatting via standard go fmt.

[deefdragon]

@jba I feel like the argument of "we can add vet checks for it" should be a serious warning as to potential problems, not a catch-all. We should do what we can to avoid requiring vet warnings to safely develop go imo.

[willfaught]

For what it's worth, here's a stab at an alternative to the unstructured k/v design func(string, ...any):

func (*Logger) Any(string, any) *Logger
func (*Logger) Bool(string, bool) *Logger
func (*Logger) Int(string, int) *Logger
func (*Logger) String(string, string) *Logger
// ...

func (*Logger) With(...Field) *Logger

func (*Logger) Log(Level, string)
func (*Logger) LogDepth(int, Level, string)

func (*Logger) Error(string)
func (*Logger) Warn(string)
func (*Logger) Info(string)
func (*Logger) Debug(string)

func (*Logger) Namespace(string) *Logger // Or perhaps "Wrap" instead of "Namespace"

func Any(string, any) Field
func Bool(string, bool) Field
func Int(string, int) Field
func String(string, string) Field
// ...

Example use:

var pkgLogger *slog.Logger = companyLogger.Namespace("mypkg")

// ...

var endpointLogger *slog.Logger = pkgLogger.Namespace("myendpoint")

// ...

if err != nil {
    var errLogger = endpointLogger.Any("save error", err).Int("userid", user.ID)

    errLogger = errLogger.Bool("prod", env.Prod)
    errLogger.Error("cannot save") // Logs with "save error", "userid", and "prod" keys

    // ...

    if err != nil {
        errLogger.Any("cleanup error", err).Error("cannot cleanup") // Logs with "save error", "userid", "prod", and "cleanup error" keys
    }

    // ...
}

// ...

if err != nil {
    endpointLogger.With(
        slog.Any("error", err),
        slog.Bool("baz", baz),
        slog.Int("bar", bar),
        slog.String("foo", foo),
    ).Error("cannot do thing")
}

Highlights:

• Specifying keys and values is separate from specifying the level and message.
• Specifying k/v's per line and per Logger is the same.
• Every level convenience method has the same func(string) signature.
• You can avoid allocations with typical use without having to deal with Field directly.
• Compile-time checks for well-formed code. No linting required.
• Error values are treated like any other interface value with any.

[willfaught]

@deltamualpha:

As much of a cop-out as it is to make a recourse to calling something "unidiomatic"... fluent interfaces are very unidiomatic for the standard library.

Is the builder pattern un-idiomatic? The fluent design seems just like it with an extra chaining convenience. Although, in this case, the "chaining" is the attachment of fields to a logger, which the original design also has.

This also has the (perhaps intentional) effect of making log.Info(sliceOfStrings...), where the slice is built up over time and then dropped into the logger, impossible.

I think it would be []any in that case, right?

This design accommodates the same accumulation scenario with Logger.With and []Field. Instead of accumulating strings, you accumulate Fields.

I guess the fluent part could be left out, and only have Logger.With(...Field). Logger.Bool/etc. are just conveniences for doing that anyway. But I wouldn't favor that, for the same reason we want Info() and Warn() conveniences for Log().

---

@jba:

For example, you need a name for the type returned by Logger.With, although no one should ever be using that type directly.

Doesn't your own Logger.With return a Logger? I'm not sure what you mean.

And to support logging a message with no attributes, you have to repeat all the methods of that type on Logger.

What's being repeated? Methods like Bool? Where are they being repeated? There's only one logger type in this design: Logger. I'm not sure what you mean.

As with the current proposal, in the common case.

I meant that they both happen in the same place/way. In your design, we can specify fields for the logger with Logger.With, and for the line with Logger.Log/Info/etc. In this design here, there's only one place to specify fields (Logger.With/Bool/etc.), so there isn't API duplication.

[smlx]

We hear the folks who would rather do the extra typing to construct Attrs for more compile-time checking.

Yes, please! 😄

[jba]

For example, you need a name for the type returned by Logger.With, although no one should ever be using that type directly.

Doesn't your own Logger.With return a Logger? I'm not sure what you mean.

OK, I didn't read your API carefully enough. If you have With return *Logger, then it has to allocate a new *Logger, and that means every line like

logger.With(...).Info(...)

will have to allocate, and performance will suffer.

As an exercise, you might want to implement your design and benchmark it against Zap. The Zap project has some good benchmarks that you can adapt.

When I said "you need a name for the type returned by Logger.With," I was thinking of an API where With returns an intermediate object that avoids allocating. It is possible to get good performance like that, but as I said, the API is messy.

So in short, I agree your design is elegant, but I don't see how to make it fast.

[daenney]

It ends up being somewhat redundant and confusing when you look at the godoc.

If a fluent API is the right fit (regardless of whether that's the case here specifically), we should not be avoiding that "because the godoc ain't pretty". Godoc needs fixing then, it's a valid way to design a package's API. It strikes me as really peculiar that the documentation generator's output would be a factor in the API design at this level.

[ldemailly]

As an exercise, you might want to implement your design and benchmark it against Zap. The Zap project has some good benchmarks that you can adapt.

Note that there are more benchmarks and loggers in https://github.com/imkira/go-loggers-bench

[AndrewHarrisSPU]

😱 should this be "panics if the value is not a time.Time"?

func (a Attr) Time() time.Time
    Time returns the Attr's value as a time.Time. It panics if the value is not a time.Duration.

--

I really like the Record/Handler/Logger arrangement. I think it's really sensible for thinking about and capturing middle-end structure.

I'm unclear on whether to consider Record's [ time, level, depth, msg ] as Attrs - I think they are all morally Attrs, but are not emitted by the Record.Attrs method?

For consistency, this could be useful (I don't think slog's Error severity should be the only place to expect "err"-keyed Attrs):

func (r *Record) AddErr(err error)
	Appends Any("err", err) as with Logger.Error

Bikeshedding, could imagine renaming Logger.LogDepth, Logger.LogAttrsDepth -> Logger.Recur, Logger.RecurAttrs.
I think the Logger.Log... methods are crowded in documentation, and log.Recur( n, ... ) seems straightforward.

The verbosity design doesn't feel immediately intuitive - Why decimal instead of coarse/fine bits? Why use negative verbosities like "WARN-2"? How does this interact with tracing?

[jba]

Should this be "panics if the value is not a time.Time"?

Thanks, fixed in the source.

I think they are all morally Attrs

Yes, but for efficiency they're separate. It is tempting to have a more unified design, but then handlers have to walk the sequence of Attrs to find certain keys (so they can format the record as TIMESTAMP LEVEL MESSAGE ..., for example), and it starts getting expensive.

The verbosity design doesn't feel immediately intuitive

It's an attempt to accommodate logr.

[AndrewHarrisSPU]

Why "Recur"? I don't get the name.

My thinking is that (all? almost all?) Depth calls are paying attention to some kind of recurrence. Maybe Depth is better.

It's also an attempt to set the LogDepth methods apart, the way they involve the call stack feels disjoint to me - it's always a little clever, and in various usages can be layers of cleverness. The methods LogDepth, LogAttrsDepth are more involved and finicky than their siblings Log and LogAttr, or their cousins Debug, Info, Warn, Error.

This is pretty minor and bike-sheddy :) I can move on.

[jhenstridge]

I think the main use case for the Depth calls is for logging within utility functions (e.g. a helper that logs some information about an http.Request). The call site of the Log calls within the utility function is not particularly interesting, but the next frame up is. So the utility function uses LogDepth() to record that.

[jba]

Right, either a helper or a complete wrapper of this API. For instance, maybe you prefer to write Log(ctx, level, msg) instead of FromContext(ctx).Log(level, msg). We noticed a lot of wrappers in the wild.

[AndrewHarrisSPU]

I was probably mistaken about how LogDepth would be commonly used. In the smallest way, nonplussed about the lexicographical closeness LogDepth in the docs or autocomplete, etc.

[jhenstridge]

It looks like this interface would integrate fairly well with systemd-journald's native protocol. In particular, the Handler.With() method looks like it'd make it possible to serialise the common attributes once and use it as a prefix for each log message datagram.

One question that I didn't see mentioned is how duplicate attributes should be handled. The API obviously doesn't do anything to prevent them, but is there any behaviour a handler is expected to implement? For example, consider calls like:

l.Info("message", "foo", "value1", "foo", "value2")
l2 := l.With("foo", "value1")
l2.Info("message", "foo", "value2")

Are these log messages valid? Is the handler expected to log both values for the attribute? Is it allowed to log only one? If it does only log one value, is it free to choose which one? If it logs multiple values, is it expected to preserve the order?

I realise that the answer might be "it is implementation defined", but if that's the case it should probably be spelled out what behaviour callers can rely on.

[jba]

I think it does have to be left up to the implementation. For speed a Handler must be allowed to ignore duplicates.

You can always wrap a Handler in another that pre-processes the list of Attrs however it wants.

[adamdecaf]

I worry that adding this to the standard library would be done so with an API that doesn't fully satisfy the intended use-cases. There are already several popular and tested libraries available for people to use. The Go team has often said that multiple community solutions are better than one lackluster "official" package, so why does the Go team need to create/maintain/improve a package when there are multiple better alternatives already in the wild?

[jba]

why does the Go team need to create/maintain/improve a package when there are multiple better alternatives already in the wild

One reason: to create a common backend API so an application's logs are consistent with each other. See the third paragraph of "What does success look like?" in the top post.

[firelizzard18]

The Go team has often said that multiple community solutions are better than one lackluster "official" package, so why does the Go team need to create/maintain/improve a package when there are multiple better alternatives already in the wild?

That is opinion, not fact. In my opinion, this proposal is a great compromise and is in no way lackluster and the community solutions are different but not inherently better. Plus the reason Jonathan mentioned.

[WillRubin]

so why does the Go team need to create/maintain/improve a package when there are multiple better alternatives already in the wild?

Was just reading about this elsewhere. The answer was, for a package designer, if you pick a logger for your package and a half dozen other package designers pick different loggers then a user of all these packages will have the joy of having a half dozen different loggers imported into their applications.

[sagikazarmark]

I guess that's a valid question, but as @WillRubin pointed out not having a standard solution has its own problems. Also, it's not like those existing solutions will have to be replaced, though some level of consolidation is expected in the ecosystem.

[jba]

This discussion has served its purpose: we now have a proposal. Further discussion should happen there. The implementation is now feature-complete (though lacking polish) and ready for you all to hammer on.

I want to thank everyone who participated here for their thoughtfulness, enthusiasm, wisdom and professionalism. It was a pleasure for me to work with you all to improve the design. As a result of this discussion, we added several features, including a way to group attributes into composites (Group), a method for replacing a value with another (LogValue), and a way to inject contexts into loggers (WithContext). We also made a number of other changes, like improving the mapping from levels to integers (only twice, but don't worry, there's still time).

The API has changed in a number of small ways as well, partly due to this discussion but also during code review. Rather than edit the top post with the new godoc, I thought it would be more useful if I provided a diff of the current godoc and the one at the top. If you want to see the new godoc on its own, scroll to the end of the design doc (review in progress).

--- old
+++ new
@@ -1,6 +1,25 @@
 package slog // import "golang.org/x/exp/slog"
 
 
+CONSTANTS
+
+const (
+	// TimeKey is the key used by the built-in handlers for the time
+	// when the log method is called. The associated Value is a [time.Time].
+	TimeKey = "time"
+	// LevelKey is the key used by the built-in handlers for the level
+	// of the log call. The associated value is a [Level].
+	LevelKey = "level"
+	// MessageKey is the key used by the built-in handlers for the
+	// message of the log call. The associated value is a string.
+	MessageKey = "msg"
+	// SourceKey is the key used by the built-in handlers for the source file
+	// and line of the log call. The associated value is a string.
+	SourceKey = "source"
+)
+    Keys for "built-in" attributes.
+
+
 FUNCTIONS
 
 func Debug(msg string, args ...any)
@@ -18,11 +37,11 @@
 func LogAttrs(level Level, msg string, attrs ...Attr)
     LogAttrs calls Logger.LogAttrs on the default logger.
 
-func NewContext(ctx context.Context, l *Logger) context.Context
+func NewContext(ctx context.Context, l Logger) context.Context
     NewContext returns a context that contains the given Logger. Use FromContext
     to retrieve the Logger.
 
-func SetDefault(l *Logger)
+func SetDefault(l Logger)
     SetDefault makes l the default Logger. After this call, output from the
     log package's default Logger (as with log.Print, etc.) will be logged at
     InfoLevel using l's Handler.
@@ -33,38 +52,46 @@
 
 TYPES
 
-type Attr struct {
+type AtomicLevel struct {
 	// Has unexported fields.
 }
-    An Attr is a key-value pair. It can represent some small values without an
-    allocation. The zero Attr has a key of "" and a value of nil.
+    An AtomicLevel is a Level that can be read and written safely by multiple
+    goroutines. The default value of AtomicLevel is InfoLevel.
 
-func Any(key string, value any) Attr
-    Any returns an Attr for the supplied value.
-
-    Any does not preserve the exact type of integral values. All signed integers
-    are converted to int64 and all unsigned integers to uint64. Similarly,
-    float32s are converted to float64.
-
-    However, named types are preserved. So given
+func (a *AtomicLevel) Level() Level
+    Level returns r's level.
 
-        type Int int
+func (a *AtomicLevel) Set(l Level)
+    Set sets the receiver's level to l.
 
-    the expression
+func (a *AtomicLevel) String() string
 
-        log.Any("k", Int(1)).Value()
+type Attr struct {
+	Key   string
+	Value Value
+}
+    An Attr is a key-value pair.
 
-    will return Int(1).
+func Any(key string, value any) Attr
+    Any returns an Attr for the supplied value. See [Value.AnyValue] for how
+    values are treated.
 
-func Bool(key string, value bool) Attr
+func Bool(key string, v bool) Attr
     Bool returns an Attr for a bool.
 
-func Duration(key string, value time.Duration) Attr
+func Duration(key string, v time.Duration) Attr
     Duration returns an Attr for a time.Duration.
 
-func Float64(key string, value float64) Attr
+func Float64(key string, v float64) Attr
     Float64 returns an Attr for a floating-point number.
 
+func Group(key string, as ...Attr) Attr
+    Group returns an Attr for a Group Value. The caller must not subsequently
+    mutate the argument slice.
+
+    Use Group to collect several Attrs under a single key on a log line, or as
+    the result of LogValue in order to log a single value as multiple Attrs.
+
 func Int(key string, value int) Attr
     Int converts an int to an int64 and returns an Attr with that value.
 
@@ -72,96 +99,73 @@
     Int64 returns an Attr for an int64.
 
 func String(key, value string) Attr
-    String returns a new Attr for a string.
+    String returns an Attr for a string value.
 
-func Time(key string, value time.Time) Attr
-    Time returns an Attr for a time.Time.
+func Time(key string, v time.Time) Attr
+    Time returns an Attr for a time.Time. It discards the monotonic portion.
 
-func Uint64(key string, value uint64) Attr
+func Uint64(key string, v uint64) Attr
     Uint64 returns an Attr for a uint64.
 
-func (a Attr) AppendValue(dst []byte) []byte
-    AppendValue appends a text representation of the Attr's value to dst.
-    The value is formatted as with fmt.Sprint.
-
-func (a Attr) Bool() bool
-    Bool returns the Attr's value as a bool. It panics if the value is not a
-    bool.
-
-func (a Attr) Duration() time.Duration
-    Duration returns the Attr's value as a time.Duration. It panics if the value
-    is not a time.Duration.
-
 func (a1 Attr) Equal(a2 Attr) bool
     Equal reports whether two Attrs have equal keys and values.
 
-func (a Attr) Float64() float64
-    Float64 returns the Attr's value as a float64. It panics if the value is not
-    a float64.
-
-func (a Attr) Format(s fmt.State, verb rune)
-    Format implements fmt.Formatter. It formats a Attr as "KEY=VALUE".
-
-func (a Attr) HasValue() bool
-    HasValue returns true if the Attr has a value.
-
-func (a Attr) Int64() int64
-    Int64 returns the Attr's value as an int64. It panics if the value is not a
-    signed integer.
-
-func (a Attr) Key() string
-    Key returns the Attr's key.
-
-func (a Attr) Kind() Kind
-    Kind returns the Attr's Kind.
-
 func (a Attr) String() string
-    String returns Attr's value as a string, formatted like fmt.Sprint.
-    Unlike the methods Int64, Float64, and so on, which panic if the Attr is of
-    the wrong kind, String never panics.
-
-func (a Attr) Time() time.Time
-    Time returns the Attr's value as a time.Time. It panics if the value is not
-    a time.Duration.
-
-func (a Attr) Uint64() uint64
-    Uint64 returns the Attr's value as a uint64. It panics if the value is not
-    an unsigned integer.
-
-func (a Attr) Value() any
-    Value returns the Attr's value as an any. If the Attr does not have a value,
-    it returns nil.
-
-func (a Attr) WithKey(key string) Attr
-    WithKey returns an attr with the given key and the receiver's value.
 
 type Handler interface {
-	// Enabled reports whether this handler is accepting records
-	// at the given level.
+	// Enabled reports whether the handler handles records at the given level.
+	// The handler ignores records whose level is lower.
 	Enabled(Level) bool
 
-	// Handle processes the Record.
+	// Handle handles the Record.
+	// It will only be called if Enabled returns true.
 	// Handle methods that produce output should observe the following rules:
-	//   - If r.Time() is the zero time, do not output it.
-	//   - If r.Level() is Level(0), do not output it.
-	Handle(Record) error
-
-	// With returns a new Handler whose attributes consist of
-	// the receiver's attributes concatenated with the arguments.
-	With(attrs []Attr) Handler
-}
-    A Handler processes log records produced by Logger output. Any of the
-    Handler's methods may be called concurrently with itself or with other
-    methods. It is the responsibility of the Handler to manage this concurrency.
+	//   - If r.Time is the zero time, ignore the time.
+	//   - If an Attr's key is the empty string, ignore the Attr.
+	Handle(r Record) error
+
+	// WithAttrs returns a new Handler whose attributes consist of
+	// both the receiver's attributes and the arguments.
+	// The Handler owns the slice: it may retain, modify or discard it.
+	WithAttrs(attrs []Attr) Handler
+
+	// WithGroup returns a new Handler with the given group appended to
+	// the receiver's existing groups.
+	// The keys of all subsequent attributes, whether added by With or in a
+	// Record, should be qualified by the sequence of group names.
+	//
+	// How this qualification happens is up to the Handler, so long as
+	// this Handler's attribute keys differ from those of another Handler
+	// with a different sequence of group names.
+	//
+	// A Handler should treat WithGroup as starting a Group of Attrs that ends
+	// at the end of the log event. That is,
+	//
+	//     logger.WithGroup("s").LogAttrs(slog.Int("a", 1), slog.Int("b", 2))
+	//
+	// should behave like
+	//
+	//     logger.LogAttrs(slog.Group("s", slog.Int("a", 1), slog.Int("b", 2)))
+	WithGroup(name string) Handler
+}
+    A Handler handles log records produced by a Logger..
+
+    A typical handler may print log records to standard error, or write them to
+    a file or database, or perhaps augment them with additional attributes and
+    pass them on to another handler.
+
+    Any of the Handler's methods may be called concurrently with itself or
+    with other methods. It is the responsibility of the Handler to manage this
+    concurrency.
 
 type HandlerOptions struct {
-	// Add a "source" attributes to the output whose value is of the form
+	// Add a "source" attribute to the output whose value is of the form
 	// "file:line".
 	AddSource bool
 
-	// Ignore records with levels above LevelRef.Level.
-	// If nil, accept all levels.
-	LevelRef *LevelRef
+	// Ignore records with levels below Level.Level().
+	// The default is InfoLevel.
+	Level Leveler
 
 	// If set, ReplaceAttr is called on each attribute of the message,
 	// and the returned value is used instead of the original. If the returned
@@ -169,7 +173,7 @@
 	//
 	// The built-in attributes with keys "time", "level", "source", and "msg"
 	// are passed to this function first, except that time and level are omitted
-	// if zero, and source is omitted if AddSource is false.
+	// if zero, and source is omitted if AddSourceLine is false.
 	ReplaceAttr func(a Attr) Attr
 }
     HandlerOptions are options for a TextHandler or JSONHandler. A zero
@@ -193,37 +197,43 @@
     NewJSONHandler creates a JSONHandler that writes to w, using the default
     options.
 
-func (h JSONHandler) Enabled(l Level) bool
-    Enabled reports whether l is less than or equal to the maximum level.
+func (h *JSONHandler) Enabled(level Level) bool
+    Enabled reports whether the handler handles records at the given level.
+    The handler ignores records whose level is lower.
 
 func (h *JSONHandler) Handle(r Record) error
     Handle formats its argument Record as a JSON object on a single line.
 
-    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
-    and the value is output in RFC3339 format with millisecond precision.
+    If the Record's time is zero, the time is omitted. Otherwise, the key is
+    "time" and the value is output as with json.Marshal.
 
-    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
-    and the value of Level.String is output.
+    If the Record's level is zero, the level is omitted. Otherwise, the key is
+    "level" and the value of Level.String is output.
 
-    If the AddSource option is set and source information is available,
-    the key is "source" and the value is output as "FILE:LINE".
+    If the AddSource option is set and source information is available, the key
+    is "source" and the value is output as "FILE:LINE".
 
     The message's key is "msg".
 
     To modify these or other attributes, or remove them from the output,
     use [HandlerOptions.ReplaceAttr].
 
-    Values are formatted as with encoding/json.Marshal.
+    Values are formatted as with encoding/json.Marshal, with the following
+    exceptions:
+      - Floating-point NaNs and infinities are formatted as one of the strings
+        "NaN", "+Inf" or "-Inf".
+      - Levels are formatted as with Level.String.
 
-    Each call to Handle results in a single, mutex-protected call to
-    io.Writer.Write.
+    Each call to Handle results in a single serialized call to io.Writer.Write.
 
-func (h *JSONHandler) With(attrs []Attr) Handler
+func (h *JSONHandler) WithAttrs(attrs []Attr) Handler
     With returns a new JSONHandler whose attributes consists of h's attributes
     followed by attrs.
 
+func (h *JSONHandler) WithGroup(name string) Handler
+
 type Kind int
-    Kind is the kind of an Attr's value.
+    Kind is the kind of a Value.
 
 const (
 	AnyKind Kind = iota
@@ -234,6 +244,8 @@
 	StringKind
 	TimeKind
 	Uint64Kind
+	GroupKind
+	LogValuerKind
 )
 func (k Kind) String() string
 
@@ -242,13 +254,39 @@
     the less important or severe the event.
 
 const (
-	ErrorLevel Level = 10
-	WarnLevel  Level = 20
-	InfoLevel  Level = 30
-	DebugLevel Level = 31
+	DebugLevel Level = -4
+	InfoLevel  Level = 0
+	WarnLevel  Level = 4
+	ErrorLevel Level = 8
 )
+    The level numbers below don't really matter too much. Any system can map
+    them to another numbering scheme if it wishes. We picked them to satisfy
+    three constraints.
+
+    First, we wanted the default level to be Info, Since Levels are ints,
+    Info is the default value for int, zero.
+
+    Second, we wanted to make it easy to work with verbosities instead of
+    levels. Verbosities start at 0 corresponding to Info, and larger values are
+    less severe Negating a verbosity converts it into a Level.
+
+    Third, we wanted some room between levels to accommodate schemes with
+    named levels between ours. For example, Google Cloud Logging defines a
+    Notice level between Info and Warn. Since there are only a few of these
+    intermediate levels, the gap between the numbers need not be large.
+    Our gap of 4 matches OpenTelemetry's mapping. Subtracting 9 from an
+    OpenTelemetry level in the DEBUG, INFO, WARN and ERROR ranges converts it to
+    the corresponding slog Level range. OpenTelemetry also has the names TRACE
+    and FATAL, which slog does not. But those OpenTelemetry levels can still be
+    represented as slog Levels by using the appropriate integers.
+
     Names for common levels.
 
+func (l Level) Level() Level
+    Level returns the receiver. It implements Leveler.
+
+func (l Level) MarshalJSON() ([]byte, error)
+
 func (l Level) String() string
     String returns a name for the level. If the level has a name, then that
     name in uppercase is returned. If the level is between named values, then an
@@ -257,62 +295,72 @@
         WarnLevel.String() => "WARN"
         (WarnLevel-2).String() => "WARN-2"
 
-type LevelRef struct {
-	// Has unexported fields.
+type Leveler interface {
+	Level() Level
 }
-    A LevelRef is a reference to a level. LevelRefs are safe for use by multiple
-    goroutines. Use NewLevelRef to create a LevelRef.
+    A Leveler provides a Level value.
 
-    If all the Handlers of a program use the same LevelRef, then a single Set on
-    that LevelRef will change the level for all of them.
+    As Level itself implements Leveler, clients typically supply a Level value
+    wherever a Leveler is needed, such as in HandlerOptions. Clients who need to
+    vary the level dynamically can provide a more complex Leveler implementation
+    such as *AtomicLevel.
 
-func NewLevelRef(l Level) *LevelRef
-    NewLevelRef creates a LevelRef initialized to the given Level.
-
-func (r *LevelRef) Level() Level
-    Level returns the LevelRef's level. If LevelRef is nil, it returns the
-    maximum level.
+type LogValuer interface {
+	LogValue() Value
+}
+    A LogValuer is a Value that transforms itself into a different Value (or,
+    using the Group feature, a group of Attrs) at the moment it is logged.
 
-func (r *LevelRef) Set(l Level)
-    Set sets the LevelRef's level to l.
+    This mechanism may be used to defer expensive operations until they are
+    needed, or to expand a single value into a sequence of components.
 
 type Logger struct {
 	// Has unexported fields.
 }
-    A Logger generates Records and passes them to a Handler.
-
-    Loggers are immutable; to create a new one, call New or Logger.With.
+    A Logger records structured information about each call to its Log, Debug,
+    Info, Warn, and Error methods. For each call, it creates a Record and passes
+    it to a Handler.
+
+    To create a new Logger, call New or a Logger method that begins "With".
+
+func Ctx(ctx context.Context) Logger
+    Ctx retrieves a Logger from the given context using FromContext. Then it
+    adds the given context to the Logger using WithContext and returns the
+    result.
 
-func Default() *Logger
+func Default() Logger
     Default returns the default Logger.
 
-func FromContext(ctx context.Context) *Logger
+func FromContext(ctx context.Context) Logger
     FromContext returns the Logger stored in ctx by NewContext, or the default
     Logger if there is none.
 
-func New(h Handler) *Logger
+func New(h Handler) Logger
     New creates a new Logger with the given Handler.
 
-func With(attrs ...any) *Logger
+func With(args ...any) Logger
     With calls Logger.With on the default logger.
 
-func (l *Logger) Debug(msg string, args ...any)
+func (l Logger) Context() context.Context
+    Context returns l's context.
+
+func (l Logger) Debug(msg string, args ...any)
     Debug logs at DebugLevel.
 
-func (l *Logger) Enabled(level Level) bool
-    Enabled reports whether l emits log records at level.
+func (l Logger) Enabled(level Level) bool
+    Enabled reports whether l emits log records at the given level.
 
-func (l *Logger) Error(msg string, err error, args ...any)
+func (l Logger) Error(msg string, err error, args ...any)
     Error logs at ErrorLevel. If err is non-nil, Error appends Any("err",
     err) to the list of attributes.
 
-func (l *Logger) Handler() Handler
+func (l Logger) Handler() Handler
     Handler returns l's Handler.
 
-func (l *Logger) Info(msg string, args ...any)
+func (l Logger) Info(msg string, args ...any)
     Info logs at InfoLevel.
 
-func (l *Logger) Log(level Level, msg string, args ...any)
+func (l Logger) Log(level Level, msg string, args ...any)
     Log emits a log record with the current time and the given level and
     message. The Record's Attrs consist of the Logger's attributes followed by
     the Attrs specified by args.
@@ -324,65 +372,82 @@
         an Attr.
       - Otherwise, the argument is treated as a value with key "!BADKEY".
 
-func (l *Logger) LogAttrs(level Level, msg string, attrs ...Attr)
+func (l Logger) LogAttrs(level Level, msg string, attrs ...Attr)
     LogAttrs is a more efficient version of Logger.Log that accepts only Attrs.
 
-func (l *Logger) LogAttrsDepth(calldepth int, level Level, msg string, attrs ...Attr)
+func (l Logger) LogAttrsDepth(calldepth int, level Level, msg string, attrs ...Attr)
     LogAttrsDepth is like Logger.LogAttrs, but accepts a call depth argument
     which it interprets like Logger.LogDepth.
 
-func (l *Logger) LogDepth(calldepth int, level Level, msg string, args ...any)
+func (l Logger) LogDepth(calldepth int, level Level, msg string, args ...any)
     LogDepth is like Logger.Log, but accepts a call depth to adjust the file
     and line number in the log record. 0 refers to the caller of LogDepth;
     1 refers to the caller's caller; and so on.
 
-func (l *Logger) Warn(msg string, args ...any)
+func (l Logger) Warn(msg string, args ...any)
     Warn logs at WarnLevel.
 
-func (l *Logger) With(attrs ...any) *Logger
-    With returns a new Logger whose handler's attributes are a concatenation of
-    l's attributes and the given arguments, converted to Attrs as in Logger.Log.
+func (l Logger) With(args ...any) Logger
+    With returns a new Logger that includes the given arguments, converted to
+    Attrs as in Logger.Log. The new Logger's handler is the result of calling
+    WithAttrs on the receiver's handler.
+
+func (l Logger) WithContext(ctx context.Context) Logger
+    WithContext returns a new Logger with the same handler as the receiver and
+    the given context.
+
+func (l Logger) WithGroup(name string) Logger
+    WithGroup returns a new Logger that starts a group. The keys of all
+    attributes added to the Logger will be qualified by the given name.
 
 type Record struct {
+	// The time at which the output method (Log, Info, etc.) was called.
+	Time time.Time
+
+	// The log message.
+	Message string
+
+	// The level of the event.
+	Level Level
+
+	// The context of the Logger that created the Record. Present
+	// solely to provide Handlers access to the context's values.
+	// Canceling the context should not affect record processing.
+	Context context.Context
+
 	// Has unexported fields.
 }
-    A Record holds information about a log event.
+    A Record holds information about a log event. Copies of a Record share
+    state. Do not modify a Record after handing out a copy to it. Use
+    Record.Clone to create a copy with no shared state.
 
-func NewRecord(t time.Time, level Level, msg string, calldepth int) Record
-    NewRecord creates a new Record from the given arguments. Use Record.AddAttr
+func NewRecord(t time.Time, level Level, msg string, calldepth int, ctx context.Context) Record
+    NewRecord creates a Record from the given arguments. Use Record.AddAttrs
     to add attributes to the Record. If calldepth is greater than zero,
-    Record.SourceLine will return the file and line number at that depth.
+    Record.SourceLine will return the file and line number at that depth,
+    where 1 means the caller of NewRecord.
 
     NewRecord is intended for logging APIs that want to support a Handler as a
-    backend. Most users won't need it.
-
-func (r *Record) AddAttr(a Attr)
-    AddAttr appends a to the list of r's attributes. It does not check for
-    duplicate keys.
+    backend.
 
-func (r *Record) Attr(i int) Attr
-    Attr returns the i'th Attr in r.
+func (r *Record) AddAttrs(attrs ...Attr)
+    AddAttrs appends the given attrs to the Record's list of Attrs.
 
-func (r *Record) Attrs() []Attr
-    Attrs returns a copy of the sequence of Attrs in r.
+func (r Record) Attrs(f func(Attr))
+    Attrs calls f on each Attr in the Record.
 
-func (r *Record) Level() Level
-    Level returns the level of the log event.
+func (r Record) Clone() Record
+    Clone returns a copy of the record with no shared state. The original record
+    and the clone can both be modified without interfering with each other.
 
-func (r *Record) Message() string
-    Message returns the log message.
+func (r Record) NumAttrs() int
+    NumAttrs returns the number of attributes in the Record.
 
-func (r *Record) NumAttrs() int
-    NumAttrs returns the number of Attrs in r.
-
-func (r *Record) SourceLine() (file string, line int)
+func (r Record) SourceLine() (file string, line int)
     SourceLine returns the file and line of the log event. If the Record
     was created without the necessary information, or if the location is
     unavailable, it returns ("", 0).
 
-func (r *Record) Time() time.Time
-    Time returns the time of the log event.
-
 type TextHandler struct {
 	// Has unexported fields.
 }
@@ -393,38 +458,144 @@
     NewTextHandler creates a TextHandler that writes to w, using the default
     options.
 
-func (h TextHandler) Enabled(l Level) bool
-    Enabled reports whether l is less than or equal to the maximum level.
+func (h *TextHandler) Enabled(level Level) bool
+    Enabled reports whether the handler handles records at the given level.
+    The handler ignores records whose level is lower.
 
 func (h *TextHandler) Handle(r Record) error
     Handle formats its argument Record as a single line of space-separated
     key=value items.
 
-    If the Record's time is zero, it is omitted. Otherwise, the key is "time"
-    and the value is output in RFC3339 format with millisecond precision.
+    If the Record's time is zero, the time is omitted. Otherwise, the key is
+    "time" and the value is output in RFC3339 format with millisecond precision.
 
-    If the Record's level is zero, it is omitted. Otherwise, the key is "level"
-    and the value of Level.String is output.
+    If the Record's level is zero, the level is omitted. Otherwise, the key is
+    "level" and the value of Level.String is output.
 
-    If the AddSource option is set and source information is available,
-    the key is "source" and the value is output as FILE:LINE.
+    If the AddSource option is set and source information is available, the key
+    is "source" and the value is output as FILE:LINE.
 
     The message's key "msg".
 
     To modify these or other attributes, or remove them from the output,
     use [HandlerOptions.ReplaceAttr].
 
-    Keys are written as unquoted strings. Values are written according to their
-    type:
-      - Strings are quoted if they contain Unicode space characters or are over
-        80 bytes long.
-      - If a value implements [encoding.TextMarshaler], the result of
-        MarshalText is used.
-      - Otherwise, the result of fmt.Sprint is used.
+    If a value implements encoding.TextMarshaler, the result of MarshalText is
+    written. Otherwise, the result of fmt.Sprint is written.
 
-    Each call to Handle results in a single, mutex-protected call to
-    io.Writer.Write.
+    Keys and values are quoted with strconv.Quote if they contain Unicode space
+    characters, non-printing characters, '"' or '='.
 
-func (h *TextHandler) With(attrs []Attr) Handler
+    Keys inside groups consist of components (keys or group names) separated by
+    the Unicode middle dot character, '·'. No further escaping is performed.
+    If it is necessary to reconstruct the group structure of a key even in the
+    presence of middle dots inside components, use [HandlerOptions.ReplaceAttr]
+    to escape the keys.
+
+    Each call to Handle results in a single serialized call to io.Writer.Write.
+
+func (h *TextHandler) WithAttrs(attrs []Attr) Handler
     With returns a new TextHandler whose attributes consists of h's attributes
     followed by attrs.
+
+func (h *TextHandler) WithGroup(name string) Handler
+
+type Value struct {
+	// Has unexported fields.
+}
+    A Value can represent (almost) any Go value, but unlike type any,
+    it can represent most small values without an allocation. The zero Value
+    corresponds to nil.
+
+func AnyValue(v any) Value
+    AnyValue returns a Value for the supplied value.
+
+    Given a value of one of Go's predeclared string, bool, or (non-complex)
+    numeric types, AnyValue returns a Value of kind String, Bool, Uint64, Int64,
+    or Float64. The width of the original numeric type is not preserved.
+
+    Given a time.Time or time.Duration value, AnyValue returns a Value of kind
+    TimeKind or DurationKind. The monotonic time is not preserved.
+
+    For nil, or values of all other types, including named types whose
+    underlying type is numeric, AnyValue returns a value of kind AnyKind.
+
+func BoolValue(v bool) Value
+    BoolValue returns a Value for a bool.
+
+func DurationValue(v time.Duration) Value
+    DurationValue returns a Value for a time.Duration.
+
+func Float64Value(v float64) Value
+    Float64Value returns a Value for a floating-point number.
+
+func GroupValue(as ...Attr) Value
+    GroupValue returns a new Value for a list of Attrs. The caller must not
+    subsequently mutate the argument slice.
+
+func Int64Value(v int64) Value
+    Int64Value returns a Value for an int64.
+
+func IntValue(v int) Value
+    IntValue returns a Value for an int.
+
+func StringValue(value string) Value
+    String returns a new Value for a string.
+
+func TimeValue(v time.Time) Value
+    TimeValue returns a Value for a time.Time. It discards the monotonic
+    portion.
+
+func Uint64Value(v uint64) Value
+    Uint64Value returns a Value for a uint64.
+
+func (v Value) Any() any
+    Any returns the Value's value as an any.
+
+func (v Value) Bool() bool
+    Bool returns the Value's value as a bool. It panics if the value is not a
+    bool.
+
+func (a Value) Duration() time.Duration
+    Duration returns the Value's value as a time.Duration. It panics if the
+    value is not a time.Duration.
+
+func (v1 Value) Equal(v2 Value) bool
+    Equal reports whether two Values have equal keys and values.
+
+func (v Value) Float64() float64
+    Float64 returns the Value's value as a float64. It panics if the value is
+    not a float64.
+
+func (v Value) Group() []Attr
+    Group returns the Value's value as a []Attr. It panics if the Value's Kind
+    is not GroupKind.
+
+func (v Value) Int64() int64
+    Int64 returns the Value's value as an int64. It panics if the value is not a
+    signed integer.
+
+func (v Value) Kind() Kind
+    Kind returns the Value's Kind.
+
+func (v Value) LogValuer() LogValuer
+    LogValuer returns the Value's value as a LogValuer. It panics if the value
+    is not a LogValuer.
+
+func (v Value) Resolve() Value
+    Resolve repeatedly calls LogValue on v while it implements LogValuer,
+    and returns the result.
+
+func (v Value) String() string
+    String returns Value's value as a string, formatted like fmt.Sprint.
+    Unlike the methods Int64, Float64, and so on, which panic if the Value is of
+    the wrong kind, String never panics.
+
+func (v Value) Time() time.Time
+    Time returns the Value's value as a time.Time. It panics if the value is not
+    a time.Time.
+
+func (v Value) Uint64() uint64
+    Uint64 returns the Value's value as a uint64. It panics if the value is not
+    an unsigned integer.

[firelizzard18]

Negating a verbosity converts it into a Level

Would you expand on this? It's not clear from context what this means.

[jba]

Clarified, I hope. https://go-review.googlesource.com/c/exp/+/444683

[firelizzard18]

That does help, thanks. I left a long-winded comment on the CL.

[firelizzard18]

For anyone else wondering what happened to Marshaler/MarshalLog, it is now called LogValuer/LogValue

[natefinch]

I think what this proposal really needs, and what would help a lot of go programs, is an improvement to go's type inference so that if the function signature is

Log(msg string, kvps map[string]any]

Then the compiler should allow

logger.Log("error!", {"user":userID, "action":actionType})

And it can understand that as a map literal.

[Davincible]

Maps have intentionally not been supported due to the performance overhead as was mentioned in the proposal but perhaps parsing them as top-level fields could be nice if it doesn't add overhead when you don't use them. Currently, they seem to be treated as groups

[natefinch]

Then it could still be

logger.Log(msg, {key, val1}, {key1, val2})

If it were

type Kvp struct {
   Key string
   Val any
}

func Log(msg string, kvp ...Kvp)

If only the inference were better.

[cbandy]

If only the inference were better.

I believe that is https://go.dev/issue/12854

[Davincible]

You can do exactly this if kvp implements the LogValuer interface.

[jba]

Let's continue the discussion on the proposal issue.

[cschneider4711]

Hi, nice design, thanks for the great proposal.

I especially like the convenience level functions to be used consistently in many codebases. Just as an idea to provide some feedback for discussion: Do you think it would make sense to include another named level in these convenience functions: "Security"?

From my IT-Security related work I experience many situations, where security logs are helpful, either in proactively detecting certain security-relevant conditions (mostly done by SIEM tools analyzing logs and monitoring events) and sometimes also during forensic analysis (but then of course also more debug-like logs are used as well).

I know that creating such thing individually with this proposed API would be easy (thx also to the gaps in the numbering scheme), but especially the consistency used among many different codebases is where Go really shines. As in the recent releases security also was a major part (thinking about the great addition of fuzzing in the whole toolchain ecosystem), I think that incorporating a convenience logging function "Security" for security-events directly in the top-level API would generate more security-relevant thoughts within development teams of how to also think about certain security-relevant conditions in their software (and how to log these).

Mostly I can imagine situations or events like login failure events, missing permissions for some actions, invalid data provided (especially when not possible by accident via a client, so a potential hacker bypassed a client-side convenience check and then the server-side input validation kicks in), etc.

The downside of this thought would be to tend to "overload" the deliberately short list of convenience functions for log-levels (who knowns what will be next proposed convenience level?). On the upside one could argue that security should be ubiquitous enough in projects to find a good fit in most projects.

One idea of how this "security event log" level could be handled special (like the error level is taking an error arg for example) could be to ease counting and grouping of such events from logs by having a clearer distinction from the other logs as being a potential security-event and possibly by taking a "type" string argument, which can be provided (freely) by the caller, enabling a grouping of certain security event types. Thereby derivations from usual patterns could easier be detected by tools specifically watching for those consistently-logged security-events in the overall logs. And when the language-idiomatic logging framework possibly gets/has a security-event kind of convenience logging level, integrating this output with intrusion detection or SIEM solutions parsing these logs would be easer.

As said, just an idea to create room for further discussion on the pros and cons about this. Keep up the good work, and I really look forward to the new logging proposal being brought to life.

[cschneider4711]

As meanwhile this discussion ticket is considered finished, I move the above comment to the official proposal https://go.dev/issue/56345

Please further discuss/comment there... thx

[cschneider4711]

And on another perspective three more ideas related to logging that came into my mind right now, again, just to inspire discussion of the pros and cons:

Correlation-ID for a Request

Especially for distributed backend systems (with many microservices or asynchronous flows) it could possibly be helpful to propagate in the architecture a unique identifier for the request (like a UUID created on an API gateway) and having this appended/prefixed on every log created under that thread. Something like the Java-style Log MDC (Message Diagnostic Context), which was essentially a ThreadLocal kind of stack to add some prefix stuff that is tied to one particular request and taken in front of any log statement thereunder automatically. Using a middleware to create and push/pop this could be easily done in most http routing frameworks, allowing for easier analysis of single requests under heavy load. Possibly the Context part of this proposal can be used for this by the popular http frameworks, but I wonder how this ID might also propagate into developer-direct usages of this logging proposal automatically? (I have to read the spec proposal more on this, I admit)

Ring-Buffer style of Debug-preserving Appending on Error-Conditions

This idea is partly taken from the Java Util Logging, where a memory-based appender-like construct was used to solve one of the most annoying and sometimes pressing problems of not having enough detailed logs to analyze a certain production error condition (often leading into attempts to re-create these conditions on test systems to gather otherwise not available debug logs):

The idea of a ring-buffer based logging would be to have a ring-buffer (of size n) in-memory, containing logs (each log entry is an entry in the ring-buffer) and it has two level-thresholds defined upon creation:

Let's take for example "Info" and "Error" as the two level-thresholds of the ring-buffer:

• When log-events below the lower level-threshold are logged, they are added to the ring-buffer (like "Debug" logs), but not written to any appending output.
• When a log-event higher-or-equal to the lower level-threshold but also lower than the upper threshold gets logged (for example here "Info" or "Warn") then it gets logged to the appending target (file for example) and added to the ring-buffer.
• When a log-event higher-or-equal to the upper threshold gets logged (for example here "Error") then the complete ring-buffer gets sent to the appender (i.e. written to file or similar).

This design allows to always have the last n detail log statements before a critical event (like error) occurred and now these n logs include the debug logs before the error condition occurred, containing exactly these developer-valuable information of what caused that error or led to this. Otherwise debug logs are not sent to the appender. This level-threshold based ring-buffer implementation (as an optional use of course) plays very well with the numeric-based ordering of even custom levels.

This might be helpful in analyzing production issues, but on the downside might impact performance, as the debug log string is always created (but at least not always sent to appenders like a file etc., which happens only on errors or such high-threshold conditions).

Also some overlapping in logs of timestamps need to be addressed, depending on the appending strategy used: If only the output of the ring-buffer (i.e. the elements removed at the end) are further logged (when conforming to the threshold of, say, "Info" or higher), then a hard crash would possibly loose these n not-yet logged elements. To avoid this, the appending strategy of appending higher-or-equal to the defined threshold (say "Info") statements directly upon entry into the ring-buffer can be used. But then, when on an error-condition the complete ring-buffer (including "Debug") gets logged, there might be duplicates of those.

Secret Filter for Logs (PII, Credentials, Tokens, etc.)

To avoid the (unfortunately often seen) problem of accidentally logging sensitive information like Personally Identifiable Information (PII) or credentials of backend systems or tokens or such, which are sometimes logged as part of error messages etc. by accident it would be helpful for a logging API to provide some kind of global hooks to plug-in any secret masking plugins. These could be created and configured by the individual dev teams, possibly taking regulatory requirements into account. Examples could be to redact email addresses, credit card numbers, session/token identifiers, cloud access keys, api-tokens, etc. based on a list of regex or better simpler filter conditions (as performance otherwise might be an issue).

The idea would be to have an easy way of hooking this into the logging API and also some kind of reference implementation to optionally use for the most common use-cases defined above.

Again, sorry for the long post, just my 2 cents about some issues I've had with different logging implementations

[cschneider4711]

As meanwhile this discussion ticket is considered finished, I move the above comment to the official proposal https://go.dev/issue/56345

Please further discuss/comment there... thx

[tigrannajaryan]

Correlation-ID for a Request

This is what OpenTelemetry does/plans to do. See here the discussion about how the Trace/Span Context is intended to be passed to the logger: #54763 (reply in thread)

[Davincible]

@cschneider4711 replying here for a more casual discussion;

Secret Filter for Logs

I think you could do this in a wrapped handler, where you first validate all fields and then pass it on to the underlying handler for processing.

While very much agree with the suggestion, how would you standardize this? i.e. recognize what should or shouldn't be masked

[yassinm]

I wonder if it would be possible if the runtime/compiler could eliminate logging (basically make logging a zero cost operation) via dead code elimination in one shape or form ? I want to stress that what is important is not how you do it but just being able to do it.

example:

• Something similar to java final fields. The compiler can infer if a variable is (boolean) false during the lifetime of the application it can eliminate all code blocks dependent on that variable.

[Davincible]

It currently is basically eliminated. The only cost is a single if statement to check if the level is enabled.

[yassinm]

I am assuming you are saying slog.Info("hello", "name", "Al") would actually incur the function call but come back right away ?

1. is this cost significant in a tight loop ?
2. Is there plan to ultimately get rid of that function call as well via dead code elimination or that is too hard to support now ?

[jba]

If you really care about efficiency at that level, write

if slog.Default().Enabled(slog.InfoLevel) {
    slog.Info(...)
}

That will be quite fast. But so will the original call, since it effectively makes that same check early on. I haven't benchmarked it, but I think the difference will be small.

There's no particular compiler effort directed at making these logging calls go away. General compiler improvements may have that as a consequence.

[pohly]

I think the difference will be small.

That's what we thought about the go-logr API, too, but unfortunately the compiler is currently not smart enough to first do the "is enabled" check and then prepare parameters. Therefore the explicit if avoids one allocation and is measurably faster. See #56873 and #53465.

[ttys3]

can this add support for integrate with opentelemetry tracing ？

since it already support WithContext, how can I intergrate this with opentelemetry tracing to log the trace id in the json logs ?

with zap, a simple custom wrapper will work.
but I do not know how to do this with slog.

some purpose of do this:

1. when it is an error, report to tracing as an Error, so we can know it is an error in the tracing dashboard (like grafana or something)
2. log a structed field named "trace_id" or something extracted from the span, so we can ref to tracing data from the log data.

typical usage:

ctxWithSpan, newSpan := otel.Tracer("my-tracer-name").Start(ctx, spanName)
defer newSpan.End()

myCtxLog := slog.WithContext(ctxWithSpan)

myCtxLog.Info("I hope this log print a field named 'trace_id' and has the trace id of my new created span")

myCtxLog.Info("how to do it with slog?")

myCtxLog.Error("this is an error, I hope this also can mark this span as an error",  errors.New("example error"))

currently I have to do this, if I need the trace_id:

const TraceIDKey = "trace_id"

type TracingHandler struct {
	handler slog.Handler
}

func NewTracingHandler(h slog.Handler) *TracingHandler {
	// avoid chains of TracingHandlers.
	if lh, ok := h.(*TracingHandler); ok {
		h = lh.Handler()
	}
	return &TracingHandler{h}
}

// Enabled implements Handler.Enabled by reporting whether
// level is at least as large as h's level.
func (h *TracingHandler) Enabled(level slog.Level) bool {
	return h.handler.Enabled(level)
}

func traceID(ctx context.Context) string {
	if span := trace.SpanContextFromContext(ctx); span.HasTraceID() {
		return span.TraceID().String()
	}
	return ""
}

// Handle implements Handler.Handle.
func (h *TracingHandler) Handle(r slog.Record) error {
	if traceID := traceID(r.Context); traceID != "" {
		// r.AddAttrs will only affect current record, not the Logger's
		// r.AddAttrs(slog.String(TraceIDKey, traceID))
		h.handler = h.handler.WithAttrs([]slog.Attr{slog.String(TraceIDKey, traceID)})

		// this also can only affect current record
		if r.Level >= slog.LevelError {
			span := trace.SpanFromContext(r.Context)
			if span.IsRecording() {
				span.SetStatus(codes.Error, r.Message)
			}
		}
	}
	return h.handler.Handle(r)
}

// WithAttrs implements Handler.WithAttrs.
func (h *TracingHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
	return NewTracingHandler(h.handler.WithAttrs(attrs))
}

// WithGroup implements Handler.WithGroup.
func (h *TracingHandler) WithGroup(name string) slog.Handler {
	return NewTracingHandler(h.handler.WithGroup(name))
}

// Handler returns the Handler wrapped by h.
func (h *TracingHandler) Handler() slog.Handler {
	return h.handler
}

the problem with my TracingHandler:

using h.handler = h.handler.WithAttrs I will got duplicated trace_id
using r.AddAttrs(slog.String(TraceIDKey, traceID)) , continue logging will not have trace_id attribute

1. I have to use slog.With and then got WithContext to set the ctx
2. after a second .With, the ctx got lost. so the second .Error will not set the tracing span to Error as expected
3. if continue use ctxLog without call .With, I got duplicated trace_id, is there quick and effective way to see if the key prev exists ?

	ctxWithSpan, newSpan := otel.Tracer("my-tracer-name").Start(ctx, "hello.Slog")
	defer newSpan.End()

	ctxLog := slog.With("foo", "bar").WithContext(ctxWithSpan)
	ctxLog.Info("hello world")
	ctxLog.With("foo", "bar").Error("have a nice day", io.ErrClosedPipe)
	ctxLog.Error("have a nice day", io.ErrClosedPipe)

[jba]

Logger.With losing the context is a bug. Thanks for finding it. Fixed in https://go.dev/cl/459615.

I think your Handle method should add the trace ID to the record using Record.AddAttrs, then call h.handler.Handle(r).
You shouldn't be calling WithAttrs in Handle.
You say that if you do that, "continue logging will not have trace_id attribute." But I think that if you keep using the same ctxLog, then each log event will call its Handle method, which will add the traceID each time.

[ttys3]

yes, with https://go.dev/cl/459615 merged, seems all my problem has gone.

using r.AddAttrs(slog.String(TraceIDKey, traceID)) works good for now.

I put the whole codes here in case somebody need it:

https://github.com/ttys3/slogsimple/blob/7852a7cbfa14d72f85367ecd0d13d5236fe550fb/tracing_handler.go#L29

here is the fixed version (using AddAttrs):

// Handle implements Handler.Handle.
func (h *TracingHandler) Handle(r slog.Record) error {
	span := trace.SpanFromContext(r.Context)
	if span.IsRecording() {
		if r.Level >= slog.LevelError {
			span.SetStatus(codes.Error, r.Message)
		}
		if spanCtx := span.SpanContext(); spanCtx.HasTraceID() {
			r.AddAttrs(slog.String(TraceIDKey, spanCtx.TraceID().String()))
		}
	}
	return h.handler.Handle(r)
}

[xorcare]

Hello, I have read the discussions but did not notice one important topic.

Function FromContext uses global logger inside itself which by default writes in os.Stderr when using this approach there will be a lot of garbage output in tests.

For example if we have the following handler:

func handle(ctx context.Context) {  
   logger := slog.FromContext(ctx)  
   logger.Info("some kind of request is received")  
}

We wrote tests on it, and ran them go test .

func Test_handle(t *testing.T) {  
   for i := 0; i < 10 i++ {  
      handle(context.Background())  
   }  
}  
  
func Test_handle_fail(t *testing.T) {  
   handle(context.Background())  
   t.Fail()  
}

I expect to see error information in the Test_handle_fail test and one log entry like this:

> go test .
2023/01/03 19:36:58 INFO some kind of request is received
--- FAIL: Test_handle_fail (0.00s)
FAIL
FAIL slog 0.179s
FAIL

But I'll be disappointed to see:

> go test .
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
2023/01/03 19:37:54 INFO some kind of request is received
--- FAIL: Test_handle_fail (0.00s)
FAIL
FAIL slog 0.259s
FAIL

The ability to set the default logger slog.SetDefault() indirectly leads to clogged output.

In my opinion it's not necessary to give an opportunity to set default value of logger, and the ability to get values from context should be taken out in separate packages, or even do it through creational pattern, for example like this:

type contextKey struct{}  
  
func NewLogManager(logger *slog.Logger) *LogManager {  
   return &LogManager{  
      defaultLogger: logger,  
   }  
}  
  
type LogManager struct {  
   defaultLogger *slog.Logger  
}  
  
// NewContext returns a context that contains the given Logger.// Use FromContext to retrieve the Logger.
func (m *LogManager) NewContext(ctx context.Context) context.Context {  
   return context.WithValue(ctx, contextKey{}, m.defaultLogger)  
}  
  
// FromContext returns the Logger stored in ctx by NewContext, or the default  
// Logger if there is none.  
func (m *LogManager) FromContext(ctx context.Context) *slog.Logger {  
   if l, ok := ctx.Value(contextKey{}).(*slog.Logger); ok {  
      return l  
   }  
   return m.defaultLogger  
}

The main problems with the default global logger (IMHO)

• Complicates testing and clutters test output
• Difficult to understand how to substitute a logger for a test
• Introduces non-explicit dependencies

[pohly]

If you want per-test output, then there is (currently) no way around creating one logger per test that writes through testing.T.Log. This can be done whether there is a global default logger or not.

For go-logr, I wrote https://github.com/kubernetes/klog/blob/main/ktesting/testinglogger.go which uses klog text formatting. Using it looks like this:

func TestSomething(t *testing.T) {
   logger, ctx := ktesting.NewTestContext(t)
   // Use logger or FromContext(ctx) for all log output inside the test.
}

There's a similar logger in go-logr/logr itself.

[xorcare]

The problem is not that it cannot be done, the problem is that it is not explicit because it is facilitated by the proposed API of the slog package with the func SetDefault(l *Logger) functions.

[pohly]

it is facilitated by the proposed API of the slog package with the func SetDefault(l *Logger) functions.

I don't see a connection. Are saying that the existence of that API makes some people believe that nothing further needs to be done to get per-test output?

Or do you prefer to get no output in unit tests unless something is done in each test? Then developers might not become aware that they are missing log messages when they don't instrument their unit tests.

[AndrewHarrisSPU]

the existence of that API makes some people believe that nothing further needs to be done to get per-test output?

I think the per-test question is what's being asked, but maybe also tangentially - can the execution of a Go program become surprisingly noisy as a result of slog? I think the answer is yes - it's a symptom that isn't unique to slog (it feels similar to how the original log and testing packages interact), but it's a new library with a new surface area around configuration.

[pohly]

So the question is whether slog should output something by default or be silent unless explicitly configured?

I can see why third-party logging libraries should be silent by default, to avoid printing something that the binary isn't prepared to handle when that library (unexpectedly?) becomes a dependency. Even that is debatable, because tracking dependencies can catch this. But for the standard library I find it reasonable to produce output by default because going forward, all Go programs should be prepared to handle such output if it is a concern.

[biohazduck]

I've been using the golang.org/x/exp/slog package for some time and I'd like to give back some user experience:

I'd like to output the Default Loggers output to stdout and a file. So far I've figured out I need to create new logger. It feels like I have to jump through many hoops to be able to just add another io.Writer. Then I have to figure out how to use TextHandler... Only to then attempt to give up and just copy the slog source code only to realize there's a whole lot of source code behind the default logging... Then I get worried if it's worth investing time into getting this muxed output to work or if I should just use the log package that has a SetOutput package level function...

[BenjamenMeyer]

this is a common patttern at least in Python and I'd love to see the same pattern in Golang: log stdout, log local file, and send to a network service.

it would be extremely helpful to be able to support multiple writers

[jba]

@biohazduck, did you try passing an io.MultiWriter to a TextHandler?

[soypat]

Yes! But the default text handler prints the time, error level and message as Attrs, unlike the default implementation which prints them pretty like I like them

[jba]

Try log.SetOutput(mw), where mw is your MultiWriter.

[soypat]

Yes, that worked! Thank you @jba

[BenjamenMeyer]

As I noted in #54763 (reply in thread) a common and very useful pattern is to be able to log to multiple outputs, even more importantly it is helpful to give each output its own formatter.

Looking at Python, they have a LogHandler for the output and a LogFormatter for generating the format. It would be useful to generate this same kind of functionality here; however, it seems like the LogHandler here conflates those two functions. Could we split those apart so we can get more versatility?

[jba]

Handler is an interface, so you could certainly write one that does what you want.
Given how big the API is already, and the uncertainty of what the best way to do this is, we're going to leave it for later (if ever).
I think I make a similar point on the proposal issue, which is where discussion should now take place.

[BenjamenMeyer]

@jba understood - I'll read up there and see how I can contribute to the discussion.

[nikandfor]

I've never shared my approach to logging, but I thought a lot about it. So it may be a good time to share it now.

What are the key points I came to:

• Logs are just events with some properties. None of the properties are special. But some can be treated specially when rendered to the user.
• Events can be used not only for logs but as traces, metrics, and anything, which makes a code much less cumbersome. We use a single API in function and multiple handlers in the background.
• Encoding must be as efficient as possible not to think about where to emit an event or save resources.
• I also wanted as lightweight API as I can get, which is Printw(message string, kvPairs ...interface{}) (with default keys: time, caller) or generalized version (without default keys) Event(kvs ...interface{}).
• To avoid allocations, I used two things. First, is a hack preventing allocations in any case.
• The second: record type is []byte, which makes any properties not escape to the heap. And handlers are of type io.Writer. The most straightforward handler is a file. And then, you read the file and make processing in a different process.
• Log levels are usually challenging to choose anyway, so they are optional and intended for those who like them. (other events are info, technically)
• To reduce the number of unnecessary logs at runtime, I used a similar to glog.V(Level) approach, but debug topic is used instead of level.

So in short it looks like

func main() {
    var w io.Writer = // could be file, packet socket, handlers like Sentry, multiple destinations...
    tlog.DefaultLogger = tlog.New(w)
    
    tlog.SetVerbosity("db_client,dump_conn") // value come from flags usually

    // ...
}

func Process(ctx context.Context, req *Request) (err error) {    
    span := tlog.SpawnFromContext(ctx, "tracing_span_name", "req_type", req.Type, "client", req.ClientIP)
    defer func() {
        span.Finish("err", err)
    }()
    // ...
    span.Printw("some work done", "more", "info_available", "save", 17, "any", map[string]any{"type": "here"})
    // ...
    span.V("detailed").Printw("will emit if only verbosity 'detailed' is set")
    
    if span.If("heavy_debug") {
        // make heavy calc
        span.Printw("heavy message", "calc", "result")
    }
}

PS: Span start and span finish are two different events with the same ID (as like as any span.Printw). The span events are not stored in the memory.

The full code is here: https://github.com/nikandfor/tlog
godoc: https://pkg.go.dev/github.com/nikandfor/tlog
examples: https://github.com/nikandfor/tlog/blob/master/examples/complex/main.go

[granpahacker]

Thanks