Logging Standards

Use structured logging · LOG-01.1 · MUST · DEV

Consistency and readability are crucial in our logs. Providing strong discoverability speeds up development and aides in diagnosing problems.

Unstructured logging example (❌)

console.log("User logged in at 2024-08-02T12:34:56Z");

Structured logging example (✅)

console.log(JSON.stringify(new Log(
  timestamp: "2024-08-02T12:34:56Z",
  level: "info",
  message: "User logged in",
  userId: 12345
));

Messages are clear and concise · LOG-01.2 · MUST · DEV

Each log message should be one or two sentences, which relay information relating to the level of the log.

e.g.

Messages are templated · LOG-01.3 · MUST · DEV

Use templates for log messages instead of interpolation to ensure consistency and parsability.

What bad looks like (❌)

// Bad
_logger.LogInformation($"Processing Payment {paymentIdToProcess}");

This example creates a log from a simple string, making it less structured.

What good looks like (✅)

// Good
_logger.LogInformation("Processing Payment {paymentIdToProcess}", paymentIdToProcess);

This version uses a templated string, resulting in an enriched log with a custom property named paymentIdToProcess. This allows for better querying and analysis in the logging back-end.

Advantages of Templated Strings

1. Consistency: Ensures that the overall structure of each message remains consistent.

2. Queryability: Allows for specific and general searches. For example, you can query for:

   • “Processing Payment {paymentIdToProcess}” to get all occurrences of this log.
   • “Processing Payment 412” to find specific instances.

3. Enhanced Context: You can pass complex objects instead of primitive types, enabling richer log details. Ensure these objects do not include Personal Identifiable Information (PII) unless approved by the client.

For example, if using Application Insights, you can search for “paymentIdToProcess” or “paymentIdToProcess 412” to pinpoint specific logs more effectively.

Use meaningful and engineer-friendly messages · LOG-01.4 · MUST · DEV

State the implication of the codes execution before or after the log line and the action an engineer would need to take (if any).

For example:

// Bad - vague with no explanation:
_logger.LogError("An error occured while trying to process the Payment {paymentId}", paymentId);

// Good - says specifically why:
_logger.LogError("Payment {paymentId} failed during processing because the customer's payment provider was not configured.", paymentId);

Avoid re-using user-facing error messages. Logs are read by engineers, the steps of which one would take to resolve the problem would differ greatly. “Please contact your system administrator” would help the user but not an engineer.

Never log sensitive data unless absolutely necessary · LOG-01.5 · MUST · DEV

Information such as passwords, application secrets (e.g. API Keys) and Personal Identifiable Information (PII) must only be logged if an agreement is in place with the client.

Logging sensitive data must be subject to agreements with the client · LOG-01.5.1 · MUST · DEV

Such an agreement must be made with security considerations regarding access-control and the right to be forgotten.

Logs must be stored using secure solutions to protect client data. Application insights is a common option for this, but engineers are not restricted to this, so long as the alternative solution has access-control.

Use the same logging provider for the front-end, back-end or data layers respectively.

e.g. For a .NET backend, if one class implements logging via the ILogger Microsoft.Extensions.Logging , then other classes on the backend should use the same logging provider.

Whichever logging provider the UI uses (e.g. Sentry) then this should be used for all logging on the front end.

Whether logs are from the UI or the server, they should be viewable in the same monitoring solutions (e.g. Application Insights).

Use appropriate log levels · LOG-04.1 · MUST · DEV

Logs must have an assigned log level which describes how that log is intended to provide value.

Avoid excessive logging that may cause performance degradation · LOG-04.2 · MUST · DEV

Logging should be performed judiciously throughout a user workflow to ensure that observability is seamlessly integrated without impacting performance. Excessive logging can lead to performance degradation and increased costs. Techniques such as Adaptive Sampling can automatically remove redundant log lines, which may result in loss of effort when too much logging has been implemented.

Debug Logs

If a log line is valuable for debugging but not essential for the production environment, consider using the Debug level. This helps keep production logs concise while still providing detailed information for development and troubleshooting.

Projects must have a client-agreed retention policy · LOG-04.3 · MUST · DEV

Every project should establish a log retention policy agreed upon with the client. This policy defines how long logs will be stored before they are archived or deleted. This ensures compliance with legal, operational, and business requirements.

Ensure log storage compliance with data location policies · LOG-04.4 · MUST · DEV

Logs must be stored in compliance with internal policies and applicable legislation regarding data location. Specifically, storing logs in non-UK/EU data centers may not be compliant and could lead to legal and regulatory issues. Always verify and adhere to data storage regulations relevant to your project to ensure compliance.

Don't repeat messages · LOG-04.5 · SHOULD · DEV

Each log message should contain information specific to the logic being executed around it. Ensure that each log line includes unique details that provide context for the execution.

Repeated log messages can hinder an engineer’s ability to find useful information during searches. Avoid duplication to maintain the effectiveness of your logging system.

Signpost significant application events · LOG-04.6 · SHOULD · DEV

Logging should be added to important workflow moments. This leaves breadcrumbs which can later be used to diagnose failure points in the application.