Managing Certificates and PKI

1. Understanding Certificate Authority

Aspect	Description	Notes
Root CA	`Self-signed certificate`	Trusted authority that issues intermediate CA certificates
Intermediate CA	`Issued by Root CA`	Used to issue end-entity certificates (e.g., service certs)
Certificate Chain	`Root CA → Intermediate CA → Service Cert`	Chain of trust required for validation
Certificate Lifecycle	`Issue → Renew → Revoke`	Managed by CA or automated system (e.g., Citadel, Vault)
Trust Store	`Root CA certificate`	Installed in services to validate certificates

Example: CA Certificate Structure


-----BEGIN CERTIFICATE-----
MIIFazCCA1KgAwIBAgIJAK5V2fB3W9z8MA0GCSqGSIb3DQEBBQUAMFgxCzAJBgNV
...
-----END CERTIFICATE-----

Note: In service meshes like Istio, the CA is often embedded in the control plane (e.g., Citadel or external PKI like HashiCorp Vault).

2. Configuring Self-Signed Certificates

Property	Description	Default Value	Notes
certificates.enabled	Enables or disables certificate generation	`false`	Set to `true` to enable self-signed certificate generation
certificates.duration	Duration of the certificate validity period	`2160h`	Valid time in hours (e.g., 90 days)
certificates.keySize	Size of the private key in bits	`2048`	Recommended size for security (e.g., 2048 or 4096)
certificates.caCertPath	Path to the CA certificate file	`/etc/certs/ca.crt`	Used to sign generated certificates
certificates.caKeyPath	Path to the CA private key file	`/etc/certs/ca.key`	Required for signing certificates

Example: Enable Self-Signed Certificates

certificates:
  enabled: true
  duration: 2160h
  keySize: 2048
  caCertPath: /etc/certs/ca.crt
  caKeyPath: /etc/certs/ca.key

Note: Self-signed certificates are useful for testing and development environments but should not be used in production without proper CA setup.

3. Integrating External CA

Configuration Option	Description	Example Value	Notes
ca.bundle.path	Path to the CA certificate bundle file	`/etc/certs/ca-bundle.crt`	Must be readable by the service mesh component
ca.bundle.type	Type of CA bundle (e.g., PEM, DER)	`PEM`	Default is PEM; DER requires base64 encoding
ca.key.path	Path to the CA private key file	`/etc/certs/ca.key`	Must be securely stored and accessible only to authorized users
ca.key.password	Password for encrypted CA key (if applicable)	`secret123`	Optional, used only if the key is encrypted

Example: Configuring External CA in Istio

apiVersion: istio.io/v1beta1
kind: IstioOperator
metadata:
  name: istio-control-plane
spec:
  values:
    global:
      caBundlePath: /etc/certs/ca-bundle.crt
      caKeyPath: /etc/certs/ca.key
      caKeyType: PEM

Note: Ensure that the CA certificate and key are properly secured and accessible to the service mesh components. Misconfigurations can lead to certificate validation failures.

Warning: Using an external CA introduces additional security considerations. Ensure that the CA is managed securely and that certificate rotation policies are in place.

4. Managing Certificate Chain

Property	Description	Values
caBundle	Root CA certificate bundle used to validate certificates	`string`
certChain	Intermediate certificate chain for TLS termination	`string`
key	Private key associated with the certificate	`string`

Example: Certificate Chain Configuration

apiVersion: v1
kind: Secret
metadata:
  name: tls-cert
type: kubernetes.io/tls
data:
  ca.crt: <root-ca-bundle>
  tls.crt: <certificate-chain>
  tls.key: <private-key>

Note: Ensure the certificate chain includes all intermediate certificates in the correct order for validation.

5. Configuring Root Certificates

Property	Description	Default Value	Notes
ca.root.cert	Root certificate file path	`/etc/certs/root.crt`	Must be in PEM format
ca.root.key	Root private key file path	`/etc/certs/root.key`	Must be in PEM format
ca.root.duration	Root certificate validity duration	`8760h`	Valid time in hours or ISO 8601 format
ca.root.issuer	Issuer name for root certificate	`ServiceMesh CA`	Used in certificate subject

Example: Root Certificate Configuration

ca:
  root:
    cert: /etc/certs/root.crt
    key: /etc/certs/root.key
    duration: 8760h
    issuer: ServiceMesh CA

Note: Root certificates must be securely stored and backed up to prevent service disruption.

Warning: Changing root certificates requires reissuing all intermediate and leaf certificates.

6. Setting Certificate Validity Period

Property	Value	Description
validityPeriod	`8760h`	Duration for which the certificate is valid (default 1 year)
renewalWindow	`24h`	Time before expiration to start renewal process

Example: Configuring Certificate Validity

certificate:
  validityPeriod: "8760h"
  renewalWindow: "24h"

Note: Validity period must be less than the CA's maximum allowed duration.

7. Implementing Certificate Signing Requests

Field	Description	Example
spec.csr	The Certificate Signing Request in PEM format	`-----BEGIN CERTIFICATE REQUEST-----\nMIHDM...=`
spec.duration	Duration for which the certificate is valid	`2160h`
spec.usages	Key usages for the certificate	`[digital signature, key encipherment, server auth]`

Example: Creating a CSR with istioctl

istioctl x create-readonly-csr <service-account> --duration 2160h

Note: The CSR must be signed by a trusted CA to be valid in the mesh.

Warning: Ensure that the CSR is properly formatted and includes all required fields.

8. Managing Certificate Storage

Storage Type	Description	Features	Notes
File-based	Certificates stored as files on disk	Simple, local storage	Not suitable for distributed environments
Secrets Management	Using tools like Vault, K8s Secrets	Encrypted storage, access control	Requires integration with service mesh
CA Backend	Centralized CA server (e.g., HashiCorp Vault)	Automated issuance, revocation	High availability and redundancy
In-memory	Certificates held in memory during runtime	Fast access, volatile storage	Not persistent across restarts

Example: Storing Certificates in Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: mesh-ca-cert
type: Opaque
data:
  ca.crt: <base64-encoded-ca-certificate>

Note: Ensure secrets are properly secured with RBAC policies to prevent unauthorized access.

Warning: File-based storage is not recommended for production due to lack of encryption and access control.

Feature	File-based	Secrets Management	CA Backend
Encryption	None (unless manually encrypted)	Yes, via secret store	Yes, via CA
Access Control	OS-level permissions	Yes, via RBAC	Yes, via CA policies
Scalability	Low	Medium	High
Persistence	Yes	No (unless backed by persistent storage)	Yes, via CA logs

9. Setting Up Certificate Monitoring

Example: Prometheus Metrics Configuration

scrape_configs:
  - job_name: 'istio-mesh'
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

Configuration	Description	Default Value	Notes
prometheus.io/scrape	Enables scraping of metrics	false	Set to true to enable monitoring
prometheus.io/path	Path to scrape metrics from	/metrics	Typically /metrics for Istio
prometheus.io/port	Port to scrape metrics from	15014	Default Istio Prometheus port

Note: Ensure that the Prometheus instance has proper RBAC permissions to access Kubernetes pods and their annotations.

Example: Certificate Expiry Alert

groups:
  - name: certificate-alerts
    rules:
      - alert: CertificateExpirySoon
        expr: istio_certificate_expiration_seconds < 86400
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Certificate expires in less than 24 hours"
          description: "The certificate for {{ $labels.instance }} is set to expire in less than 24 hours."

Certificate Monitoring Best Practices

Monitor certificate expiration times using Prometheus metrics
Set up alerts for certificates expiring within 24 hours
Use istio_certificate_expiration_seconds metric to track expiration
Ensure Prometheus has access to Istio pods and their annotations

10. Troubleshooting Certificate Errors

Error Type	Description	Common Causes	Solution
Certificate Not Trusted	Certificate is not recognized by the trust store	Self-signed certs, expired certs, or missing intermediate CA	Install the root/intermediate CA in trust store or use a valid certificate from a recognized CA
Certificate Mismatch	Hostname or SAN does not match the certificate	Incorrect DNS name in cert, mismatched service names	Ensure certificate includes the correct SANs and matches the service's FQDN
Expired Certificate	Certificate has passed its validity date	Auto-renewal failed or manual renewal missed	Renew the certificate with a new valid expiration date
Certificate Chain Issue	Missing or incorrect intermediate certificates	Partial chain, misconfigured CA	Ensure full certificate chain is provided and correctly ordered
Time Skew	System clock is out of sync with certificate validity period	Server time is not synchronized (e.g., NTP issues)	Synchronize system clocks using NTP or similar

Example: Checking Certificate Validity with OpenSSL

openssl x509 -in /path/to/cert.pem -text -noout

Example: Verifying Certificate Chain

openssl s_client -connect service.example.com:443 -showcerts

Note: Use istioctl proxy-config secret in Istio to inspect certificate status on proxies.

Warning: Misconfigured certificate chains can lead to TLS handshake failures and service disruption.

11. Implementing Certificate Rotation Automation

Example: Automating certificate rotation with cert-manager

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-cert
spec:
  secretName: example-tls
  duration: 2160h
  renewBefore: 72h
  issuerRef:
    name: letsencrypt-prod
    kind: Issuer

Property	Description	Default Value
duration	How long the certificate is valid for	`2160h`
renewBefore	How long before expiration to start renewal	`72h`
secretName	Name of the Kubernetes secret to store the certificate	None

Note: Cert-manager automatically handles renewal based on renewBefore and duration settings.

Example: Using Vault for certificate rotation

{
  "name": "vault-cert-rotation",
  "type": "pki",
  "config": {
    "max_ttl": "8760h",
    "default_lease_ttl": "2160h"
  },
  "mount_point": "pki"
}

Key Steps for Certificate Rotation

Configure automatic renewal with renewBefore or similar
Set appropriate duration and max_ttl values
Use a CA or issuer that supports automated rotation
Ensure service mesh components can reload certificates without downtime

12. Managing Certificate Revocation

Certificate Field	Description	Relevance to Revocation
Serial Number	Unique identifier for each certificate issued by a CA	Used to identify and revoke specific certificates in CRLs or OCSP responses
Issuer	DN (Distinguished Name) of the CA that issued the certificate	Required to verify the authenticity of the revocation request
Subject	DN of the entity the certificate was issued to	Used to confirm identity during revocation process
Revocation Reason	Code indicating why a certificate was revoked (e.g., key compromise, CA compromise)	Helps in tracking and auditing certificate revocations

Example: Certificate Revocation Request (CRL)

Issuer: /C=US/O=MyCA/CN=MyRootCA
Serial Number: 0x1234567890abcdef
Revocation Date: 2026-04-05T10:00:00Z
Reason: keyCompromise

Note: CRLs are typically updated every few hours or minutes depending on the service mesh's configuration and security policies.

Warning: Ensure that all services in the mesh have mechanisms to detect and respond to revoked certificates promptly to maintain secure communication.