Skip to content

Latest commit

 

History

History
116 lines (75 loc) · 5.07 KB

loki_operator.md

File metadata and controls

116 lines (75 loc) · 5.07 KB

Deploying NetObserv with the Loki Operator

This page provides a quick walk-through for setting up the Loki Operator with NetObserv. You can find here more documentation (or there for OpenShift).

The Loki Operator integrates a gateway that implements multi-tenancy & authentication with Loki for logging.

NetObserv requires to use a specific tenant for Loki, named network, which uses a specific tenant mode implemented in Loki Operator 5.6+. For that reason, NetObserv is not compatible with prior version of the Loki Operator.

Installing Loki Operator

Install the Loki operator using Operator Hub. If using OpenShift, open the Console and navigate to Administrator view -> Operators -> OperatorHub.

Search for loki. You should find Loki Operator in Red Hat catalog.

Install the operator with the default configuration.

Hack script

We provide a hack script that uses AWS S3 storage, to run the steps described below. It assumes you have the AWS CLI installed with credentials configured. It will create a S3 bucket and configure Loki with it.

The first argument is the bucket name, second is the AWS region. Example:

./hack/loki-operator.sh netobserv-loki eu-west-1

If you choose to run it, you can ignore the following steps.

Namespace

For simplicity, this guide assumes Loki is deployed in the same namespace as NetObserv.

If it doesn't already exist, create the netobserv namespace:

kubectl create ns netobserv

External storage

Loki operator requires an external storage, such as Amazon S3. Check the documentation to set it up. Make sure you create the secret in netobserv namespace. Take note of the storage configuration you need to set in LokiStack, as mentioned in the linked documentation.

Example with AWS S3, using aws CLI:

S3_NAME="netobserv-loki"
AWS_REGION="eu-west-1"
AWS_KEY=$(aws configure get aws_access_key_id)
AWS_SECRET=$(aws configure get aws_secret_access_key)

aws s3api create-bucket --bucket $S3_NAME  --region $AWS_REGION --create-bucket-configuration LocationConstraint=$AWS_REGION

kubectl create -n netobserv secret generic lokistack-dev-s3 \
  --from-literal=bucketnames="$S3_NAME" \
  --from-literal=endpoint="https://s3.${AWS_REGION}.amazonaws.com" \
  --from-literal=access_key_id="${AWS_KEY}" \
  --from-literal=access_key_secret="${AWS_SECRET}" \
  --from-literal=region="${AWS_REGION}"

LokiStack

Then create a LokiStack in netobserv namespace. When using OpenShift, navigate to:

Administrator view -> Operators -> Installed Operators -> Loki Operator -> LokiStack -> Create LokiStack

  • Name it loki (any name is fine, but you need to adapt the URLs below accordingly)
  • Choose the size. While not suitable for production, 1x.demo is OK for testing / demo (or 1x.extra-small in operator 5.7). Note that very small clusters (e.g. 3 worker nodes) require 1x.demo, see troubleshooting section below.
  • Set Object Storage -> Secret as noted above.
  • Set Tenants Configuration -> Mode to openshift-network.

This will create gateway, distributor, compactor, ingester, querier and query-frontend components.

NetObserv configuration

Once the Loki stack is up and running, you need to configure NetObserv to communicate to Loki through its gateway service:

  loki:
    mode: LokiStack
    lokiStack:
      name: loki

User access

You then need to define ClusterRoleBindings for allowed users or groups, such as this one for a user named test. This can also be done from the CLI:

oc adm policy add-cluster-role-to-user netobserv-loki-reader test

Cluster admins do not need this role binding.

Troubleshooting

  • Logs are by default --log.level=warn. You can set --log.level=debug in gateway.go and opa_openshift.go to get more logs.

  • AWS region not set for deploy-example-secret.sh If aws configure get region returns blank, the shell will fail. You can force region using aws configure --region us-east-1 for example.

  • Insufficient CPU or memory If your pods hang in Pending state, you should double check their status using oc describe We recommand to use size: 1x.extra-small but this still requires a lot of resources. You can decrease them in internal/manifests/internal/sizes.go and set 100m for each CPUs and 256Mi for each Memories

  • Certificate errors in Gateway logs Check ZeroSSL.com CA with acme.sh

  • Running custom Loki query bypassing the gateway:

oc exec -it netobserv-plugin-d894b4544-97tq2 -- curl --cert /var/loki-status-certs-user/tls.crt  --key /var/loki-status-certs-user/tls.key  --cacert /var/loki-status-certs-ca/service-ca.crt -k -H "X-Scope-OrgID: network"  https://loki-query-frontend-http:3100/loki/api/v1/label/DstK8S_Namespace/values