Hello everyone,
We are pleased to announce that Cryostat 4.2.0 is now here!
cryostat.io/harvester-period and cryostat.io/harvester-max-files labels are now available and can be specified in the application's deployment.
For further details on these items, check the GitHub Discussions release announcement.
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 4.1.x with OLM, then you may have already been upgraded to 4.2.0, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>Feel free to skip this historical recap of how we arrived at where we are now and jump directly to the current capabilities and features.
Originally, Cryostat stored all data as files written to its local filesystem, within directories like
/opt/cryostat.d/recordings.d. This was decided when the only data to be stored were Flight Recording Archives and
custom Event Templates. Over time Cryostat grew new features and new types of data to store: Stored Credentials and
Automated Rules, Discovery Plugins (Cryostat Agents) and persisting Targets and Discovery Nodes, and most recently
Thread Dumps and Heap Dumps.
In the Cryostat 2.x series a small h2 database was added, which was simply backed by a file within the local
filesystem, which would house Stored Credentials, Targets, Discovery Nodes, and Discovery Plugins. This took care of
most of the “small” data, but the “large data” Flight Recording archives were still simply written to a local
filesystem path. All of these data types could survive Cryostat being temporarily scaled down or restarting due to a
crash, but the data being kept within a local filesystem (in Kubernetes, a PersistentVolumeClaim) meant that it
might not persist if Cryostat was uninstalled completely. This also made it more difficult or infeasible for users to
have Cryostat export captured data to object storage providers - users
deploying Cryostat to Docker might need to do something convoluted like mount an
S3-backed FUSE volume mount to their machine and then share that as a volume mount to the container. Kubernetes users
could do something similar to configure their object storage provider with a CSI driver to provide PersistentVolumes,
which could be selected for the PersistentVolumeClaim used by Cryostat. But, even after going through these extra
steps, that data in the PersistentVolumeClaim needs to be carefully handled so that it isn’t tied to the Cryostat
deployment’s lifecycle, and sharing the data between services is difficult when it’s written “raw” into a PVC.
So, as part of the big rewrite and re-architecture of Cryostat 3.0, the database was moved from file-backed h2 on
local filesystem to a Postgres container, and other object storage was moved from direct local filesystem to using the
AWS S3 SDK and a cryostat-storage container (a “fork” of
SeaweedFS which just layers on a custom entrypoint script and container
build). At this point, however, these containers and their Kubernetes Deployments are tightly integrated into the
Cryostat installation - the database and object storage had their own PersistentVolumeClaims to back their own
data storage, and without resorting to FUSE mounts or CSI drivers and PersistentVolume storage class selectors it
still wouldn’t be possible to have Cryostat export Flight Recordings to your own object storage provider of choice.
Still, this was an important step to get us where we are today, and also opened the door for easier sharing of data
between Cryostat’s components.
Cryostat 4.0 did further work toward loosening the coupling of database and object storage containers to the Cryostat installation by splitting them into their own Kubernetes Deployment objects tied to the same Custom Resource, but went no further.
Finally, in Cryostat 4.1 the Cryostat Custom Resource in Kubernetes/OpenShift and the Cryostat Helm Chart both allow
for “external” storage to be configured. If no such configuration is made then the Cryostat Operator or Helm Chart will
default to deploying the same cryostat-storage as usual, to act as a “batteries included” S3-compatible object
storage. However, if you would rather have Cryostat use some other object storage provider for Flight Recordings, you
can now do so easily. Cryostat 4.1 also adds two new types of data that can be captured - Thread Dumps and Heap Dumps.
Heap Dumps in particular tend to be rather large binary files, so users expecting to make heavy use of this feature
should appreciate this new External Storage feature, too. When the Operator or Helm Chart see these configurations they
will configure the Cryostat installation to connect to this external storage provider and skip the cryostat-storage
Deployment entirely.
The new “Docs” section of this website contains a setup guide with much of the same information as below. The goal of this blog post is to apply that same information to a concrete scenario with an actual object storage provider selected, and to demonstrate different Cryostat installations with equivalent configurations.
I’ll illustrate this capability by giving example configurations to hook up the Helm Chart, Operator Custom Resource, and Compose installations to an external object storage provider.
I will use a Backblaze B2 account in these examples, but the general structure will be the same for any provider. You should do your own analysis of the available commercial S3-compatible object storage providers as well as the available open source self-hosted object storage providers to determine which best suits your needs.
Go ahead and create an account with your chosen object storage service provider, or prepare an account in your self-hosted storage solution. You will need to take note of the following pieces of information:
AWS_ACCESS_KEY_ID or similar - this is the equivalent of a username or serviceaccount nameAWS_SECRET_ACCESS_KEY or similar - this is the equivalent of a password or auth tokencryostat-storage) may not be set
up to support storage bucket resolution by virtual host/subdomain, but only by path. If your storage provider supports
virtual host access then you should generally choose to use it.I will use the following parameters for this demo, based on my real Backblaze B2 account but modified/redacted to not actually expose my account:
AWS_ACCESS_KEY_ID will be represented by $AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEY will be represented by $AWS_SECRET_ACCESS_KEYhttps://s3.us-east-005.backblazeb2.comus-east-005abcd1234-apiVersion: v1
stringData:
STORAGE_ACCESS_KEY: $AWS_ACCESS_KEY_ID
STORAGE_ACCESS_KEY_ID: $AWS_SECRET_ACCESS_KEY
kind: Secret
metadata:
name: s3cred
type: Opaque
substituting the two values for your actual account values. Save this as s3cred.yml.
Secret object in your Cryostat installation namespace: kubectl create -f s3cred.yml.$ helm install \
--set storage.storageSecretName=s3cred \
--set storage.provider.url=https://s3.us-east-005.backblazeb2.com \
--set storage.provider.region=us-east-005 \
--set storage.provider.usePathStyleAccess=false \
--set storage.provider.metadata.storageMode=metadata \
--set storage.buckets.names.archivedRecordings=abcd1234-archivedrecordings \
--set storage.buckets.names.archivedReports=archivedreports \
--set storage.buckets.names.eventTemplates=abcd1234-eventtemplates \
--set storage.buckets.names.jmcAgentProbeTemplates=abcd1234-jmcagentprobetemplates \
--set storage.buckets.names.threadDumps=abcd1234-threaddumps \
--set storage.buckets.names.heapDumps=abcd1234-heapdumps \
cryostat ./charts/cryostat
feel free to add other configuration values as desired, ex. --set reports.replicas=1 or
--set core.discovery.kubernetes.enabled=true --set core.discovery.kubernetes.namespaces='{mynamespace}'.
The storage.storageSecretName setting tells the Helm Chart the name of the s3cred Secret which we created, where
it will expect to find the STORAGE_ACCESS_KEY and STORAGE_ACCESS_KEY_ID key-value pairs. These will be used to
configure Cryostat’s S3 API client. The storage.provider.url is the S3 API endpoint. The storage.provider.region
should be self-explanatory. storage.provider.usePathStyleAccess=false configures Cryostat to use virtual host access
since Backblaze B2 supports it, and storage.provider.metadata.storageMode=metadata configures Cryostat to use the
Object Metadata API since Backblaze B2 does not support Object Tagging. The storage.buckets.names.* values set the
globally unique bucket names to use for various different types of data which Cryostat may store. Each of these are
placed into separate buckets so that you can choose to configure different bucket-level policies for different types of
data - storage quotas, object lifecycles, versioning, encryption, storage classes, etc.
This will look rather similar to the previous Helm example.
apiVersion: v1
stringData:
ACCESS_KEY: $AWS_ACCESS_KEY_ID
SECRET_KEY: $AWS_SECRET_ACCESS_KEY
kind: Secret
metadata:
name: s3cred
type: Opaque
substituting the two values for your actual account values. Save this as s3cred.yml.
Secret object in your Cryostat installation namespace: kubectl create -f s3cred.yml.apiVersion: operator.cryostat.io/v1beta2
kind: Cryostat
metadata:
name: cryostat-sample
spec:
objectStorageOptions:
secretName: s3cred
provider:
url: https://s3.us-east-005.backblazeb2.com
region: us-east-005
usePathStyleAccess: false
metadataMode: metadata
storageBucketNameOptions:
archivedRecordings: abcd1234-archivedrecordings
archivedReports: abcd1234-archivedreports
eventTemplates: abcd1234-eventtemplates
heapDumps: abcd1234-heapdumps
jmcAgentProbeTemplates: abcd1234-jmcagentprobetemplates
threadDumps: abcd1234-threaddumps
Refer back to the Helm example for a line-by-line explanation of what each of these configuration properties means. Of course, you can also combine these properties with other Custom Resource properties.
Following my previous Cryostat in Compose post, let’s simply build on that
foundation and use the Cryostat smoketest script’s -s ext (“storage external”) flag to generate a Compose YAML
manifest:
$ cd cryostat
$ export AWS_ACCESS_KEY_ID=replaceme
$ export AWS_SECRET_ACCESS_KEY=replaceme
$ export S3_ENDPOINT=https://s3.us-east-005.backblazeb2.com
$ export S3_REGION=us-east-005
$ export S3_PATH_STYLE_ACCESS=false
The smoketest.bash script generates default bucket names which include the bucket base name (ex. “archives”), the
first few characters of AWS_ACCESS_KEY_ID (this is not considered secret information), the S3_REGION, and a few
random characters as an instance ID. Don’t worry about finding those generated bucket names and manually creating them
- Cryostat will automatically check if the buckets already exist and try to create them if they don’t when it starts.
$ ./smoketest.bash -n -s ext > cryostat-compose.yml
$ for i in *.tar.gz ; do \
f=$(echo $i | cut -d. -f1); podman volume create $f; podman volume import $f $f.tar.gz; \
done
$ podman compose -f cryostat-compose.yml up
Depending on the object storage provider you choose, you may gain additional integration points for the data Cryostat exports to storage. In Backblaze B2 for example, you can set up Event Notifications as a webhook-like system to notify another application when things change. You might choose to have B2 send a notification to another service of yours whenever Cryostat uploads a new Flight Recorder or Heap Dump file to archives, in case you have some additional analysis tooling that you want to build a pipeline to.
Various object storage providers also implement concepts like object lifecycles and bucket storage quotas. You should consider how heavily you use Cryostat and how much data your usage generally produces and set quotas accordingly to avoid accidentally using too much storage space and racking up a large bill. Using object lifecycles to manage old data, especially large objects like Flight Recorder files and Heap Dumps, can also be very helpful. You might choose to move files from standard storage into deep/cold storage after one week, and delete them entirely after one month, for example. Think carefully about how these features might interact with Cryostat’s Automated Rules and periodic archival if you choose to use them.
An interesting side-effect of using external storage in this way arises from the fact that Cryostat does not maintain any separate metadata about what files it has uploaded to the storage buckets. Cryostat simply queries the storage provider for the current bucket contents as needed. This ensures Cryostat is resilient to external modifications of the bucket contents - which may be due to object lifecycle policies, or other applications, or users interacting directly with the storage provider console or API - but also means that you can connect two or more Cryostat instances to the same set of buckets using the same credentials. When Cryostat A pushes a Flight Recorder file to the shared archives, that same file will become visible in Cryostat B’s view of the archives, although Cryostat B will not produce any notification for you that this has happened.
So one interesting use-case you might explore is to install a Cryostat A alongside your applications in Kubernetes and have it export data to a shared storage, then run a Cryostat B in Compose on your local machine hooked up to the same storage. These two Cryostat instances will not share a database so they will not see the same discovered targets, or have the same Automated Rules, etc., but they will share Recording Archives, Thread Dump Archives, and Heap Dump Archives. Just be sure to select the “All Archives” tab of each of these views.
You can then use features like View in Grafana or Automated Analysis (see Step 6) and have the computation done on your local machine instead of by the Cryostat instance installed in your cloud environment.
]]>Although most of this website and other external documentation about Cryostat focuses on how to install and use it in a Kubernetes or OpenShift environment, none of that is actually required for Cryostat itself. Cryostat could be run as a bare metal JVM process, and it can certainly be run as an OCI container in a Docker/Podman environment - in fact, day-to-day Cryostat development testing mostly takes place in Podman containers and doesn’t touch Kubernetes at all!
Since Cryostat 3.0 the development testing setup has relied on YAML docker-compose definition files to piece together
an installation. This is done by the smoketest.bash
script and various YAML base files. The smoketest script
supports a small suite of command line switches and environment variables to customize exactly what gets deployed,
which is obviously useful in development to test various scenarios (like different S3-like object storage providers).
It is also possible to run the script in a dry-run mode and have it print out the final Docker Compose YAML definition,
rather than continuing to actually deploy it. There are a few configuration volumes that are also created and mounted
by the script, such as the one that contains the proxy config and htpasswd file for the oauth2-proxy container.
There is a walkthrough of this process and some sample configuration files that can be downloaded
on the project’s GitHub Wiki page.
In particular, you might want to try customizing the Cryostat Compose YAML using the -s ext flag, which allows you
to hook up Cryostat to an external S3-like object storage provider. This could be another self-hosted Docker Compose
project you’re running locally, or it could be a commercial service - it’s up to you to decide what fits best.
Java, containers, Docker/Podman, and observability are all common things nowadays. In the same way that there are many people, teams, and businesses deploying Java in containers to Kubernetes clouds every day - or multiple times per day - and needing a way to profile and observe how those Java VMs are performing, there are people, teams, and businesses running their workloads without Kubernetes. So let’s put together a scenario where someone might be deploying a Java- based application in their home lab, or on their small office NAS, or in the corner of their startup’s garage, running Compose instead of Kubernetes.
This example will use Podman and podman-compose, but should also work with little to no modification with Docker and Docker Compose.
Let’s put together a compose.yml for a sample application. In practice this might be a Compose file you download from
the project you’re trying to deploy, or it could be a Compose file you’ve written yourself to deploy the thing you’re
developing. I’ll use Cryostat’s quarkus-petclinic
testing sample, which is based on this demo quarkus-petclinic.
name: petclinic-cryostat
services:
quarkus-petclinic:
image: ${QUARKUS_PETCLINIC_IMAGE:-quay.io/redhat-java-monitoring/quarkus-petclinic:latest}
hostname: quarkus-petclinic
depends_on:
quarkus-petclinic-db:
condition: service_healthy
ports:
- "10011:10011"
environment:
QUARKUS_HTTP_HOST: 0.0.0.0
QUARKUS_HTTP_PORT: 10011
QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://quarkus-petclinic-db:5432/petclinic
restart: always
healthcheck:
test: curl --fail http://localhost:10010 || exit 1
interval: 10s
retries: 3
start_period: 30s
timeout: 5s
quarkus-petclinic-db:
image: "quay.io/sclorg/postgresql-15-c9s"
deploy:
resources:
limits:
cpus: "1"
memory: 128m
environment:
- POSTGRESQL_USER=developer
- POSTGRESQL_PASSWORD=developer
- POSTGRESQL_DATABASE=petclinic
healthcheck:
test: ["CMD-SHELL", "pg_isready --dbname $$POSTGRES_DB --username $$POSTGRES_USER"]
interval: 5s
timeout: 5s
retries: 6
ports:
- "5432:5432"
This just describes a simple Quarkus-based webserver application in one container, and a Postgres database in a second container, with a bit of configuration to tell the application how to hook up to that database.
Save this YAML as petclinic.yml, then run podman compose up -f petclinic.yml in a terminal to spin up the
application. Wait a few moments for the two containers to come up and become ready, then go to
http://localhost:10011 and ensure it is running. Then return to your terminal and press
Ctrl-c to shut it down for now.
As I mentioned above, there is a more complete description of the Cryostat setup process on the project’s GitHub Wiki page with sample configuration volume files.
Download the .tar.gz configuration file volumes and import them as instructed. Copy the main cryostat-compose.yml
YAML file content and save it locally. Then, run podman compose -f cryostat-compose.yml up -d, wait for the containers
to pull, start, and become healthy, then go to https://localhost:8443 to ensure Cryostat is
up and running in the background.
Now that we have Cryostat up and running, let’s get our application running again and in a way that Cryostat is able
to see it and collect that precious performance data. Modify your petclinic.yml to add some container labels and
enable JMX:
name: petclinic-cryostat # this must match the name: property of the petclinic.yml
services:
quarkus-petclinic:
labels:
io.cryostat.discovery: "true"
io.cryostat.jmxHost: "quarkus-petclinic" # this must match the service's name
io.cryostat.jmxPort: "11223" # this must match the jmxremote.port and jmxremote.rmi.port number configured below
environment:
JAVA_OPTS_APPEND: >-
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=11223
-Dcom.sun.management.jmxremote.rmi.port=11223
-Djava.rmi.server.hostname=quarkus-petclinic # this must also match the service's name
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
...
This is only a demo, so we’re skipping JMX authentication and TLS setup. You should always configure this on your
production workloads, and check the guides on this site for instructions on how to configure Cryostat to
work with JMX+TLS and JMX+auth.
Depending on your Java application, framework, base image, entrypoint script, etc. you may need to change the
JAVA_OPTS_APPEND environment variable name or use a different configuration entirely.
Let’s start up this application again and run it in the background: podman compose -f petclinic.yml up -d. Go back
to Cryostat at https://localhost:8443/topology and
check out the Topology view. You should see a target JVM named
compose-quarkus-petclinic-1.
podman compose -f cryostat-compose.yml down can be used to tear down the Cryostat installation. You can also pass
--volumes if you’d like to delete the persistent volumes that were attached to this installation, which contain
captured data such as archived Flight Recording files.
Likewise, podman compose -f petclinic.yml down [--volumes] can be used to tear down the quarkus-petclinic
application.
Let’s say you have followed this guide and installed Cryostat 4.1.0, which is the current latest version. Later down the line you notice that Cryostat 4.1.1 or Cryostat 4.2.0 has been released, and you want to upgrade your Cryostat-in-Compose installation. How would you do that?
Using the same cryostat-compose.yml file from before, do podman compose -f cryostat-compose.yml down.
Then follow the guide on getting a Cryostat Compose file
again, and replace the cryostat-compose.yml to ensure that all of the appropriate updates have been made to container
version tags, environment variables, etc. Then, podman compose -f cryostat-compose.yml up -d will spin up the updated
deployment, apply any database migrations, and restore your Cryostat functionality.
Hello everyone,
Cryostat 4.1.0 has landed. It’s a minor release in semantic versioning terms, but it’s a big release in terms of functionality and features. Read on to find out what’s new.
Deployment YAML.
Automated Rules view is now under the Flight Recorder section, under the Capture subsection, as the Automated Rules item - or Flight Recorder/Capture/Automated Rules. Recordings has been split into Flight Recorder/Capture/Recordings and Flight Recorder/Analyze/Archives, and Events has been split into Flight Recorder/Capture/Events and Flight Recorder/Capture/Instrumentation.
Dashboard card, and that same card also now contains the same thread and heap dump controls as this new navigable view. Thread and heap dumps are stored in new, separate buckets in Cryostat's object storage, and have similar Targets, All-Targets, and All-Archives viewing modes as Archived Recordings. They can also be downloaded, deleted, and labeled like archived recordings, but there are no analysis tools for these data types within the Cryostat UI.
cryostat-storage S3-compatible object storage instance, as opposed to simply written to local filesystem within the Cryostat container. In Cryostat 4.1 the configuration knobs to choose your own S3-compatible object storage are now finally exposed, so when you install a Cryostat instance you can choose the batteries-included cryostat-storage option, or opt for your own choice of object storage provider. This may be your own self-hosted object storage which you manage or it can be a commercial service provider. For long-lived production Cryostat installations it is highly recommended that you should choose a commercial service provider or your own self-managed object storage, rather than relying on the simple object storage included with Cryostat. Various internal improvements have also been made for the communication between Cryostat, cryostat-storage, cryostat-reports, and jfr-datasource to optimize the transfer of files between containers.
ConfigMaps or Secrets. It is now also possible to define the following resource types declaratively:
-Dcryostat.agent.smart-trigger.config.path=/path/to/file or the environment variable CRYOSTAT_AGENT_SMART_TRIGGER_CONFIG_PATH=/path/to/file.
Automated Analysis Reports.
Topology view indicates Automated Analysis scores on discovered targets using a status badge displaying the maximum (most critical) analysis score. However, some of these analysis scores are static, configuration-based scores which may not be of particular interest in all cases - for example, the Passwords in Environment Variables score. In many cases it is acceptable to include passwords in environment variables, so if this is the only detected "issue" with a target JVM, it may not be desirable for this to reflect as a critical status on the Topology view. You can use this new Settings configuration to control which analysis results are used for determining the Topology view status badges.
Recordings into Flight Recorder/Capture/Recordings and Flight Recorder/Analyze/Archives and the new Flight Recorder/Analyze,Automated Reports and autoanalyze=true feature described above, the workflow for manually generating an Automated Analysis report for a target JVM has been reworked. The new workflow guides you through creating a recording (with selected default values) if required, then creates, archives, and deletes a Snapshot recording with the `autoanalyze=true` label to trigger the automated analysis. The results will be viewable in the new Analyze panel in the Flight Recorder/Capture/Recordings view as well as at Flight Recorder/Analyze/Automated Reports, and will also be cached and connected to the archived recording copy viewable at Flight Recorder/Analyze/Archives. This reduces the number of clicks to get to an automated analysis report for a target application, as well as ensuring the results are easy to both view immediately and reference back to in the future.
Archived Recording with the label autoanalyze=true is created, Cryostat will automatically perform automated analysis of that recording. The results will be cached and viewable in the new Flight Recorder/Analyze/Automated Reports view. Hint: labels applied to Active Recordings are copied to Archived Recordings, so the act of archiving any active recording with this label applied will trigger this behaviour. You will notice that the active recording creation form and Automated Rule creation form have new controls to enable this feature, which is turned on by default. Cryostat will also automatically perform this function for "external recordings" - ones which Cryostat observes are available in a target JVM, which Cryostat did not initiate. Such recordings are often present due to the use of the -XX:StartFlightRecording JVM flag.
Automated Rule from the Cryostat UI as a basis for a new rule, you would need to either:
Edit action which allows you to edit a Rule in-place (with some limitations), and a new Copy action which allows you to create a new Rule using the existing Rule as a template with the same configuration.
Dashboard and Topology views contain a component which displays details about a selected JVM target. This also includes components where a miniature Topology view is included, such as Automated Rules and Security. Other views, such as Flight Recorder/Capture/Recordings provided no way to view JVM details beyond the alias and connection URL via the target selector dropdown. There is a new info button beside the dropdown which, when clicked, summons a modal dialog containing the same details component so that this information is visible anywhere a target selection has been made.
Recordings view had a tab for target-specific archived recordings, and the old Archives view had a tab for All Targets. These were extremely similar and redundant. The new Flight Recorder/Capture/Recordings view focuses only on active recordings and removes the target-specific archives tab. The new Flight Recorder/Analyze/Archives view has a Targets tab with a target selector dropdown. If a target is selected this behaves just like the old Recordings view and its target-specific archives tab. If the target selection is cleared then this behaves just like the old Archives view and its All Targets tab.
Frameworks row. This includes a chart for Hibernate ORM JFR events, and a chart and a table for Quarkus REST JFR events. Cryostat 4.0 previously shipped Preset Event Templates, Preset Automated Rules, and a Match Expression feature to help you take capture these JFR event types, so this new feature provides a convenient way for you to visualize this captured data.
For further details on these items, check the GitHub Discussions release announcement.
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 4.0.x with OLM, then you may have already been upgraded to 4.1.0, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>Hello everyone,
We are pleased to announce the release of Cryostat 4.0.1! This is a minor bugfix release addressing some issues uncovered with the previous 4.0.0 feature release.
Automated Rules error handling was improvedDiscovery only Custom Targets should be deletableDiscovery handles Kubernetes Services (Endpoints) which have nil targetslogging goes to stdout by default, not stderr, can be configured using Agent properties overrides arguments, and passes down log level to the Red Hat Insights clientshutdown avoids a potential NPErecording queries include recordings started by the Agent’s own Harvester facilitystartup performs callback DNS resolution within the discovery registration loop, not only once on startupautoconfig allows control of the Agent’s log level, and disables Agent logging by defaultsecrets/configmaps are hashed and appended to Deployment pod templates so that configuration updates like renewed certificates cause a rolloutreports sidecars are correctly configured so that they handle automated analysis report generationsproxy handles connection failures to upstream Cryostat instances gracefully, avoiding proxy restartdashboard correctly updates MBean Metrics chartsvolumes configuration by the entrypoint script respects the underlying implementation’s enforced limitsFor further details on these items, check the GitHub Discussions release announcement.
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 4.0.0 with OLM, then you may have already been upgraded to 4.0.1, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>Hello everyone,
It’s time for another Cryostat release. This time, we are very proud to present Cryostat 4.0, a new major version with again some significant under-the-hood changes, and some exciting new OpenShift-specific features.
quarkus-jfr extension (along with quarkus-rest), or has hibernate-jfr on its classpath, then you can take advantage of the new Preset Event Templates that ship with Cryostat. These are open to community contributions for any other popular frameworks and libraries that provide JFR events. Simply start a recording using the relevant preset template and begin capturing these framework-specific events.jfrEventTypeIds(target): string[] function available in the matchExpression Common Expression Language scope. Pass in the target object also in scope and this function returns a list of JFR event type IDs for that target JVM. This enables functionality such as that seen in the new quarkus and hibernate Automated Rules that ship with Cryostat 4.0, where the rules’ expressions are satisfied by target JVMs that support the relevant JFR event types. This makes it even quicker and easier to get started capturing data from applications using these frameworks and libraries..spec.template.metadata.labels and add cryostat.io/namespace: ${cryostat-cr-ns} and cryostat.io/name: ${cryostat-cr-name} labels (replacing the ${var} expressions with corresponding values). This prompts the Cryostat Operator to patch your Deployment to include Volumes containing the Cryostat Agent JAR and required TLS certificates, to include a JAVA_TOOL_OPTIONS environment variable containing the -javaagent flag to statically attach the Agent to your application, and to include other environment variables to configure the Agent to communicate with the specified Cryostat instance./api/v1 through /api/v3 path prefixes are no longer used, and all API endpoints are now prefixed with /api/v4. Many of these are fully backward compatible with how they behaved in 3.0.jfr-datasource for analysis in Grafana, or performing automated analysis on recordings. When a client request for one of these operations comes in, Cryostat will immediately send back a response containing a Job ID. At some later point in time, a WebSocket notification will be sent containing the same Job ID and indicating the success/failure of that job and any particular details about it. For automated analysis report generation in particular, the first time a request is sent the response will contain a Job ID, but when the job completes then Cryostat will hold the result in a cache for some time. The client should send a follow-up request after the WebSocket notification is received, and this time Cryostat will respond with the actual report JSON document. This differs from 3.0 and prior Cryostat versions, where the client would send a request and Cryostat would send no response at all until the entire operation completed - which could result in the client timing out and abandoning the request.reports.replicas. Simply --set reports.replicas=1 (or any positive integer you choose) when installing or upgrading the Helm Chart.storage.storageSecretName and authentication.cookieSecretName. When these are specified the Helm Chart does not need to do a dynamic lookup of any self-generated Secrets already in the cluster, so the installation is more declarative and can be more easily managed with GitOps-style workflows. Be sure to create the Secrets first before referencing them in the Helm Chart installation.cryostat-agent-init container image is now available. This is a small container image simply containing the Cryostat Agent -shaded JAR. This can be used in multistage OCI container image builds as a potentially simpler way to insert the JAR, rather than using a Maven dependency. This is also what the Operator uses for autoconfiguration to mount a Volume containing the JAR.ubi9 and JDK 21. Cryostat 3.0 containers used ubi8 and JDK 17.simple-json-datasource.To accompany this release I have recorded several demo/tutorial videos. These illustrate some of the new features as well as highlighting existing Cryostat functionality that seemed to connect naturally:
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
Note: the Helm Chart 4.0.0 release will be a bit delayed this time around. Please watch the GitHub repository for updates if you require the Helm Chart.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 3.0.0 with OLM you will need to perform a manual upgrade to 4.0.0. You should back up all data (Automated Rules, Custom Event Templates, Stored Credentials, Archived Recordings), perform the upgrade, then restore all of your data. We plan that the next release will allow a seamless in-place upgrade once again.
Please reach out to the Cryostat mailing list or GitHub Discussions with any questions or comments.
]]>Hello everyone,
We are pleased to announce the release of Cryostat 3.0.1! This is a minor bugfix release addressing some issues uncovered with the previous 3.0.0 feature release.
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 3.0.0 with OLM, then you may have already been upgraded to 3.0.1, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>Hello everyone,
It’s time for another Cryostat release. This time, we are pleased to present Cryostat 3.0, a new major version with some significant under-the-hood changes.
cryostat-db (lightly customized Postgres) database instead of H2 file-based database, and more information is kept in the database: the encrypted target credentials keyring, Automated Rules definitions, discovered targets, and discovery plugins are all persisted in this database. In this release the database schema is still in an early state and it is not yet recommended to attempt to run Cryostat with a separately managed database deployment.cryostat-storage container (a lightly customized SeaweedFS) for a “batteries included” storage solution. If you roll your own deployment then you can swap this out for any S3-compatible storage provider you prefer or may already use.openshift-oauth-proxy or oauth2_proxy) in the Pod. Only this proxy is exposed to cluster traffic via a Service. This means that all API requests to Cryostat, as well as all visitors to the Cryostat Web UI or Grafana dashboard flow through this proxy. The proxy handles user sessions to control access to the application, providing unified access control and user sessions for both the Cryostat Web UI and Grafana dashboard. Both of these UIs are accessible via the same Ingress or Route and present the same TLS certificate. When deployed on OpenShift, the auth proxy will use its Namespace (the Cryostat installation Namespace) to perform Role-Based Access Control checks for user authentication and authorization by integrating with the OpenShift cluster SSO provider, very similar to what previous Cryostat releases directly implemented. Additionally, the auth proxy can be optionally configured with an htpasswd file to enable Basic authentication. On OpenShift this allows for defining additional user accounts that may access the Cryostat application beyond those with OpenShift SSO RBAC access, whereas on other Kubernetes it is the primary authentication mechanism.Specifically for the Cryostat Operator there are a few items of note:
v1beta1/Cryostat and v1beta1/ClusterCryostat custom resources are now unified in the v1beta2/Cryostat CR. When creating this CR you may specify an optional list of target namespaces. The installation namespace - the CR’s .metadata.namespace - is the location where the Operator will create the Cryostat Deployment, and on OpenShift this namespace will also be used by the auth proxy for RBAC to control which users have access to the Cryostat application. The default user Role required is create pods/exec in the Cryostat CR’s installation namespace, but this can be configured using the .spec.authorizationOptions.openShiftSSO.accessReview property. The list of target namespaces is used to control which namespaces the Cryostat instance should monitor for new target applications to appear within, the same as before. Taken together with point 1 above it is now simpler to install the Operator and configure your Cryostat instance(s): simply install the Operator cluster-wide, then create one or more Cryostat CRs to correspond to namespaces or groups of namespaces that contain your applications..spec.networkOptions.coreConfig.externalHost CRD property available, which can be used to set a custom domain for the OpenShift Route which the Operator creates for your Cryostat deployment.For the Cryostat Helm Chart, there are new configuration values that can be set:
openshift-oauth-proxy. If disabled (as default) then oauth2_proxy is deployed instead. Both proxies may be configured to enable Basic authentication (see next item), but for users deploying onto OpenShift, the openshift-oauth-proxy also implements integration with the OpenShift cluster SSO.openshift-oauth-proxy this is in addition to the OpenShift SSO, whereas when deploying oauth2_proxy this is the only out-of-the-box supported user authentication mechanism.SubjectAccessReview for testing user/client access via OpenShift SSO can be configured with this property.cryostat-db instance, and a cryostat-storage instance as outlined at the top of this post. There will be only one Service, and one Ingress/Route, pointing to the auth proxy.This is not an exhaustive list of all the new configuration values, only a highlight of a few that control new visible features. Please check the chart’s updated README for a full listing.
In this release there is only one new feature implemented and one enhancement.
The minor enhancement first: when the Cryostat Agent starts up it also starts a tiny embedded webserver, which it uses to service requests made by the Cryostat server. This embedded webserver secures itself using Basic authentication. Previously, the Basic username was always user and the password was randomly generated and 24 ASCII characters in length. In this release both of these are configurable: the username can be overridden (but defaults to user), and the generated password length can be changed (but defaults to 24 characters). The generated password also draws from a larger character set.
The new feature is Dynamic Attach. In short, this means that the Cryostat Agent can be attached to an already-running application JVM, as opposed to requiring attachment at JVM startup time via the -javaagent flag. This requires that the Agent JAR is available on the same host or in the same container filesystem as the application JVM, and that the user has some way to exec the Agent JAR as a separate java process to bootstrap the attachment. For example, if the application is deployed in Kubernetes, then the user can set this up by downloading the Cryostat Agent JAR locally, copying it into the application container using kubectl cp, and then run the Agent JAR using kubectl exec.
$ kubectl cp \
/path/to/cryostat-agent.jar \
-n my-namespace \
mypod:/tmp/cryostat/cryostat-agent.jar
$ kubectl exec \
-n my-namespace \
mypod -c mycontainer \
-i -t -- \
java -jar /tmp/cryostat/cryostat-agent.jar \
-Dcryostat.agent.baseuri=http://cryostat:8181 \
-Dcryostat.agent.authorization="Bearer ${MY_AUTH_TOKEN}" \
-Dcryostat.agent.callback=http://${POD_IP}:9977 \
-Dcryostat.agent.api.writes-enabled=true
Please note that the -D system properties for the Agent are placed after the -jar /path/to/cryostat-agent.jar. This ensures that they are arguments picked up by the Agent process after it starts, which it receives and uses to set the system properties again when it attaches to the application JVM process that it finds. Any other system properties, including ones passed before the -jar flag, will take effect on the separate Agent process but will not be carried over to the application JVM when the Agent attaches.
What hasn’t changed?
If you are a cluster administrator installing Cryostat you will notice some changes in the Cryostat Operator and Cryostat CR, or in the Cryostat Helm Chart configuration values. If you manually deploy Cryostat then there are a large number of setup and configuration changes required, with different and new environment variables and new additional containers to deploy (the auth proxy, database, and object storage mentioned above).
From an end user’s perspective most things remain the exact same. Despite the major version number increment and breaking configuration changes, this release did not target adding any significant new user features or modifying existing ones. Only a few noteworthy items stand out:
Security navigation item in the Cryostat Web UI no longer offers a way to upload TLS certificates directly into the Cryostat server truststore. Instead, this view only provides a list of the certificates that have been loaded. Adding new certificates to the truststore must be done by adding them to the storage volume which Cryostat reads at startup. Using the Operator this can still be done with the existing TrustedCertSecrets CR property.X-JMX-Authorization header is no longer supported. Previous versions of Cryostat accepted this header on API requests and would use the header value as basic auth credentials when opening a JMX connection to a target application. This was replaced by the encrypted database table backing the encrypted credentials keyring mechanism a few versions back, which was made the default. In 3.0 only this encrypted credentials keyring is supported, the X-JMX-Authorization header is no longer handled, and the Cryostat Web UI Settings view no longer offers an advanced configuration for selecting which mechanism to use.
Generally, the rest of the UI and features should look, feel, and behave the same way as they did in 2.4.localhost:0. This special value asks the JVM to open a local JMX connection to itself, without exposing a port to the network, so additional authentication and TLS encryption is not necessary./api/v1, /api/v2, /api/v2.1, /api/v2.2, and /api/v2.3 API endpoints - many of these will now return HTTP 3xx redirection responses to the new /api/v3 equivalents, but some will continue to serve requests directly (where the response format differs). There were no new Cryostat 2.4 API additions and so there is no corresponding /api/v2.4 prefix. /api/beta endpoints have seen some breaking changes. In all cases, /api/beta endpoints should be considered subject to change or removal in any future release, and all of the v1|v2|v2.1|v2.2|v2.3 endpoints should now be considered deprecated. These will be removed in a future 3.x release in favour of the /api/v3 equivalents that already exist, or in favour of new endpoints with this prefix which will retain the same general semantics as the older versions they replace. If you have built any additional scripts or tooling around the Cryostat API, please consider upgrading these now to be compatible with the new v3 API endpoints.Hello everyone,
We are pleased to announce the release of Cryostat 2.4.0! This is a feature release with the following key highlights compared to 2.3.1:
This was a relatively smaller feature release compared to some previous ones. That’s because more of our development effort has already shifted to the next major release: Cryostat 3.0. Stay tuned for more in-depth details about what 3.0 will entail. For now, here are two things you can expect from 3.0:
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 2.3.1 with OLM, then you may have already been upgraded to 2.4.0, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>Hello everyone,
We are pleased to announce the release of Cryostat 2.3.1! This is a minor bugfix release addressing some issues uncovered with the previous 2.3.0 feature release.
multipart/form-data:
CRYOSTAT_DISABLE_SSL was handled inconsistently and would not always disable server SSL as expected:
CRYOSTAT_DISABLE_BUILTIN_DISCOVERY would also disable Custom Targets, and could break server startup in some scenarios:
You can install Cryostat using our Kubernetes operator on OperatorHub.io or via a Helm Chart . As always, you can also run Cryostat in other environments with a little more manual setup.
If this is your first time installing Cryostat on Kubernetes, you can Get Started right here on this website.
If you had previously installed Cryostat Operator 2.3.0 with OLM, then you may have already been upgraded to 2.3.1, or else you should be able to approve and install the upgrade.
Please reach out to the Cryostat mailing list or GitHub Discussion with any questions or comments.
]]>