Skip to content

Fix shadow configs#1

Closed
Goooler wants to merge 1 commit into
dejan2609:KAFKA-18142-shadow-9.0.0-beta4from
Goooler:update-shadow
Closed

Fix shadow configs#1
Goooler wants to merge 1 commit into
dejan2609:KAFKA-18142-shadow-9.0.0-beta4from
Goooler:update-shadow

Conversation

@Goooler

@Goooler Goooler commented Dec 24, 2024

Copy link
Copy Markdown
diffuse diff --jar kafka-clients-4.1.0-SNAPSHOT-before.jar kafka-clients-4.1.0-SNAPSHOT-after.jar

OLD: kafka-clients-4.1.0-SNAPSHOT-before.jar
NEW: kafka-clients-4.1.0-SNAPSHOT-after.jar

 JAR   │ old       │ new       │ diff 
───────┼───────────┼───────────┼──────
 class │  23.2 MiB │  23.2 MiB │  0 B 
 other │ 503.3 KiB │ 503.3 KiB │  0 B 
───────┼───────────┼───────────┼──────
 total │  23.7 MiB │  23.7 MiB │  0 B 

 CLASSES │ old   │ new   │ diff      
─────────┼───────┼───────┼───────────
 classes │  3889 │  3889 │ 0 (+0 -0) 
 methods │ 56595 │ 56595 │ 0 (+0 -0) 
  fields │ 15708 │ 15708 │ 0 (+0 -0) 
diff kafka-clients-4.1.0-SNAPSHOT-before.pom kafka-clients-4.1.0-SNAPSHOT-after.pom

2a3,7
>   <!-- This module was also published with a richer model, Gradle metadata,  -->
>   <!-- which should be used instead. Do not delete the following line which  -->
>   <!-- is to indicate to Gradle or any Gradle module metadata file consumer  -->
>   <!-- that they should prefer consuming it instead. -->
>   <!-- do_not_remove: published-with-gradle-metadata -->
7d11
<   <packaging>pom</packaging>

Closes GradleUp/shadow#1115.

@Goooler

Goooler commented Dec 24, 2024

Copy link
Copy Markdown
Author

Oooops, I missed you don't want MANIFEST.MF here.

@Goooler Goooler closed this Dec 24, 2024
```diff
diffuse diff --jar kafka-clients-4.1.0-SNAPSHOT-before.jar kafka-clients-4.1.0-SNAPSHOT-after.jar

OLD: kafka-clients-4.1.0-SNAPSHOT-before.jar
NEW: kafka-clients-4.1.0-SNAPSHOT-after.jar

 JAR   │ old       │ new       │ diff
───────┼───────────┼───────────┼──────
 class │  23.2 MiB │  23.2 MiB │  0 B
 other │ 503.3 KiB │ 503.3 KiB │  0 B
───────┼───────────┼───────────┼──────
 total │  23.7 MiB │  23.7 MiB │  0 B

 CLASSES │ old   │ new   │ diff
─────────┼───────┼───────┼───────────
 classes │  3889 │  3889 │ 0 (+0 -0)
 methods │ 56595 │ 56595 │ 0 (+0 -0)
  fields │ 15708 │ 15708 │ 0 (+0 -0)
```

```diff
diff kafka-clients-4.1.0-SNAPSHOT-before.pom kafka-clients-4.1.0-SNAPSHOT-after.pom

2a3,7
>   <!-- This module was also published with a richer model, Gradle metadata,  -->
>   <!-- which should be used instead. Do not delete the following line which  -->
>   <!-- is to indicate to Gradle or any Gradle module metadata file consumer  -->
>   <!-- that they should prefer consuming it instead. -->
>   <!-- do_not_remove: published-with-gradle-metadata -->
7d11
<   <packaging>pom</packaging>
```
Comment thread build.gradle
xml.asNode().appendNode('dependencies')
}
def dependenciesNode = xml.asNode().get('dependencies').get(0)
def dependenciesNode = xml.asNode().get('dependencies') ?: xml.asNode().appendNode('dependencies')

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Goooler

Goooler commented Dec 24, 2024

Copy link
Copy Markdown
Author

@dejan2609

Everything looks good for now. I'm not sure why the packaging node is missing, but it should be a small thing.

@dejan2609

dejan2609 commented Dec 24, 2024

Copy link
Copy Markdown
Owner

💯
Thank you @Goooler ! I will integrate your solution.

@Goooler Goooler closed this Dec 25, 2024
@Goooler Goooler deleted the update-shadow branch February 25, 2025 01:38
dejan2609 pushed a commit that referenced this pull request Feb 12, 2026
…ker provisioning (apache#21394)

## Summary

Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were
ignored during system test worker provisioning, and refactors
`vagrant/base.sh` to support flexible JDK versions without code changes.

---

## Problem

The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that
caused JDK version parameters to be ignored:

| Bug | Problem |
|-----|---------|
| **#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to
`17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 |
| **#2: `--jdk-arch` ignored** | Architecture parameter was passed but
never used in the S3 download URL |

---

## Solution

- Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex
- Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH`
- Update S3 path to use `/jdk/` subdirectory
- Add logging for debugging

---

## Changes

### `vagrant/base.sh`

| Change | Description |
|--------|-------------|
| **Input validation** | Added regex validation for `JDK_MAJOR` and
`JDK_ARCH` with sensible defaults |
| **Dynamic construction** | `JDK_FULL` is now constructed from
`JDK_MAJOR` and `JDK_ARCH` if not explicitly provided |
| **Updated S3 path** | Changed URL from
`/kafka-packages/jdk-{version}.tar.gz` to
`/kafka-packages/jdk/jdk-{version}.tar.gz` |
| **Logging** | Added debug output for JDK configuration |
| **Backward compatibility** | Vagrantfile can still pass `JDK_FULL`
directly; the script validates and uses it if valid |

---

## S3 Path Change

### Old Path
```
s3://kafka-packages/jdk-{version}.tar.gz
```

### New Path
```
s3://kafka-packages/jdk/jdk-{version}.tar.gz
```

### Available JDKs in `s3://kafka-packages/jdk/`

| File | Version | Architecture |
|------|---------|--------------|
| `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 |
| `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 |
| `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 |
| `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 |
| `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 |
| `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 |
| `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 |
| `jdk-17-linux-x64.tar.gz` | 17 | x64 |
| `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 |
| `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 |
| `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 |
| `jdk-25-linux-x64.tar.gz` | 25 | x64 |
| `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 |
| `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 |
| `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 |
| `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 |
| `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 |

---

## Future JDK Releases

> **IMPORTANT: No code changes required for future Java major/minor
releases!**

The validation regex supports all version formats:
- **Major versions**: `17`, `25`, `26`
- **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1`
- **Legacy format**: `8u144`, `8u202`

### Adding New JDK Versions

To add support for a new JDK version (e.g., JDK 26, 25.0.3):

1. Download the JDK tarball from Oracle/Adoptium
2. Rename to follow naming convention:
`jdk-{VERSION}-linux-{ARCH}.tar.gz`
3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz
s3://kafka-packages/jdk/`
4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}`

No modifications to `base.sh` or any other scripts are needed.

---

## Benefits

| Before | After |
|--------|-------|
| `--jdk-version` ignored | ✅ Correctly uses specified version |
| `--jdk-arch` ignored | ✅ Correctly uses specified architecture |
| Only major version support | ✅ Full version support (e.g., `25.0.2`) |
| Code change needed for new JDK | ✅ Just upload to S3 and pass version
|

---

## Testing

Tested with different JDK versions to confirm the fix works correctly:

| Test | JDK_MAJOR | Expected | Actual | Result | Test Report |
|------|-----------|----------|--------|--------|-------------|
| JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ |
| JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ |


---

## Backward Compatibility

- **Vagrantfile**: Continues to work as before
- **Existing workflows**: Default behavior unchanged (JDK 17 on x64
architecture)
- **No breaking changes**: All existing configurations continue to work

---

 Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants