Skip to content

KAFKA-9726: Add IdentityReplicationPolicy to MirrorMaker2 (#10652)#1

Merged
ivanyu merged 1 commit into
2.8-2021-06-28from
identity-replication-policy
Jul 2, 2021
Merged

KAFKA-9726: Add IdentityReplicationPolicy to MirrorMaker2 (#10652)#1
ivanyu merged 1 commit into
2.8-2021-06-28from
identity-replication-policy

Conversation

@mdedetrich

@mdedetrich mdedetrich commented Jul 1, 2021

Copy link
Copy Markdown

This PR contains a cherry pick of apache@93f5737 without all of the tests (due to merge conflicts)

This new policy enables active/passive, one-way replication without renaming topics, similar to MM1. This implementation is described in KIP-382 (adopted), originally as "LegacyReplicationPolicy".

This enables operators to migrate from MM1 to MM2 without re-architecting their replication flows, and enables some additional use-cases for MM2. For example, operators may wish to "upgrade" their Kafka clusters by mirroring everything to a completely new cluster. Such a migration would have been difficult with either MM1 or MM2 previously.

When using IdentityReplicationPolicy, operators should be aware that MM2 will not be able to detect cycles among replicated topics. A misconfigured topology may result in replicating the same records back-and-forth or in an infinite loop. However, we don't prevent this behavior, as some use-cases involve filtering records (via SMTs) to prevent cycles.

Reviewers: Mickael Maison <mickael.maison@gmail.com>

Co-authored-by: Ryanne Dolan <rdolan@twitter.com>
Co-authored-by: Matthew de Detrich <mdedetrich@gmail.com>
Co-authored-by: Ivan Yurchenko <ivanyu@aiven.io>
@ivanyu ivanyu merged commit dcae2d4 into 2.8-2021-06-28 Jul 2, 2021
@ivanyu ivanyu deleted the identity-replication-policy branch July 2, 2021 04:25
jeqo pushed a commit that referenced this pull request Sep 27, 2023
This change introduces some basic clean up and refactoring for forthcoming commits related to the revised fetch code for the consumer threading refactor project.

Reviewers: Christo Lolov <lolovc@amazon.com>, Jun Rao <junrao@gmail.com>
AnatolyPopov pushed a commit that referenced this pull request Feb 16, 2024
This change introduces some basic clean up and refactoring for forthcoming commits related to the revised fetch code for the consumer threading refactor project.

Reviewers: Christo Lolov <lolovc@amazon.com>, Jun Rao <junrao@gmail.com>
aiven-anton pushed a commit that referenced this pull request May 22, 2025
… for aborted txns (apache#17676) (#1 7733)

Reviewers: Jun Rao <junrao@gmail.com>
tvainika pushed a commit that referenced this pull request Apr 17, 2026
…ker provisioning (apache#21394)

## Summary

Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were
ignored during system test worker provisioning, and refactors
`vagrant/base.sh` to support flexible JDK versions without code changes.

---

## Problem

The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that
caused JDK version parameters to be ignored:

| Bug | Problem |
|-----|---------|
| **#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to
`17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 |
| **#2: `--jdk-arch` ignored** | Architecture parameter was passed but
never used in the S3 download URL |

---

## Solution

- Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex
- Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH`
- Update S3 path to use `/jdk/` subdirectory
- Add logging for debugging

---

## Changes

### `vagrant/base.sh`

| Change | Description |
|--------|-------------|
| **Input validation** | Added regex validation for `JDK_MAJOR` and
`JDK_ARCH` with sensible defaults |
| **Dynamic construction** | `JDK_FULL` is now constructed from
`JDK_MAJOR` and `JDK_ARCH` if not explicitly provided |
| **Updated S3 path** | Changed URL from
`/kafka-packages/jdk-{version}.tar.gz` to
`/kafka-packages/jdk/jdk-{version}.tar.gz` |
| **Logging** | Added debug output for JDK configuration |
| **Backward compatibility** | Vagrantfile can still pass `JDK_FULL`
directly; the script validates and uses it if valid |

---

## S3 Path Change

### Old Path
```
s3://kafka-packages/jdk-{version}.tar.gz
```

### New Path
```
s3://kafka-packages/jdk/jdk-{version}.tar.gz
```

### Available JDKs in `s3://kafka-packages/jdk/`

| File | Version | Architecture |
|------|---------|--------------|
| `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 |
| `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 |
| `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 |
| `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 |
| `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 |
| `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 |
| `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 |
| `jdk-17-linux-x64.tar.gz` | 17 | x64 |
| `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 |
| `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 |
| `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 |
| `jdk-25-linux-x64.tar.gz` | 25 | x64 |
| `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 |
| `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 |
| `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 |
| `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 |
| `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 |

---

## Future JDK Releases

> **IMPORTANT: No code changes required for future Java major/minor
releases!**

The validation regex supports all version formats:
- **Major versions**: `17`, `25`, `26`
- **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1`
- **Legacy format**: `8u144`, `8u202`

### Adding New JDK Versions

To add support for a new JDK version (e.g., JDK 26, 25.0.3):

1. Download the JDK tarball from Oracle/Adoptium
2. Rename to follow naming convention:
`jdk-{VERSION}-linux-{ARCH}.tar.gz`
3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz
s3://kafka-packages/jdk/`
4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}`

No modifications to `base.sh` or any other scripts are needed.

---

## Benefits

| Before | After |
|--------|-------|
| `--jdk-version` ignored | ✅ Correctly uses specified version |
| `--jdk-arch` ignored | ✅ Correctly uses specified architecture |
| Only major version support | ✅ Full version support (e.g., `25.0.2`) |
| Code change needed for new JDK | ✅ Just upload to S3 and pass version
|

---

## Testing

Tested with different JDK versions to confirm the fix works correctly:

| Test | JDK_MAJOR | Expected | Actual | Result | Test Report |
|------|-----------|----------|--------|--------|-------------|
| JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ |
| JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ |


---

## Backward Compatibility

- **Vagrantfile**: Continues to work as before
- **Existing workflows**: Default behavior unchanged (JDK 17 on x64
architecture)
- **No breaking changes**: All existing configurations continue to work

---

 Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants