[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update by richard67 · Pull Request #21 · richard67/joomla-cms

richard67 · 2021-12-04T20:14:06Z

Pull Request for Issue # .

Replaces #20 .

Summary of Changes

This pull request (PR) is just to document the changed script "build/deleted_file_check.php" which I am currently using to generate the lists of deleted files and folders and renamed files for updating script.php.

The changes in the script are:

Not create the result files but append the results of the current comparison to the particular files if they aren't already included in that file from previous comparisons.
This allows to execute a sequence of comparisons, see description below on how I use the changed script with a shell script to completely rebuild the lists.
When a file or a folder once had been removed so it is included in results of previous comparisons and now with the current "to" version is added back, it is removed from the previous results.
For an example see [4.0] Remove _com_joomlaupdate.scss from deleted files in script.php joomla/joomla-cms#36161 .
Prepend each comparison with a comment which contains the from and to version.
The version numbers are read from the manifest XML files "administrator/manifests/files/joomla.xml" of the "from" and "to" directories.
Fix help doc and comments to reflect past changes in the 4.0-dev branch.

Testing Instructions

Currently I use following Bash script to generate the lists for the 4.0-dev branch:

#!/usr/bin/env bash

# Function for download and unpack of a full package if not done before
function fetchZipDownload {
	if [ ! -d ./tmp/deleted_files_check/$1 ]
	then
		if [ ! -f ./tmp/deleted_files_check/${1}.zip ]
		then
			curl -L -o ./tmp/deleted_files_check/${1}.zip $2
		fi
		if [ -f ./tmp/deleted_files_check/${1}.zip ]
		then
			mkdir -p ./tmp/deleted_files_check/$1
			unzip ./tmp/deleted_files_check/${1}.zip -d ./tmp/deleted_files_check/$1
		fi
	fi
}

BASEDIR=$(dirname "$(realpath $0)")
cd $BASEDIR

# Build current version if there is no result present from a previous build
if [ ! -d ./tmp/deleted_files_check/current ]
then
	if [ ! -f ./tmp/packages/*Full_Package.zip ]
	then
		php ./build.php --remote=HEAD --exclude-gzip --exclude-bzip2
	fi
	if [ -f ./tmp/packages/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/current
		unzip `ls -1 ./tmp/packages/*Full_Package.zip` -d ./tmp/deleted_files_check/current
	fi
fi

# Clone and build previous major version if there is no result present from a previous build
if [ ! -d ./tmp/deleted_files_check/previous ]
then
	if [ ! -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/previous-package
		mkdir -p ./tmp/deleted_files_check/previous-build
		cd ./tmp/deleted_files_check/previous-build
		git clone -b 3.10-dev https://github.com/joomla/joomla-cms.git .
		php ./build/build.php --remote=HEAD --exclude-gzip --exclude-bzip2
		cd $BASEDIR
		if [ -f ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ]
		then
			cp ./tmp/deleted_files_check/previous-build/build/tmp/packages/*Full_Package.zip ./tmp/deleted_files_check/previous-package/
		fi
	fi
	if [ -f ./tmp/deleted_files_check/previous-package/*Full_Package.zip ]
	then
		mkdir -p ./tmp/deleted_files_check/previous
		unzip `ls -1 ./tmp/deleted_files_check/previous-package/*Full_Package.zip` -d ./tmp/deleted_files_check/previous
	fi
fi

rm -rf ./tmp/deleted_files_check/previous-build

# Fetch and unpack zip downloads of the current major version's past releases if not present from a previous run
fetchZipDownload 4.0.0-beta1 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta/Joomla_4.0.0-beta1-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta2 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta2/Joomla_4.0.0-beta2-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta3 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta3/Joomla_4.0.0-beta3-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta4 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta4/Joomla_4.0.0-beta4-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta5 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta5/Joomla_4.0.0-beta5-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta6 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta6/Joomla_4.0.0-beta6-Beta-Full_Package.zip
fetchZipDownload 4.0.0-beta7 https://github.com/joomla/joomla-cms/releases/download/4.0.0-beta7/Joomla_4.0.0-beta7-Beta-Full_Package.zip
fetchZipDownload 4.0.0-rc1   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc1/Joomla_4.0.0-rc1-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc2   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc2/Joomla_4.0.0-rc2-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc3   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc3/Joomla_4.0.0-rc3-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc4   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc4/Joomla_4.0.0-rc4-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc5   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc5/Joomla_4.0.0-rc5-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0-rc6   https://github.com/joomla/joomla-cms/releases/download/4.0.0-rc6/Joomla_4.0.0-rc6-Release_Candidate-Full_Package.zip
fetchZipDownload 4.0.0       https://github.com/joomla/joomla-cms/releases/download/4.0.0/Joomla_4.0.0-Stable-Full_Package.zip
fetchZipDownload 4.0.1       https://github.com/joomla/joomla-cms/releases/download/4.0.1/Joomla_4.0.1-Stable-Full_Package.zip
fetchZipDownload 4.0.2       https://github.com/joomla/joomla-cms/releases/download/4.0.2/Joomla_4.0.2-Stable-Full_Package.zip
fetchZipDownload 4.0.3       https://github.com/joomla/joomla-cms/releases/download/4.0.3/Joomla_4.0.3-Stable-Full_Package.zip
fetchZipDownload 4.0.4       https://github.com/joomla/joomla-cms/releases/download/4.0.4/Joomla_4.0.4-Stable-Full_Package.zip

# Start with comparing the previous major release build with the current version's build.
rm -f ./deleted_files.txt
rm -f ./deleted_folders.txt
rm -f ./renamed_files.txt
php ./deleted_file_check.php --from=./tmp/deleted_files_check/previous --to=./tmp/deleted_files_check/current

# Compare past beta or RC or stable releases of the current version from old to new
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta1 --to=./tmp/deleted_files_check/4.0.0-beta2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta2 --to=./tmp/deleted_files_check/4.0.0-beta3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta3 --to=./tmp/deleted_files_check/4.0.0-beta4
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta4 --to=./tmp/deleted_files_check/4.0.0-beta5
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta5 --to=./tmp/deleted_files_check/4.0.0-beta6
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta6 --to=./tmp/deleted_files_check/4.0.0-beta7
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-beta7 --to=./tmp/deleted_files_check/4.0.0-rc1
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc1   --to=./tmp/deleted_files_check/4.0.0-rc2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc2   --to=./tmp/deleted_files_check/4.0.0-rc3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc3   --to=./tmp/deleted_files_check/4.0.0-rc4
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc4   --to=./tmp/deleted_files_check/4.0.0-rc5
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc5   --to=./tmp/deleted_files_check/4.0.0-rc6
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0-rc6   --to=./tmp/deleted_files_check/4.0.0
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.0       --to=./tmp/deleted_files_check/4.0.1
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.1       --to=./tmp/deleted_files_check/4.0.2
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.2       --to=./tmp/deleted_files_check/4.0.3
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.3       --to=./tmp/deleted_files_check/4.0.4

# Finish with comarison between the latest release and the current build
php ./deleted_file_check.php --from=./tmp/deleted_files_check/4.0.4       --to=./tmp/deleted_files_check/current

When keeping the intermediate results from comparing older 4.0 versions since Beta 1, it only needs to do the first comparison from current 3.10-dev to current 4.0-dev and then the last 2 comparisons. The 2nd last because something might have been forgotten in script.php when the last release was made, and the last one for the last release and the current branch.

At the end I can take the complete content of each file, intend it by 3 tabs and copy it into script PHP and can see the changes in Git.

Then it needs only 3 manual steps:

Add back the intermediate 4.0.0 update SQL scripts which I had removed between 2 beta versions when combining the update SQL script, so the comparisons don't know about them.
Move deletion of folder "/libraries/vendor/paragonie/random_compat" from section "// From 3.10 to 4.0" to section "// 4.0 from Beta 7 to RC 1". The reason is that we have different child folders of that being deleted when updating from 3.10 or when updating from 4.0 Beta 7, so we get the folder 2 times, and the duplicate check added in this PR here will add it do the 1st and not to the 2nd section, but in this case it should be in the 2nd.
Remove section comments for empty sections

This can be automated later with further improvements, but the question now is if what I am doing here is the right way to go.

For 4.1-dev I run the following shell script after having run the previously mentioned one for 4.0-dev:

#!/usr/bin/env bash

# Copy results from 4.0-dev
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_files.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/deleted_folders.txt ./
cp -f /home/richard/lamp/public_html/joomla-cms-4.0-dev/build/renamed_files.txt ./

# From 4.0-dev to 4.1-dev - ignore results from previous comparisons
php ./deleted_file_check.php \
    --from=/home/richard/lamp/public_html/4.0-compare-releases/Joomla_4.0.5-rc5-dev-Development-Full_Package \
    --to=/home/richard/lamp/public_html/4.1-compare-releases/Joomla_4.1.0-dev-Development-Full_Package

Actual result BEFORE applying this Pull Request

I have to identify and remove duplicates myself from the right section of the deleted files and folder lists in script.php.

I have to remove files or folders from the lists which once had been removed and later been added back between releases.

Expected result AFTER applying this Pull Request

Files and folders which have been identified to be removed when updating from the previous major (or minor) release on another branch, e.g. in case of J4 from 3.10, are not to be removed again when updating from a previous version of the current release to be checked.

When some file or folder is removed between 2 versions and later added back with another, later version, the tool is able to handle that yet and remove them from the lists.

Documentation Changes Required

There is no documentation yet on how to use deleted_file_check.php.

Remove files and folders from the lists if they once were removed and later were added back.

richard67 · 2021-12-18T13:44:51Z

Closing in favour of #22 .

* Use the new option names * Update KnownButtons.php * Update ToolbarPresets.php

Recipe Plugin

…g-and-cleanup Call cleanup method and trigger actionlog

… tag name instead of the tag id (joomla#46174) * update sql * [CMS PR 46174] Update the update SQL scripts and for new installations supports.sql (#21) * Names quotes for column names * Remove some tabs * Use separate lines for closing brackets * Fix SQL error "operator does not exist" * Append only once to array * Update supports.sql for new installations

richard67 mentioned this pull request Dec 4, 2021

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #20

Closed

richard67 force-pushed the 4.0-dev-deleted-files-and-folders-improvements-3 branch from f165cc6 to 52de98c Compare December 9, 2021 11:03

richard67 added 8 commits December 18, 2021 12:58

Let deleted files check append and not replace results

38c9d6d

Append only if some results

e309376

Report both versions at the end

df3c17a

Fix "to" parameter and fix wrong help and comments

989155b

One more code comment fix

28cd8d1

Prepend comment with versions from manifest XML files

c627d7e

Handle files and folders which have been added back

91905b8

Remove files and folders from the lists if they once were removed and later were added back.

Update documentation and fix some comments

9c35e19

richard67 force-pushed the 4.0-dev-deleted-files-and-folders-improvements-3 branch from 91ce94e to 9c35e19 Compare December 18, 2021 11:58

richard67 mentioned this pull request Dec 18, 2021

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update #22

Closed

richard67 closed this Dec 18, 2021

richard67 pushed a commit that referenced this pull request Oct 21, 2022

Use the new option names (#21)

0ce9557

* Use the new option names * Update KnownButtons.php * Update ToolbarPresets.php

richard67 pushed a commit that referenced this pull request Jul 14, 2023

Merge pull request #21 from joomla-projects/recipe

9c5260f

Recipe Plugin

richard67 pushed a commit that referenced this pull request Mar 16, 2025

Merge pull request #21 from joomla-projects/feature/trigger-aciton-lo…

3d04988

…g-and-cleanup Call cleanup method and trigger actionlog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update#21

[4.0] Improve the generator tool for lists of files and folders to be removed or renamed on update#21
richard67 wants to merge 8 commits into4.0-devfrom
4.0-dev-deleted-files-and-folders-improvements-3

richard67 commented Dec 4, 2021 •

edited

Loading

Uh oh!

richard67 commented Dec 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

richard67 commented Dec 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Testing Instructions

Actual result BEFORE applying this Pull Request

Expected result AFTER applying this Pull Request

Documentation Changes Required

Uh oh!

richard67 commented Dec 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

richard67 commented Dec 4, 2021 •

edited

Loading