Git Conversion Lazarus
From Free Pascal wiki
Jump to navigationJump to search
This page has the detailed steps used to convert the Lazarus SVN repro to git.
SubGIT
The initial conversion will be done using TMate Subgit.
Install
Unpack subgit to
~/subgit/subgit-3.3.11
Required
An author.txt file
svn-name - git-name <email>
Run / main repro
cd ~/subgit/subgit-3.3.11 ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz.git gedit laz.git/subgit/config
And add the following (the first line, replaces an existing setting)
[svn]
trunk = trunk:refs/heads/main
gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
[translate]
eols = false
otherProperties = false
ignores = false
The translate options prevent .gitignore and .gitattributes. Those can be translated, but will later be replaced...
cp authors.txt laz.git/subgit/authors.txt
./bin/subgit install laz.git
Without skipping the gitignore/gitattributes, there will be an error at revision 47330. The process will hang. nterrupt it and edit
gedit lazarus.git/svn/.metadata
and set branches-maxRev and tags-maxRev to 47332 (it should be at 47330) https://stackoverflow.com/questions/18918215/skip-revisions-when-installing-subgit-repository Continue with
./bin/subgit install laz-bin.git
Stop the daemon
./bin/subgit shutdown laz-bin.git/
Run / binaries repro
./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz-bin.git
gedit laz-bin.git/subgit/config
[svn]
gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
trunk = binaries:refs/heads/main
cp authors.txt laz-bin.git/subgit/authors.txt
follow the steps above
Run / website repro
./bin/subgit configure https://svn.freepascal.org/svn/lazaruswebsite laz-web.git
gedit laz-web.git/subgit/config
[svn]
gitCommitMessage = %message\\n\\ngit-svn-id: %branch@%revision -\\n
trunk = trunk:refs/heads/main
cp authors.txt laz-web.git/subgit/authors.txt
follow the steps above
Part 2
- Updating gitignore
- Prefixing all issue references with #
- translating svn revisions to git hashes
Set a git user
git config --local user.email "lazarus@lazarus-ide.org" git config --local user.name "lazarus"
Clean out left overs from subgit
git for-each-ref --format="%(refname)" refs/svn/ | xargs -n 1 git update-ref -d
Clean left overs from binaries
There are a few tags, that lead to binaries (gdb.exe) ending up in the data
git tag -d binaries_0_9_26 git tag -d binaries_0_9_26_2 git tag -d binaries_0_9_28 git tag -d binaries_0_9_30 git tag -d binaries_0_9_30_2 git tag -d binaries_0_9_30_2RC1 git tag -d binaries_1_0
Updating gitignore/gitattributes / Prefix issues with #
The below combines the 2 steps. It can be run in 2 separate calls.
It will run for about one hour.
FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --index-filter ' cp ~/subgit/subgit-3.3.11/.gitignore ~/subgit/subgit-3.3.11/.gitattributes . ; git add .gitignore .gitattributes ' --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f -- --all
FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f -- --all
Cleanup
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d git reflog expire --expire=now --all ;git gc --prune=now
Translating references to svn rev (r12345) to #hash
This must be the very last filter-branch. As each filter-branch creates new hashes for the commits, and that would invalidate the hash refs created here.
- The first batch will update only commits it branch main.
- This will also only be references to other commits in main.
- Merges from branches to main can not be updated.
- The 2nd batch will update only commits in branches. This means the hashes for the main branch will be left intact.
- This will update merges from main to fixes
To verify the 2nd batch does not update the main branch compare the hash for git log -n1 main. It should not be changed by the 2nd batch.
rm -rf /tmp/rmap mkdir /tmp/rmap git rev-list main | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "git-svn-id: .*@[0-9][0-9]* " | sed "s/.*@\([0-9]*\) .*/\1/" ) ; echo $0 > /tmp/rmap/$a' git rev-list main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict;
unless ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print; next; }
my %f; foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
my \$rvl=\$1;
foreach my \$a (split / *[\/,] */, \$rvl) {
unless (\$f{\$a}){
\$f{\$a}=q/#/;
my(\$a1, \$a2) = split / *- */, \$a;
my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>; chomp \$c; }
my \$h; if(\$c && (open \$h, q/</, qq#../map/\$c#)) {
\$c=<\$h>; chomp \$c; \$c = q/#/.substr \$c, 0,10;
if (\$a2) {
my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>; chomp \$c2; }
my \$h; if(\$c2 && (open \$h, q/</, qq#../map/\$c2#)) { \$c2=<\$h>; chomp \$c2;
\$c = \$c.q/-#/.substr( \$c2, 0,10);
}
else { \$c = \$c.q/..# / unless \$c eq q/#/; warn q/MAP2 FAIL /.\$c2 }
}
\$f{\$a}=\$c;
}
else {warn q/MAP1 fail/.\$a;}
}}
}
foreach my \$k (keys %f) { my \$v = \$f{\$k};
unless (\$v eq q/#/) { s/([ r,\/]\${k}\b)/\$1 \$v/g; }
}
warn \$_ if keys %f;
print; "
' -f -- --all
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
rm -rf /tmp/rmap mkdir /tmp/rmap git rev-list --all | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "git-svn-id: .*@[0-9][0-9]* " | sed "s/.*@\([0-9]*\) .*/\1/" ) ; echo $0 > /tmp/rmap/$a' git rev-list main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict;
if ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print; next; }
my %f; foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
my \$rvl=\$1;
foreach my \$a (split / *[\/,] */, \$rvl) {
unless (\$f{\$a}){
\$f{\$a}=q/#/;
my(\$a1, \$a2) = split / *- */, \$a;
my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>; chomp \$c; }
if (\$c) { unless ( -e qq#/tmp/rmap/m_\$c# ) {
my \$h; if(open \$h, q/</, qq#../map/\$c#) { \$c=<\$h>; chomp \$c; }
}}
if(\$c ) {
\$c = q/#/.substr \$c, 0,10;
if (\$a2) {
my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>; chomp \$c2; }
if (\$c2) { unless ( -e qq#/tmp/rmap/m_\$c2# ) {
my \$h; if(open \$h, q/</, qq#../map/\$c2#) { \$c2=<\$h>; chomp \$c2; }
}}
if(\$c2) { \$c = \$c.q/-#/.substr( \$c2, 0,10); }
else { \$c = \$c.q/..# / unless \$c eq q/#/; warn q/MAP2 FAIL /.\$c2 }
}
\$f{\$a}=\$c;
}
else {warn q/MAP1 fail/.\$a;}
}}
}
foreach my \$k (keys %f) { my \$v = \$f{\$k};
unless (\$v eq q/#/) { s/([ r,\/]\${k}\b)/\$1 \$v/g; }
}
warn \$_ if keys %f;
print; "
' -f -- --all
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d git reflog expire --expire=now --all ;git gc --prune=now rm -rf /tmp/rmap
Updating tags
This must be done after all filter-branches
- Annotate all tags.
- Create some more tags
This needs the git user being set
git for-each-ref --format="%(refname:short)" refs/tags/ | xargs -L1 /bin/sh -c 'git tag -f -a $0 -m $0 $0 '
git rev-list --in-commit-order --reverse fixes_2_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_3" -m "main to version 2.3" $0' git rev-list --in-commit-order --reverse fixes_2_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_1" -m "main to version 2.1" $0' git rev-list --in-commit-order --reverse fixes_1_8..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_9" -m "main to version 1.9" $0' git rev-list --in-commit-order --reverse fixes_1_6..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_7" -m "main to version 1.7" $0' git rev-list --in-commit-order --reverse fixes_1_4..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_5" -m "main to version 1.5" $0' git rev-list --in-commit-order --reverse fixes_1_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_3" -m "main to version 1.3" $0' git rev-list --in-commit-order --reverse fixes_1_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_1" -m "main to version 1.1" $0' git rev-list --in-commit-order --reverse fixes_0_9_30..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_31" -m "main to version 0.9.31" $0' git rev-list --in-commit-order --reverse fixes_0_9_28..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_29" -m "main to version 0.9.29" $0' git rev-list --in-commit-order --reverse fixes_0_9_26..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_27" -m "main to version 0.9.27" $0' git rev-list --in-commit-order --reverse fixes_0_9_24..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_25" -m "main to version 0.9.25" $0' git rev-list --in-commit-order --reverse main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_0" -m "main init" $0' git rev-list --in-commit-order --reverse main..fixes_2_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-2_2" -m "fixes 2.2 branch" $0' git rev-list --in-commit-order --reverse main..fixes_2_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-2_0" -m "fixes 2.0 branch" $0' git rev-list --in-commit-order --reverse main..fixes_1_8 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_8" -m "fixes 1.8 branch" $0' git rev-list --in-commit-order --reverse main..fixes_1_6 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_6" -m "fixes 1.6 branch" $0' git rev-list --in-commit-order --reverse main..fixes_1_4 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_4" -m "fixes 1.4 branch" $0' git rev-list --in-commit-order --reverse main..fixes_1_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_2" -m "fixes 1.2 branch" $0' git rev-list --in-commit-order --reverse main..fixes_1_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-1_0" -m "fixes 1.0 branch" $0' git rev-list --in-commit-order --reverse main..fixes_0_9_30 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_30" -m "fixes 0.9.30 branch" $0' git rev-list --in-commit-order --reverse main..fixes_0_9_28 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_28" -m "fixes 0.9.28 branch" $0' git rev-list --in-commit-order --reverse main..fixes_0_9_26 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_26" -m "fixes 0.9.26 branch" $0' git rev-list --in-commit-order --reverse main..fixes_0_9_24 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-fixes-0_9_24" -m "fixes 0.9.24 branch" $0' git rev-list --in-commit-order --reverse main..free-sparta | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-free-sparta" -m "free-sparta branch" $0' git rev-list --in-commit-order --reverse main..gtk-splitup | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-gtk-splitup" -m "gtk-splitup branch" $0' git rev-list --in-commit-order --reverse main..lcl-smartlink | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-lcl-smartlink" -m "lcl-smartlink branch" $0' git rev-list --in-commit-order --reverse main..listviewsortindicator | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-listviewsortindicator" -m "listviewsortindicator branch" $0' git rev-list --in-commit-order --reverse main..macosfullkeyboardaccess | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "t-macosfullkeyboardaccess" -m "macosfullkeyboardaccess branch" $0'
Finish
edit HEAD to point to main
gedit HEAD
git remote add origin https://gitlab.com/freepascal.org/lazarus-ide/lazarus_test_conversion_X.git
git push --all origin git push --tags origin
- On gitlab
- Settings / Repository
- set default gitlab branch
- no autoclose issues on default branch // decide later - set "reject unverified users"
- protect branches * // no force push
- protect tags: fixes* lazarus* release* main* // maintainers only
- set default gitlab branch
- Settings / Integrations
- External wiki
- Settings / General
- Disable Wiki, Snippets, Papges, Requirements, Operations, Service Desk, Container, Packages
For now also CI, until we use it - Merge request approvals: Enable "Prevent users from modifying MR approval rules"
- set the avatar
- Add "description"
- Make public
- Disable "Users can request access"
- Disable Wiki, Snippets, Papges, Requirements, Operations, Service Desk, Container, Packages
Comparing data before/after filter-branch
With subject line / Will show changes in subject
git log --pretty='format:%ae %an %cd %s %d' --graph --all > ../log.txt
Without subject line / Should have no diff
- This compares the graph and committers => Thus it will show if all commits are present, and if all branches are correct
git log --pretty='format:%ae %an %cd %d' --graph --all > ../log.txt
- compare the changes to files (gitignore / gitattributes will diff for the top commit(s)) / --graph can be added too
- (slow)
git log --all --pretty='%ae %an %cd %d' --stat > ../statlog.txt
- Compare the overall amount of lines in the log.
git log --all | wc -l
Make sure you do that after "clean up" in each step.
Otherwise the internal subgit or filter-branch revs will go into the compare.
Also "removing the binaries" introduces a diff.
Data
Please mail me for any changes. Do not make changes here. Otherwise the changes will not make it into the conversion.
.gitignore
**/*.o **/*.ppu **/*.exe **/*.bak **/*.diff **/*.patch **/units/ **/lib/ **/*;[1-9] **/*.~* **/*.*~ **/*.fpcunit.ini **/*.lps **/*.compiled /docs/chm/**/*.chm /docs/chm/**/*.xct /docs/chm/**/*.kwd /docs/chm/**/*.txt /libeay32.dll /ssleay32.dll /lazarus.cfg ide/revision.inc !**/patches/*.patch !**/patches/*.diff
.gitattributes
* text=auto !eol **/*.pp text **/*.pas text **/*.inc text **/*.lfm text **/*.lrs text **/*.lps text **/*.lpr text **/*.lpi text **/*.lpk text **/*.lpl text **/*.lrj text **/*.dpr text **/*.dfm text **/*.pot text **/*.po text **/*.rc text **/*.txt text **/*.txt.sample text **/*.ini text **/*.cfg text **/*.iss text **/*.isl text **/*.xml text **/*.xsl text **/*.html text **/*.css text **/*.php text **/*.js text **/.htaccess text **/*.patch text **/*.md text **/*.bat text **/*.sh text **/*.compiled text **/*.bdsproj text **/*.plist text **/*.h text **/*.c text **/*.cpp text **/*.in text **/*.xpm text **/*.svg text **/Makefile text **/Makefile.compiled text **/Makefile.fpc text **/*.exe -text **/*.dll -text **/*.so -text **/*.gif -text **/*.jpg -text **/*.ico -text **/*.bmp -text **/*.res -text **/*.ttf -text **/*.cur -text **/*.db3 -text **/*.dbf -text **/*.mbf -text **/*.mdx -text **/*.fbk -text **/*.odt -text **/*.ods -text **/*.odg -text **/*.odp -text **/*.xls -text **/*.pdf -text **/*.zip -text **/*.rar -text **/*.tar -text **/*.tgz -text **/*.gz -text **/*.icns -text **/*.dcr -text components/aggpas/**/*.ppm -text **/*.app/Contents/MacOS/* -text
