Overwrite does not apply -> let's just overwrite always

[andrew-glenn]

Describe the problem
When performing a copier update on a fresh repository, copier prompts to overwrite files, but doesn't actually overwrite them.

Template

https://github.com/andrew-glenn/example-copier-template

To Reproduce

❯ mkdir s32
❯ cd s32
❯ copier copy gh:andrew-glenn/example-copier-template $(pwd)
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+9be7fdc
 identical  .
    create  example-file.txt
    create  LICENSE
    create  .copier-answers.yml


❯ date > example-file.txt
❯ git init . && git add -A . && git commit -m1

Initialized empty Git repository in /private/tmp/s32/.git/
[main (root-commit) 53f6c0b] 1
 3 files changed, 206 insertions(+)
 create mode 100644 .copier-answers.yml
 create mode 100644 LICENSE
 create mode 100644 example-file.txt

❯ copier update
No git tags found in template; using HEAD as ref
Updating to template version 0.0.0.post1.dev0+9be7fdc
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+9be7fdc
 identical  .
  conflict  example-file.txt
 Overwrite example-file.txt? [Y/n] Y
 identical  LICENSE
 identical  .copier-answers.yml


❯ cat example-file.txt
Fri Aug 12 14:45:44 CDT 2022

Expected behavior
Copier overwrites the file it prompts me for.

Environment

• OS: OSX 12.5
• Copier version: copier 6.1.0
• Python version: Python 3.9.10
• Installation method: pypi

Additional context
I wrapped git to log the command it was performing. The following logs correspond to this code-block. I've truncated the path of the tempfiles for readability.

Code:

copier/copier/main.py

Lines 697 to 708 in 141ddf1

	git("init", retcode=None)
	git("add", ".")
	git("config", "user.name", "Copier")
	git("config", "user.email", "copier@copier")
	# 1st commit could fail if any pre-commit hook reformats code
	git("commit", "--allow-empty", "-am", "dumb commit 1", retcode=None)
	git("commit", "--allow-empty", "-am", "dumb commit 2")
	git("config", "--unset", "user.name")
	git("config", "--unset", "user.email")
	git("remote", "add", "real_dst", "file://" + subproject_top)
	git("fetch", "--depth=1", "real_dst", "HEAD")
	diff_cmd = git["diff-tree", "--unified=1", "HEAD...FETCH_HEAD"]

Logs:

git -C (...)/copier.main.update_diff.2od63kww init
git -C (...)/copier.main.update_diff.2od63kww add .
git -C (...)/copier.main.update_diff.2od63kww config user.name Copier
git -C (...)/copier.main.update_diff.2od63kww config user.email copier@copier
git -C (...)/copier.main.update_diff.2od63kww commit --allow-empty -am dumb commit 1
git -C (...)/copier.main.update_diff.2od63kww commit --allow-empty -am dumb commit 2
git -C (...)/copier.main.update_diff.2od63kww config --unset user.name
git -C (...)/copier.main.update_diff.2od63kww config --unset user.email
git -C (...)/copier.main.update_diff.2od63kww remote add real_dst file:///private/tmp/s32
git -C (...)/copier.main.update_diff.2od63kww fetch --depth=1 real_dst HEAD
git -C (...)/copier.main.update_diff.2od63kww diff-tree --unified=1 HEAD...FETCH_HEAD --inter-hunk-context=-1

Running the git diff at the end shows that HEAD is the upstream template, and FETCH_HEAD is the local project.

❯ git -C (...)/copier.main.update_diff.2od63kww diff-tree --unified=1 HEAD...FETCH_HEAD --inter-hunk-context=-1
diff --git a/example-file.txt b/example-file.txt
index 257cc56..59cd8b3 100644
--- a/example-file.txt
+++ b/example-file.txt
@@ -1 +1 @@
-foo
+Fri Aug 12 14:45:44 CDT 2022

In the case of an overwrite, I believe the order should be reversed, so that HEAD overwrites FETCH_HEAD, not maintaining the status-quo, and negating the affirmation to overwrite the file from upstream template.

Am I missing something?

[andrew-glenn]

Also - I'm more than happy to submit a PR if it's determined this is not the intended behavior. Just looking to brainstorm and verify I'm not missing anything here, as well as determine a path forward.

[yajo]

Hi! Thanks for the report and sorry for taking so long to answer. Following your steps I managed to reproduce the problem.

First of all, we have to keep in mind Copier is behaving "correctly". I mean that it is supposed to respect your changes in example-file.txt.

The smart update process is "smart" because it will only update something if it has changed in the template. Otherwise, Copier understands that you changed that file in your project because you had to, and live happy with that.

So, if your expectations are that Copier should put foo into example-file.txt, then please read carefully how the update process works. To achieve that, you should recopy the project instead of updating:

copier copy gh:andrew-glenn/example-copier-template .

However, there's a real bug: Copier shouldn't ask you to overwrite something that it's not going to overwrite.

The problem comes from a design dated before the smart update system was introduced in Copier 3.

This fix is to drop this "overwrite file?" form, as explained in #343 (comment). Basically asking the user is useless in updates, because of 2 reasons:

1. It's buggy (you saw this).
2. Git is better deciding what changes to keep or discard than that form.

Also it's useless in copies because you're supposed to be copying something for the 1st time, so you should want everything overwritten.

So copier should overwrite always, period.

The tasks to do would be something around this:

1. Remove the "overwrite" option, as it is unnecessary. Some code to remove would probably be:

   copier/copier/main.py

   Line 145 in fc6bb03

   overwrite: bool = False

   copier/copier/main.py

   Lines 223 to 254 in fc6bb03

   	def _solve_render_conflict(self, dst_relpath: Path):
   	"""Properly solve render conflicts.
   	It can ask the user if running in interactive mode.
   	"""
   	assert not dst_relpath.is_absolute()
   	printf(
   	"conflict",
   	dst_relpath,
   	style=Style.DANGER,
   	quiet=self.quiet,
   	file_=sys.stderr,
   	)
   	if self.match_skip(dst_relpath):
   	printf(
   	"skip",
   	dst_relpath,
   	style=Style.OK,
   	quiet=self.quiet,
   	file_=sys.stderr,
   	)
   	return False
   	if self.overwrite or dst_relpath == self.answers_relpath:
   	printf(
   	"overwrite",
   	dst_relpath,
   	style=Style.WARNING,
   	quiet=self.quiet,
   	file_=sys.stderr,
   	)
   	return True
   	return bool(ask(f" Overwrite {dst_relpath}?", default=True))

   copier/copier/cli.py

   Lines 163 to 166 in fc6bb03

   	overwrite: cli.Flag = cli.Flag(
   	["-w", "--overwrite"],
   	help="Overwrite files that already exist, without asking.",
   	)

2. Change lots of tests 😆

I'm more than happy to submit a PR

That'd be great! But let me advice you this is not easy task, as this feature is used very often. Do you still want to do it?

[rousseldenis]

@yajo What's the status of this as I've the same problem for OCA repos?

[yajo]

For now, nothing changed apart from what I already explained in #741 (comment).

The important part is that Copier is behaving correctly. It's just a UX problem, and the plan is to remove the overwrite question and just overwrite always. After all, updates are only allowed in git-tracked projects, and you can always review the diff afterwards with git, which is more comfortable and powerful than anything else I could possibly do.

As a workaround, you can pass --overwrite in CLI, to avoid the UX problem.