From danfe at nsu.ru  Wed Dec  1 07:17:47 2010
From: danfe at nsu.ru (Alexey Dokuchaev)
Date: Wed, 1 Dec 2010 18:17:47 +0600
Subject: [mojosetup] Simple patches for handling hardlinks and GNU
	././@LongLink entries
Message-ID: <20101201121747.GA88745@regency.nsu.ru>

I found that MojoSsetup tar backend lacks support for hard links and
long names (GNU tar extension).  Attached as two quick patches I've come
up with.  Then can probably be improved, but seem to work.  Only tested
on Unix.

./danfe
-------------- next part --------------
Index: scripts/mojosetup_mainline.lua
===================================================================
--- scripts/mojosetup_mainline.lua	(revision 3289)
+++ scripts/mojosetup_mainline.lua	(revision 3290)
@@ -471,6 +471,18 @@
 end
 
 
+local function install_hardlink(dest, lndest, manifestkey)
+    if not MojoSetup.platform.hardlink(dest, lndest) then
+        MojoSetup.logerror("Failed to create hardlink '" .. dest .. "'")
+        MojoSetup.fatal(_("Hardlink creation failed!"))
+    end
+
+    -- !!! FIXME: no checksum will be recorded (is it needed?)
+    manifest_add(MojoSetup.manifest, dest, manifestkey, "file", perms, nil, nil)
+    MojoSetup.loginfo("Created hardlink '" .. dest .. "' -> '" .. lndest .. "'")
+end
+
+
 -- !!! FIXME: we should probably pump the GUI queue here, in case there are
 -- !!! FIXME:  thousands of dirs in a row or something.
 local function install_directory(dest, perms, manifestkey)
@@ -566,9 +578,11 @@
         install_directory(dest, perms, manifestkey)
     elseif ent.type == "symlink" then
         install_symlink(dest, ent.linkdest, manifestkey)
+    elseif ent.type == "hardlink" then
+        install_hardlink(dest, MojoSetup.destination .. "/" .. ent.linkdest, manifestkey)
     else  -- !!! FIXME: device nodes, etc...
         -- !!! FIXME: should this be fatal?
-        MojoSetup.fatal(_("Unknown file type in archive"))
+        MojoSetup.fatal(_("Unknown file type in archive") .. ": " .. ent.type)
     end
 end
 
Index: archive_tar.c
===================================================================
--- archive_tar.c	(revision 3289)
+++ archive_tar.c	(revision 3290)
@@ -262,9 +262,11 @@
         ar->prevEnum.type = MOJOARCHIVE_ENTRY_FILE;
     else if (type == TAR_TYPE_DIRECTORY)
         ar->prevEnum.type = MOJOARCHIVE_ENTRY_DIR;
-    else if (type == TAR_TYPE_SYMLINK)
+    else if (type == TAR_TYPE_SYMLINK || type == TAR_TYPE_HARDLINK)
     {
-        ar->prevEnum.type = MOJOARCHIVE_ENTRY_SYMLINK;
+        // ugly assignment of type below allows to unify both symlink and
+        // hardlink handling under one ``if (...) { ... }'' clause
+        ar->prevEnum.type = TAR_TYPE_SYMLINK - type + MOJOARCHIVE_ENTRY_SYMLINK;
         memcpy(scratch, &block[TAR_LINKNAME], TAR_LINKNAMELEN);
         scratch[TAR_LINKNAMELEN] = '\0';  // just in case.
         ar->prevEnum.linkdest = xstrdup((const char *) scratch);
Index: platform_unix.c
===================================================================
--- platform_unix.c	(revision 3289)
+++ platform_unix.c	(revision 3290)
@@ -565,6 +565,12 @@
 } // MojoPlatform_symlink
 
 
+boolean MojoPlatform_hardlink(const char *src, const char *dst)
+{
+    return (link(dst, src) == 0);
+} // MojoPlatform_hardlink
+
+
 boolean MojoPlatform_mkdir(const char *path, uint16 perms)
 {
     // !!! FIXME: error if already exists?
Index: platform.h
===================================================================
--- platform.h	(revision 3289)
+++ platform.h	(revision 3290)
@@ -49,6 +49,11 @@
 //  syscall! Returns true if link was created, false otherwise.
 boolean MojoPlatform_symlink(const char *src, const char *dst);
 
+// Create a hardlink in the physical filesystem. (src) is the NAME OF THE LINK
+//  and (dst) is WHAT IT POINTS TO. This is backwards from the unix link()
+//  syscall! Returns true if link was created, false otherwise.
+boolean MojoPlatform_hardlink(const char *src, const char *dst);
+
 // Read the destination from symlink (linkname). Returns a pointer
 //  allocated with xmalloc() containing the linkdest on success, and NULL
 //  on failure. The caller is responsible for freeing the returned pointer!
Index: fileio.h
===================================================================
--- fileio.h	(revision 3289)
+++ fileio.h	(revision 3290)
@@ -55,6 +55,7 @@
     MOJOARCHIVE_ENTRY_FILE,
     MOJOARCHIVE_ENTRY_DIR,
     MOJOARCHIVE_ENTRY_SYMLINK,
+    MOJOARCHIVE_ENTRY_HARDLINK,
 } MojoArchiveEntryType;
 
 // Abstract archive interface. Archives, directories, etc.
Index: lua_glue.c
===================================================================
--- lua_glue.c	(revision 3289)
+++ lua_glue.c	(revision 3290)
@@ -989,6 +989,8 @@
             typestr = "dir";
         else if (entinfo->type == MOJOARCHIVE_ENTRY_SYMLINK)
             typestr = "symlink";
+        else if (entinfo->type == MOJOARCHIVE_ENTRY_HARDLINK)
+            typestr = "hardlink";
         else
             typestr = "unknown";
 
@@ -1069,6 +1071,14 @@
 } // luahook_platform_symlink
 
 
+static int luahook_platform_hardlink(lua_State *L)
+{
+    const char *src = luaL_checkstring(L, 1);
+    const char *dst = luaL_checkstring(L, 2);
+    return retvalBoolean(L, MojoPlatform_hardlink(src, dst));
+} // luahook_platform_hardlink
+
+
 static int luahook_platform_mkdir(lua_State *L)
 {
     const int argc = lua_gettop(L);
@@ -1830,6 +1840,7 @@
             set_cfunc(luaState, luahook_platform_issymlink, "issymlink");
             set_cfunc(luaState, luahook_platform_isfile, "isfile");
             set_cfunc(luaState, luahook_platform_symlink, "symlink");
+            set_cfunc(luaState, luahook_platform_hardlink, "hardlink");
             set_cfunc(luaState, luahook_platform_mkdir, "mkdir");
             set_cfunc(luaState, luahook_platform_installdesktopmenuitem, "installdesktopmenuitem");
             set_cfunc(luaState, luahook_platform_uninstalldesktopmenuitem, "uninstalldesktopmenuitem");
-------------- next part --------------
Index: archive_tar.c
===================================================================
--- archive_tar.c	(revision 3299)
+++ archive_tar.c	(revision 3303)
@@ -157,6 +157,8 @@
 #define TAR_TYPE_BLOCKDEV '4'
 #define TAR_TYPE_DIRECTORY '5'
 #define TAR_TYPE_FIFO '6'
+#define TAR_TYPE_LONGLINK 'K'
+#define TAR_TYPE_LONGNAME 'L'
 
 static boolean is_ustar(const uint8 *block)
 {
@@ -199,9 +201,10 @@
     size_t fnamelen = 0;
     int type = 0;
 
+    MojoArchive_resetEntry(&ar->prevEnum);
+longlink:
     memset(scratch, '\0', sizeof (scratch));
 
-    MojoArchive_resetEntry(&ar->prevEnum);
     if (info->input != NULL)
         fatal("BUG: tar entry still open on new enumeration");
 
@@ -217,6 +220,7 @@
             return NULL;  // !!! FIXME: fatal() ?
         zeroes = (memcmp(block, scratch, sizeof (block)) == 0);
     } // while
+    zeroes = true;  // allow reentrancy for ././@LongLink handling
 
     // !!! FIXME We should probably check the checksum.
 
@@ -229,19 +233,25 @@
     if (ar->prevEnum.filesize % 512)
         info->nextEnumPos += 512 - (ar->prevEnum.filesize % 512);
 
-    // We count on (scratch) being zeroed out here!
-    // prefix of filename is at the end for legacy compat.
-    if (ustar)
-        memcpy(scratch, &block[TAR_FNAMEPRE], TAR_FNAMEPRELEN);
-    fnamelen = strlen((const char *) scratch);
-    memcpy(&scratch[fnamelen], &block[TAR_FNAME], TAR_FNAMELEN);
-    fnamelen += strlen((const char *) &scratch[fnamelen]);
-
+    // Non-zero fnamelen means we're dealing with long name, which should be
+    // already set on previous interation
     if (fnamelen == 0)
-        return NULL;   // corrupt file.  !!! FIXME: fatal() ?
+    {
+        // We count on (scratch) being zeroed out here!
+        // prefix of filename is at the end for legacy compat.
+        if (ustar)
+            memcpy(scratch, &block[TAR_FNAMEPRE], TAR_FNAMEPRELEN);
+        fnamelen = strlen((const char *) scratch);
+        memcpy(&scratch[fnamelen], &block[TAR_FNAME], TAR_FNAMELEN);
+        fnamelen += strlen((const char *) &scratch[fnamelen]);
 
-    ar->prevEnum.filename = xstrdup((const char *) scratch);
+        if (fnamelen == 0)
+            return NULL;   // corrupt file.  !!! FIXME: fatal() ?
+    }
 
+    if (ar->prevEnum.filename == NULL)
+        ar->prevEnum.filename = xstrdup((const char *) scratch);
+
     type = block[TAR_TYPE];
     if (type == 0)  // some archivers do the file type as 0 instead of '0'.
         type = TAR_TYPE_FILE;
@@ -269,7 +279,23 @@
         ar->prevEnum.type = TAR_TYPE_SYMLINK - type + MOJOARCHIVE_ENTRY_SYMLINK;
         memcpy(scratch, &block[TAR_LINKNAME], TAR_LINKNAMELEN);
         scratch[TAR_LINKNAMELEN] = '\0';  // just in case.
-        ar->prevEnum.linkdest = xstrdup((const char *) scratch);
+        if (ar->prevEnum.linkdest == NULL)
+            ar->prevEnum.linkdest = xstrdup((const char *) scratch);
+    }
+    else if (type == TAR_TYPE_LONGLINK || type == TAR_TYPE_LONGNAME)
+    {
+        // Such a header (with name `././@LongLink') is GNU way of storing
+        // file long file names (>=100 chars); contents (data) will be the
+        // name of the *next* file or link in the archive
+        if (!ar->io->seek(ar->io, info->curFileStart))
+            return NULL;
+        if (ar->io->read(ar->io, block, sizeof (block)) != sizeof (block))
+            return NULL;  // !!! FIXME: fatal() ?
+        if (type == TAR_TYPE_LONGNAME)
+            ar->prevEnum.filename = xstrdup((const char *) block);
+        else
+            ar->prevEnum.linkdest = xstrdup((const char *) block);
+        goto longlink;
     } // else if
 
     return &ar->prevEnum;

From danfe at nsu.ru  Thu Dec  2 01:37:03 2010
From: danfe at nsu.ru (Alexey Dokuchaev)
Date: Thu, 2 Dec 2010 12:37:03 +0600
Subject: [mojosetup] Simple patches for handling hardlinks and GNU
	././@LongLink entries
In-Reply-To: <20101201121747.GA88745@regency.nsu.ru>
References: <20101201121747.GA88745@regency.nsu.ru>
Message-ID: <20101202063703.GA52528@regency.nsu.ru>

On Wed, Dec 01, 2010 at 06:17:47PM +0600, Alexey Dokuchaev wrote:
>          -- !!! FIXME: should this be fatal?
> -        MojoSetup.fatal(_("Unknown file type in archive"))
> +        MojoSetup.fatal(_("Unknown file type in archive") .. ": " .. ent.type)

This part is probably superfluous and just a debuggging leftover: no
need to reiterate that unknown file type is, literally, "unknown".

./danfe

From fgouget at codeweavers.com  Mon Dec  6 13:09:13 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Mon, 06 Dec 2010 19:09:13 +0100
Subject: [mojosetup] Upgrade problems with MojoSetup
Message-ID: <4CFD26C9.10501@codeweavers.com>


I'm running into a couple of problems when upgrading an application with 
MojoSetup.

So let's say that application was first installed into /opt/myapp' by 
the MojoSetup installer: running install-myapp-1.0.bin
This first installation created a bunch of files and directories, including:
    /opt/myapp/bin
    /opt/myapp/bin/oldfile

Then version 2.0 of myapp comes out and the user upgrades simply by 
running install-myapp-2.0.bin.


The first problem is that obsolete files don't get deleted. So even 
'oldfile' does not exist anymore in myapp 2.0, after going through the 
above '/opt/myapp/bin/oldfile' will still exist.


The second problem is that after running install-myapp-2.0.bin, 
MojoSetup has completely forgotten that it created '/opt/myapp/bin' when 
version 1.0 was installed. So when you next uninstall it does not delete 
the directories.


Both problems happen because the manifest created when 1.0 was installed 
is ignored and overwritten by the 2.0 install. Ignoring it means we 
don't know which files are no longer relevant and should be deleted. And 
it also means that since we don't put existing directories into the 
manifest, the new manifest does not list those created by 1.0, so that 
they don't get uninstalled either.

So I'm tempted to change that but I would like to know what other people 
think. Surely I'm not the first one to hit this problem?

Is the standard solution to force an uninstall of the old version before 
installing the new one? If so shouldn't MojoSetup complain before 
overwriting the old manifest?


-- 
Fran?ois Gouget
fgouget at codeweavers.com


From icculus at icculus.org  Sun Dec 12 17:05:51 2010
From: icculus at icculus.org (Ryan C. Gordon)
Date: Sun, 12 Dec 2010 17:05:51 -0500
Subject: [mojosetup] Upgrade problems with MojoSetup
In-Reply-To: <4CFD26C9.10501@codeweavers.com>
References: <4CFD26C9.10501@codeweavers.com>
Message-ID: <4D05473F.6050606@icculus.org>


> Is the standard solution to force an uninstall of the old version before
> installing the new one? If so shouldn't MojoSetup complain before
> overwriting the old manifest?

It's not something we've hit before (all the MojoSetup installs I know 
of are either single shot, or use the .xml manifest with loki_update).

I think what you're describing (realizing there's an earlier version, 
and uninstalling it before installing the new one) is feasible, and 
probably not that hard. I'll take a look at doing this soon.

--ryan.


From icculus at icculus.org  Sun Dec 12 17:06:27 2010
From: icculus at icculus.org (Ryan C. Gordon)
Date: Sun, 12 Dec 2010 17:06:27 -0500
Subject: [mojosetup] Simple patches for handling hardlinks and
 GNU	././@LongLink entries
In-Reply-To: <20101201121747.GA88745@regency.nsu.ru>
References: <20101201121747.GA88745@regency.nsu.ru>
Message-ID: <4D054763.8040402@icculus.org>


> I found that MojoSsetup tar backend lacks support for hard links and
> long names (GNU tar extension).  Attached as two quick patches I've come
> up with.  Then can probably be improved, but seem to work.  Only tested
> on Unix.

I'll take a look at this soon, too.

--ryan.



From fgouget at codeweavers.com  Wed Dec 15 11:40:00 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Wed, 15 Dec 2010 17:40:00 +0100
Subject: [mojosetup] Upgrade problems with MojoSetup
In-Reply-To: <4D05473F.6050606@icculus.org>
References: <4CFD26C9.10501@codeweavers.com> <4D05473F.6050606@icculus.org>
Message-ID: <4D08EF60.1080200@codeweavers.com>

Le 12/12/2010 23:05, Ryan C. Gordon a ?crit :
>
>> Is the standard solution to force an uninstall of the old version before
>> installing the new one? If so shouldn't MojoSetup complain before
>> overwriting the old manifest?
>
> It's not something we've hit before (all the MojoSetup installs I know
> of are either single shot, or use the .xml manifest with loki_update).

I believe we used loki_update in the early days of Loki. However, iirc, 
that requires having a 'first time' installer and an 'upgrade installer' 
which is a usability issue (people don't know which one to get) and 
twice the number of installers to test.


> I think what you're describing (realizing there's an earlier version,
> and uninstalling it before installing the new one) is feasible, and
> probably not that hard. I'll take a look at doing this soon.

The problem I see with doing a regular uninstall (which I picture as 
transparently invoking 'mojosetup --uninstall' early in the process) is 
that it means losing the 'rollback' capability. In fact it means never 
doing upgrades.

I have identified three issues that I believe need fixing for smooth 
upgrades:
  * Correctly remove directories on uninstall after an upgrade, which 
means not losing track of them in the manifest on upgrades.
  * Remove obsolete files on upgrade; that is files installed by the 
previous version which are not present in the new one.
  * Not ask for permission to overwrite files installed by the previous 
version: this requires the user to either answer hundreds of questions 
or give a blanket Yes/No, both of which defeat the purpose of this 
question. The only files we should ask about are those we (including our 
previous incarnations) did not create. This can essentially be worked 
around by setting 'allowoverwrite = true' in config.lua, though it again 
defeats the purpose of this part of the code.

All three issues come about because we don't read the old manifest on 
upgrades so we don't know what we did before. Then it's a small matter 
of tweaking the code to take into account the extra knowledge this gains 
us (that's the part I'm wrestling with now).

The attached file contains the code I'm using to read the old manifest. 
I couldn't get load_manifest() to do my bidding so I rolled my own in 
pure Lua. I like it because it should be robust in the face of broken 
manifest files. But maybe there's a downside to that approach.


-- 
Fran?ois Gouget
fgouget at codeweavers.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_oldpackage.lua
Type: text/x-lua
Size: 804 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101215/47427da5/attachment.bin>

From fgouget at codeweavers.com  Thu Dec 16 11:14:21 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Thu, 16 Dec 2010 17:14:21 +0100
Subject: [mojosetup] Upgrade problems with MojoSetup
In-Reply-To: <4D08EF60.1080200@codeweavers.com>
References: <4CFD26C9.10501@codeweavers.com> <4D05473F.6050606@icculus.org>
	<4D08EF60.1080200@codeweavers.com>
Message-ID: <4D0A3ADD.4090806@codeweavers.com>

Le 15/12/2010 17:40, Francois Gouget a ?crit :
[...]
> * Correctly remove directories on uninstall after an upgrade, which
> means not losing track of them in the manifest on upgrades.
> * Remove obsolete files on upgrade; that is files installed by the
> previous version which are not present in the new one.

I attached a pair of patches that solve these two issues.
  * mojo-1-upgrade-obsolete.diff
    Delete obsolete files and directories on upgrade.

    We try to be careful when loading the old Lua manifest so that a
    corrupted file does not crash the installer and does not clobber our
    state information.

  * mojo-2-upgrade-dirs.diff
    Don't forget directories created by older versions on upgrade.

    Otherwise they won't be deleted on uninstall which will result in an
    unclean uninstallation.


The first patch also contains the code for reading the old manifest file 
so it's needed by the second one.
Let me know if the approach seems sound to you.

-- 
Fran?ois Gouget
fgouget at codeweavers.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mojo-1-upgrade-obsolete.diff
Type: text/x-patch
Size: 2299 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101216/33dce5c0/attachment.bin>

From fgouget at codeweavers.com  Thu Dec 16 11:22:24 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Thu, 16 Dec 2010 17:22:24 +0100
Subject: [mojosetup] Open the log file in append mode.
Message-ID: <4D0A3CC0.4050208@codeweavers.com>


My installer invokes some external shell scripts that output their log 
information to the same file as MojoSetup. But without append mode, 
anything MojoSetup writes after invoking the scripts ends up in the 
middle of the file.


-- 
Fran?ois Gouget
fgouget at codeweavers.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mojo-logappend.diff
Type: text/x-patch
Size: 791 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101216/1470c94b/attachment.bin>

From fgouget at codeweavers.com  Thu Dec 16 11:27:52 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Thu, 16 Dec 2010 17:27:52 +0100
Subject: [mojosetup] Warn about rollback errors only once.
Message-ID: <4D0A3E08.6070400@codeweavers.com>


Of course we should not get rollback errors... (unless some directory 
disappeared for instance). But if we do the error message is the same 
for every file so after the first dozen or so the user would normally 
assume that the program got into an infinite loop and try to kill it 
abruptly.

So since the error just reports about not being able to restore 'some 
files' anyway it may as well be shown only once.

Another option would be to count how many files we could not restore and 
just show that summary at the end. However I thought having the count is 
of dubious value and that being warned of trouble early was more valuable.

-- 
Fran?ois Gouget
fgouget at codeweavers.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mojo-rollback.diff
Type: text/x-patch
Size: 1246 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101216/cbf4eb00/attachment.bin>

From fgouget at codeweavers.com  Thu Dec 16 19:16:54 2010
From: fgouget at codeweavers.com (Francois Gouget)
Date: Fri, 17 Dec 2010 01:16:54 +0100
Subject: [mojosetup] Upgrade problems with MojoSetup
In-Reply-To: <4D0A3ADD.4090806@codeweavers.com>
References: <4CFD26C9.10501@codeweavers.com> <4D05473F.6050606@icculus.org>
	<4D08EF60.1080200@codeweavers.com>
	<4D0A3ADD.4090806@codeweavers.com>
Message-ID: <4D0AABF6.2060402@codeweavers.com>

Le 16/12/2010 17:14, Francois Gouget a ?crit :
[...]
> I attached a pair of patches that solve these two issues.
> * mojo-1-upgrade-obsolete.diff
> Delete obsolete files and directories on upgrade.
>
> We try to be careful when loading the old Lua manifest so that a
> corrupted file does not crash the installer and does not clobber our
> state information.
>
> * mojo-2-upgrade-dirs.diff
> Don't forget directories created by older versions on upgrade.
>
> Otherwise they won't be deleted on uninstall which will result in an
> unclean uninstallation.

Some notes about these patches:
* I opted to keep the content of the old manifest around and have the
   oldfiles entries point to it, in case it was needed, potentially by
   the application's lua scripts. I have not found a use for that though
   so it may be better to just treat oldfiles as a set to minimize
   memory usage.

* I forgot to reset oldpackage and oldfiles to nil once we're done.

* The patch treats a failure to read the old manifest as non-fatal.
   I don't know if that's the right thing to do or not (as long as we
   don't fail when the file does not exist).

* I'll use 'hg export' next time.


I'm also attaching a couple more patches that build on this to fix the 
remaining issues for your review and to have a more complete set.


-- 
Fran?ois Gouget
fgouget at codeweavers.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mojo-3-upgrade-overwrite.diff
Type: text/x-patch
Size: 832 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101217/92b14a8a/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mojo-4-upgrade-uninstaller.diff
Type: text/x-patch
Size: 2388 bytes
Desc: not available
URL: <http://icculus.org/pipermail/mojosetup/attachments/20101217/92b14a8a/attachment-0001.bin>

