Cross-platform packaging and deployment with par
This is a tutorial on \s-1PAR\s0, first appeared at the 7th Perl Conference. The \s-1HTML\s0 version of this tutorial is available online as <http://search.cpan.org/perldoc?PAR::Tutorial>
% sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101" Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1. BEGIN failed--compilation aborted at sshnuke.pl line 1.
Q: \*(L"Help! I can't run your program!\*(R"
A1: Install Perl & \*(C`perl -MCPAN -e'install(...)'\*(C'
How do we know which modules are needed?
New versions of \s-1CPAN\s0 modules may break \*(C`sshnuke.pl\*(C'
A2: Install Perl & \*(C`tar zxf my_perllib.tgz\*(C'
Possibly overwriting existing modules; not cross-platform at all
A3: Use the executable generated by \*(C`perlcc sshnuke.pl\*(C'
Impossible to debug; \*(C`perlcc\*(C' usually does not work anyway
Do what \s-1JAR\s0 (Java Archive) does for Perl
Aggregates modules, scripts and other files into a Zip file
Easy to generate, update and extract
Version consistency: solves forward-compatibility problems
Developed by community: \*(C`[email protected]\*(C'
\s-1PAR\s0 files can be packed into self-contained scripts
Automatically scans perl script for dependencies
Bundles all necessary 3rd-party modules with it
Requires only core Perl to run on the target machine
\s-1PAR\s0 also comes with \*(C`pp\*(C', the Perl Packager: % pp -o sshnuke.exe sshnuke.pl # stand-alone executable!
\s-1PAR\s0 files are just Zip files with modules in it
Any Zip tools can generate them: % zip foo.par Hello.pm World.pm # pack two modules % zip -r bar.par lib/ # grab all modules in lib/
To load modules from \s-1PAR\s0 files: use PAR; use lib "foo.par"; # the .par part is optional use Hello;
This also works: use PAR "/home/mylibs/*.par"; # put all of them into @INC use Hello;
Use \*(C`par.pl\*(C' to run files inside a \s-1PAR\s0 archive: % par.pl foo.par # looks for 'main.pl' by default % par.pl foo.par test.pl # runs script/test.pl in foo.par
Same thing, with the stand-alone \*(C`parl\*(C' or \*(C`parl.exe\*(C': % parl foo.par # no perl or PAR.pm needed! % parl foo.par test.pl # ditto
The \s-1PAR\s0 loader can prepend itself to a \s-1PAR\s0 file:
\*(C`-b\*(C' bundles non-core modules needed by \*(C`PAR.pm\*(C': % par.pl -b -O./foo.pl foo.par # self-contained script
\*(C`-B\*(C' bundles core modules in addition to \*(C`-b\*(C': % parl -B -O./foo.exe foo.par # self-contained binary
Recursively scan dependencies with \*(C`scandeps.pl\*(C': % scandeps.pl sshnuke.pl # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN 'Crypt::SSLeay' => '0', # X # 'Net::HTTP' => '0', # # 'Crypt::SSLeay::X509' => '0', # S # Crypt::SSLeay 'Net::HTTP::Methods' => '0', # S # Net::HTTP 'Compress::Zlib' => '0', # X # Net::HTTP::Methods
Scan an one-liner, list all involved files: % scandeps.pl -V -e "use Dynaloader;" ... # auto/DynaLoader/dl_findfile.al [autoload] # auto/DynaLoader/extralibs.ld [autoload] # auto/File/Glob/Glob.bs [data] # auto/File/Glob/Glob.so [shared] ...
Combines scanning, zipping and loader-embedding: % pp -o out.exe src.pl # self-contained .exe % out.exe # runs anywhere on the same OS
Bundle additional modules: % pp -o out.exe -M CGI src.pl # pack CGI + its dependencies, too
Pack one-liners: % pp -o out.exe -e 'print "Hi!"' # turns one-liner into executable
Generate \s-1PAR\s0 files instead of executables: % pp -p src.pl # makes 'source.par' % pp -B -p src.pl # include core modules
Command-line options are almost identical to \*(C`perlcc\*(C''s
Also supports \*(C`gcc\*(C'-style long options: % pp --gui --verbose --output=out.exe src.pl
Small initial overhead; no runtime overhead
Dependencies are POD-stripped before packing
Loads modules directly into memory on demand
Shared libraries (DLLs) are extracted with File::Temp
Works on Perl 5.6.0 or above
Tested on Win32 (\s-1VC++\s0 and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, \s-1AIX\s0, Solaris, HP-UX, Tru64...
A common question: > I have used pp to make several standalone applications which work > great, the only problem is that for each executable that I make, I am > assuming the parl.exe is somehow bundled into the resulting exe.
The obvious workaround: You can ship parl.exe by itself, along with .par files built by "pp -p", and run those PAR files by associating them to parl.exe.
On platforms that have \*(C`ln\*(C', there is a better solution: % pp --output=a.out a.pl b.pl # two scripts in one! % ln a.out b.out # symlink also works % ./a.out # runs a.pl % ./b.out # runs b.pl
Of course, there is no cross-platform binary format
Pure-perl \s-1PAR\s0 packages are cross-platform by default
However, \s-1XS\s0 modules are specific to Perl version and platform
Multiple versions of a \s-1XS\s0 module can co-exist in a \s-1PAR\s0 file
Suppose we need \*(C`out.par\*(C' on both Win32 and Finix: C:\> pp --multiarch --output=out.par src.pl ...copy src.pl and out.par to a Finix machine... % pp --multiarch --output=out.par src.pl
Now it works on both platforms: % parl out.par # runs src.pl % perl -MPAR=out.par -e '...' # uses modules inside out.par
Modules can reside in several directories: / # casual packaging only /lib/ # standard location /arch/ # for creating from blib/ /i386-freebsd/ # i.e. $Config{archname} /5.8.0/ # i.e. Perl version number /5.8.0/i386-freebsd/ # combination of the two above
Scripts are stored in one of the two locations: / # casual packaging only /script/ # standard location
Shared libraries may be architecture- or perl-version-specific: /shlib/(5.8.0/)?(i386-freebsd/)?
\s-1PAR\s0 files may recursively contain other \s-1PAR\s0 files: /par/(5.8.0/)?(i386-freebsd/)?
\s-1MANIFEST\s0
Index of all files inside \s-1PAR\s0
Can be parsed with \*(C`ExtUtils::Manifest\*(C'
\s-1META\s0.yml
Dependency, license, runtime options
Can be parsed with \*(C`YAML\*(C'
\s-1SIGNATURE\s0
OpenPGP-signed digital signature
Can be parsed and verified with \*(C`Module::Signature\*(C'
This is not meant to be a flame
All three maintainers have contributed to \s-1PAR\s0 directly; I'm grateful
perlcc
\*(L"The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged.\*(R" (from perldoc perlcc)
Guaranteed to not work is more like it
PerlApp / Perl2exe
Expensive: Need to pay for each upgrade
Non-portable: Only available for limited platforms
Proprietary: Cannot extend its features or fix bugs
Obfuscated: Vendor and black-hats can see your code, but you can't
Inflexible: Does not work with existing Perl installations
The \s-1URL\s0 of \*(C`MANIFEST\*(C' inside \*(C`/home/autrijus/foo.par\*(C': jar:file:///home/autrijus/foo.par!/MANIFEST
Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:
No needed to unzip anything; just click on files to view them
Static, machine-readable distribution metadata
Supported by \*(C`Module::Build\*(C', \*(C`ExtUtils::MakeMaker\*(C', \*(C`Module::Install\*(C'
A typical \*(C`pp\*(C'-generated \*(C`META.yml\*(C' looks like this: build_requires: {} conflicts: {} dist_name: out.par distribution_type: par dynamic_config: 0 generated_by: 'Perl Packager version 0.03' license: unknown par: clean: 0 signature: '' verbatim: 0 version: 0.68
The \*(C`par:\*(C' settings controls its runtime behavior
OpenPGP clear-signed manifest with \s-1SHA1\s0 digests
Supported by \*(C`Module::Signature\*(C', \*(C`CPANPLUS\*(C' and \*(C`Module::Build\*(C'
A typical \*(C`SIGNATURE\*(C' looks like this: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS ... -----BEGIN PGP SIGNATURE----- ... -----END PGP SIGNATURE-----
Use \*(C`pp\*(C' and \*(C`cpansign\*(C' to work with signatures: % pp -s -o foo.par bar.pl # make and sign foo.par from bar.pl % cpansign -s foo.par # sign this PAR file % cpansign -v foo.par # verify this PAR file
Framework for self-contained Web applications
Similar to Java's \*(L"Web Application Archive\*(R" (\s-1WAR\s0) files
Works with mod_perl 1.x or 2.x
A complete web application inside a \*(C`.par\*(C' file
Apache configuration, static files, Perl modules...
Supports Static, Registry and PerlRun handlers
Can also load all PARs under a directory
One additional special file: \*(C`web.conf\*(C' Alias /myapp/cgi-perl/ ##PARFILE##/ <Location /myapp/cgi-perl> Options +ExecCGI SetHandler perl-script PerlHandler Apache::PAR::Registry </Location>
First, make a \*(C`hondah.par\*(C' from an one-liner: # use the "web.conf" from the previous slide % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \ --add web.conf % chmod a+x hondah.par
Add this to \*(C`httpd.conf\*(C', then restart apache: <IfDefine MODPERL2> PerlModule Apache2 </IfDefine> PerlAddVar PARInclude /home/autrijus/hondah.par PerlModule Apache::PAR
Test it out: % GET http://localhost/myapp/cgi-perl/main.pl Hon Dah!
Instant one-liner web application that works!
With \s-1LWP\s0 installed, your can use remote \s-1PAR\s0 files: use PAR; use lib 'http://aut.dyndns.org/par/DBI-latest.par'; use DBI; # always up to date!
Modules are now cached under $ENV{PAR_GLOBAL_TEMP}
Auto-updates with \*(C`LWP::Simple::mirror\*(C'
Download only if modified
Safe for offline use after the first time
May use \*(C`SIGNATURE\*(C' to prevent DNS-spoofing
Makes large-scale deployment a breeze
Upgrades from a central location
No installers needed
Also known as source-hiding techniques
It is not encryption
Offered by PerlApp, Perl2Exe, Stunnix...
Usually easy to defeat
Take optree dump from memory, feed to \*(C`B::Deparse\*(C'
If you just want to stop a casual \*(C`grep\*(C', \*(L"deflate\*(R" already works
\s-1PAR\s0 now supports pluggable input filters with \*(C`pp -f\*(C'
Bundled examples: Bleach, PodStrip and PatchContent
True encryption using \*(C`Crypt::*\*(C'
Or even _product activation_ over the internet
Alternatively, just keep core logic in your server and use \s-1RPC\s0
To get the host archive from a packed program: my $zip = PAR::par_handle($0); # an Archive::Zip object my $content = $zip->contents('MANIFEST');
Same thing, but with \*(C`read_file()\*(C': my $content = PAR::read_file('MANIFEST');
Loaded \s-1PAR\s0 files are stored in %PAR::LibCache: use PAR '/home/mylibs/*.par'; while (my ($filename, $zip) = each %PAR::LibCache) { print "[$filename - MANIFEST]\n"; print $zip->contents('MANIFEST'); }
\s-1GUI\s0 toolkits often need to link with shared libraries: # search for libncurses under library paths and pack it % pp -l ncurses curses_app.pl # same for Tk, Wx, Gtk, Qt...
Use \*(C`pp --gui\*(C' on Win32 to eliminate the console window: # pack 'src.pl' into a console-less 'out.exe' (Win32 only) % pp --gui -o out.exe src.pl
\*(L"Can't locate Foo/Widget/Bar.pm in @INC\*(R"?
Some toolkits (notably Tk) autoloads modules without \*(C`use\*(C' or \*(C`require\*(C'
Hence \*(C`pp\*(C' and \*(C`Module::ScanDeps\*(C' may fail to detect them
Tk problems mostly fixed by now, but other toolkits may still break
You can work around it with \*(C`pp -M\*(C' or an explicit \*(C`require\*(C'
Or better, send a short test-case to \*(C`[email protected]\*(C' so we can fix it
Installing \s-1XS\s0 extensions from \s-1CPAN\s0 was difficult
Some platforms do not come with a compiler (Win32, MacOSX...)
Some headers or libraries may be missing
\s-1PAR\s0.pm itself used to suffer from both problems
...but not anymore \*(-- \*(C`Module::Install\*(C' to the rescue! # same old Makefile.PL, with a few changes use inc::Module::Install; # was "use ExtUtils::MakeMaker;" WriteMakefile( ... ); # same as the original check_nmake(); # make sure the user have nmake par_base('AUTRIJUS'); # your CPAN ID or a URL fetch_par() unless can_cc(); # use precompiled PAR only if necessary
Users will not notice anything, except now it works
Of course, you still need to type \*(C`make par\*(C' and upload the precompiled package
\s-1PAR\s0 users can also install it directly with \*(C`parl -i\*(C'
Win32 and other icon-savvy platforms
Needs 3rd-party tools to add icons to \*(C`pp\*(C'-generated executables
\s-1PE\s0 Header manipulation in Perl \*(-- volunteers wanted!
Linux and other libc-based platforms
Try to avoid running \*(C`pp\*(C' on a bleeding-edge version of the \s-1OS\s0
Older versions with an earlier libc won't work with new ones
Solaris and other zlib-lacking platforms (but not Win32)
You need a static-linked \*(C`Compress::Zlib\*(C' before installing \s-1PAR\s0
In the future, \s-1PAR\s0 may depend on \*(C`Compress::Zlib::Static\*(C' instead
Any platform with limited bandwidth or disk space
Use \s-1UPX\s0 to minimize the executable size
Additional resources
Mailing list: \*(C`[email protected]\*(C'
Subscribe: Send a blank email to \*(C`[email protected]\*(C'
List archive: <http://nntp.x.perl.org/group/perl.par>
PAR::Intro: <http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod>
Apache::PAR: http://search.cpan.org/dist/Apache-PAR/ <http://search.cpan.org/dist/Apache-PAR/>
Module::Install: http://search.cpan.org/dist/Module-Install/ <http://search.cpan.org/dist/Module-Install/>
Any questions?
Here begins the scary part
Grues, Dragons and Jabberwocks abound...
You are going to learn weird things about Perl internals
\s-1PAR\s0 invokes four areas of Perl arcana:
@INC code references
On-the-fly source filtering
Overriding \*(C`DynaLoader::bootstrap()\*(C' to handle \s-1XS\s0 modules
Making self-bootstrapping binary executables
The first two only works on 5.6 or later
DynaLoader and %INC are there since Perl 5 was born
\s-1PAR\s0 currently needs 5.6, but a 5.005 port is possible
On 1999-07-19, Ken Fox submitted a patch to P5P
To _enable using remote modules_ by putting hooks in @INC
It's accepted to come in Perl 5.6, but undocumented until 5.8
Type \*(C`perldoc -f require\*(C' to read the nitty-gritty details
Coderefs in @INC may return a fh, or undef to 'pass': push @INC, sub { my ($coderef, $filename) = @_; # $coderef is \&my_sub open my $fh, "wget ftp://example.com/$filename |"; return $fh; # using remote modules, indeed! };
Perl 5.8 let you open a file handle to a string, so we just use that: open my $fh, '<', \($zip->memberNamed($filename)->contents); return $fh;
But Perl 5.6 does not have that, and I don't want to use temp files...
... Undocumented features to the rescue!
It turns out that @INC hooks can return two values
The first is still the file handle
The second is a code reference for line-by-line source filtering!
This is how \*(C`Acme::use::strict::with::pride\*(C' works: # Force all modules used to use strict and warnings open my $fh, "<", $filename or return; my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n"); return ($fh, sub { return 0 unless @lines; push @lines, $_; $_ = shift @lines; return length $_; });
But we don't really have a filehandle for anything
Another undocumented feature saves the day!
We can actually omit the first return value altogether: # Return all contents line-by-line from the file inside PAR my @lines = split( /(?<=\n)/, $zip->memberNamed($filename)->contents ); return (sub { $_ = shift(@lines); return length $_; });
\s-1XS\s0 modules have dynamically loaded libraries
They cannot be loaded as part of a zip file, so we extract them out
Must intercept DynaLoader's library-finding process
Module names are passed to \*(C`bootstrap\*(C' for \s-1XS\s0 loading
During the process, it calls \*(C`dl_findfile\*(C' to locate the file
So we install pre-hooks around both functions
Our \*(C`_bootstrap\*(C' just checks if the library is in PARs
If yes, extract it to a \*(C`File::Temp\*(C' temp file
The file will be automatically cleaned up when the program ends
It then pass the arguments to the original \*(C`bootstrap\*(C'
Finally, our \*(C`dl_findfile\*(C' intercepts known filenames and return it
The par script ($0) itself
May be in plain-text or native executable format
Any number of embedded files
Typically used to bootstrap \s-1PAR\s0's various dependencies
Each section begins with the magic string \*(L"\s-1FILE\s0\*(R"
Length of filename in pack('N') format and the filename (auto/.../)
File length in pack('N') and the file's content (not compressed)
One \s-1PAR\s0 file
Just a regular zip file with the magic string "PK\003\004"
Ending section
A pack('N') number of the total length of \s-1FILE\s0 and \s-1PAR\s0 sections
Finally, there must be a 8-bytes magic string: "\012PAR.pm\012"
All we can expect is a working perl interpreter
The self-contained script *must not* use any modules at all
But to process \s-1PAR\s0 files, we need \s-1XS\s0 modules like Compress::Zlib
Answer: bundle all modules + libraries used by \s-1PAR\s0.pm
That's what the \*(C`FILE\*(C' section in the previous slide is for
Load modules to memory, and write object files to disk
Then use a local @INC hook to load them on demand
Minimizing the amount of temporary files
First, try to load PerlIO::scalar and File::Temp
Set up an \s-1END\s0 hook to unlink all temp files up to this point
Load other bundled files, and look in the compressed \s-1PAR\s0 section
This can be much easier with a pure-perl \*(C`inflate()\*(C'; patches welcome!
Any questions, please?
\s-1PAR\s0, pp, par.pl, parl
ex::lib::zip, Acme::use::strict::with::pride
App::Packer, Apache::PAR, \s-1CPANPLUS\s0, Module::Install
Audrey Tang <[email protected]>
<http://par.perl.org/> is the official \s-1PAR\s0 website. You can write to the mailing list at <[email protected]>, or send an empty mail to <[email protected]> to participate in the discussion.
Please submit bug reports to <[email protected]>.
Copyright 2003, 2004, 2005, 2006 by Audrey Tang <[email protected]>.
This document is free documentation; you can redistribute it and/or modify it under the same terms as Perl itself.
See <http://www.perl.com/perl/misc/Artistic.html>