Announcing pts clang A portable LLVM Clang C and C compiler release running on Linux i386 and Linux x86 64

Announcing pts clang A portable LLVM Clang C and C compiler release running on Linux i386 and Linux x86 64

This blog post announces pts-clang, a portable LLVM Clang C and C++ compiler release running on Linux i386 and Linux x86_64.

pts-clang is a portable Linux i386 version of the clang tool of the LLVM Clang compiler, version 3.3. C and C++ compilation is supported, other frontends (such as Objective C) were not tested. The tool runs on Linix i386 and Linux amd64 systems. Its libc-independent, all code is statically linked to the binary. It also contains statically linked linker (ld), so installing binutils is not necessary.

See all details in the most recent README.

Trying it out

If you dont have root access and you dont have the libc headers (e.g. in the libc6-dev package, sample file /usr/include/stdio.h) installed, you can still try pts-clang, with pts-xstatic. See the details in this blog post.

If you dont have /usr/include/stdio.h, install the libc development package:

$ sudo apt-get install libc6-dev

To be able to compile both i386 and amd64 binaries, and you have a recent Linux distribution (e.g. Ubuntu Precise) you can install the libc6-dev package for both architectures:

$ sudo apt-get intall libc6-dev:i386 libc6-dev:x86_64

Download and try pts-clang like this:

$ cd /tmp
$ rm -f pts-clang-latest.sfx.7z
$ wget http://pts.50.hu/files/pts-clang/pts-clang-latest.sfx.7z
$ chmod +x pts-clang-latest.sfx.7z
$ ./pts-clang-latest.sfx.7z -y # Creates the pts-clang directory.
$ cat >>hw.c <<END
#include <stdio.h>
int main(void) {
 return !printf("Hello, %s! ", "World");
}
$ pts-clang/bin/clang -s -O2 -W -Wall hw.c
$ ./a.out
Hello, World!
END

Use clang++ for compiling C++ code, but for that you have to install one of the libstdc++...-dev packages first.

Does pts-clang create portable executables?

By default (without the -xstatic or -xermine flags), the executables created by pts-clang are just as portable as those generated by gcc or clang. They are dynamically linked (unless -static is specified), thus they depend on the system libraries (e.g. /lib/libc.so.6).

If the -static flag is specified, then the executable becomes statically linked, but this doesnt provide real portability, because for calls such as getpwent(3) (getting info of Unix system users) and gethostbyname(3) (DNS resolution), glibc loads files such as libnss_compat.so, libnss_dns.so. On the target system those libraries may be incompatible with your binary, so you may get a segfault or unintended behavior. pts-xstatic solves this, because it uses uClibc.

If the -xstatic flag is specified, pts-xstatic is used to create a portable statically linked, Linux i386 executable, linked against uClibc.

If the -xermine flag is specified, Ermine is used to pack library and other dependencies to a single, portable executable. This can be even more portable than -xstatic, because Ermine can pack locale files, gconv libraries etc. Ermine is a Linux ELF portable executable creator: it takes a dynamically linked ELF executable, discovers its dependencies (e.g. dynamic libraries, NSS libaries), and builds a protable, statically linked ELF executable containing all the dependencies. See the features, licensing information and get Ermine from here. The result can be even more portable than -xstatic, because Ermine can pack locale files, gconv libraries etc. Not all the packing is automatic: use -xermine,... to specify packing flags to Ermine.

Portability improvements

• pts-clang is portable (libc-independent): all shipped binaries are either statically linked. (The clang binary is packed with Ermine, the file is a statically linked executable, which contains a dynamically linked executables and its dependencies (e.g. libc etc.) with itself.) The system libraries are not used for running the compiler (but they are used for linking the output file, except when -xstatic is specified).
• A statically linked linker (ld, GNU gold) binary is provided, so GNU binutils is not a requirement for compilation on the host system.
• Some other optional, statically linked binutils tools (ar, ranlib and strip) are also provided for convenience in the pts-static-binu binary release, see more info in its README. These tools can be used for auxiliary tasks such as building static libraries.

Because of these portability improvemenets, its easy to run pts-clang in a chroot environment.

C++11 and C++0x compatibility

Please note that even though Clang 3.3 supports C++11, much of that is implemented in the C++ standard library (GCCs libstdc++ or Clangs libc++) header files, and no attempt is made in pts-clang to provide the most up-to-date C++ standard library. With -xstatic, an old libstdc++ (the one from gcc-4.4.3) is provided, and without -xstatic the systems default libstdc++ will be used, which can be older than C++11.

Author, copyright and recompilation

The binaries here were created by P�ter Szab�, using existing LLVM Clang and uClibc cross compiler and other binaries, and writing some custom trampoline code. See the details in the GitHub repository.

All software mentioned in this blog post is free software and open source, except for Ermine.

Thanks to Ermine

The author of pts-clang is grateful and says thank you to the author of Ermine, who has provided a free-of-charge Ermine license, using which the portable clang.bin binary was created from the official Clang binary release (which is libc-dependent).

If you want to create portable Linux executables (and you dont care too much about file size), give Ermine a try! Its the most comfortable, easy-to-use, and comprehensive tool available.

Installation instructions for the old version (v1)

Download the old version (v1) from here: http://pts.szit.bme.hu/files/pts-clang/pts-clang-xstatic-bin-3.3-linux-i386-v1.sfx.7z You can extract it with 7z (in the p7zip package), but you can also make it executable and run it, because its a self-extracting archive.

Clang itself is a cross-compiler, so it can generate object files for many architectures and operating systems (see the -target flag), but you also need a system-specific linker (not included) to build binaries or libraries. In newer versions (v2 and later), a copy of the GNU gold linker is also incuded.

This release introduces a non-standard command-line flag -xstatic which enables the Linux i386 target with static linking using the bundled uClibc library. The required .h and .a files, as well as a portable GNU ld linker binary are also included for the use of this flag. In newer versions (v2 and later) the files required by -xstatic are available in a separate download, see the details in this blog post.

Installation instructions for the even old version (v0)

Download version v0 from here: pts-clang-xstatic-bin-3.3-linux-i386.sfx.7z. You can extract it with 7z (in the p7zip package), but you can also make it executable and run it, because its a self-extracting archive.

More info

See the most recent README for full installation instructions, usage details, full feature list etc.

Read full post »

Announcing Bens Corner! NGR returning to its Roots

Announcing Bens Corner! NGR returning to its Roots

Hey everyone, hows it been? I know its been quite some time since Ive made a "personal" post like this, but I wanted to take some time today to explain some things that will be changing here at NGR -- or rather, to announce that things will be slowly returning to the way they once were. You see, back when NGR first opened four years ago, we were all about the people. Sure, we would share news articles and what not, but the sort of things you would see here most of the time were reviews, top 10 lists, discussion topics, and just personal posts in general. Instead of writing about the same old news story you may have seen hundreds of times already, it was a blog about us, the writers, and you, the community. Instead of posting about that new game coming out everyone already knew about, we would share our thoughts and feelings on it, and rather than simply writing about whatever popped into our heads (or into the news), we would look at what you guys wanted to see, and do it if possible. A lot of our older enteries actually came from questions we were personally asked, and others actually came from discussions we had with our readers. This is the sort of thing we would love to bring back, and it is the reason Im making this post today.

Starting later today Ill be opening a new blog series here on NGR. The series will be called "Bens Corner," and it will be me personally returning to our old style. Of course well still be posting news articles, reviews, and whatever else youve come to expect, but this series will truely take this blog back to our roots. Itll be a series of personal entries from me, and they will not reflect the views of NGR in general. Ill be sharing stories, discussing things I currently have on my mind, and hopefully Ill be able to get you guys involved again as well. Im willing to answer any questions youd like to know the answers to, and Ill glady discuss anything you are interested in as well (if possible of course).

With all of that being said, Id just like to remind you guys that these posts will be my own personal opinions, and at some point I may say something you disagree with. I just ask that you guys respect my opinions, but at the same time feel free to disagree with me and leave it in the comments! Its just one of the many ways you guys can get involved, and I would love to see your feedback! (Just like old times.)

Read full post »

Announcing uevalrun self contained computation sandbox for Linux

Announcing uevalrun self contained computation sandbox for Linux

This blog post is to announce uevalrun.

uevalrun is a self-contained computation sandbox for Linux, using User-mode Linux for both compilation and execution of the program to be sandboxed. The program can be written in C, C++, Python, Ruby, Perl or PHP. uevanrun enforces memory limits, timeouts and output size limits in the sandbox. The primary use case for uevalrun is evaluation of solution programs submitted by contestants of programming contests: uevalrun compiles the solution, runs it with the test input, compares its output against the expected output, and writes a status report.

For your convenience, here is a (non-updated) copy of the documenation of uevalrun. See project home page for the most up-to-date version.

Installation

A 32-bit (x86, i386) or 64-bit (x86_64, amd64) Linux system is required with enough RAM for both compilation and execution, plus 6 MB memory overhead. The Linux distribution doesnt matter, because uevalrun uses statically linked binaries, and its self-contained: it contains all the tools it needs, including the C and C++ compilers (from the uClibc gcc-4.1.2 toolchain).

uevalrun doesnt need root privileges: it runs as a simple user.

uevalrun needs about 160MB of disk space, most of which is consumed by the C compiler (31MB extracted + 17MB compressed), the scripting language interpreters (9MB compressed), and the virtual disk images (77MB, uncompressed). After removing all temporary files, 80MB will be enough.

Download and compile:

$ svn co http://pts-mini-gpl.googlecode.com/svn/trunk/uevalrun uevalrun
$ cd uevalrun
$ ./make # This downloads some more files during compilation.

Try it (the italic parts are displayed by the program):

$ (echo #! perl; echo print "Hello"x2, " ") >foo.pl
$ echo HelloHello >expected.txt
$ ./uevalrun -M 10 -T 9 -E 9 -s foo.pl -t /dev/null -e expected.txt
...
@ result: pass
$ echo /**/ int main() { return!printf("HelloHello "); } >foo.c
$ ./uevalrun -M 10 -T 9 -E 9 -U 19 -N 32 -C 9 
 -s foo.c -t /dev/null -e expected.txt
...
@ result: pass

How to use

This section has not been written yet. If you have questions, dont hesitate to ask!

Requirements

Security is the most important requirement of uevalrun, followed by high performance and usability.

The sandboxing must be secure, more specifically:

• Programs inside the sandbox must not be able to communicate with the outside world, except for their stdin and stdout. So they dont have access to the filesystem (except for some read-only access to some whitelisted system libraries and binaries), and they cant do network I/O. If uevalrun is used for programming contest submission evaluation, this restriction prevents the program from finding and reading the file containing the expected output.
• Sandboxed programs must not be able to use more system resources (memory, disk, CPU) than what was allocated for them.
• Sandboxed programs must not be running for a longer period of time than what was allocated.
• Even the compilation of programs to be run (executed) inside the sandbox must be sandboxed (possibly with different system resource allocation), to prevent the attacker from submitting a program source code which exploits a bug in the compiler.
• The sandbox must be reverted to its initial state before each compilation and execution, so sandboxed programs wont be able to gather information about previous compilations and executions. For programming contests, this restriction prevents the program from reading other contestants submissions or their output.
• Timeouts (both real time and user time) must be enforced outside the sandbox, so the program will be reliably killed if it runs for too long time, even if it tampers with time measurement inside the sandbox.
• Sanboxed programs must not be able to lie about their success, their performance or the correctness of their output by writing special marker characters to their stdout or stderr.

The sandbox must be fast and waste few resources, more specifically:

• Reverting the sandbox to its initial, empty state must be very fast (preferably faster than 1 second).
• Starting a program inside the sandbox must be very fast (preferably faster than 1 second).
• Running a program inside the sandbox must not be much slower than running the same program outside the sandbox. If the program is CPU-intensive, it may run 1.1 times slower (i.e. 10% slower) inside than outside. If the program is I/O-intensive, it may run 10 times slower inside than outside. This requirement is intentionally quite loose on I/O virtualization performance.
• Running a program inside the sandbox must not need more than 10MB more memory than running the same program outside.
• Running a program inside the sandbox must not need any disk space, except for the disk space needed by the program binary, the test input and the expected output, all stored as files outside the sandbox.
• Compilation inside the sandbox must require only a reasonable amount of temporary disk space (for the file to be compiled, the temporary files, and the output binary).

The sandbox must be easy to use and versatile, more specifically:

• Sandboxed programs must be able to accept any 8-bit binary input (stdin).
• Sandboxed programs must be able to write any 8-bit binary output to their stdout.
• Multiple sandboxed programs must be able to run at the same time on the same host system, without affecting each other.
• If a sandboxed program fails (e.g. because it writes to its stdout different from what was expected, or it does a memory access violation, it reaches a timeout, it exceeds its allocated memory etc.), the proper failure reason has to be reported (e.g. ``wrong answer must not be reported if the program times out or vice versa).
• The sandboxing software must have as little few system dependencies as possible. It must be able to run in a restricted (e.g. chroot) environment, it must not depend on system libraries or configuration. It must work on 32-bit (x86, i386) and 64-bit (x86_64, amd64) Linux systems, on any Linux distirubution.
• Sandboxed programs can be written in C, C++, Python, Ruby, Perl and PHP, or in any language whose compiler can produce a 32-bit (i386) Linux statically linked executable binary.

Design

To fullfill the requirements, the following fundamental building blocks are used.

User-mode Linux(UML) is used for virutalization: both compilation and execution is performed in a UML guest, reading data from the host system using virtual block devices (ubd), and writing its output to its /dev/tty1 (con0), which can be read by a process on the host. A UML guest kernel (currently 2.6.31.1) tailered to the sandboxing requirements (security and performance) is configured and compiled. Some kernel patches are applied to increase security, reliability and performance. (Please note that these patches apply to the UML guest kernel only, so the host remains unpatched, and rebooting is not needed either.) Networking and the virtual file system are disabled in the UML guest kernel for increased security. All unnecessary drivers and features are removed from the UML guest kernel to get fast boot times and to reduce the overhead.

The underlying virtualization technology used by UML is ptrace(), which doesnt need root privileges or a kernel module or kernel modifications in the host. As an alternative to UML, Seccomp could be used for sendboxing, but thats quite cumbersome, because the sandboxed process cannot allocate memory for itself (see the Google Chrome Seccomp sandbox for a generic solution), but its still prohibitively cumbersome to sandbox GCC this way (with its requirements for forking and creating temporary files). Most sandboxing approaches on Linux require root privileges for a short amount of time (for chroot, PID namespaces (see clone(2)) and other namespaces). Most virtualization approaches (like KVM, Xen, VirtualBox, except possibly for QEMU) need loading of kernel modules or patching the kernel, and support only slow guest boot times.

With UML, its possibly to boot the UML guest, run a hello-world binary, and halt the UML guest in less than 0.02 second (sustained). For that speed, one needs a (guest) kernel patch (included and automatically applied in uevalrun) which skips the Calibrating delay loop kernel startup step, which usually takes 0.33 second. The memory overhead of UML is also quite low (6 MB with a stripped-down kernel).

All software running inside and outside UML is written in C (for high performance) and compiled with uClibcand linked statically to avoid depending on the libc on the host, and to avoid the need for installing a libc in the guest. (The total size of these custom-built binaries is less than 100kB.) There is no Linux distribution installed the guest – only a few custom binaries are copied, like a custom /sbin/init, which mounts /proc, sets up file descriptors and resource limits, and starts the sandboxed program, and then halts the guest. The /lib directory in the guest is empty, except for compilation, where /lib contains libc.a, libstdc++.a etc.

BusyBox (statically linked with uClibc) is used as a Swiss Army Knife tool for scripting and automation (of the build and the setup process), both inside and outside UML. Please note, however, that once all the binaries are built and the filesystem images are created, BusyBox is not used anymore for running sandboxed programs and evaluating their output (except for temporary filesystem creation) because of performance reasons, but work is done by the small and fast custom C tools just compiled.

Sandboxed program sources can be written in C (using a large subset of uClibc as the system library), C++ (using uClibc and libstdc++), Python (using all the built-in Python modules), Ruby (using just the built-in Ruby modules and classes which are implemented in C), Perl (using only the few dozen most common modules), or PHP (using just the built-in PHP functions which are implemented in C).

For C and C++ compilation, a GCC 4.1.2 toolchain is used which is based on uClibc and produces statically linked executables.

Interpreters for Python, Ruby, Perl and PHP are provided as precompiled, self-contained, single, statically linked, 32-bit (i386) Linux executables. Its easy to copy them to the /bin directory of the UML guest.

All tools used for building the software are either shipped with the software, or the software downloads them during compilation. All binary tools are statically linked, 32-bit (i386) Linux executables. This provides maximum host system compatibility and reproducible behavior across systems.

In our speed measurements, CPU-intensive programs running inside UML run 1% slower than outside UML, but I/O-intensive programs (such as C++ compilation with GCC) can be about 6 times slower.

The Minix filesystemis used in the UML guests for both read-only filesystems (e.g. the root filesystem with the scripting language interpreters) and read-write filesystems (such as /tmp for the temporary assembly files created by GCC). The Minix filesystem was chosen instead of ext2 and other usual Linux filesystem, because it has less size overhead, its codebase is smaller, and it has equally good performance for the small and mostly read-only filesystems it is used for.

License

uevanrun has been released under the GNU GPL v2.

Bugs? Problems? Contact the author. Send feedback!

Please add your feedback to the issue tracker. All feedbacks are welcome.

Read full post »

Announcing mmshget mmsh MMS over HTTP video stream downloader and reference implementation

Announcing mmshget mmsh MMS over HTTP video stream downloader and reference implementation

This blog post is to announce mmshget, a command-line Python script to download streaming videos of the mmsh:// (MMS-over-HTTP) protocol, in .wmv (or .asf) format. mmshget can also be used as an easy-to-understand, simple, client-side, partial reference implementation of the mmsh:// protocol.

Download the Python script or see the source tree.

mmshget is inspired by and similar to mimms, but it is smaller, easier to understand and has less features. mimms supports both seekable and live streams (mmshget supports seeakable streams only), and mimms additionally supports non-HTTP versions of the MMS protocol (mmshget supports only mmsh://, the HTTP version). mimms depends on the C library libmms, mmshget is implemented in pure Python (needs Python 2.4 later only).

Read full post »

Announcing intalg integer and number theory algorithms in pure Python 2 x

Announcing intalg integer and number theory algorithms in pure Python 2 x

This is the initial release announcement for pybasealgo (fast implementation of basic algorithms in pure Python 2.x, minimum version:) and the first set of algorithms, intalg (integer and number theory algorithms). The algorithm implementations can be used in Project Euler and programming contest problem solving. These are free software under the GPL. Click on the links above to download the source.

intalg contains the following algorithms:

• integer bit count (bit_count)
• square root rounding down (sqrt_floor)
• k^th root rounding down (root_floor)
• bounded prime list builder using the sieve of Eratosthenes (primes_upto, first_primes_moremem, first_primes)
• bounded prime generator using the sieve of Eratosthenes (yield_primes_upto, yield_first_primes)
• unbounded prime generator using the sieve of Eratosthenes (yield_primes, yield_composites)
• high precision upper bound for natural and base 2 logarithms (log2_256_more, log_more)
• greatest common divisor using Euclidean algorithm (gcd)
• primality test using the Rabin-Miller test with fast parameters (is_prime)
• finding the next prime (next_prime)
• computation of binary coefficients (choose)
• integer factorization to prime divisors (factorize, yield_slow_factorize, finder_slow_factorize)
• computation of the number of divisors (divisor_count)
• computation of the sum of divisors (divisor_sum)
• random number generation using the Mersenne twister MT19937, without using floating point numbers (MiniIntRandom)
• Pollards Rho algorithm aiding integer factorization (pollard)
• Brents algorithm aiding integer factorization (pollard)
• simplification of fractions (simplify_nonneg)
• RLE: run-length encoding (rle)
• conversion of fraction of very large integers to float (fraction_to_float)

Design principles:

• Pure Python: Provides full functionality using only built-in Python modules.
• Fast: Fast implementations of fast algorithms, e.g. Newton iteration for square root and k^th root, sieve of Eratosthenes for prime number generation, Brents algorithm for integer factorization. Typically faster than most of the sample code and other libraries found online. (If you find something faster, please let me know by posting a comment.) Not as fast as C code (e.g. pygmp).
• Correct: Many Python code examples (e.g. Brents algorithm) found on the net are buggy (especially in corner cases) or are too slow. pybasealgo strives to be correct. Production use (i.e. I use this library for solving Project Euler problems) and unit tests help this.
• Simple: Maintains a balance between speed and implementation simplicity. It implements sophisticated and fast algorithms in a simple way, but it doesnt contain the fastest possible algorithm if its very complex (e.g. for integer factorization, pyecm is 1472 lines long, our intalg contains Brents algorithm, which is slower but much simpler).
• No-float: Doesnt use floating point numbers, not even for temporary computations (e.g. math.log(...)). This is to avoid rounding errors, which make the correctness of an algorithm hard to prove and they make some computations nondeterministic on some architectures and systems. And also to support arbitrarily large numbers: floats have a maximum (e.g. all IEEE 754 64-bit are less than 2e308), so they cannot be used in calculations with very large numbers. Python uses temporary floating point numbers for random integer generation, so pybasealgo provides an integer-only replacement.
• Arbitrarily large integers: All algorithms work with small integers (of type int) and arbitrarily large integers (of type long). There is no automatic truncation of the results.
• Deterministic: Every function returns the same value for the same input, and it runs at the same speed. (Except if some return values are cached in memory, then subsequent calls in the same process will be faster.) This makes debugging and benchmarking a lot easier. Determinism is very important in a programming contest situation: if your program succeeds for you, you dont want it to fail or time out when the judges rerun it. Even those algorithms which use random numbers (e.g. Pollards Rho algorithm and Brents algorithm for integer factorization) are deterministic by default, because they use pseudorandom numbers with a seed which depends only on the input.

Have fun programming and Python and I wish you great success in Project Euler and in programming contests.

Read full post »

Announcing pts xstatic A tool for creating small statically linked Linux i386 executables with any compiler

Announcing pts xstatic A tool for creating small statically linked Linux i386 executables with any compiler

This blog post announces pts-xstatic, a convenient wrapper tool for compiling and creating portable, statically linked Linux i386 executables. It works on Linux i386 and Linux x86_64 host systems. It wraps an existing compiler (GCC or Clang) of your choice, and it links against uClibc and the other base libraries included in the pts-xstatic binary release.

See the most recent README for all details.

C compilers supported: gcc-4.1 ... gcc-4.8, clang-3.0 ... clang-3.3. C++ compilers supported: g++ and clang++ corresponding to the supported C compilers. Compatible uClibc C and C++ headers (.h) and precompiled static libraries (e.g. libc.a, libz.a, libstdc++.a) are also provided by pts-xstatic. To minimize system dependencies, pts-xstatic can compile with pts-clang (for both C and C++), which is portable, and you can install it as non-root.

As an alternative of pts-xstatic, if you want a tiny, self-contained (single-file) for Linux i386, please take a look at pts-tcc. With pts-xstatic, you can create faster and smaller statically linked executables, with the compiler of your choice.

As an alternative for pts-xstatic and uClibc, see diet libc and its diet tool (which is an alternative of the xstatic tool), with which you can create even smaller binaries.

Motivation

0. Available uClibc GCC toolchain binary releases are very old, e.g. the i686 release contains gcc-4.1.2 compiled on 2009-04-11.
1. With uClibc Buildroot, the uClibc version is tied to a specific GCC version. Its not possible to compile with your favorite preinstalled C or C++ compiler version, and link against your favorite uClibc version. pts-xstatic makes this possible.
2. libstdc++ is not easily available for uClibc, and its a bit cumbersome to compile. pts-xstatic contains a precompiled version.

Minimum installation

If you want to install try pts-xstatic quickly, without root access, without installing any dependencies, and without changing any settings, this is the easiest way:

$ cd /tmp
$ rm -f pts-xstatic-latest.sfx.7z
$ wget http://pts.50.hu/files/pts-xstatic/pts-xstatic-latest.sfx.7z
$ chmod +x pts-xstatic-latest.sfx.7z
$ ./pts-xstatic-latest.sfx.7z -y # Creates the pts-xstatic directory.
$ rm -f pts-clang-latest.sfx.7z
$ wget http://pts.50.hu/files/pts-clang/pts-clang-latest.sfx.7z
$ chmod +x pts-clang-latest.sfx.7z
$ ./pts-clang-latest.sfx.7z -y # Creates the pts-clang directory.
$ cat >>hw.c <#include <stdio.h>
int main(void) {
 return !printf("Hello, %s! ", "World");
}
END
$ pts-xstatic/bin/xstatic pts-clang/bin/clang -s -O2 -W -Wall hw.c && ./a.out
Hello, World!
$ strace -e open ./a.out
Hello, World!
$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped
$ ls -l a.out
-rwxr-xr-x 1 pts pts 16888 Jan 2 23:17 a.out

Compare the file size with statically linking against regular (e)glibc:

$ gcc -static -m32 -o a.big -s -O2 -W -Wall hw.c && ./a.big
Hello, World!
$ strace -e open ./a.big
Hello, World!
$ file a.big
a.big: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=0x37284f286ffeecdb7ac5d77bfa83ade4310df098, stripped
$ ls -l a.big
-rwxr-xr-x 1 pts eng 684748 Jan 2 23:20 a.big

FYI with diet libc, the generated a.out file is only 8668 bytes long.

See full installation instructions in the most recent README.

Does pts-xstatic create portable executables?

pts-xstatic creates portable, statically linked, Linux ELF i386 executables, linked against uClibc. By default, these executables dont need any external file (not even the file specified by argv[0], not even the /proc filesystem) to run. NSS libraries (the code needed for e.g. getpwent(3) (getting info of Unix system users) and gethostbyname(3) (DNS resolution)) are also included. The executables also work on FreeBSD in Linux mode if the operating system field in the ELF header frm SYSV to Linux.

As an alternative to pts-xstatic: gcc -static (or clang -static) doesnt provide real portability, because for calls such as getpwent(3) (getting info of Unix system users) and gethostbyname(3) (DNS resolution), glibc loads files such as libnss_compat.so, libnss_dns.so. On the target system those libraries may be incompatible with your binary, so you may get a segfault or unintended behavior. pts-xstatic solves this, because it uses uClibc.

It can be useful to embed locale files, gconv libraries, arbitrary data and configuration files needed by the program, Neither `gcc -static, pts-xstatic or statifier can do it, but Ermine can. Ermine is not free software, but you can get a free-of-charge time-limited trial, and you can ask for a discount for noncommercial use. See all details here, and give it a try!

More info

See the most recent README for full installation instructions, usage details, full feature list etc.

Read full post »

Announcing mplaylist Audio playlist player using mplayer with checkpointing

Announcing mplaylist Audio playlist player using mplayer with checkpointing

This blog post is the formal announcement of mplaylist, and audio playlist player using mplayer, with checkpointing.

mplaylist is Python script which can play audio playlists (.m3u files), remembering the current playback position (file and time) even when killed, so it will resume playback at the proper position upon restart. The playback position is saved as an .m3u.pos file next to the .m3u file. mplaylist uses mplayer for playing the audio files.

mplayer needs Python and a Unix system with mplayer installed. (It may be easy to port to Windows, but it has not been tried.) Download the script directly from here. There is no GUI. You have to start mplayback from the command-line, in a terminal window.

The reason why I wrote mplaylist is that I needed the following features and I couldnt easily find an audio player for Ubuntu which had all of them:

• It supports multiple playlists.
• It remembers the playback position (file and time) for each playlist.
• Preferably, it remembers playback position even when the process is killed.
• Lets the user adjust playback speed, without changing the pitch.

mplaylist supports all these features. Checkpointing (i.e. remembering the playback position) works even if both the mplayer and mplaylist processes are killed with kill -9 (SIGKILL). If you have a journaling filesystem with block device barriers properly set up, checkpointing also works if you unplug the power cable.

Please note that mplaylist is not only for music files. It works excellently for playing back (series of) audio books and (series of) talks.

Read full post »

Announcing ssweb single shot webserver in Python

Announcing ssweb single shot webserver in Python

This blog post announces ssweb, a single-shot HTTP server library in Python 2.x. Single-shot means is that the webserver accepts only a single incoming HTTP connection, handles it, and then exits immediately.

Such a server can be used as a redirect target in the web-based OAuth authentication protocol, to receive the freshly generated OAuth token. In this case the program is a command-line tool running a HTTP server for a short amount of time, to make the passwordsless login more convenient for the user, without having to manually copy-paste token. If it doesnt sound useful, then its most probably not useful for you right now.

Example usage:

$ wget -O ssweb.py http://raw.githubusercontent.com/pts/ssweb/master/ssweb.py
$ python ssweb.py
URL: http://127.0.0.1:40464/foo
{res_body: Shot., req_head: GET /foo HTTP/1.0 Host: 127.0.0.1:40464 User-Agent: Python-urllib/1.17 , client_addr: (127.0.0.1, 40026), res_head: HTTP/1.0 200 OK Content-Type: text/plain Content-Length: 5 , req_body: }
Shot.
$ python ssweb.py x
Please visit URL: http://127.0.0.1:59872/foo
... (after visiting the URL in Firefox)
{res_body: Shot., req_head: GET /foo HTTP/1.1 Host: 127.0.0.1:59872 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive , client_addr: (127.0.0.1, 50702), res_head: HTTP/1.0 200 OK Content-Type: text/plain Content-Length: 5 , req_body: }

Read full post »

Announcing flickrurlget Flickr photo downloader from command line in batch

Announcing flickrurlget Flickr photo downloader from command line in batch

This blog post is an announcement of flickrurlget, a command-line tool for Unix, written in Python 2.x that can be used to download photos from Flickr in batch. flickrurlget itself doesnt download photos, but it generates a list of raw photo URLs which can be downloaded with a download manager (even with `wget -i).

Download and install from source.

There are many photo download tools for Flickr (including command-line tools, GUI tools, Firefox extensions and Chrome extensions), none of which I tried having the feature set of flickrurlget:

• Can get highest-resolution (if possible, original) image URLs in:
  • a Flickr photo
  • a photostream of a Flickr user (i.e. all photos of the user)
  • the favorite photos of a Flickr user
  • an album (photoset) of a Flickr user
  • a Flickr group
  • a gallery of a Flickr user
• Batch operation: started from the command-line with a bunch of Flickr page URLs, and it discovers all the photos in there without any interaction.
• Can get anybodys photos (not only your own).
• Can get non-public and adult (non-safe) photos as well (after loggig in).
• Doesnt need a GUI or a web browser for most of the operations.

There is the command-line tool flickr_download (source code and PyPi projet page), which is also written in python. Here is how flickrurlget is better than flickr_download:

• Takes Flickr page URLs and figures out all parameters automatically.
• Can download groups and galleries (in addition to user photostreams and photosets) as well.
• Has user ID, group ID and photoset ID discovery: the user doesnt have to do manual ID lookups before running the tool.
• Uses only standard Python modules, no other dependencies.
• Better compatibility with old Python: works with 2.7, 2.6, 2.5 and 2.4; not only 2.7.

Flickr has a nice, full-featured REST API (works with both XML and JSON), its documentation is available here. flickrurlget uses this API. It can also optionally OAuth to log the user in.

Read full post »

Announcing Portable MariaDB Small portable binary MariaDB distribution for Linux

Announcing Portable MariaDB Small portable binary MariaDB distribution for Linux

Portable MariaDB is a small, portable binary distribution of the SQL server MariaDB (Montys fork of MySQL) for Linux i386 (32-bit). Only the mysqld binary and a versatile init script are included. Portable MariaDB can be run by any user in any directory, it doesnt try to access any mysqld data or config files outside its directory. Portable MariaDB can coexist with regular mysqld (MySQL or MariaDB) and other instances of Portable MariaDB on a single machine, as long as they are not configured to listen on the same TCP port. The only dependency of Portable MariaDB is glibc 2.4 (available in Ubuntu Hardy or later).

The most up-to-date documentation of Portable MariaDB is here.

The sources are here.

Why use Portable MariaDB?

• Its small (not bloated). Fast to dowload, fast to extract, fast to install. Quick size comparison: mariadb-5.2.9-Linux-i686.tar.gz is 144 MB, the corresponding Portable MariaDB .tbz2 is less than 6 MB.
• Its portable: does not interfere with other MySQL server installations on the same machine.
• Its self-contained and consistent: copy the database and the configuration in a single directory from one machine to another.

Installation

To run Portable MariaDB, you need a Linux system with glibc 2.4 (e.g. Ubuntu Hardy) or later. 32-bit and 64-bit systems are fine. For 64-bit systems you need the 32-bit compatibility libraries installed. You also need Perl.

 $ cd /tmp # Or any other with write access.
 $ BASE=https://raw.githubusercontent.com/pts/portable-mariadb/master/release
 $ #OLD: wget -O portable-mariadb.tbz2 $BASE/portable-mariadb-5.2.9.tbz2
 $ wget -O portable-mariadb.tbz2 $BASE/portable-mariadb-5.5.46.tbz2
 $ tar xjvf portable-mariadb.tbz2
 $ chmod 700 /tmp/portable-mariadb # For security.
 $ /tmp/portable-mariadb/mariadb_init.pl stop-set-root-password

Usage

For security, dont do anything as root.

 $ cd /tmp/portable-mariadb
 $ ./mariadb_init.pl restart
 Connect with: mysql --socket=/tmp/portable-mariadb/mysqld.sock --user=root --database=test --password
 Connect with: mysql --host=127.0.0.1 --user=root --database=test --password

Feel free to take a look at /tmp/portable-mariadb/my.cnf, make modifications, and restart mysqld so that the modifications take effect.

Security

By default, connections are accepted from localhost (Unix domain socket and TCP) only, all MySQL users are refused (except if a password has been set for root above), and root has unrestricted access. Unix permissions (such as the chmod 700 above) are protecting against data theft and manipulation on the file level.

It is strongly recommended to change the password of root to a non-empty, strong password before populating the database.

Java support

Java clients with JDBC (MySQL Connector/J) are fully supported. Please note that Java doesnt support Unix doman socket, so make sure in my.cnf that mysqld listens on a TCP port. Please make sure you have ?characterEncoding=UTF8 specified in your JDBC connection URL, otherwise some non-ASCII, non-Latin-1 characters would be converted to ?.

Unicode support

Just as with MariaDB. All encodings and collations are supported. The latin1 encoding is the default, which can be changed in my.cnf.

Language support

All natural languages (of MariaDB) are supported for error messages. Set the `language flag in my.cnf accordingly. English is the default.

Read full post »