Building XS Modules for 64bit ActivePerl on Windows

Here I'll discuss how to build extensions for 64bit ActivePerl for Windows using freely available tools.

First let's discuss which perl for Windows you need if you haven't decided that yet.

For most general use and development you want a 32bit ActivePerl or Strawberry Perl. ActivePerl has the advantage of some difficult to build modules available as easy to install PPM packages in the ActiveState repositories or some of the third-party repositories. In other ways the two are similar, ActivePerl will now automatically install the MinGW toolchain and dmake when you need it to compile an extension, the tools Strawberry Perl comes with, or you can just type ppm install MinGW.

The reason you would generally prefer a 32bit perl on Windows is that many modules, especially some Win32:: modules are not compatible with 64bit perls yet; and in the case of ActivePerl, there are far more PPMs available for the 32bit builds.

Your choice of whether to use a 32bit or 64bit perl may be dictated by factors other than convenience and compatibility however, you may need to link the interpreter into a native process of a certain bitness, or link a .dll library of a certain bitness into the perl interpreter. Or you may need to use either a 32bit or 64bit ODBC driver which will only work with that respective bitness of perl. And so on.

Once you've decided that you need to use a 64bit perl, you have a choice between Strawberry Perl and ActivePerl. Strawberry Perl comes with MinGW-w64 and dmake bundled in the distribution and pre-setup for you, so you can skip the setup I describe here for ActivePerl. If you choose to use ActivePerl, for the availability of (some) PPMs or other reasons, read on.

For 64bit ActivePerl you have the choice of two freely available tools to build extensions, the Windows SDK and the MinGW-w64 suite, setup for both is described below.

The Windows SDK

The freely available Windows SDK (current version) contains everything you need including a set of compilers. If however you are getting the Windows 8 SDK you will also need to get Visual Studio Express 11 for the compilers, as the Windows 8 SDK no longer includes a set of compilers.

Now you need to modify your environment to make the SDK tools available to perl (CPAN installations, etc..) I recommend using Rapid Environment Editor, or you can make .cmd file and modify your cmd.exe shortcut to add the /K option and the path to your .cmd file with your settings. I attempted, with some measure of success, to make a system of switching between different perls in a single cmd.exe session in this article.

The settings you need for the Windows 7 SDK are as follows (modify to taste:)

set PATH=C:\Perl64\site\bin;C:\Perl64\bin;C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\x64;C:\Windows\system32;C:\Windows
set INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE;C:\Program Files\Microsoft SDKs\Windows\v7.1\Include;C:\Program Files\Microsoft SDKs\Windows\v7.1\Include\gl
set LIB=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Lib\amd64;C:\Program Files\Microsoft SDKs\Windows\v7.1\Lib\X64
set LIBPATH=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Lib\amd64

Now launch cpan in cmd.exe and try to install an XS module such as Class::XSAccessor to test everything out.

If you get link errors about not being able to find bufferoverflowU.lib, edit C:\Perl64\lib\Config_heavy.pl and remove all instances of bufferoverflowU.lib. This shouldn't happen with the newer ActivePerl builds however.

The settings for the Windows 8 SDK with the Visual Studio Express 11 compilers should be similar, but I have not tried that yet, so don't know the details.

MinGW-w64

The ActivePerl MinGW mode mostly works with MinGW-w64, although it is meant for the 32bit builds with older mingw.org suites so some tweaks are necessary.

First download the WIN64 native compiler suite from the MinGW-w64 download page in the WIN64 Downloads -> Personal Builds -> rubenvb -> release section. The file will be named something like x86_64-w64-mingw32-gcc-4.7.0-release-win64_rubenvb.7z. Unpack with 7-zip or WinRAR everything below the top level directory in the archive to C:\MinGW_w64 .

Modify your PATH to add C:\MinGW_w64\bin . See the part about environment settings in the previous section, I recommend Rapid Environment Editor.

Set the LIBRARY_PATH environment variable as follows:

set LIBRARY_PATH=C:\MinGW_w64\x86_64-w64-mingw32\lib

Now install dmake. We'll use a 32bit binary from CPAN so make sure that WOW64 is enabled in your OS if you are using an older server OS. Download the dmake dist from CPAN, open the archive, go into the dmake folder, and extract dmake.exe and the startup subfolder into C:\Perl64\site\bin . Then run:

icacls c:\perl64\site\bin\dmake.exe /grant your_user_name:(rx)

Next edit C:\Perl64\lib\Config_heavy.pl and remove all instances of bufferoverflowU.lib.

That's all there is to it. Now with gcc.exe in your PATH ActivePerl will go into mingw mode when it needs to build an extension, try it now by running cpan Class::XSAccessor or some such extension.

Note that if you will package Perl applications with built extensions, you will need to include the libgcc and libstdc++ (if there were any C++ extensions) DLLs with your package, to be run from the same directory as the perl interpreter or somewhere in PATH. This is also true for the new versions of the 32bit mingw.org toolchain. The alternative is to include

-static-libgcc -static-libstdc++

in your LDDLFLAGS along with the default $Config{lddlflags}. I'm going to submit a bug report to ActiveState to make these options default for newer gccs.

Read more...

Last modified: 2012-5-28 (月) at 10:51 am

Linux Backup with Back In Time

Here I will describe how to do scheduled automatic backups to an external hard drive on a Debian/Ubuntu Linux computer with the Back In Time system.

First install the package:

sudo aptitude install backintime-gnome

Preparing the Hard Drive

Plug in your USB/ESATA/FireWire hard drive into your linux computer, then do a:

dmesg | tail

to note the block device, such as /dev/sdd.

Next you will want to partition it. Use cfdisk or gparted.

Make one primary partition of type Linux (83) that is the size of the drive.

Next format it with your favorite file system.

Use a command such as:

sudo mkfs.ext4 /dev/sdd1

Now let's mount it. We are going to mount it under /media so that the updatedb cron job does not index it and clutter your locate db with your backup files. We are also going to mount it by UUID so that the order you plug in your devices, the presence of hubs, etc. does not affect mounting.

First make the mount point:

sudo mkdir /media/backup

Then get the UUID of the new partition:

sudo blkid /dev/sdd1

Now add the mount point to /etc/fstab using the UUID from the previous step:

UUID=e3b623e7-2049-46cf-89aa-1152c38e08e8 /media/backup ext4 defaults,nofail

Now mount it!

sudo mount /media/backup

Remember to mount it when you plug your hard drive in.

If you know of a way to automount an fstab entry, please email me so I can add it here...

We are going to make the backup directory one level below the mount point so that having the drive disconnected tells Back In Time to not run the backup job, otherwise it would run the backup job on the root file system!

sudo mkdir /media/backup/snapshots

Setting up Back In Time

Now you're ready to run the Back In Time configuration program. Run:

ssh -X root@localhost backintime-gnome

Click the Settings button.

For "Where to save snapshots" select /media/backup/snapshots.

For "Schedule" I use "Every Day" at 03:00. Choose a time your machine is likely to be idle and the hard drive is likely to be plugged in.

Click the Include tab. Add folders such as /home and /etc and any others you want to back up.

Press "Ok" and close the program.

Press Ctrl-C to close the ssh session.

And you're done!

I also highly recommend installing anacron:

sudo aptitude install anacron

especially on a laptop, because Back In Time runs from cron. This way if your laptop is off or asleep when the job is supposed to run, your backup will still run at some future time.

Final Thoughts

Backups can seriously save your arse. Hard drives and SSDs fail. Alot.

Now Mac OS X and Windows have attractive built-in backup options, Time Machine and Windows Backup, respectively. Time Machine is unmatched in power and simplicity. Windows Backup is not as shiny, but still does the job admirably.

With Back In Time, we now have a nice backup solution for Linux as well.

Enjoy the comfort of knowing that your data is safe.

Read more...

Last modified: 2012-1-31 (火) at 11:30 pm

Using SQL Anywhere on Mac OS X

Here I will go through the steps necessary to set up and use SQL Anywhere on Mac OS X.

Download and Install SQL Anywhere

It is available here: http://www.sybase.com/detail?id=1016644.

Open the package and double click the Install SQL Anywhere icon, select all components.

At the end click the Check for Updates button, it will take you to the EBF download. You will need to create a MySybase account to download the EBF package. Download the package, doubleclick the Install SQL Anywhere icon and select the update product option.

Set up Your User Environment

Add the following to your ~/.bashrc:

export SQLANYWHERE_HOME=/Applications/SQLAnywhere12/System
export DYLD_LIBRARY_PATH=$SQLANYWHERE_HOME/lib64:$SQLANYWHERE_HOME/lib32:$DYLD_LIBRARY_PATH
export DYLD_BIND_AT_LAUNCH=1
export PATH=$PATH:$SQLANYWHERE_HOME/bin64:$SQLANYWHERE_HOME/bin32

For the next step, open a new terminal session for the environment changes to take effect.

Create a Database

Execute the following to create a new database:

cd
mkdir sqlanywhere
cd sqlanywhere
mkdir logs
dbinit -t ./logs/hlaghdb.log -p 4096 -dba DBA,SQL hlaghdb.db

Set up the Database Server

Create a file called ~/Library/LaunchAgents/com.ianywhere.sqlanywhere.plist, put the following in that file, changing /Users/rkitover to your actual home directory and the database file name to your actual database file:

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd" >
<plist version='1.0'>
<dict>
<key>Label</key><string>com.ianywhere.sqlanywhere</string>
<key>WorkingDirectory</key><string>/Users/rkitover/sqlanywhere</string>
<key>EnvironmentVariables</key>
    <dict>
        <key>DYLD_LIBRARY_PATH</key>
        <string>/Applications/SQLAnywhere12/System/lib64:/Applications/SQLAnywhere12/System/lib32</string>
        <key>DYLD_BIND_AT_LAUNCH</key>
        <string>1</string>
    </dict>
<key>ProgramArguments</key>
<array>
        <string>/Applications/SQLAnywhere12/System/bin32/dbsrv12</string>
        <string>-ud</string>
        <string>/Users/rkitover/sqlanywhere/hlaghdb.db</string>
</array>
<key>Debug</key><false/>
<key>Disabled</key><true/>
<key>KeepAlive</key><true/>
</dict>
</plist>

NOTE: we do not give the database server a name in this example. This is because for some reason when I tried to use -n option to dbsrv12 it responded with Communication error. Try running dbsrv12 from the command line with the -n option to see if it works for you, if it does add it to the ProgramArguments in the .plist file. E.g.:

dbsrv12 -ud -n SQLANYWHERE hlaghdb.db

If this works, add the following to the .plist file after dbsrv12:

<string>-n</string>
<string>SQLANYWHERE</string>

The following examples will all assume the database server has no name.

Now execute the following to load the service and start the database server:

launchctl load -w ~/Library/LaunchAgents/com.ianywhere.sqlanywhere.plist
launchctl start com.ianywhere.sqlanywhere

You should see the database server in ps:

$ ps aux | grep dbsrv
rkitover       51654   0.3  0.2  1627072  14000   ??  Ss    1:02PM   0:03.07 /Applications/SQLAnywhere12/System/bin32/dbsrv12 -ud /Users/rkitover/sqlanywhere/hlaghdb.db

It will be started across reboots.

launchctl stop doesn't work with this plist, so if you need to restart the server, just kill the process and do a launchctl stop followed by a launchctl start.

The SQL Anywhere Applications

Open the SQLAnywhere12 folder under Applications in Finder.

The applications we're interested in are Sybase Central and Interactive SQL.

The Sybase Central application will allow you to configure all aspects of your database. To use it, open the application, double click on the SQL Anywhere 12 plugin, right click inside the blank window and choose Connect, choose Connect to a running database on this computer, enter the username and password you used with the dbinit command, which in this example is DBA and SQL respectively.

The Interactive SQL application is a handy GUI for testing SQL statements on your database. To login, open the application, choose Database authentication, choose Connect to a running database on this computer, enter your username and password and press Connect. Inside the application, enter your SQL statements in the SQL statements window and press F5 to execute them; you will see your results in the results window.

Using sqsh

If you would prefer a commandline utility to Interactive SQL for testing SQL statements, you can use the sqsh utility from MacPorts (http://www.macports.org/.)

The sqsh utility is an interactive SQL utility for databases using the TDS protocol, such as Sybase ASE and Microsoft SQL Server. SQL Anywhere also uses the TDS protocol, so the utility works for these databases as well.

If you do not yet have an interfaces file, follow these steps to set one up.

In your .bashrc add the following:

export SYBASE=~/sybase

In this directory create a file called interfaces with the following:

SQLANYWHERE
  master tcp ether localhost 2638
  query tcp ether localhost 2638

Now you can launch sqsh like so:

sqsh -S SQLANYWHERE -U DBA -P SQL

To use it, type in a SQL statement, it can span multiple lines, then type go on a line by itself to execute it. This may take some getting used to if you're used to tools such as sqlplus or dbish, where SQL statements are terminated by a semicolon. You can use dbish with SQL Anywhere as well, I will describe how to do so further down.

Setting up ODBC

If you have UnixODBC from MacPorts (http://www.macports.org/,) you can set up the included ODBC drivers with it.

In /opt/local/etc/odbcinst.ini add the following:

[SQLAnywhere]
Description = SQL Anywhere
Driver = /Applications/SQLAnywhere12/System/lib64/libdbodbc12.dylib
Setup = /Applications/SQLAnywhere12/System/lib64/libdbodbc12.dylib
FileUsage = 1

Now lets test it out (take out the -S flag to cpanm if you use local::lib):

curl -L http://cpanmin.us | perl - -S App::cpanminus
cpanm -n -S DBI
cpanm -n -S DBD::ODBC
perl -MDBI -le 'my $dbh = DBI->connect("dbi:ODBC:driver=SQLAnywhere", "DBA", "SQL"); print for $dbh->selectrow_array("select 42")'

Installing the Perl Driver

First install DBI:

curl -L http://cpanmin.us | perl - -S App::cpanminus
cpanm -n -S DBI

Then build and install the Perl library:

cp -a /Applications/SQLAnywhere12/sdk/perl /tmp/sqlany
cd /tmp/sqlany
perl Makefile.PL
make
sudo make install
cd
rm -rf /tmp/sqlany

Test it out:

perl -MDBI -le 'my $dbh = DBI->connect("dbi:SQLAnywhere:", "DBA", "SQL"); print for $dbh->selectrow_array("select 42")'

Using DBI::Shell (dbish)

If you don't like Interactive SQL or sqsh, you can try the Perl module DBI::Shell for a commandline SQL utility.

First install it:

curl -L http://cpanmin.us | perl - -S App::cpanminus
cpanm -n -S DBI::Shell

Then launch it. I would recommend using the ODBC driver, especially if you use UUID columns:

dbish dbi:ODBC:driver=SQLAnywhere DBA SQL

To use the Perl binding:

dbish dbi:SQLAnywhere: DBA SQL

SQL statements are terminated by semicolons, type /quit to exit the shell and /help to see what commands are available.

If you would like to save the readline history of your sessions and use it for completion, you can wrap it with rlwrap. To do so add this alias to your ~/.bashrc:

alias sqlany="rlwrap -a -N -t dumb -i -f ~/.sqlanywhere_dbish_history -H ~/.sqlanywhere_dbish_history -s 30000 dbish dbi:ODBC:driver=SQLAnywhere DBA SQL"

and initialize the history file:

touch ~/.sqlanywhere_dbish_history

then you can type sqlany to launch the shell.

DBIx::Class ORM Support

There is complete support for SQL Anywhere in DBIx::Class and DBIx::Class::Schema::Loader (the Catalyst create=static helper.)

That's All

Hope you found this guide helpful. Feel free to leave comments on the blog or email me if you have questions.

Read more...


Using readline with Oracle SQL*Plus

By default, Oracle's SQL*Plus has no support for history, command line editing, etc.

This problem can be solved with the rlwrap utility. There is a Debian package, or you can easily compile it for any system.

Source is here: http://utopia.knoware.nl/~hlub/rlwrap/.

Once installed, add this alias to your .bashrc:

alias sqlplus="rlwrap -i -f ~/.sqlplus_history -H ~/.sqlplus_history -s 30000 sqlplus"

This defines a history file and uses it as your completion wordlist, and sets the history size to 30000 entries.

Before running it for the first time, do a:

touch ~/.sqlplus_history

That's all there is to it, and now you have readline support in your SQL*Plus!

Also check out another Oracle utility called yasql. You can find more information on it here: http://www.orafaq.com/wiki/YaSQL.

Read more...


My cmd.exe Profile

I wanted to set up a cmd.exe environment similar to my msys .bashrc for msysGit (http://code.google.com/p/msysgit/), to set up environment variables and doskey macros (aliases.)

This post will show you some techniques for doing this.

As Larry Wall said, "it's easier to port a shell than a shell script."

I use Console2 for my various command prompts (msysgit, powershell, cmd.exe...) you can get it here: https://sourceforge.net/projects/console/.

First, how do we tell cmd.exe to run a script on startup? There are two ways, the way I use is with the /K option to run a batch file and continue. Another option is to set a registry key, see cmd /? for what registry key to set.

Make a file called .cmd_profile.cmd in your %HOME%, then configure your cmd.exe tabs in Console2 to something like this:

c:\windows\system32\cmd.exe /K c:\Users\rkitover\.cmd_profile.cmd

Or make a shortcut on your desktop with this command line.

Or set that registry to the location of your profile script.

My .cmd_profile.cmd looks like this:

@echo off

REM Presumably we will only use real consoles with this profile.
REM And perl readline only works with TERM=dumb .
REM Set this to 'cygwin' to work with cygwin utilities.
set TERM=dumb

doskey use_strawberry64=^
    set PATH=c:\strawberry\perl\site\bin;c:\strawberry\perl\bin;c:\strawberry\c\bin;%PATH%^
    $Tcall remove_path_dups^
    $Tcall strawberry_maint_macros

doskey use_activeperl64=^
    set PATH=C:\Perl64\site\bin;C:\Perl64\bin;C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64;%PATH%^
    $Tcall remove_path_dups^
    $Tset LIB=C:\Program Files\Microsoft SDKs\Windows\v7.1\Lib\x64;C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib\amd64^
    $Tcall activestate_maint_macros

doskey use_activeperl32=^
    set PATH=C:\Perl\site\bin;C:\Perl\bin;c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin;c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools;c:\Windows\Microsoft.NET\Framework\v4.0.30319;c:\Windows\Microsoft.NET\Framework\v3.5;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\VCPackages;c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools;c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin;%PATH%^
    $Tcall remove_path_dups^
    $Tset LIB=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\lib^
    $Tset LIBPATH=c:\Windows\Microsoft.NET\Framework\v4.0.30319;C:\Windows\Microsoft.NET\Framework\v3.5;C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\LIB^
    $Tcall activestate_maint_macros

call exec_macro use_activeperl32

Some features to notice here:

line continuation

The line continuation and escape character for certain shell metacharacters in batch files is ^.

doskey command separator

The command separator in doskey macros is $T, see doskey /? for more information.

environment variable expansion

Environment variables are expanded immediately when the batch file is executed, not when the command using them runs. Here the %PATH% I use is the %PATH% my cmd.exe starts with, always. This is not what happens in my .bashrc, but it suits my purposes in this case well enough that I haven't fixed it yet.

There are a couple of supporting .cmd utilities to go along with this profile. I put them into C:\Users\rkitover\bin and put that directory in my %PATH%.

exec_macro.cmd
@echo off

for /f "usebackq tokens=*" %%x in (`doskey /macros ^| perl -ne "next unless s/^%*=//; s/\s*\$T/\n/g; print"`) do %%x
remove_path_dups.cmd
@echo off

for /f "usebackq tokens=*" %%x in (`perl -le "for (split /;/, $ENV{PATH}) { $p{$_} || do { $p{$_}++; push @p, $_ }; } print join q{;}, @p"`) do set PATH=%%x
strawberry_maint_macros.cmd
@echo off

doskey pclean=perl Makefile.PL$Tdmake realclean$Tdel /F MANIFEST

doskey pinst=perl Makefile.PL$Tdmake$Tdmake install$Tcall exec_macro pclean

doskey pprepare=rmdir /S inc $Tdel /F MANIFEST$Tperl Makefile.PL$Tdmake manifest$Tperl Makefile.PL$Tdmake

doskey pdist=call exec_macro pprepare$Tdmake dist

doskey pupload=call exec_macro pprepare$Tdmake upload
activeperl_maint_macros.cmd
@echo off

doskey pclean=perl Makefile.PL$Tnmake realclean$Tdel /F MANIFEST

doskey pinst=perl Makefile.PL$Tnmake$Tnmake install$Tcall exec_macro pclean

doskey pprepare=rmdir /S inc $Tdel /F MANIFEST$Tperl Makefile.PL$Tnmake manifest$Tperl Makefile.PL$Tnmake

doskey pdist=call exec_macro pprepare$Tnmake dist

doskey pupload=call exec_macro pprepare$Tnmake upload

These use perl, I'll fix them to use something native like powershell or wsh at some point, but I don't know those languages yet.

Read more...

Last modified: 2011-2-1 (火) at 6:10 am