What's on my Pen Drive ?

Updated December 2006

It's been a while since I updated this page, but I've been busy. Yes, a lame excuse, but how does this sound?

• A complete web server..
• .. with PHP 5
• .. and a SQLITE database
• .. and a query system
• .. and no software installation
• ... in under 20Mb ?

In fact you can plug into a networked (Windows) PC and all your friends and colleagues can share. Cool enough ?

Project B2 - Overview

As you know, a lot happens in the IT world in two years - software gets bigger, hardware gets smaller and cheaper, and pen drives are still pen drives.. My tiny 32Mb stick was still being used with all the good stuff featured below, but I figured it was time to do something a bit more whizzy. And so "Project B2" was born: it would be a set of software that would be useful and cool at the same time. B2 is based around the XITAMI web server software. It's small, cheap (free), powerful, and supports PHP. It's also got a GUI front end and ascii-text config files. Perfect.

Yes, I did try Apache as well, and yes, that would have given me the same functionality, but the vital difference was that on the Windows version of Apache at the time it wouldn't work for me without using drive letters, and if this was to be a truly useful tool it would need to be capable of being plugged into different PCs with wacky first-free-drive letters. Xitami it was then, and y'know I haven't looked back as it's worked extremely well..

So, it was all very well having a cool web server, but what could I do with it? Well hows about a database that I could link to using my favourite PHP? Again, the same technology decisions, and I ended up using the tried and tested SQLite binary/text database. Once again the advantage was the freedom from drive-letter slavery and it's also a very good piece of software. In fact it's so good i've used it at work for an internal system that keeps records of client software keys. It's also very fast once it's up and running, and I find that my browser frequently struggles to keep up with the volume of data that SQLite and PHP output. I've been testing my latest configuration over the past week or so with a book database containing 242,000 records that occupy a 40Mb database file and have to tell you it rocks! I'd put more records in but i'm having to borrow data and that's all I can get right now. Maybe more later.. As regards loading data into a database i've written a simple-ish Perl script which uses a SQLite DBI to load data. I'll publish that as well: it runs under ActiveState Perl for Windows, but as it's Perl it also works with *IX and Linux etc. Fast and reliable. And free. I like that word!

I've also devised an Isbn/EAN13 conversion script in PHP which will be particularly useful if you're in the book trade, otherwise not :-) Remember that January date folks!

As regards startup and sharing I tried to make it as non-technical as possible, bearing in mind that salespeople might be using it (no offence eh?), and although i've encountered varying degrees of success I can't get the thing to autostart the web server on insertion. The compromise is that once it's inserted and given itself a drive letter, double-clicking the drive letter will start the web server. Once nice feature i've devised is that will use Windows ipconfig utility to tell you the PC IP address, display it on screen and then paste it into your clipboard. You can then tell anyone else that wants to use it that IP address and port number and you're ready to rock.

So how does that overview sound? I'll be going through setting up the project on a step-by-step basis over the coming weeks, so stay tuned. If you want to contact me about it, please use this link.

Oh, and why Project "B2" ? Well, it's going to Be useful 2 someone (I hope)..

Updated again December 2004

From the Squeezing-blood-out-of-a-stone department ..  I've recovered over 40% of the space occupied by EXE files, thanks to a cool tool called UPX. Click here for more info !

Updated December 2004

Life on the move isn't all about meetings, troubleshooting and having large lunches (especially not in my case). In case you want to wind down in the evening, or just while you're waiting for that train, I've located some Flash games that are under 1Mb each, and are free. Click here to check them out..

Updated May 2004

As BT seem to have started chopping and changing my IP address about a bit recently it was becoming a real pain to remember to edit the batch files for the latest address, so I have automated it. Click here to see how it's done..

Updated March 2004

I needed some extra storage, so I splashed out on a 64Mb JMTEK Pro memory stick. The "Pro" means that you get software bundled with it :-) There's an email package that seems to work well, a security ZIP feature, a lock feature and a backup feature. The only one that seems to be truly useful to me right now is the email package. I'm not particularly keen on the lock feature as it seems to lock the PC's access to the pendrive, and not the pendrive itself. What I really want is to be able to encrypt everything on the drive itself without having to do anything on the PC - after all, that's the whole idea of portable computing.

The biggest problem I had though was to get Windows 98SE to recognise the pendrive at all. While the supplied mini-CD held all the drivers, installing them didn't seem to make any difference, and I couldn't find the answer on the JMTEK site. After soome determined Googling I found that a diagnostic tool: USBview - was to be found on the Win98SE installation CD under reskit\tools\diagnose. Running this showed the pendrive manufacturer to be TGE and you can download the driver from them. Please note that you still have to install JMTEK's drivers first - the TGE ones then seem to complete the missing link.

All in all not a bad device, but I personally feel that if you buy it for the bundled software you'll be disappointed. I know that all these devices are sourced from Asia, and I know that price margins are incredibly tight, but if they could just spend a little more on getting software properly translated, and really think about how the devices are going to be used, then they'd end up selling more. I want to be able to have a password-protected data area which is self-contained on the device - not dependant on what can be run on the PC. Maybe next year eh ?

Anyway, please read on ......

On the road

If you spend any amount of time on the road in the IT industry, you'll undoubtedly have equipped yourself with a software "toolkit" that you use on a regular basis. Those readers of a "certain age" will recall fond memories of carrying a box of floppies loaded with diagnostic and test software that would get hauled out at every site. They might also remember that the further the distance travelled the more likely the chance of floppy failure. Sod's Law in action. It was just part of the job.

Ancient History

A bit later on came the CD-ROM. This offered a lot more scope for carrying a large amount of software and floppies were consigned to the bin, although in the early days it was sometimes a case of turning up on site only to find that the PC in question didn't have a CD-ROM drive, even though the MD's PC did.. Ho hum.

Well, as you know over the years the CD-ROM has all but replaced the floppy and life is much easier. But sometimes it isn't. True, the CD is not affected by magnets, and there's no wear as such, but hey, you wouldn't believe how easily they scratch. I sometimes think that the CD-case manufacturers have a deal with CD-makers to sell more CDs this way. Also, as a generation of PCs gets older, their CD-ROM drives start to wear out, and guess what - CD-recorders will happily load 700Mb of data onto a CD, but old drives will not read right to the edge of the disc where that all-important bit of data is found !

Enter The Pen Drive

I can't remember exactly when the Pen Drive burst onto the market, but initially it wasn't widely used as USB ports hadn't started to appear on many computers. It didn't take long though for manufacturers to realise the advantages of the USB system (IMHO as important a development as CD-ROM) which started to appear on even the cheapest PC.

As an early adopter I threw caution to the wind and splashed out on a "no name" Pen Drive, with a stonking 32Mb of storage. I loaded the drivers for Windows 98 and plugged it in - instant, fast storage - very cool. Now.. what could I do with it ? I have to admit that in the early days my pen drive spent a lot of time in my bag just languishing: it was a solution without a problem. I later decided to load my installation software onto it and from that point it travelled all over the place helping me with software installations. Ironically it didn't get used as much as expected as my clients tend not to have recent PCs, so. back to the CD-ROM.

An Idea

Just recently, I read an article somewhere on the 'net in which an industry pundit suggested that in the (near) future, PCs would simply act as processing and display engines, and we would all have a memory stick with all our "stuff" on it. Well, hey, this was a great idea, but me being me I didn't want to wait for the future, so I set about looking for a way to do it myself.

  I determined that these would be my requirements:

1. A Word Processor - MS Word compatible - for writing reports;
2. A Spreadsheet Application - MS Excel compatible - for converting data in emergencies, and moer important, keeping my expenses up to date;
3. Secure Shell (SSH) application - to remotely manage my system;
4. Some sort of development software - I sometimes need to write ad-hoc stuff off site;
5. Web browser ?
6. Email client ?
7. My favourite "toolkit" applications;
8. Any client software I need to install;
9. Storage space, for "stuff".

In my opinion, not too much to ask as this is what I would normally need to carry around, but the challenge was to find software that was self-contained. It was important not to use any of the client's software other than Windows. It was also vital not to have to fiddle with the registry of any machine I was using, and a lot of companies are now denying users access to the registry, which is a very good idea. So basically that precluded the old "I'll just install this copy of word and write my report, if that's ok?". This was going to be a real challenge !

I initially looked at Java, but the storage requirements were just too great. Shame really as this would have a good way to do it. I then thought about DOS applications. There were a couple of things out there that might have fitted the bill, but really I couldn't rely on DOS support being present in Windows for too much longer, so that was ruled out. Next up was a set of applicaions for Windows 3.1: these tended to use INI files instead of the registry, but amazingly, anything useful was still being offered as either shareware or commercial software and my tight-wad principles prevented me from spending any money. So I was a bit stuck really, and I started googling to find a solution.

Eureka !

Half an hour later, I was starting to find applications. I'll go through the list above to tell you what I found.

You can click on any of the pictures to see a larger version..

Word Processor

Well, MS-word compatibility was important, so I downloaded a copy of MS-Word 5.5. What a great piece of software ! It's a DOS application but behaves very well indeed under Windows. The download comes as a self-extracting EXE (but can be renamed ZIP) and is about 3.3Mb in size. Bear in mind that once the file has been unpacked that even on a 32Mb pen drive you can still carry around all the setup files in case you need to add a feature ! Try doing that with Office 2003 !

The setup is very straightforward, and on modern PCs completes very quickly. It asks questions about modifying configuration files, so I'd suggest answering "no" to this. the only problem you are likely to have is getting it to talk to a printer, but if you just want to write a report and email it back to base (which is what I do) then this setup can't be beaten. You should also note that it offers a "Save As Rich Text (RTF)" option. This is a universal format and can be read and updated on a variety of non-PC devices, so think about using this as an on-the-road format.

SpreadSheet

Again, some sort of Microsoft compatibility was important, and my original idea of using something like the famous AsEasyAs DOS spreadsheet was still being sold as commercial. Besides, it didn't have an "Excel" save option. A bit more seaching revealed the oddly-named but powerful SphygmicApparently this means '..of or relating to the pulse' Spreadsheet application. This is in fact a 16-bit Windows application and is Excel 4 compatible. The nice thing about it is that although the installation program saves the DLLs and OCXs to the usual Windows folders you can move them back to then same folder as the application itself, and this fits in with my everything on the pendrive plan.

The Sphygmic software also saves to it's own .VTS format which as far as I know is not supported by anything else. It obviously doesn't do everything the latest version of Excel does, but it's not far off. It only emphasises the bloat that Microsoft have seen fit to add in order to "enhance" their product !

Secure Shell

In some respects, this is my most important application as it provides me with Secure Shell access to my own system. An added bonus is that it incorporates additional (separate) applications to provide Telnet and secure FTP. I use puTTY all the time, and it's never let me down: it's an extremely stable package.

puTTY normally stores parameters in the Windows registry, and it would appear from the web site that a number of people have asked if it's possible to run it standalone on a floppy disk. Well, it's possible, but there is a bit of a compromise. You have to run it with as many command-line parameters as possible, and also accept the default font and colour - a rather washed-out off-white Courier New (well at least it's not Times New Roman, thank goodness). the author publishes a couple of batch files that add and remove registry entries on the PC, but this goes against my "self-contained" philosophy. Here's how my solution works for me:

\SSH\puTTY\putty.exe -ssh -P 22 -a -x -2 -i \SSH\puTTY\your privatekeyname.ppk username@yourhost

.. and it's best run from a .BAT file. On starting up, puTTY will pop up a command shell window that in turn runs an SSH character-mode window. You can safely close the first window once you have logged in. I'll give the author Mr Tatham 9 out of 10 for this one !

Development Software

Oh boy, well this was a challenge, and it also hints at some of the old tools I use on a regular basis. I decided that if I had to develop software on the road it was only going to be for emergency use. We're not talking about a sophisticated GUI application here. Let's just say that if an important client had some valuable data that need converting I'd like to be able to do that rather than lose a client. Also, if a data extract needed to be done, then a small program that stripped out a couple of megabytes of garbage would be very useful. Yes folks, the software that I built my early career on is Borland's Turbo Basic 1.1 ! Now, I'm not going to post a download URL as it's commercial software, and it's registered to me, but it's vintage 1987 and it still works, and very quickly too.

Then there is the Pacific 'C' development environment / system for DOS. This is a complete system and has it's IDE and help system. I have to admit now that I'm no 'C' programmer but aided by the help system and examples I was able to write a small "stub" program to fix a problem that one of our users had.

Armed with Turbo Basic and Pacific C how could I go wrong ?

Web Browser

Why a web browser, you may ask.. Well, keeping strictly to the self-contained plan it seems a shame to have use someone else's Internet Explorer, and you have the added risk that they will be able to retrieve whatever it is you were looking at from the cache afterwards. Finding such an application proved quite difficult, and many of the so-called self-contained apps that I found actually use IE's own core, so finding "OffByOne" proved to be a real find.

It's completely self-contained, and actually supports SSL as well. The only minus point I found was that it attempts to write an INI file to C:\WINDOWS but I have emailed the authors to see if there is a way round that. It doesn't support absolutely everything that IE or Netscape does, but weighing in at just over 1Mb it's great for road warriors !

Email Client

While it's true that I could have used web-based email to send and receive email, it's often more convenient to use your own application if you have particularly sensitive content or want a local record of stuff that you send and/or receive. Once again, self-contained was the order of the day and I was fortunate to be able to find Ultrafunk's Popcorn.

This is freeware and supports one account. If you register it you can have unlimited accounts, otherwise the functionality is exactly the same. It's very cool as it will retrieve headers directly from your POP3 server and you can then decide whether to download the messages concerned. This is an ideal way of getting rid of SPAM - you don't actually read the messages so effectively the spammer has wasted his/her money because you never read it !

I use it with my Intech account and my Nokia 7110 - at 9600 baud time is money and Popcorn helps me in that respect.

Toolkit Applications

The majority of these I have picked up over time, but are useful to me on a daily basis. Apart from "list" it's all freeware stuff.

UPX.exe	Ultimate Packer for eXecutable
gzip.exe	GNU unzip application
sed.exe	DOS stream editor
list.com	Vernon Buerg's fantastic DOS/binary file display application
playwav.exe	A command-line .WAV player (who needs MP3s ?)
portmon.exe	Windows-based serial and parallel port monitor
superscan.exe	Port scanner :-) For testing vulnerability
Unix2Dos.exe, Dos2Unix.exe	Converts unix line endings to dos and vice versa
wmailto.exe	Command-line email client

Conclusion

Well, as you can see, even with all that software on the pen drive I still have over 40Mb free, which should be ample to write a substantial report, develop a few bits of software and maybe even store a few music WAVs to listen to.. This is only a 64Mb device and by the time I get round to replacing it the price of 128Mb devices will have dropped substantially as manufacturers get tooled up for mass-produced 1Gb versions.

Other exciting developments include rugged pens, which are waterproof and are able to be stepped on without damage. We're seeing pens that incorporate password-protected resizeable areas or are encryptable so that you can store confidential data on them. We're even seeing tiny pens that can utilise a Sony memory stick slot, and these can be read on non-Sony devices by using an adapter.

Finally, bootable pen drives are starting to appear so that in the near future you'll be able to boot into a copy of your favourite operating system straight from the pen. Or how about one that incorporates a web browser with enhanced privacy ! Now how cool is that ?

Talking of privacy, how about a pen drive as a Biometric Scanner ! So, what are you going with all your old floppies now that you've migrated to a little pen ? You could try here for a solution !

I created a folder called "IP" (for want of a better name). It might be important to keep it short as some PCs have a real problem with small environment sizes.

In that folder I created a file with a name of the IP address of my machine. This is a dummy example, not my real address:

..and here's the SSH batch file amended for dynamic IP:

cd \IP

for %%i in ( * ) DO SET IP=%%i

\SSH\puTTY\putty.exe -ssh -P 22 -a -x -2 -i \SSH\puTTY\your privatekeyname.ppk username@%IP%

Now how simple is that ? all you have to remember is that every time the IP address changes, you should delete the old address file in there and create a new one, then the for loop will only find one value.

I was actually looking around for some (Macromedia) Flash games for another project and found these at http://www.neave.com/games/. They fit my criteria admirably as they don't need any external files to run. They also don't take up a lot of space. The author states that ..These games were hand-coded by Neave and are free to use for non-commercial purposes without my prior permission. All copyrights belong to their respective owners, and not Neave Games... If you fancy trying some highly-addictive retro games, these are the ones for you !

Now tell me honestly, do you remember this, and if so, how much a week did you spend on it ?

Had an email this morning from one of the guys at the UPX team suggesting I might want to pack my EXEs to save space. They describe UPX as:

... a free, portable, extendable, high-performance executable packer for several different executable formats. It achieves an excellent compression ratio and offers very fast decompression. Your executables suffer no memory overhead or other drawbacks because of in-place decompression.

I duly downloaded the software and ran through my list of EXE files on the now uncomfortably full pen drive. It was as simple as upx -9 -v *.exe and this amazing tool had quickly shrunk my EXEs and saved me megabytes ! I was particularly pleased to see that the wildcard worked: it's nice to see developers thinking of real users :-) In all I've recovered a whopping 6Mb so now I can squeeze on a little bit more software or maybe even some more documents. Thanks guys !

In practical terms, not all EXE files could be packed. You'll see from the listing below that there are different types of EXE, and PLAYWAV.EXE (which is small anyway) was not a supported type. However, the bigger EXEs could be compressed and that's what mattered.

Read all about UPX and download it from http://upx.sourceforge.net/ !

UPX 1.25w        Markus F.X.J. Oberhumer & Laszlo Molnar        Jun 29th 2004



        File size         Ratio      Format      Name

   --------------------   ------   -----------   -----------

     78336 ->     31744   40.52%    win32/pe     FCODE32.EXE

     40448 ->     18432   45.57%    win32/pe     ftp.exe

     39354 ->     21678   55.08%     dos/exe     MUNPACK.EXE

upx: PLAYWAV.EXE: CantPackException: unsupported exe format

    237568 ->     90624   38.15%    win32/pe     PORTMON.EXE

    520247 ->    181248   34.84%    win32/pe     procexp.exe

     44032 ->     17568   39.90%    win32/pe     regfind.exe

    192512 ->     80896   42.02%    win32/pe     Regmon.exe

     22104 ->     12160   55.01%     dos/exe     SED.EXE

     98304 ->     44544   45.31%    win32/pe     Tcpview.exe

    188416 ->     86528   45.92%    win32/pe     TDIMON.EXE

    407552 ->    162816   39.95%    win32/pe     Unix2Dos.exe

     91648 ->     38912   42.46%    win32/pe     WMAILTO.EXE

   --------------------   ------   -----------   -----------

   1960521 ->    787150   40.15%                 [ 12 files ]



Packed 13 files: 12 ok, 1 error.

Before UPX:   Total Files Listed: 73 File(s)     15,360,945 bytes

After  UPX:   Total Files Listed: 74 File(s)      9,605,256 bytes

Other Links!

We had some comments at http://www.metafilter.com/mefi/31640

© Peter Garner, 2004