Thanks for posting your code!
Some things to think about for your encryption algorithm:
The absolute maximum number of possible encryptions you can do is 2^32 or about 4billion because you 'only' have 32 bits of seed. That might sound like a lot but consider that modern ciphers will use 1024 or more bits of key, each bit doubling the size of the problem.
The rng in blitzmax is probably not that good and wastes bits of your key - for instance if it uses the standard C runtime rng as described in the C Standard, then there are only 32768 possible keys. With your decryption program I could generate all the possible decryptions of your file in just a few milliseconds! If I had any idea what was in there, I could easily then write a program to find the ones I'm interested in, eg. those with plain text or those with jpg headers in them.
Also, if the rng uses a poor algorithm, then I can encrypt a lot of pre-known text (something like 'the quick brown fox' a hundred times, or a million zeroes) with your algorithm using different keys and do some statistical analysis on the resultant files and from that work out the fact that many rngs have very short repeating loops in the cycle of numbers that they generate. From that it's reasonably easy to deduce the algorithm.
Another problem is each time you call the rng, you discard all but the bottom 8 bits (0-255) of the result. Again if the rng algorithm is poor you might find those 8 bits are not that random.
A whole field of the research in cryptography is to build rngs with bigger spatial qualities and keys so they can't be analysed in this way.
Please don't take this as criticism, take it as food for thought to give you some ideas how even with the tools at hand you can quickly improve your program. It's already easily good enough to deter any casual hackers!

Nick, can you remember Parabellum's Blitz encryption code?
Jim