Python to Google Sheets to Unity to JSON resource to runtime - nothing but net

I just uploaded a Unity app at http://bigdicegames.com/CWGSparks/index.html that draws this picture:

Which, if you've been following along, and are a little forgiving with my projection, could look like a highway map of the US. It's incomplete, and it's low res - the only vertices are at cities, and I have just shy of 500 cities in my database. Also, I've been manually connecting up cities based on my interest in them, so there's a lot more cities in Texas, Florida, and California, and a lot more highways that will someday be coming in to this picture.

Previous posts have talked about doing pathfinding from city to city, and I've been using Python for that part of the project. To help me visualise what's going on, I use Reportlab to draw PDFs, which sometimes I print out.

This is all intended to be in service of a game. (A computer game? Maybe playable on the web? I also have ideas of a choose-your-own-branching path gamebook, but that doesn't require as much pathfinding.) So, it makes sense to start porting the data into Unity.

In talking to my friends about this project, I talk about having a highway database. That's clearly putting on airs, as really what I have is two Python arrays of tuples. One array is the city array, and each line in the file has a city name and lat/lon coordinates. The other array is the road array, which links up cities by name. Simple as that.

Well, yeah, but a little gross. There's no error checking, and it's super easy to get duplicates in there. Greeley, Colorado was in my "database" twice, with two different spellings. That seems like it's cleaned up now.

Python's great, but it's not the best data repository. I've been kind of itching to move my data to a Google Sheets spreadsheet. This ties in with some desire I have to try to embrace a data-driven, "Entity/Component/System" (ECS) style for a personal project of some size.

Aside: http://bigdicegames.com/ECSteroids/index.html is a small game I made, trying to play around with ECS maybe a year and a half, two years ago. arrow keys steer, CTRL shoots.

So, anyway, I wanted to port my Python-based data to Google Sheets. That's super easy, I just wrote out a CSV file by hand for each of my tables. If I actually had to think about commas inside my data values, I'd probably go so far as to import csv from Python and use a writer provided. But no time for that.

So, I imported my CSV into a Google Sheets, um, sheet. Looks great, all my data is there, and even some fancier stuff, where I generate a unique key for city and roads, as if I was using a real database.

For what it's worth, a lot of those cities were scraped from Wikipedia's article on most populous cities in the US. So that got me started, but I've been adding extra cities to give my roads useful places to turn. The original Wikipedia article had 314 cities, and I've been adding extra places so that now the spreadsheet has 498 lines. Anchorage, Alaska and Honolulu, Hawaii are problematic, but I'll get to that later.

So, the data's in Google's hands now. Fine. I want to be able to use it inside Unity. I poked around a number of similar tools, and ended up using Google Sheets for Unity Lite, which does the job I need right now. There's a non-lite version on itch.io, but it's unclear to me what the value added is.

Watching and rewatching A YouTube video where a guy is using GSFUL to import data, I managed to pull the data out of Google Sheets, and down into JSON data in my app. Cool, so far.

Except I don't really want the users to be hammering Google Sheets to pull data at runtime. I'd much prefer to bundle that data into the app itself.

Unity has several ways to make this sort of thing work, but what I chose to do is that my Google Sheets pull happens as an editor extension, pulling the data as JSON, and then I write it into my Assets/Resources/JSON folder. "Resources" are data that get bundled up inside the app through Unity magic.

A little dead end banging around, trying to read from the Resources folder directly, I ended up using Resources.Load<TextAsset>, like so:

And the JSON is now in a CityArray and RoadArray object, each of which is a simple (if annoying) wrapper around the array of JSON objects for cities and roads. Boom.

At that point, I just used Vectrosity to draw the lines. There was a little bit of math to jam the map onto the screen rectangle, but that's just a bounding box and some scaling. Nothing fancy.

I sort of want to put some virtual cars on the roads driving from state to state. (No papers, Vasili!) That can come later.

Also, maybe a little more Texas, it looks lonely.

I agree with Google Maps on Cannonball Route

As a test of my contraction hierarchy code, and of an expanding database I'm building of US highways, I asked both Google Maps and my code to find a path from New York, NY to Los Angeles, CA. 

Looks like Google recommends going through Indianapolis, to Oklahoma City, then due West.

 My map isn't as extensive as Google Maps, but let's see what it decides to do:

New York, NY -> Indianapolis, IN -> Oklahoma City, OK, and on west to Los Angeles, CA. 

Better than I expected. I'll take it!

Contraction Hierarchies vs A*

You may recognize the above map as a crude highway map covering (parts of) California, Nevada, Utah, Wyoming, and Colorado.

You might even have heard of a project that I've got on my back burner, a branching, choose your own path, adventure book and/or computer game, which I might be calling "Sparks and Rusty" or "Sparks and the Wheelman" or some other goofy pairing that evokes 1980s action road adventures.

In this adventure, you have a semi tractor, and are asked to use it to tow a semi trailer from Mountain View, CA to Colorado Springs, CO. The interesting (hopefully) bit is that the semi trailer houses a supercomputer, on which is running a super-intelligent, sentient AI. That's "Sparks".

So, yeah, get from point A to point B, with your newfound buddy / damsel, across presumably hostile territory.

Also, as this is a Cars With Guns joint, the hostile territory will have lots of violent folks on the highways. Murderous cycle gangs. Folks with machine guns mounted on the hood of their cars. Ambushes. Gorse bushes. Perhaps not the last one.

If you're familiar with the Steve Jackson Games gamebook "Convoy", you're thinking along the right tracks - it's an inspiration, to be sure.

So, you've got a map, and you've got a destination. It should be easy to pathfind to the destination, right?

Well, sure. Let's first of all get out of the way that my map is fairly small - 70 cities, some of which I added, just to make the highways bend in approximately the right places (I'm looking at you, Muddy Gap, WY). So, it's not too hard to exhaustively search that, Dijkstra is actually pretty good for this sort of thing.

Also, let's concede that my city-to-city distance is an approximation, using pythagoras and a mercator projection onto a flat plane. Real highways are wigglier, and this is coming up on big enough to actually care about the curvature of the Earth. All of that can be refined later, if I need to.

So hey, A* (pronounced "ay-star"), that's a thing, too, right? Yep, sure is. Can be faster than Dijkstra, and is fewer characters, though it might be problematic as a filename, or a class name, depending on your OS and your language.

So, I wrote a quick little A* implementation - I've got "AI Engineer" on my resume for a reason, I should be able to knock this out in a page or less of Python, right?

Turns out, sure, something like that. And A* tells me that the way to navigate my map to get from Mountain View to Colorado Springs is:

['Mountain View, CA', 'Pleasanton, CA', 'Walnut Creek, CA', 'Vacaville, CA', 'Sacramento, CA', 'Placerville, CA', 'Reno, NV', 'Ely, NV', 'Cove Fort, UT', 'Green River, UT', 'Grand Junction, CO', 'Rifle, CO', 'Lawson, CO', 'Denver, CO', 'Colorado Springs, CO']

Which is maybe not what I would have chosen, if I was just wandering the highways of the Western US without a map by my side, I might have gone through Reno, staying on I-80 through West Endover, and turn right at Fort Collins, CO, and down to Colorado Springs. But I trust this is shorter. It doesn't take cycle gangs or refuelling into account, which players will need to think about.

Sidebar anecdote: years ago, my father asked me how Google Maps can do continent-scale navigation so quickly. I shrugged and said I didn't have access to the Google Maps source code, but I imagined that if somebody asked Google Maps for turn-by-turn navigation from, let's say Bremerton, WA to Boston, MA, Google Maps would be smart enough to know that you can get from Seattle, WA to Boston, MA, by taking I-90 (pretty much) without turns, and so that's a subproblem that can be "shortcut". Not saying that you're taking a shorter physical path, but not considering all segments of I-90 incrementally; that's thousands of potential exits off the interstate that you might not care about.

And, in fact, you actually can do a little better than saying on I-90, if for no better reason than to detour around Chicago and Minneapolis. At least, that's what Google Maps says right now. So, it has some knowledge of times when the interstate isn't the best answer.

I've taken some online courses to keep my brain full, and one of the AI courses I took had a short section on highway navigation, with a visiting guest from Microsoft/Bing Maps, talking about "Contraction Hierarchies" / Node Labels.

The short version of the "Node Labels" technology is that you store at every location a "label", which I'll just call a "dictionary" of intermediate locations, along with the shortest distance to that intermediate location. The amazing bit is that this dictionary ends up averaging something like the log of the number of points in your map. So, I look up Bremerton, and find a bunch of cities including St Louis, New Orleans, and Chicago. I look up Boston, and find a different bunch of cities, including New Orleans, Memphis, Miami, and Chicago. And so I find the intersection of these dictionaries, and find that I can route from Bremerton to Boston via Chicago in 3096 miles, or Bremerton to New Orleans to Boston in 4250 miles. I'll take Chicago today, thank you.

If you've got these labels preprocessed for a bunch of locations, you can do this midpoint thing very fast.

I'm a little fuzzy on what the Microsoft guy said (if he did) about how you turn this in to navigation - maybe he was just presenting a "distance oracle", which is still something.

Maybe you recursively do this, and find a midpoint between Bremerton and Chicago, and so on and so on. I'd be concerned, though - in my example, Chicago is in my "label" for Bremerton, so there might be edge-case issues going on, where I can't get a good divide-and-conquer solution.

Or, maybe inside your label, you store not only distances, but a path. That's a lot more data.

Or, instead of storing all of this label information, you store some small information in the cities on your map that helps you reconstruct the label quickly. That seems good, especially since I'm not actually in the business of doing server-side computation of this stuff for lots of queries per second.

And it's this small information approach that Contraction Hierarchies uses to give you a solution.

Imagine I assign each city in my map a unique integer index. Could be assigned randomly, could be alphabetical, it doesn't matter at this point how I come up with these indices. Now, in that order, I'm going to start simplifying my map, removing one city at a time, which should make things incrementally simpler. But I still want the remaining map to contain correct information about shortest paths. This is where "shortcuts" come in. Again, shortening the computation work, not shortening the physical, map, distance.

So, let's say I decide that I'm looking at my map, and I decide to simplify it to remove Bozeman, MT.
May map says that I can go from Belgrade, MT to Bozeman, MT, and from Bozeman, MT to Livingston, MT. I would add in a shortcut from Belgrade to Livingston, and add a note on the shortcut saying "also, to take this shortcut, you'll pass through Bozeman". And we do this for each city in our graph, adding shortcuts where the city is required for shortest path calculations to remain correct.

So, we make a new graph, which we'll call G* which has all the original cities and all the original edges, but also these new shortcuts. Depending on the connectedness of your graph, a lot of cities can go away and not have any shortcuts at all.

The neat trick at this point is to navigate G* by only going "up" in indices, including using our new shortcuts. So, for a known good path from Bremerton to Boston, we will have contracted the various cities along the way in some order, which you can visualize as a jagged mountain range - each contraction yields a shortcut, allowing us to "fill in" a little valley between neighboring points (that are later in the contraction order, so therefore are "higher" on our mountain visualization). So, the points late in the contraction order become "hubs" that traffic want to go through.

So, we figure out these best hubs, reachable from our start location and our destination location, each going up. In the Bremerton and Boston example, we can imagine that Chicago is a good hub (O'Hare is maybe not a great experience for air travellers, but as a highway crossroads, it's useful), contracted late in the preprocessing, and so "up" from both Bremerton and Boston.

And the path to get from Bremerton to Chicago has a few shortcuts along the way, probably including when we contracted Bozeman, so when we come to unroll our directions, we replace shortcuts with paths through their contracted cities.

And that's just about it.

So, I wrote a Contraction Hierarchy solution for my game map (Mountain View to Colorado Springs, if you recall). I put in Mountain View, and asked it where I could get to going "up" my graph, and it said:

St. George, UT
Walnut Creek, CA
Bakersfield, CA
Modesto, CA
Salinas, CA
Vacaville, CA
Paso Robles, CA
Fort Collins, CO
Mountain View, CA
Reno, NV
Pleasanton, CA
Tonopah, NV

I asked it where I could get to by going "up" from Colorado Springs, and the list was:
Denver, CO
Cove Fort, UT
Pueblo, CO
Colorado Springs, CO
Walsenburg, CO
Rock Springs, WY
Salt Lake City, UT
Fort Collins, CO
Reno, NV
St. George, UT
Tonopah, NV

For each of these destinations, I got the city, a real highway distance, and a path with shortcuts to get to that city.

So, I found the places I could get to from my start and from my destination, which narrowed things down to:

(1092.933032328154, 'Reno, NV')
(1187.1179961380471, 'St. George, UT')
(1329.8134461278964, 'Fort Collins, CO')
(1570.1329202911515, 'Tonopah, NV')

That's the cities along with their combined distance (start to city to destination), so it looks like our travellers are going through Reno. My two graph searches give me shortcutted paths, which combine to look like:

['Mountain View, CA', 'Pleasanton, CA', 'Walnut Creek, CA', 'Vacaville, CA', 'Reno, NV', 'Cove Fort, UT', 'Denver, CO', 'Colorado Springs, CO']

which seems about right, though Vacaville to Denver is a pretty short distance on the page, and a pretty big chunk of our travel distance. That's the shortcuts, so let's unwrap those:

['Mountain View, CA', 'Pleasanton, CA', 'Walnut Creek, CA', 'Vacaville, CA', 'Sacramento, CA', 'Placerville, CA', 'Reno, NV', 'Ely, NV', 'Cove Fort, UT', 'Green River, UT', 'Grand Junction, CO', 'Rifle, CO', 'Lawson, CO', 'Denver, CO', 'Colorado Springs, CO']

Which is the same solution that we got from A*. This is reassuring. If they were different, I'd go back and try to figure out what's going on. I can imagine that if we were trying to find shortest paths across Manhattan, there might be many routes with similar (map) distances, and in that case, maybe A* and CH would give different answers. One thing about A* is that there's a heuristic function that is used to prioritize expanding paths based on expected distance remaining, and if I got that wrong, I could see A* giving a slightly wrong answer.

But they're the same, and they're both lightning fast on my map of 70 cities. I would expect that if I had a lot bigger of a map, I might start caring, but for my game, it probably doesn't matter.

One thing that might make me go for A* in my game is that my map might be dynamic enough that I won't want to redo the processing - let's say our heroes hear that a bridge is out, and the road from Reno to Rachel, NV is blocked. (Or, maybe there's some other reason why the U. S. Army is rerouting traffic around Rachel?) In that case, using A* on the (dynamic) map data might be more convenient.

Spoiler: it's probably aliens at Area 51. Maybe some got out, maybe the army is bringing new ones in, maybe there's a shipment of alien materials that overturned. I've calculated a path from Rachel, NV, to Devil's Tower, WY, just in case.


This stuff isn't super hard, but it does take a little work to get right. I watched (and re-watched, and read and reread the slides from) a German AI course on navigation:

http://ad-wiki.informatik.uni-freiburg.de/teaching/EfficientRoutePlanningSS2012

Particularly, lectures 6 and 7.


So, Dad, if I wanted to go from Bremerton to Boston, or vice versa, I'd use shortcuts.

On making a book, when one isn't really the author - the process behind "Shapeshifting"

Over the summer of 2019, I put together a book, using OpenAI's GPT-2 algorithm, as running on http://talktotransformer.com . It's currently available in physical and electronic formats, and literally dozens of copies have been printed, and over a hundred copies have been downloaded. These are not terrific numbers (yet), but just getting the book onto Amazon is an accomplishment.

This post talks about much of the process in making this book, including many of the pain points that I would try to avoid if I did something similar again.

The Influences

I've been trying to go back and identify what pieces other people's AI projects banged together in my head to spark this process. Janelle Shane's "AI Weirdness" blog was certainly one piece; she posts an experiment each week, ranging from naming ice cream flavors to Dungeons and Dragons spells. So that feels like it probably set my mind in motion.

I also read about "Prismatic Corpse", a rewrite of D&D crowdsourced from group memory. I signed up to be a part of that game jam, but didn't get around to submitting anything - probably because I decided I was too deep in working on the book by the submission deadline. Also, I was concerned that the participants might be offended at my use of AI in their presumably human project.

Around the same time, I read "Maze Rats", a print-and-play RPG that's super light, with a lot of mileage from a couple of 2d6 table lookups.

Also, years ago, I read some of John Hodgman's books, including one with hundreds of hobo names. Cumulatively, this stuff's a riot.

I've also had a long-term project simmering on the back burner, to make a computer role-playing game, and that's a somewhat terrifying project, involving many moving parts, and lots of different dimensions of creativity.

The Book

What I ended up delivering, and this feels out of order, but it helps to understand where I ended up, which was roughly where I was aiming. With this in mind, a lot of the rest of the process makes a little more sense (and I'll take whatever sense anybody can find).

The final book is over 100 pages, with a cover that evokes mid-1970s role playing games, and internal content looking like early 1980s rulebooks. I cut my RPG teeth on J. Eric Holmes' Basic Dungeons and Dragons (the edition with a monochrome blue cover), which was a step up from the "little brown books", themselves a rewrite of Gygax and Arneson's home rules.

In my experience, each of these are pretty spindly skeletons, evoking what role playing could be, rather than prescribing how you had to play the game. The players (including the dungeon master) had to contribute a great deal to make the experience work.

I find myself describing Shapeshifting as somewhere between a parody and an homage, something not really a rules system, but more than just an accessory or players aide. It's probably correctly placed in some gray area between each of these points.

There are descriptions of attributes, including strength, dexterity, and luck - but no real rules on what those attributes do in gameplay. There's no discussion of whether you roll 3d6 six times in order, or if you do some sort of point-buy system. Somehow, attributes are important, though.

Also, there's discussion of different races, from Humans to Elves to Dwarven to Half-Elk to Half-Halflings. No stat modifiers, just flavor text. Usually, when I encounter flavor text in a game, it's the stuff that I ignore, but with Shapeshifting, it's 100% of the book.

Well, not quite 100%. There's a few tables (hat tip to Maze Rats) for weapons, armor, monsters, and magic items. If you get nothing else out of the book, maybe a big table of over 1000 monsters is useful to you. Granted, a lot of the monsters are weird, so be prepared for the tone not to match your existing campaign. (If my monsters fit into your campaign, write me and let me know, I'm very interested.)

In order to make my tables line up on clean page boundaries, I got a bunch of stock art (some free, some paid for) and used it to shim the otherwise ragged ends of my pages.

The Technology 

Most of the technology required for generating the text came in the form of OpenAI's GPT-2 text generation engine. This is a neural net that was trained on a corpus of text found on the Internet, found by following links from any post on Reddit with three or more upvotes.

The text generation software is what's called a "Transformer" (not the Cybertronian robots), and I don't entirely understand how it works. But you don't have to! Adam King posted an implementation of the algorithm at talktotransformer.com that allows a user to prompt the AI with a sentence, a list, or even just a sentence fragment, and the AI will continue the text from where the prompt leaves off.

So, I entered an awful lot of half sentences that I figured would belong in a RPG book, like "Elves are a race that..." and "When a player has initiative...", and the AI would give me a couple paragraphs at a time of text more or less on-topic. Mostly not on-topic, so I'd try again. The stuff that seemed usable would go into a big text file, and I'd keep going. In time, I had enough for a few chapters, and I'd focus on another part of what I needed for my book.

For the big lists (again, thanks to Janelle Shane), it turns out that GPT-2 loves making lists. So, I'd give it a list like:
1) sword
2) dagger
3) mace
and the AI would proceed to give me a long list of weapons, complete with numbers. The ordered list tag was one of the first things I learned when learning HTML, and it's strongly represented in GPT-2's training data, it would seem.

I would categorize stuff that worked - sometimes I'd get armor in with my weapons, or magic items in with my monsters, but collating and curating was easy, if a little numbing.

I proceeded to organize my lists to try to help the de-duplication process, as well as giving players a little opportunity to fudge the rolls, or roll on a convenient neighborhood (rolling 2d6 around the shark section of monsters, maybe).

The Formatting

Having most of the text, most of the lists, I opened up Scribus, an open-source desktop publisher, and proceeded to dive in to doing layout. I briefly considered using Python tools to generate a PDF by hand, but I decided that I wanted more variability and uneven layout, which Scribus afforded me. Here, I was aided by some documents by Sine Nomine Publishing that talked about how TSR's layout changed over time. Turns out, there was a lot of variability, so I felt free to take inspiration, rather than follow a strict set of layout rules.

Scribus makes it easy to create layouts of page layouts to reuse (heading here, page number there, column A, column B), which went a long way to making stuff flow together with a minimal level of cohesion and professional layout.

One of the tricky bits that I wanted to get right was to have inline dice images on all my dice tables. I was prepared to use my copious drawing skills to generate dice images, but I couldn't figure out a way to get Scribus to flow an inline image as part of paragraph text. (If you've ever written tutorial text for a console game, and are required to embed an icon of the XBox "X" button, you'll know what I'm talking about.)

I never figured out embedded images, but what I did find was a font with dice, so all I needed to do was to dump my table content into a text file, run a python script on it to turn it into a CSV with the die roll "index" in the first column, and the rest in the next column - except I didn't end up using commas to separate the values, because I had some phrases with commas inside them. I think I ended up using tab delimiters.

I then wrote a Python plugin script to import my CSV (TSV?) in, line-by-line, switching to the dice font as needed, and then back to the body text font. This took a few seconds to run for the longer tables, but was basically painless.

The Last Bits

I knew I also wanted to have a sample adventure, which included a right-angles corridors and rooms level, and a caves and caverns level.

For the corridors-and-rooms level, I wrote a Python script inspired by a lot of "Wave Function Collapse" samples that I've seen around, which honestly, I find a little annoying, because I was doing this sort of stuff decades ago, calling it "constraint propagation". The fancy bit that WFC has going for it is that, if you do it right, you can give it something that looks like your desired output, and it magically extrapolates that single input into infinite outputs. Sort of. It works better if your input is easy to parse into tiles, and then it's able to identify duplicated tiles, which lets it infer the relative frequencies of which tiles can go next to which tiles.

I tried a few existing implementations, and didn't get anywhere, so I wrote my own constraint propagation implementation from scratch, using a set of hand-drawn tiles in a little grid journal I happened to have with me when I was waiting for my car to get serviced.

Boom, instant dungeon crawl. Except that I had Escher-like stairway loops that bothered me. I proceeded to add more information to tiles, including that this stairway tile had an entrance that might come in on level <x>, and then the other side of the tile might exit on level <x+1>. Which was more or less fine, but the constraints were a lot tighter than most WFC samples, so I'd get trapped in inconsistencies, and the system would recognize a dead end and give up.

I tried adding in backtracking, but that wasn't going well, either. I used up lots of memory and lots of time, and still wasn't getting good results.

So, I just threw away all the stairway tiles. I sort of miss them, but not enough to go back to rewrite my dungeon generator right now.

After getting the dungeon generator working, I wrote a super simple generator that made ragged shapes that I assert are caverns, and then the generator connected them with jagged shapes that I tell you are cave passageways. After the long fight with getting the corridors-and-room level working nicely, this went super quick. Or, maybe, my standards had been lowered. But it looked like a lot of what I've seen as one page dungeons. So, in it went.

The Publishing

At this point, I had a PDF (really a couple different PDFs, one that had the whole book, including front and back covers, and fancy maps on the inside covers, one PDF with just the outside cover art, one PDF of just the inside) and I was ready to start uploading to places to get them into the hands of people.

I uploaded to DriveThruRPG, an obvious place for a person with an RPG PDF that they want to get into the hands of people who might want to read it. This was right around the time of GenCon, and they said "hey, we'll review your submission, but c'mon, it's GenCon", which was totally fair. I marked it as "pay what you want", with a suggested $5 price tag.

And then I turned my attention to Amazon. I pushed my PDF(s) up to them, going through their multi-page submission web wizard process. Seems like I got pulled backwards and re-entered information more than once. It's Amazon, they aren't in the job of making good web UX, right?

And, I got to the end of the flow, and got a button to request my submission be reviewed. So, bam, sent it off.

And while I waited for those things, I thought about maybe also doing a Kindle eBook. Turns out, you can upload a PDF which they'll turn into a Kindle eBook, but they prefer you upload a proprietary Kindle formatted version, to take advantage of the Kindle platform's features. Which makes sense, just feels a little up-sell-y.

Oh, and to author the Kindle formatted version, you have to use their desktop software. Maybe there could be a web version of it? But it's Amazon, they aren't in the job of making good web development tools. So, I downloaded the Windows version of the software. Onto my Linux machine. I didn't really expect it to work. Also, it did not work. So, I dusted off an underpowered, overused Mac laptop, and downloaded their Kindle authoring tool for that. And cut and pasted the text in, bit by bit. The pictures mostly didn't survive, but most of the pictures were there to make the page layout fit well, so in a land of auto-flowing text, I let go my responsibility for worrying about page breaks.

I did keep my two dungeon maps - they were more important to the content (in as much as anything really is important). I did a pass to add in keyword linking, because it's a Kindle feature.

And I uploaded that version. And, perhaps not entirely surprisingly, it was the Kindle eBook that got approved and available for downloading first. A Friday in early August, if my memory serves me, which I don't actually trust. And not too long after, the Drive Thru RPG download went live. The paperback version went live, long enough for me to order a copy, and then it went to "ON HOLD", with a little message "to find out why your book is on hold, contact us", with a link going to a big page of FAQs about the publishing process, but not any super obvious discussion of what would make a book be put on hold.

I figured some flag got tripped somewhere in the process, perhaps having to do with DPI settings or margins. I found some means of contacting the customer service team, and asked them "hey, the UI told me to contact you, what's up?", and the customer service team said "oh, hm, we'll have to check with the tech team. We'll get back to you in like three days?". So, sure. I sat and waited, and got a response from the customer service team, relaying a message from the tech team saying "we can't print the book until he replaces the Quentin Caps font". Which, all right, I get that fonts can be tricky. No discussion about what was tricky about this particular font. Maybe it was a bad format, maybe my use of it wasn't clearly within the rights of my license, so I found a similar font, paid a somewhat reasonable amount for that font, and re-submitted. And waited. And after several days, I reached out again, asking why my book was still on hold. And again, customer service had to contact the tech team to find out why the book was stuck. And again, it was a font issue. This time, my new font. Still no indication about what the issue is, or how to fix it. No guidance on what to do other than to replace the font.

So, and if you've read this far, which presumably you have, or maybe you're skimming, because it's been a lot of words up to this point, maybe this is the one takeaway that is of use. When you're working with a print on demand service, and you give them a PDF, you can embed fonts in the PDF, or you can convert fonts into outlines. Or, and this is the grossest option, you can take the fonts into let's say Photoshop, make an image (maybe a PNG) of the text, save that out, pull it in to your desktop publishing app, and paste the image in place of the text using the problematic font.

This is bad because it increases your filesize. It's bad because the layout is all kinds of sloppy. It's bad because it's caving in to a workflow that flags problems without offering solutions.

But it worked. My book is (currently) in print. You can buy paperback or two different formats of electronic versions. Or all three. You can send copies to your favorite game master for the holidays.

I'm tempted to make a Kickstarter option for the boxed set. I haven't fully imagined what it would entail. A box, certainly. More art? Better art? A bigger adventure. Better rules? Little cardboard standee figurines? Dice? Seems like at a minimum, a project this random needs some dice.

People ask me if I've played the game, and I laugh at them. This isn't a game for playing, this is a book to laugh at. Or, if you do play it, let me know. I'd be delighted to hear.

Taking off the "editor" hat, putting on the "publisher" hat

It's not like my projects really have well-defined role boundaries; I take on projects where I get to / have to do a lot of different things. For this book, I did a small amount of writing, more editing, some art direction, page layout, and now I'm shifting in to the boring tail end of finishing up a project.

I don't mind the tedious tasks at the end of this project very much, because it marks an actual finish line, actually completing a thing, which I'm not very good at. I start a lot of things, and I get distracted, hey! Let's ride bikes!

I uploaded some version of the "release candidate" PDF to Drive Thru RPG last night, some version to Amazon this morning, a slightly different version of the PDF to FedEx/Kinko's last night, and now, to reference Alton Brown, I just walk away. This book is like scrambled eggs. Just walk away. Let it firm up, let it rest, let the heat even out so that the center is done.

Drive Thru RPG says that it'll take a while to validate my upload, and I imagine this week is worse than normal, because GenCon is right around this time.

Amazon says it may take 72 hours to do their verification, which takes me into the weekend. Maybe into next week, depending on how those 72 hours fall.

It feels good to be powerless, in a really weird way; again, this is the finish line of this project. Just walk away.


Oh, also, shout out to Danielle at the Woodinville, Washington FedEx/Kinko's for making sure that my "release candidate" draft looks the way I wanted it. It's not important, but I'm glad to have a one-off edition that I'm proud of.

Constraint Propagation, "Wave Function Collapse", Dungeon Generation

I'm still trying to get my book "Shapeshifting: The Dream of Dagrec's Legacy" finished by the end of the summer, and if at all possible, available by mid-August.

The book is intended to be an homage / parody / satire / reflection on old game systems, old game books that were evocative, but not actually sufficient to be played as written, leaving players to come up with house rules to fill in the gaps.

Also, it's being largely written by AI. More on that in a later post.

I decided that a relevant thing to add to the book was a generated dungeon. The 1st Edition Dungeon Master's Guide had some tables to help you generate a random dungeon, and there were Dungeon Geomorphs, which were map tiles that you could lay out in random assortments, if you liked.

And, of course, every roguelike that's sufficiently like Rogue will have some sort of dungeon generation system, which typically separate the world into rectangular rooms connected by a network of tunnels.

A while ago, I stumbled across people getting excited about "Wave Function Collapse", a "new" algorithm that can be used to generate plausible content, given a small test sample. The animations I looked at were pretty, but the algorithm seemed to be a repackaging of constraint propagation, which I had been using for content generation for years, and had first seen in my AI class back in 1991, where the technique was used to find a consistent labelling of edges for doing image comprehension on a set of edges representing a computer vision view on a world of blocks.

The one thing that looks like it's different in WFC is that the frequencies of patterns are recorded, and when a pattern is matched as a basis to place new content, the input frequency data is consulted to pick a plausible successor. For example, if my input had straight corridors followed by more straight corridors 9 times out of 10, and a turn 1 time out of 10, I could use that to generate corridors that resembled the input by rolling a d10 and placing a turn 1 time out of 10, otherwise continue the corridor going straight.

One hesitation I've had about the various WFC implementations I've looked at is that they largely don't implement backtracking, meaning they can get stuck in dead ends. This is apparently not a show-stopper for people to use these implementations - the algorithm seems to get stuck rarely, and if it does, just restart, and it'll be fine. Typically.

So, I wrote a simple WFC / constraint system, without implementing backtracking, and it was generating dungeon maps pretty much to my liking, resembling the image at the top of this post. One feature I had in my tileset was stairs, which I didn't give the algorithm enough information to really deal with well. The generator would create loops with 3 stairways, all going down going one way around the loop. If you're familiar with Escher's "Ascending and Descending", or the Penrose stair illusion on which it's based, that's the same idea. Not really something I wanted accidentally appearing in my dungeons.

I added in some elevation tags, and I still ran into problems. Now, what was happening was that the generator was making portions of the map at wildly different elevations, and couldn't figure out how to merge them together. And, without backtracking, there'd be no way to adjust the placed tiles, so I'd have to restart. And restart. And again.

So, I rewrote the generator from scratch, this time with backtracking. One reason I had hesitated implementing backtracking myself, and why most of the WFC implementations avoided it, was that there was a lot of partial solution state that one kept around in a backtracking solution.

But I dove in, and got a backtracking placer working. It was slow, because of all the copying I was doing. And I was doing it recursively, so I was using up the program stack. And this began to be gross, so I unrolled my recursive loop to be an iterative loop. This let me control things differently, doing reckless things to my queue of partial solutions, including throwing away a big fraction of my work when I started to get pressed for memory, and doing a random shuffle of my queue to give it an opportunity to "jump out" of dead ends. I've had fairly good results in the past from periodically shuffling my iterative work queue to get away from local maxima. It sounds goofy, but you might try it and see if it helps you.

So, I had an iterative backtracking solver, and it was slow. Part of the problem, I'm sure, was the copying I was doing. Part of the problem was that I was using Python. I'm not really willing to rewrite this whole solver in C++ for performance - and the galling thing was that my WFC generator was able to make a reasonably-sized map (~17x22) in under a minute, while my backtracking generator was struggling to finish a map maybe a quarter of that size (8x10?) in a few hours.

So, maybe I don't need stairs. The whole reason I stepped away from the WFC implementation was that I had these non-local constraints that WFC didn't deal with well.

One thing that might be faster is to implement Knuth's "dancing links" algorithm, which I never have been able to entirely wrap my head around. It seems to allow you to do backtracking search in-place, which would be handy.

I went to my tileset data and turned off all my stair tiles, and got my old WFC implementation working again. The tunnel system looks like a block of uncooked ramen noodles, but I'm willing to go with that as an insane, but legal, generated dungeon. It also reminds me of the studies where they fed spiders a variety of drugs and looked at the kinds of webs those spiders spun.

Once this was in place, I got a feature working where each tile could specify how frequently it would appear in the output. For example, in early versions, there were a LOT of pits and one-way doors. I turned those down, and the dungeons look somewhat reasonable (still with the ramen tunnels, but I'm still OK with that).

I've also added some convenience constraints, blocking out the lower left hand corner so that I can paste in a legend of the various symbols used on the map. Not all dungeon maps need that, but my hand-drawn dungeon bits might not always be obvious to the reader. Is a circle with a star inside it a pentagram or a statue? At this level, a statue. Later on, we'll talk some more.

I intend to write out the map as a CSV to make it easy to do some editorial touch ups, and then read the edited CSV back to feed to the renderer.

And then I intend to move on to some deeper caves and caverns, with yet another generator, this one less tied to grid lines.

On the sad state of inline images

In part, this post is a gratuitous effort to add juice to my "Shapeshifting RPG" project, which you can find at the Shapeshifting RPG website or at the Shapeshifting RPG GitHub repo. I really want to get that "done" in the next few weeks, which includes "writing" a few more chapters, a bunch of layout, and some iteration with DriveThruRPG to get it purchasable by the end of the summer.

If you're not familiar with the Shapeshifting RPG project, allow me to explain. I'm a big fan of computer-generated art, whether that's Harold Cohen's AARON project, or computer music, or even computer-written books.

OpenAI recently released a computer language model "GPT-2", trained on a big chunk of Internet text, which does a surprisingly solid job of generating prose passing for human-written. It still seems a little weird, but it does a subjectively better job of passing the Turing Test than, say, Weizenbaum's ELIZA.

To make it easier to interact with GPT-2, Adam King hosted a webpage https://talktotransformer.com/
which allows you to use pre-generated prompts or your own text and get a couple paragraphs of generated text.

Also, about this same time, the "Prismatic Corpse" game jam project drifted across my consciousness - people getting together to make an old-school RPG without working "together".

The ingredients were mise en placed, the table was set, it seemed like the only thing I could possibly do was to create my own RPG ruleset.

And so, for the past couple months, I've been prompting the TalkToTransformer AI with various bits of RPG rules, and it's been giving me big boluses of vaguely readable text, which I proceed to edit into chapters.

Where the AI totally wins is in generating lists of things. Hat tip to Janelle Shane, whose https://aiweirdness.com/ blog can consistently make me laugh out loud, with AI generated lists of things that begin on the weird side of plausible, and quickly descend into crazy-wrong space. So, I asked GPT-2 to make lists of monsters, and weapons, and armor, and NPCs. It loves lists. And a RPG rulebook is often a lot of pages of tables. Which are like lists.

To make my book borderline "useful", I decided to index my list of monsters, each with a 4-digit number from one to six. This gives you an easy way to roll up a foe to fight against, if you wanted to. Just roll 4d6, and look on the table for that die roll. So, all I had to do was to put a picture of four 6-sided dice in front of each element in my monster table.

I'm using Scribus to do the layout of my book, which is mostly doing what I need it to - it's WYSIWYG layout, without being a word processor, which is fine. It's even got the ability to run Python scripts, so you know I like that. I poked around a lot with trying to put an image inline into a bit of text, but it seems like Scribus prefers to have images exist outside of text flows, which is great if you want your paragraph text to flow around an image like this:

To be clear, I don't think that's how I want my text to flow, but it's super cool that Scribus makes it easy to do that. (And, somewhat amusingly, I had a hard time getting the image of the text flow to get placed where I want it in Blogger's layout. Layout is hard. Layout of images relative to text is doubly hard.)

What I ended up doing, which feels like failure, but it gets the job done, is to find a font that has the numbers 1-6 as the faces of dice, and then I add that to my list of monsters, like so:

It turns out, it wasn't completely that easy - what I ended up doing was writing one Python script to walk over a monsters.txt file that I had prepared, doing some light sanity checking and cleaning up (but, clearly, not fixing case-consistency; Ice-Eared Goblin should probably be all lower-case, like everything else). That script outputted a CSV with the indices in their own fields. Then, I had another Python script, this one inside Scribus, which pulled in the CSV and wrote out the text, one line at a time, but switching fonts along the way, switching to my dice font for the index, then switching back to the text font for the monster name. This is a surprisingly time-consuming script to run, taking several seconds to generate around 18 pages of monsters.

It works, it's sufficient for this project, but it feels like yet another hack along the same lines as hacks that I've done in game development to get, for example, the PlayStation "X" button to render as an image in a tutorial text display. This feels like it doesn't need to be this hard, especially with the Unicode Consortium spending time on frowning poop, and Apple bowing to user demands for shapely peach imagery.

Mandelbrot set on an emulated Apple ][

I figured I'd start off with a pretty picture, so that maybe you'll let me go on a discursive narrative.

When I was but a wee lad, I learned to program on my family's Apple ][. I started off with the little stuff that you do, like 

10 PRINT "HELLO, WORLD"

20 GOTO 10

and the like. I began drawing pictures on the "High Resolution Graphics" (HGR) screen, with a staggering 280x160 pixels (280x192, if you were fancy and used the second page and gave up the 4 lines of text at the bottom of the screen). But, eesh, I was playing fancy games like Space Invaders, Donkey Kong, and Pac Man in the arcades, and I wasn't getting anywhere near the performance that the professional developers were. And even games written for the Apple were a lot more capable than what I was able to get done in BASIC.

Somehow, somewhere, I learned that the professionals didn't do high-performance graphics in Applesoft BASIC. The big boys used Assembly Language. So, that's where I decided I needed to focus my energy.

As an aside, probably around this time, my dad did have Apple Pascal running on the home machine, but maybe Pascal also was sluggish in its own way. For whatever reason, I never spent much time with Pascal for the Apple ][. A few years later, the Apple IIgs came around, and that's when I learned Pascal, which led directly to C, then C++, which may account for the majority of code that I've written in my life, though there are plenty of others. C and C++ I learned on a 286 IBM PC and a 486 IBM PC Compatible, back when IBM was relevant in the PC space.

There were a few dead ends that I learned on the Apple ][, though, including GraFORTH and Apple Logo - both promising to give good graphics, in their own way. GraForth was a Forth implementation, which introduced me to stacks, which would help me with Postscript and RPN years later. And Logo was LISP without all the parentheses. So, even though the languages themselves didn't help me make cool animations, they planted seeds.

The idea fixed in my mind was to learn Assembly Language one way or another. On a family trip to the Pacific Science Center in Seattle, Washington, we passed through the gift store (back when it occupied the balcony of Building 4), and I found a book for sale titled "Assembly Language for the Applesoft Programmer". Well, that was exactly what I wanted. This was the key to writing computer games, and being successful, and avoiding actually having to "work" at a "job" - I'd write some sort of platformer for the Apple ][, and, I don't know. Profit. I didn't have all the steps worked out, but I knew I needed this book.

I begged and pleaded for Dad to buy the book for me, and he flipped through it, and decided it was a fairly substantial work, but it had a lot of programs to type in (back in the day, we typed in programs, even from magazines). So, Dad made me a deal; he'd buy me the book if I promised that I'd work through all the programming tasks, type in all the code, actually go through the whole thing. Yes, of course. 

I probably read the first chapter in the car ride home, which introduced a number of concepts. So far, so good. But then, chapter 2 instructed the reader to open up any one of a number of commercial assemblers for the Apple ][. My heart sank - I had just used up all my negotiating capital to get a $12 book, there's no way that Dad would go on to drop $200 on a commercial assembler, as well.

So, the book gathered dust on a shelf. Dad did work through a small number of the examples, using the Apple ]['s built-in monitor, the poor man's assembler toolchain.

I felt guilty about the promise I had made, and I kept the book as a reminder. As I mentioned, the Apple IIgs came along, and I learned Pascal, which introduced me to procedural programming, and I went off to college, and learned Scheme and LISP and C and C++ and CLU. Perhaps a few others, besides. 

Years later, like probably around 30 years after making that promise to work through the examples, I had experience with working with LinApple and a variety of other Apple ][ emulators, and I knew that there were disk images of various old pieces of Apple software available on the Internet, including some software that was actually legally available.

I dusted off the book, looked up the suggestions of commercial assemblers mentioned to build the samples, and discovered that Bob Sander-Cedarlof of S-C Software had made his S-C Assembler available for free, and so I was able to begin to make progress on my long-delayed promise.

I learned about the X, Y, and A registers, I learned about the Zero Page, I learned about the various flags that arithmetic operations would set. I learned a little bit of magic locations in memory. I wrote a sort routine that would organize literally dozens of pieces of data.

Yeah, this wasn't going to actually have practical benefits, but a promise is a promise, and I continued to plug through.

The Apple ][ used the Motorola 6502 CPU, which had support for addition, subtraction... branching... using 8-bit integers. As I recall, some of the addressing modes allowed you to use contiguous pairs of bytes to support 16-bit integer math. The hardware did not support multiplication, division, or floating point numbers. So, I wasn't going to have it easy if I wanted to write a 3d flight simulator or first person shooter. Not really what my goals were - I just needed to complete the book.

Well, except that I decided that I should build something of my own, based on what I had learned. And one of the programming go-tos (sigh) that I had gone back to over and over again was to write a Mandelbrot Set fractal renderer. I had done this in Pascal on the IIgs, C on the 286, and over and over again, as my own little proof to myself that I understood how various programming environments worked. In the years since, I've heard this referred to as a "coding kata".

So, pushing the (emulated) Apple ][ to render a Mandelbrot set, which requires complex algebra and looks best with lots of pixels, that was a challenge I felt was worthy of this journey.

I built multiplication of complex numbers, I wrote pixels to the screen, I got stuff sort of looking sort of right, but not quite, and debugging was a huge pain. So, I ported my assembly program back into Applesoft BASIC, and I was happy and sad to see that the BASIC program drew pictures that looked correct to me, while my assembly code was giving me a similar-looking blob on the screen, but not the recognizable bumblebee shape. Eventually, I managed to track down a flag that was being set that I didn't expect to be set, which was causing a conditional branch to happen when I didn't expect it. I rearranged the logic, and I had my assembly program drawing images to the screen, as I had imagined to begin with:

   

What you can see in that image is that I was drawing the image progressively - here, you can see every other row drawn, but initially, I drew widely-spaced pixels:

which is enough to tease the recognizable shape. Once that pass is done, I drew a lot more pixels horizontally:

Those look like solid horizontal lines, but I'll get back to that. Then, I filled in vertically:

And then I had one more pass filling in pixels horizontally:

wait, what? If you're unfamiliar with the Apple ]['s graphic screen, you'll wonder where all that white and orange came from. If you've spent time in the pixel trenches, you probably know that orange is accomplished by lighting up the odd columns of the second palette, blue is even columns of the second palette, green is odd columns of the first palette, and purple the even columns of the first palette. Because my progressive renderer was drawing only even columns for the first several passes, there was a lot of purple and a little bit of blue. No orange, no green, until later. Also, white was a result of lighting adjacent odd and even pixels. It's almost like it wasn't a 280 column display, but more like a 140 column display.

And so, tada:

 

Working to the level that I was shooting for. There's some color fringing, like the blue on the far left, the green on the edge of the orange field. But you know, that's fine. Also, I want to point out that the white circle actually looks like a circle, even though the Apple ][ did not have square pixels. The aspect ratio math wasn't hard. But kids these days, they don't even think about that sort of thing.

Earlier, I mentioned that I had an Applesoft implementation and an Assembly implementation - to my recollection, the Assembly version took me about 10x as long to write, and ran about 3x as fast. That doesn't seem like an especially fair trade. I consider, though, that I was pushing the mathematical capabilities (multiplication of fixed-point numbers on a 6502), which was going to be slow, no matter what. Performance was an important part of what set me on the road to learning Assembly, but by the time I had finished the book, even a fast program on a 64K 1MHz machine wasn't going to impress a lot of people. So, I considered myself done.

Giddy with my accomplishment, I printed out the various screenshots that you've seen up to this point. I took them to show off to Dad the next time I visited. He wasn't super impressed - I don't think he even remembered that day, three decades earlier, when I had committed myself to learning assembly. But he nodded and there was some small level of recognition that I had done something.

A related story, which I won't tell now, has to do with Dad and I collaborating on making a Pac Man clone on the Apple ][. Things went comically badly.

Going from a grayscale image to an image with transparency in GIMP 2.10.8 on Linux

I've searched for this in a bunch of places, and I've found a technique that works for me - I think my requirements are a little different from maybe what other people are looking for, so I'm writing this up for my own notes, and maybe somebody else can benefit.

Problem Statement:

I've got some black-on-white grayscale art (possibly pen-and-ink line art) that I'd like to turn into a layer of a single color, but with the grayscale value in the alpha channel.

If I was writing a Python script using PIL, I'd expect the incoming image to be a grayscale format, with 8 bits of lightness information (0 = black, 255 = white), no alpha channel, and the output image is RGBA, where every pixel is (0, 0, 0, A) where A is 255-L, L being the lightness value of the input image.

Why not make a Python script to do this? It sounds like 5 lines of Python or less, right?

Yeah, maybe.

Still, this is how I do it in GIMP. Maybe I want to do additional processing in GIMP, and having it already open might be handy. Or something.

Step 1

Ok, I'm starting with my image loaded in GIMP. I'm not going to help you get to this point.

Optionally, crop the image - for this image, I tried "Image > Crop To Content", which didn't get as tight on the left hand side as I expected, so I followed up by using the rectangular selection box from the toolbar, selected a generous box, and manually pulled in the left side to where I wanted it. And then "Image > Crop To Selection".

I suppose you could do a bunch of other stuff here, too, like resizing the image. Live your best life.

Step 2

Duplicate the layer

This can be done by right clicking on the layer in the Layers panel.

Step 3

Invert the layer. I had best results with "Colors > Linear Invert". You could try the normal invert, I suppose.

Step 4

Right click on the new layer, select "Add Layer Mask"

Select the Grayscale copy of layer. (Why is it grayed out AND selected?)

Step 5

With the paint bucket tool, fill the layer (not the mask) with black, using "Fill whole selection", making sure your selection is all or none of the image.

Step 6

No, there's no step 6. You're done. Make a sweatshirt or whatever it is that you were going to do with your grayscale Half-Elk Fighter-Master. Or whatever.

Making a G+ Sendoff

I'm in the last few hours, as I write this, of Google+, the community-oriented social media site. Google will be (will have, by the time you probably read this) closing it down. The site never got a huge following, though there were people who really enjoyed using it.

I wanted to make a send-off in one way or another. I was thinking about using my AxiDraw pen plotter machine to draw a picture to G+, but that didn't click for me. And then I thought that it'd be fun to have an Apple ][ welcome G+ into the afterlife, a place where systems go after they're shut down for good.

First, I started up LinApple, an emulator of an Apple //e, and wrote a test program into it that would print some text to the screen, one character at a time, reminiscent of reading chat on an old BBS system at maybe 1200 baud.

Aside: years ago, I bought a 1200 baud modem for my family's Apple IIgs, and Dad was suspicious of what purpose I'd put it to. I pointed out that I could dial in to the county library and reserve books. Also, there were BBS systems where I could send messages and play games, but I knew that those purposes would be less compelling to Dad. So, Dad, I bought that modem so that I could remember using it and remember using Google+ and see the similarities. Probably still not compelling.

Listing 1

As you can see here, I first clear the screen and set the cursor to the upper left corner of the screen (Line 10), then I take two lines of text, assign them to the string variable A$ and call out to a subroutine. (Lines 20-50). That subroutine iterates through the line, printing one character at a time, then calling an additional subroutine (Lines 1000-1020), which simply spins, slowing down the program, making the text scroll onto the screen at a readable rate. (Slowdown from lines 2000 to 2020).

Interestingly, there's a bug in this code - the subroutine at 2000 doesn't return explicitly. I guess that it reaches the end of memory, knows that it needs to return, and goes back to line 1030.

At 1030, we continue on to the next character in the string, then at 1040, we print nothing other than a newline, terminating our printing to that row of the screen, automagically moving to the left and down one row. Later, we'll see scrolling, as well. All handled by the system. BASIC had a lot going on that I took for granted as a kid.

So, that structure seemed to work. I set out to write a blort of text that would be long enough to be fitting, short enough to be readable. I knew I didn't want to do text editing in my emulated Apple, so I copied this code out of the emulator and onto my Linux machine. Well, it was already ON my Linux machine, just in a disk image that the Apple emulator could read, but my Linux machine couldn't usefully edit.

So, I used AppleCommander to "export" the BASIC program to a text file, which I saved as reference.

The text file I wrote in emacs, because a) of course I did, and b) emacs is pretty good for editing fixed-width text files, which is the sort of thing that I wanted. It's got word wrapping, which made my life easier, too.

So, I wrote a page and a half or so of text, saved that out as a text file.

This next bit is perhaps unnecessary, but I wrote a Python script (again, because of course I did) that read the lines of my text file and wrote out a BASIC program that would print those lines to the screen. Sort of a compiler of sorts, that compiled a text file to BASIC.

The structure of the output file was the same as my test BASIC file (Listing 1), I just had a lot more calls to the line printing subroutine; one each for each line in the text file.

Listing 2

With this, you can see a few things. For one, you can see that emacs is happier providing syntax highlighting for Python than it was for BASIC. Maybe I don't have the Applesoft BASIC major mode installed. The structure is pretty simple - read all the source text file into memory, write a preamble, write out instructions to handle each line, and then write the closing matter, particularly the subroutines.

The header is super straightforward:

Listing 3

All it does is use the HOME command to move to the top left of the screen and clear it. You'll see that I also increment the line number by a healthy 10 numbers and return that.

Kids these days with their IDEs and their structured programming languages, they don't appreciate the value of leaving a healthy amount of space between line numbers, so that you could go back and add in extra commands if you needed to later.

Listing 4

This simply takes a provided string of text (which I had read from my text file) and generate a line of BASIC that assigns that string into the A$ string variable, then on the same line (something done more in BASIC than any other language I've ever seen) invoked the print subroutine. Again, I increment the line number and return.

Listing 5

And that wraps up the BASIC output, and also the Python script. I just write out the code to print one character at a time, placing that subroutine at line 5000, because that seemed like a safe space. Just to be sure, though, I assert that my text printing body of the code hasn't pushed us past 5000 before we got here. Similarly, I have the delay subroutine at 6000.

So, I ran the program, and ran into a few bugs - I had forgotten the semicolon on "PRINT MID$(A$,I,1);" - the semicolon means "print this string, but do not move the cursor afterwards", which achieves the teletype incremental progress. I had also forgotten the carriage-return (again, the kids don't remember typing on a typewriter where there was a physical lever that would RETURN the physical cylinder (the CARRIAGE) such that the typewriter would go on from the left margin.

I fixed those things up, tweaked the text, and output my WELCOME.BAS output file.

I struggled a bit with AppleCommander, trying to get the BASIC code back onto the disk image that the emulator would use. The tricky bit was that Applesoft BASIC is stored on floppies in a "tokenized" format, not in plain text. This takes one step off the job of the interpreter at runtime, and it makes it more efficient to save BASIC programs to a 140k floppy.

So, I needed to figure out a way to take my plain-text WELCOME.BAS file and tokenize it, and store the tokenized output onto the disk image.

Turns out, AppleCommander has a command-line version with a -bas option that does exactly this. Almost exactly this. It isn't perfect about carriage return / line feed combos, so I tinkered with that, and eventually got my WELCOME.BAS onto a disk image.

I actually had some shenanigans with getting it onto the disk image that I wanted to use, which took some rebooting of my emulator, but in the end, I got the program running as expected.

I fiddled with the display options to get the green monochrome monitor effect, and set out to record the results. The first recording I did was using byzanz, a recorder that outputs an animated GIF. This has been effective enough for many of my previous projects, so I fired it up, and got pretty good results. I loaded the output into GIMP, and cropped the image to the size I wanted, and this was the outcome:

Figure 1 - Animated GIF

Good enough for a post to Google+. But I'm never sure if an animated GIF is really going to make it all the way to the viewer without being resampled, reprocessed, and ultimately de-animated. So, maybe that's not good enough. Also, somehow, there was a glitch of random white rectangles. I don't know what that was.

So, I used SimpleScreenRecorder to capture a movie (MPEG?), suitable for upload to YouTube, in case a GIF (appropriately enough, super old technology) doesn't work into the future. SimpleScreenRecorder has a lot of good options, including allowing me to tweak my recording window, turn off recording the mouse, turn off sound, all good things for this project. And, boom, I uploaded it to YouTube, and by the time that I had filled in the metadata, the upload and processing were complete, and my little bit of remembrance was ready to go live.

Figure 2 - Video on YouTube

At this point, I used the following technologies in this project:

• LinApple
• Ubuntu Linux
• emacs
• Python 2.7
• Applesoft BASIC
• AppleCommander 1.5.0 (GUI and command-line)
• GIMP
• byzanz
• simple screen recorder
• Google+
• YouTube
• Blogger

Seems like a lot, and a big part of writing this blog post was to capture the workflow in case anybody (mostly, my later self) could re-use a part of that flow for later projects.

Because I am putting myself to the fullest possible use, which is all I think that any conscious entity can ever hope to do.

Farewell, G+, we will miss you.

Run/Walk: Kirkland Shamrock 5k

The weather forecast was highs in the 60s, but that doesn't equate to temperatures during the event.

Finish photo

Another year, another St. Patrick's Day themed run. This one took us from the marina park, up the hill to the cross-Kirkland connector trail, and then down through residential roads. The hills were hilly, the temperatures were comfortable, and the roads were dry.

If you can make out a time on the clock on the finish photo, it would say something over an hour, which I want to disclaim, as we were in the second half of the third wave, so there was well over 10 minutes on the clock before we got started. 50 minutes is still not a lightning-fast time, but for a casual jog/stroll, it's fine. I'm content, if not proud.