Adventures in Image Processing

So here’s an interesting mystery / conundrum / process I recently went through in trying to create a new workflow.

For reasons or not wanting to subscribe to software, I’m still using the CS5 versions of Adobe Apps for Surfing The Deathline. The original documents are heavily constructed in Photoshop, then all the text, sound effects etc are done in InDesign, which makes them rather non-portable to other solutions.

It had been so long since I’d done a serious update of the books, that I’d forgotten parts of my workflow, and so had started some things from Scratch.

Surfing The Deathline uses .png format images for its pages. Although they take up a lot more space than .jpg versions, they have an advantage of being colour-accurate. A major problem of .jpg is that for images in black & white, a single pixel of colour will shift the white and black values away from their correct tones. So, where you get two pages butting up against each other at the spine, the greys might not match.

InDesign CS5 has no direct png output option, so the workflow is:

  1. PageExporterUtility script to output the pages as individual .pdf files
  2. Convert the .pdfs to .pngs
  3. rename the .pngs and move them to the appropriate EPUB document’s image directory.

I had created an Automator action, which took in the .pdf files, and converted them to .png, and saved them to disk. It took about 3-5 minutes to do all 236 pages.

But there was a problem with he output…

Certain pages, seemed to have red & blue fringing on their text. Going through the .pdf files, it bcame apparent that it was linked to the pages which had a specific masterpage controlling their appearance. Looking at that masterpage, the thing that suggested itself as problematic, was the page number – it was frontmost in the layering stack. So, I deleted and recreated the page number object in the masterpage, applied it to the pages, and reran the .pdf to .png workflow.

Problem solved. Almost.

My large sound effects were still showing red/blue colour fringing. After a couple of days research, it became apparent that this was caused by the system applying sub-pixel antialiasing to the .pdf file during the render.

After experimenting with commandline options for disabling it, I found out there was actually a checkbox for it in the System Preferences app. Unfortunately switching it off makes the system’s display look worse, so what I needed was a way to toggle it off, run the image processing steps, then switch it back on.

After some experimenting, and asking on forums, I was able to get an Applescript that did the job, and add it to my Automator action:

What this does is:

  1. run an Applescript to open System Preferences, test if the checkbox is ticked, untick it if it is, close System Preferences, then,
  2. run all the pdf files through the Render PDF Pages as Images function to create png files, then,
  3. move the converted versions to a new location. Then,
  4. open System Preferences, test the antialiasing setting, and switch it back on, then close System Preferences.

It was fantastic – I had a wonderful system that, once  I’d output the .pdfs from InDesign, could after selecting all of them, render them all to .png in a single right-click.

But there was a problem…

Images which crossed the spine of the EPUB book weren’t aligning correctly. Clearly, something was wrong with the way the Automator renderer was converting .pdf into .png. It didn’t matter what scale I rendered it at – even at the full native 300dpi, the problem remained.

When I compared it against doing the same process manually in Photoshop, it also became apparent that the math behind Automator’s conversion was out – files were always cropped 1-2 pixels smaller from Automator, than they were from Photoshop.

Then I started researching if there was an alternative commandline image processor in macOS – something I could call from an Applescript, to replace Render PDF Pages as Images. Thankfully, there was – SIPS, Scriptable Image Processing System.

After a bunch more research, I managed to sort out the appropriate commands, and gave SIPS a go on my pdf files. The results were the same. I tried it manually with Preview, the results were the same, again. It appears SIPS is the core image processor all these built-in macOS tools use, and it’s SIPS that has the bad math function for rendering PDF files as images.

Sips also produced pretty garbage image quality, compared to Photoshop.

So now I was looking for an alternative to SIPS, and I managed to find one – ImageMagick, a cross-platform commandline image processor. It uses Ghostscript, an opensource alternative to Postscript to render the .pdf, so everything about it will be separate from the SIPS processes. After a couple of days trying to figure out how to install it (hey, opensource projects, try making your basic documentation an educational resource for people who haven’t used your tools previously), I was able to make it work…

It delivered fantastic results, but took 30 seconds per image to process the .pdf files. In contrast, Photoshop, which was so slow I was looking for an alternative, takes 8 seconds.

You might question why I don’t use Affinity Photo, which can tear through the entire 236 pages in around 8 seconds total (gotta love that multithreaded action). Well, unfortunately, Affinity Photo’s pdf renderer can’t handle the edge effects of my InDesign speech bubbles.

So I’m back to where I began, using a ancient versions of Photoshop and InDesign, and needing to take a 35 minute break so Photoshop’s Image Processor script can do its thing, every time I want to run a set of updates from InDesign to EPUB.

Update 23 April 2021:

In experiments with image sizes for Photoshop’s scaling when it renders the PDF file to TIFF, I’ve hit upon a target size that seems to be in some sort of mathematical sweet spot for Photoshop, because the processing time has gone down to about 1 second per image, from ~8 seconds.

Surveying the catastrophe.

If there is one trend that is clear, it’s that a greater and greater percentage of economic “value” will be concentrated into fewer, and fewer people and companies. Automation will destroy more jobs, and concentrate more economic extraction to fewer individuals, than new industries will produce.

It begs the question, why a political party would base their campaign on providing “jobs”, rather than providing income? “We promise to keep you busy with tasks” – hardly the most inspiring of messages, is it?

There is too much that too many people do, which requires more skill and more time, than people have money to pay for it to be done.

UBI, at a level considerably higher than the current unemployment rate, is inevitable – large economic extractors (including wealthy individuals) are going to have to pony up radically higher taxes, and that money is going to have to go directly to people, because through Covid we’ve seen both the future of what happens when too few people have enough, and how effective “just giving people money” is in blunting that catastrophe.

So again, IF you were a political party hoping to win government, why would you double down on an obsolete paradigm?

2020 – A Wrapup

What a year.

Things were going ok for the first ten or so weeks – I bit the bullet an began gearing up to do some welding, to finally make a push to get some sculpture done. I was offered another arts residency, which would be a wonderful expansion of Noosa Mnemonic, in cooperation with the local regional art gallery…

…then the plague hit, and hit HARD.

By week 12 we entered a kind of non-time, in which things drifted and became unfixed. Public events went away, my residency was cancelled. So I ploughed ahead with my home welding setup, and designing the Studio-in-a-box.

I received an Arts Qld grant that covered my costs for the Studio-in-a-box, which was also a nice little notch to add to the artist’s cv. A big part of this year was figuring out quantity surveying for materials, and mapping out cut-charts for steel lengths.

There were some big upgrades for my protective clothing kit this year – with a new powered respirator welding helmet, and an (as-yet untested) powered water-cooling suit. When all this stuff is combined, I should be kitted up like an astronaut, and able to work safely and comfortably in more or less any conditions.

I solved a problem of flakey power supply during the storm season, by dropping a bunch of cash on a honking great UPS.

In photography my processes had some issues to overcome, adapting to a new photo processing / management tool. It produces better images, but has worse management features. So I’m effectively now working in two apps. I also updated my monitor configuration for my workstation to 3 screens. Sadly, these changes really didn’t work out too well – 2020 is definitely the year I fell out with technology. I have to learn to not allow anyone to provide an “ecosystem” to me, and instead build one myself.

I finished an update to The Metaning, which frees it from Apple Books, and a European publishing standards group released an Ebook reader for Mac, Linux & Windows that can read the fixed-layout EPUB format, so that clears the way for me to ditch Apple for selling my books, and move to direct sales.

The other major thing this year, was 15 minutes of fame, after a tweet thread I started about my time in high school was picked up and retweeted by a bunch of folks. Had a lot of people relating to me their childhood horrors from the same school.

Overall, 2020 was a year in which a lot of long-term plans and strategies played out, and produced results that weren’t all I’d hoped. Of course, across everything hangs the pall of Covid, so it’s hard to put the achievements of the year in any greater context of where I thought I’d be by the end of it.

Studio In a Box

So this is a project I’ve been working away on designing for a while now, as a way to solve a lack of accessible studio space, and the need to stay at home during Covid social-distancing.

It’s a trifold-door cabinet, with internal power supply, that will be installed in my carport, between the secondary entrance (left) and laundry (right) doors of my home. It’s designed so that all the equipment stacks inside it – at the rear is workbench space, including a drill-press station, then the UV-blocking welding screens go in front of that, and finally the welders on a trolley, and a fold-up welding table.

All the equipment within is on wheels, so it can be rolled out and the space configured, with no lifting required. All it requires me to do is move my car forward a couple of metres, but the cabinet is narrow enough that my car can fit beside it when closed.

There’s a long, narrow workbench for my drill-press and bench-grinder, as well as storage, and a table for my metal-cutting bandsaw (a quiet alternative to a drop-grinder), that sits over the Air-Compressor. The compressor is an interesting piece of kit – it’s a silenced model, that uses two small motors, rather than one large one. You can easily hold a conversation at normal speaking volume, while standing next to it.

The power supply, which will sit roughly in the middle of the cabinet, is already installed – a pair of 15 amp, and a pair of 10 amp plugs, on a 32 amp line, so I can drive both the air compressor (10) and the welder’s plasma cutter (15) at the same time. Or, I can keep both my TIG and MIG welders powered up at the same time, and alternate between them, using  MIG to tack things in place, then TIG for the finished welds.

All in all, it should be a super adaptable, and quick setup / packdown low-effort workspace.

2019 – A Wrapup.

2019 started with high hopes for diving back into EPUB development. I put huge amounts of time into redesigning my EPUB books to be able to have their text on a separate layer, so different translations could be done, and to allow the artwork to be user-selectable between finished, sketch, and thumbnail versions.

Weeks were spent trying to get the image outputting from InDesign to work correctly, and I had more or less cracked it, and knew theoretically how it would all be achieved.

Unfortunately, updates to Apple’s books platform broke the core functionality I relied upon, and all my books on the Apple Books store are now broken.

My uncle, Travor Ashton, a wonderfully generous man, sadly passed away.

I entered a major outdoor sculpture festival, and put a bunch of work in to applying for a grant to cover training costs to refresh my welding skills.

Unfortunately, my grant application was unsuccessful, and the sculpture, once repaired, displayed some significant structural weaknesses when exposed to driving wind, that meant i would be unable to install it at the event, so I had to pull out.

I had a new residency project, Noosa Mnemonic – a VR recreation of Noosa, based on people recreating in VR, places they love in Noosa, working only from memory. The goal was to have all the separate places added together into a single VR environment. I created a really interesting new VR location, and arranged for other artists to contribute locations.

Unfortunately, by the end of the year, it seemed to have become moribund, lacking for funding, and a reduced scope that makes the vision more of less moot. After breaking myself on the previous year’s major project for the Drone Racing course, I resolved to be less emotionally invested in this project, so C’est la vie.

I did some more VR outreach projects for the Library Makerspace – I really wonder if I’ve missed my true calling, because I love doing public outreach events.

My bike was serviced, and I was able to get back into riding periodically. It’s good for my mental health. The bike works better now than it ever had since new. I bought a helmet-mounted action-cam, and took to videotaping all my rides, so as to protect myself from incompetent drivers.

We sold my late father’s house, and dissolved his estate. It was finally over. I find myself on occasion missing him, missing that ability to have someone to talk to, who was largely removed from my life, but towards the end, there wasn’t much left to say. His ideas and opinions had been so poisoned by the right-wing internet, that in the end, I was left with very little in the way of happy memories about him.

I started investigating drop-shipping my photo prints as I dipped a toe into Instagram, but then Instagram changed the ability to see metrics of user interaction with posts, and it became yet another platform that one has to question if it’s worth the effort.

More upgrades and updated to my 10 year old Mac Pro tower. it’s such a dependable tank of a machine, frankensteined to hell as it is.

Toward the last third of the year, gearing up for internatonal travel to Japan, gained a sense of urgency, and I designed and had fabricated a set of camera mounting plates to let me better mount my camera on a backpack, while remaining connected to my sling-strap. The design was a pretty amazing success.

Then, I was in Japan, and three of the happiest weeks of my life. Magical country, and I wish I could live in a country mountain town, with the constant sound of running water. The cities weren’t so much my thing, but I could spend months travelling around on local trains, seeing the little farming communities.

Once I came back, there was a lot of lost time – administrative things, medical things – I saw The Sisters Of Mercy play live – a lifelong ambiton, and they were “meh”.

The year ended sitting on inflatable pool chairs with friends, drinking margaritas in the pool, watching Return of The Living Dead on a projector I’d rigged up to screen onto the side of the building.

It was a year of highs and lows, but then aren’t they all?