transparently migrate data from local file storage to key-value-stores like RIAK KV

About 2 years ago I sat down and wrote a filesystem. Well not from scratch but using the great FUSE (Filesystem in Userspace) framework. I’ve released it as open-source later on Github:

This script acts as glue between a local file storage mount point and RIAK. It is targeted at specific use cases when local mount-points need to be migrated to RIAK without changing the applications accessing those mount point. Think of it as a transparent RIAK filesystem layer with multiple options to control it’s behavior regarding local files.

riakfuse

I had a very specific purpose in mind when I wrote this: There was a local filesystem that got filled up and because of technical restrictions we were not able to resize it or even completely copy it without interrupting the service using the data stored there.

Since we were already using the RIAK Key-Value Database for certain binary loads the idea came up to also utilize this key-value concept for a filesystem.

The idea now is: You have a local filesystem that holds a lot of folders and files already and you want to gradually want to move it to new grounds.

This migration needed to happen with minimal service interruptions assuming that there is constant reading and writing happening.

In this riak-fuse project I’ve written an overlay filesystem that steps between the application and the underlying “old”-filesystem. It looks and behaves identical to the application reading and writing.

But, depending on the mode you have chosen while mounting, every file read will at first be read from the “old”-filesystem and after successfull read stored into the key-value store.

On the next read it will be read from the key-value store directly.

The same applies for writing. Riak-fuse will write either to both, local storage and key-value store or just to the key-value store.

So in a nutshell: Data is slowly but surely on each access transferred over to the key-value store and load + storage space is slowly moving over from the local storage to the key-value store.

To facilitate this there are a lot of options for this script:

This all comes with ready-to-use docker and docker-compose files for you to try out.

Also it might interest you as an extremely simplified example of how to write an actual file system module for FUSE in Python.

Disclaimer: This effectively is my first python script as well as fuse module. Don’t be too hard on your judgement.

“Around Tokyo” project

If you are interested in even some older video recordings, pictures and impressions about some areas of Tokyo this is a good time-sink for you.

Lyle Saxon has several old-fashioned looking websites (wonderful) with lots and lots of browsing content as well as a YouTube channel with recordings from earlier times:

Tokyo resident since 1984. Video material from 1990-93, as well as newer material from 2008 onwards.

The “Around Tokyo” project was and is to document life in Tokyo and the surrounding areas, as well as some material from other areas of Japan.

YouTube Channel

QR codes – how do they work?

I came across a very nice explanatory piece for QR codes. If you always wanted to know the basic principles this is a good chance to get a grasp.

QR code (abbreviated from Quick Response Code) is the trademark for a type of matrix barcode (or two-dimensional barcode) first designed in 1994 for the automotive industry in Japan. A barcode is a machine-readable optical label that contains information about the item to which it is attached. In practice, QR codes often contain data for a locator, identifier, or tracker that points to a website or application. A QR code uses four standardized encoding modes (numeric, alphanumeric, byte/binary, and kanji) to store data efficiently; extensions may also be used.[1]

Wikipedia

I am using QR codes in several of my projects – one example: Miataru uses QR codes to encode the device ID and help with the device handshake. You scan the QR code of your friend with your Miataru client app and immediately will be able to see his location in Miataru. Without the need to enter long rows of numbers.

Japanese Highway Junctions

Highways allows us to travel long distances and interchanges, or junctions, connect those highways so that traffic can pass or change direction without interruption. And in Japan, where heavy mountainous terrain and dense cities create unique constraints, interchanges are, simply put, magnificent feats of structural engineering that we sometimes can’t appreciate through the typical vantage point of a car window.

Spoon & Tamago

Of course, go ahead and browse the highway system on Google Maps.

from now on console emulation will look much better

A modder going by the handle DerKoun has released an “HD Mode 7” patch for the accuracy-focused SNES emulator bsnes. In their own words, the patch “performs Mode 7 transformations… at up to 4 times the horizontal and vertical resolution” of the original hardware.

The results, as you can see in the above gallery and the below YouTube video, are practically miraculous. Pieces of Mode 7 maps that used to be boxy smears of color far in the distance are now sharp, straight lines with distinct borders and distinguishable features. It’s like looking at a brand-new game.

ArsTechnica

procedural generated traditional Chinese landscape scrolls

{Shan, Shui}* is inspired by traditional Chinese landscape scrolls (such as this and this) and uses noises and mathematical functions to model the mountains and trees from scratch. It is written entirely in javascript and outputs Scalable Vector Graphics (SVG) format.

https://github.com/LingDong-/shan-shui-inf

This is quite impressive and I am thinking about pushing that into the header of this blog :-) It’s just too nice looking to pass on.

let AI convert videos to comic strips for you

Artificial Intelligence is used more and more to achieve tasks only humans could do before. Especially in the areas that need a certain technique to be mastered AI goes above and beyond what humans would be able to do.

In this case a team has implemented something that takes video inputs and generates a comic strip from this input. Imagine it to look like this:

Input
Output

In this paper, we propose a solution to transform a video into a comics. We approach this task using a
neural style algorithm based on Generative Adversarial Networks (GANs).

Paper
click to read the paper

They even made a nice website you can try it yourself with any YouTube Video you want:

bliss is: waking up to this

This is how my bedroom sounds at 5 am. There’s barely light here in Germany but spring can’t hold it back. There are so many birds right around the house and in the garden around the house that it’s really really really impressive to hear.

Yes. I am not living in a city. And still you can barely hear the hum of cars/trucks in the background. Sadly.

I made this recording by holding my iPhone up in the air and with the Voice Recorder app included in the OS.

Nevertheless this is quite a contrast to the last 2+ weeks in Tokyo.

Panoramic Images free (-hand)

I really like taking panoramic images whenever I can. They convey a much better impression of the situation I’ve experienced then a single image. At least for me. And because of the way they are made – stitched together from multiple images – they are most of the time very big. A lot of pixels to zoom into.

The process to take such a panoramic image is very straight forward:

  1. Take overlapping pictures of the scenery in multiple layers if possible. If necessary freehand.
  2. Make sure the pictures overlap enough but there’s not a lof of questionable movement in them (like a the same person appearing in multiple pictures…)
  3. Copy them to a PC.
  4. Run the free Microsoft Image Composite Editor.
  5. Pre-/Post process for color.

The tools used are all free. So my recommendation is the Microsoft Image Composite Editor. Which in itself was a Microsoft Research project.

Image Composite Editor (ICE) is an advanced panoramic image stitcher created by the Microsoft Research Computational Photography Group. Given a set of overlapping photographs of a scene shot from a single camera location, the app creates high-resolution panoramas that seamlessly combine original images. ICE can also create panoramas from a panning video, including stop-motion action overlaid on the background. Finished panoramas can be saved in a wide variety of image formats,

Image Composite Editor

Here’s how the stitching process of the Musashi-kosugi Park City towers night image looked like:

select images
choose projection method
crop the projection – maybe use “autocomplete”
export the final panoramic image
zoom in :-)

Purple Tentacle

There are some things that influenced us over time. I’ve never played a lot of computer games. But I’ve played adventure games. Most notably those of LucasArts.

The “Day of the Tentacle” – being the sequel to “Maniac Mansion” – was one adventure game that I have a lot of great memories of. I have played it through a lot of times since it’s release.

At the beginning of the game the main villain (the purple tentacle) of the game is making a statement:

Bernard: Ok, you’re free to go.
Green Tentacle: Thanks Bernard!
Purple Tentacle: Yes, thank you, naive human! Now I can finish taking over the world! Ha ha ha!
Green Tentacle: Wait!
Bernard: Oh, yeah. Now I remember. He’s incredibly evil, isn’t he?
Green Tentacle: Uh… I’ll try to talk him out of it.

LucasArts (June 1993). Day of the TentacleDOS.

And because of his aspiration to take over the world the picture of the scene this is being said ended up as my phones unlock screen background (and if lots of other places) ever since.

With the help of the internet, you can have it on yours as well. And since screen resolutions improve over time, have it in timeless vectors:

vectorialized by Chalda Pnuzig

The vectorized purple tentacle above has been kindly provided by: Chalda Pnuzig

archive your slice of the web

Many use and love archive.org. A service that roams the public internet and archives whatever it finds. It even creates timelines of websites so you can dive right into history.

Have a piece of history right here:

You can have something similar hosted in your own environment. There are numerous open source projects dedicated to this archival purpose. One of them is ArchiveBox.

ArchiveBox takes a list of website URLs you want to archive, and creates a local, static, browsable HTML clone of the content from those websites (it saves HTML, JS, media files, PDFs, images and more).

I’ve done my set-up of ArchiveBox with the provided Dockerfile. Every once in a while it will start the docker container and check my Pocket feed for any new bookmarks. If found it will then archive those bookmarks.

As the HTML as well as PDF and Screenshot is saved this is extremely useful for later look-ups and even full-text search indexing.

Only in Japan

When you are searching the internet for more information and things to learn about Japan you will inevitably also find John Daub and his “Only in Japan” productions. And that is a good thing!

ONLY in JAPAN is a series produced in Tokyo by one-man band John Daub.  

Only in Japan Patreon page

Back in 2018 we even where around when John announced that he is going to live-stream.

And so we met up with him and eventually even said “Hi”.

this is John in full gear

Of course it wasn’t just us who got a good picture. We were part of the live stream as well – involuntarily as we had tried very hard to not be in frame.

Video Game History

Have you ever asked yourself what those generations coming after us will know about what was part of our culture when we grew up? As much as computers are a part of my story a bit of gaming also is.

From games on tape to games on floppy disks to CDs to no-media game streaming it has been quite a couple of decades. And with the demise of physical media access to the actual games will become harder for those games never delivered outside of online platforms. Those platforms will die. None of them will remain forever.

Hardware platforms follow the same logic: Today it’s the new hype. Tomorrow the software from yesterday won’t be supported by hardware and/or operating systems. Everything is in constant flux.

Emulation is a great tool for many use-cases. But it probably won’t solve all challenges. Preserving access to software and the knowledge around the required dependencies is the mission of the Video Game History Foundation.

Video game preservation matters because video games matter. Games are deeply ingrained in our culture, and they’re here to stay. They generated an unprecedented $91 billion dollars in revenue in 2016. They’re being collected by the Smithsonian, the Museum of Modern Art, and the Library of Congress. They’ve inspired dozens of feature films and even more books. They’re used as a medium of personal expression, as the means for raising money for charity, as educational tools, and in therapy.
And yet, despite all this, video game history is disappearing. The majority of games that have been created throughout history are no longer easily accessible to study and play. And even when we can play games, that playable code is only a part of the story.

Video Game History Foundation

How to design a Transit Map

We’ve all used them. And if they are made well they really make life easier: Transit Maps.

Apart from using transit map art style to visualize a train line transit maps can be applied to a lot of data visualization needs.

Take time to consider everything about your diagram. How thick do you want the route lines to be? Are they touching, or is there a slight gap between them? Are you going to use curves or straight edges where a line changes direction? Consider your station markers – will they be ticks, dots or something else? Think about how you would like to differentiate interchange stations or transit centres as well. Consider the typeface you’re going to use for station names – it should be legible and simple. When you’ve considered all these points, you’ve given yourself a set of rules that you will use to construct the diagram. Every design decision you make should be evaluated against these rules: sometimes, you can break them if needed, but it definitely helps to have them in your head as you work.

Tutorial: How to design a Transit Map

all macOS wallpapers in 5k

Every major version of Mac OS X macOS has come with a new default wallpaper. As you can see, I have collected them all here.
While great in their day, the early wallpapers are now quite small in the world of 5K displays.
Major props to the world-class designer who does all the art of Relay FM, the mysterious @forgottentowel, for upscaling some of these for modern screens.

https://512pixels.net/projects/default-mac-wallpapers-in-5k/

hyper-realistic japanese train

There are some things that, when looked at closely, just amaze you. This picture was like that for me:

I stumbled on this picture on Twitter. At first it looks like your average japanese train. With the blue seat cushions. The adverts and hand rails. All normal, right?

If you read the text or start to zoom in you will find that this is not a picture taken with a camera. This is a drawing.

This visual art is called Hyperrealism and it takes a lot of dedication and skill when done free-hand.

The artist had posted this picture on his Twitter feed. He explains that he had drawn it whilst commuting on the train over the course of a month. He used an iPad Pro and MediBang Paint. Of course he also names the train-type: Tobu Series 5050 used by the Tobu line in the greater Tokyo area.

blocking ads and promotions on twitter

When a group of people with the same problem meets, they work together and sometimes do an experiment.

Nobody likes ads or “promotional content”.

At some point Twitter chose to push ads in the official Twitter client into every timeline and decided to make them look like normal timeline content.

It did not take long for a group of people that do not like that to meet and join forces: Since about a week now a very small group of people has taken their Twitter block lists and merged them using the Block Together service.

This experiment great since it’s completely effortless. You link your block lists once and from thereon you keep using Twitter like you always did. Whenever you see a paid promotion you “block it”. Everybody from thereon will not see promotions and timeline entries from this specific Twitter user (unless you would actively follow them).

17326 accounts blocked! Wow! I started with about 3500 before merging with others.

And the effect after about a week is just great! I cannot see a downside so far but the amount of promotion content on my timeline has shrunk to a degree where I do not see any at all.

This is a great way to get rid of content you’ve never wanted and focus on the information you want.

the interesting bit about googles game streaming

In 2012 I’ve experienced streamed game play for the first time. I was a beta-user of the OnLive service which created a bit of fuzz back then.

Last week Google had announced to step into the game streaming business as well. They’ve announce Google Stadia as the Google powered game streaming platform. It would come with it’s own controller.

3 color variants

And this controller is the most interesting bit. We have seen video live streaming. We have seen and played streamed games. But every time we needed some piece of software or hardware that brought screen, controller and player together.

The Google Stadia controllers now do not connect to the screen in front of you. The screen, by all it knows, just shows a low-latency video/audio stream.

The controller connects to your wifi and directly to the game session. Everything you input with the controller will be directly sent to the Google Stadia session in a Google datacenter. No dedicated console hardware in between. And this will make a huge difference. Because all of a sudden the screen only is a screen. And the controller will connect to the “cloud-console” far-far away. As if it was sitting right below the screen. This will make a huge difference!

Digital Daily Routine as an Experiment – “Digitaler Alltag als Experiment”

Last week we were approached by Prof. Dr. Nicole Zillien from Justus-Liebig-University in Gießen/Germany. She explained to us that she currently is working on a book.

In this book an empirical analysis is carried out on “quantified-self” approaches to real life problems.

With the lot of information and data we had posted on our personal website(s) like this blog and the “loosing weight” webpage apparently we qualified for being mentioned. We were asked if it would be okay to be named in the book or if we wanted to be pseudonymized.

Since everything we have posted online and which is publicly accessible right now can and should be quoted we were happy to give a go-ahead. We’re publishing things because we want it to spur further thoughts.

It will be out at the end of 2019 / beginning of 2020. As soon as it is out we hope to have a review copy to talk about it in this blog once again.

We do not know what exactly is being written and linked to us – we might as well end up as the worst example of all time. But well, then there’s something to learn in that as well.

css font-feature “tnum”

Oh this is so useful for my head-up-display prototype implementation:

This feature replaces numeral glyphs set on glyph-specific (proportional) widths with corresponding glyphs set on uniform (tabular) widths. Note that some fonts may contain tabular figures by default, in which case enabling this feature may not appear to affect the width of glyphs.

tabular figures: tnum

proper links when printing out the internet

Cascading Style Sheets or CSS in short are a very powerful tool to control how content is being displayed.

CSS is designed to enable the separation of presentation and content, including layout, colors, and fonts. This separation can improve content accessibility, provide more flexibility and control in the specification of presentation characteristics, enable multiple web pages to share formatting by specifying the relevant CSS in a separate .css file, and reduce complexity and repetition in the structural content.
Separation of formatting and content also makes it feasible to present the same markup page in different styles for different rendering methods, such as on-screen, in print, by voice (via speech-based browser or screen reader), and on Braille-based tactile devices. CSS also has rules for alternate formatting if the content is accessed on a mobile device.

Wikipedia

So with CSS you can differentiate between target audiences. It gives you control over the output being rendered for specific render targets.

I frequently come across content I want to read. And almost as frequently I do not have time for a longer read when I come across interesting content.

My workflow for this is: keeping some to-be-read backlog of PDF files I have printed from websites. These PDF files are automatically synced to various devices and I can read them at a later stage.

What often is frustrating to see: bad the print results of website layouts as these websites have not even thought of the remote option of being printed.

With this blog I want to support any workflow and first and foremost my own. Therefore printing this blog adds some print-audience specifics.

For example the links I am using in the articles are usually inline when you are using a browser. When you’re printing the article those links get converted and are being written out with the text. So you can have them in your print-outs without loosing information.

this is how Google Chrome shows you the print preview…

And the changes you need to apply to any webpage to instantly enable this are very simple as well! Just add this to your page stylesheet:

@media print {
a {
text-decoration: none;
}
a::after {
content: "( " attr(href) " )";
margin-left: 0.2em;
text-decoration: underline;
}
}

something is coming up…

Since 2011 we’ve got this Boogie Board in the household. It’s simply a passive LCD panel on which you can write with a plastic pen. When you do you’re interacting with the liquid crytals and you switch their state. So what was black becomes white.

So we got this tablet and it’s magnetically pinned to our fridge. And whenever we’ve booked the next trip we’re crossing off days by coloring them in a grid.

How do you do such countdowns?

Twitter Blocklists

My usual twitter use looks like this: I am scrolling through the timeline reading up things and I see an ad. I click block and never again will I see anything from this advertiser. As I’ve written here earlier.

As Twitter is also a place of very disturbing content there are numerous services built around the official block list functionality. One of those services is “Block Together“.

Block Together is designed to reduce the burden of blocking when many accounts are attacking you, or when a few accounts are attacking many people in your community. It uses the Twitter API. If you choose to share your list of blocks, your friends can subscribe to your list so that when you block an account, they block that account automatically. You can also use Block Together without sharing your blocks with anyone.

blocktogether.org

I’ve signed up and apparently this is as easy as it gets when you want to share block lists.

There seem to be more people that use Twitter like I do. For example Volker Weber wrote about his handling of “promoted content”.

My block list on Twitter currently includes 1881 accounts and these are only accounts that put paid promotions without my request into my timeline.

I’ve read that Volker has such a long list as well – maybe it’s worth sharing as Volker is one person I would trust on his decisions for such a list. (vowe is a good mother!)

how do you organize your tasks?

For about 2 years now I am using Todoist as my main task management / todo-list service.

This lead to a lot of interesting statistics and usage patterns as this service seems to integrate oh-so-nicely into a lot of daily tasks.

What kind of integration is it? Glad you asked!

At first we were using all sorts of different ways to manage task lists across the family with the main lists around everything evolved being the personal tasks and todos of each family member as well as the obvious groceries shopping list.

We’ve been happy customers of Wunderlist before but then Microsoft bought it and announced they will shut it down soon and replace it with “Todo” out of Office 365. Not being an Office 365 customer did lead to a dead-end on this path.

And then Amazon Alexa showed up and we wanted to naturally use those assistants around the house to add things to shopping and todo lists right away. Unfortunately neither Wunderlist nor the intermediate solution Toodledo were integrated with Alexa.

Then there suddenly was a window of opportunity We wanted Alexa integration and at least all the features we knew from Wunderlist and Toodledo and Todoist delivered right out of the box.

It takes todos and shopping items from Alexa, through the website, through Apps, Siri can use it and in general it’s well integrated with lots of services around. You can even send it eMails! Also we’ve never experienced syncing issues whatsoever.

And it’s the little things that really make a difference. Like that Chrome browser integration above.

You see that “Add website as task”? Yes it does exactly what you would expect. Within Chrome and two clicks you’ve added the current website URL and title as a task to any of your lists in Todoist. I’ve never been a fan of favourites / bookmarks in browsers. Because I usually do not store any history or bookmarks for longer. But I always need to add that website to a list to work through later the day. I used to send myself eMails with those links but with this is a much better solution to keep track of those links and not have them pile up over a long time.

What’s also very nice is the way Todoist generates statistics and tracks your progress over time. There’s a system in Todoist called “Karma”.

Which allows you to marvel at your progress and sun yourself in the immense productivity you’ve shown.

But hey – there’s actual value coming from this. Like if you do it for a year or two you get such nice statistics which show how you did structure your day and how you might be able to improve. Look at a simple yearly graph of how many tasks have been completed at specific times of the day.

So when most people in the office spend their time on lunch breaks I usually complete the most tasks from my task list. Also I am quite early in “before the crowd” and it shows. Lots of stuff done then.

And improvements also show. On a yearly base you can see for example how many tasks you did postpone / re-schedule when. Like those Mondays which are currently the days most tasks get postponed. What to do about that?

I see artificial people

When people think of artificial intelligence, AI, they think of Alexa, Siri, Google Home and self-driving cars.

When an AI dreams of humans it dreams up their faces. No really.

Generative Adversarial Networks (GAN) are a relatively new concept in Machine Learning, introduced for the first time in 2014. Their goal is to synthesize artificial samples, such as images, that are indistinguishable from authentic images. A common example of a GAN application is to generate artificial face images by learning from a dataset of celebrity faces. 

Style-based GANs – Generating and Tuning Realistic Artificial Faces

This is some highly impressive stuff given that the system does produce fairly believable results without lots of distortions. You can see some distortions if you click on the image below and keep refreshing. Evertime it will generate a new face for you…

You definitely will not know this person:

a new header

I had redone the header of this blog a while ago but since I was trying around some things on the template I wanted something more dynamic but without any additional dependencies.

So I searched and found:

Tim Holman did a very nice implementation of this “worm generator” with only using the HTML5 canvas tag and some math. I made some very slight changes and integrated it into the header graphic. It will react to your mouse movement and resets if you click anywhere. Give it a go!

“kachung” + shutter sound

When you take a picture with an iPhone these days it does generate haptic feedback – a “kachung” you can feel. And a shutter sound.

Thankfully the shutter sound can be disabled in many countries. I know it can’t be disabled on iPhones sold in Japan. Which kept me from buying mine in Tokyo. Even when you switch the regions to Europe / Germany it’ll still produce the shutter sound.

Anyway: With my iPhone, which was purchased in Germany, I can disable the shutter sound. But it won’t disable the haptic “kachung”.

look ma! no mirror! (yes this is an iPhone 6)

It’s interesting that Apple added this vibration to the activity of taking a picture. Other camera manufactures go out of their way to decouple as much vibration as possible even to the extend that they will open the shutter and mirror in their DSLRs before actually making the picture – just so that the vibration of the mirror movement and shutter isn’t inducing vibrations to the act of taking the picture.

With mirror less cameras that vibration is gone. But now introduced back again?

Am I the only one finding this strange?

testing video

One thing I cannot do without linking to external sources or having control over the content storage is to have videos here on the pages.

There are a couple of options to achieve this and I am evaluating some of them right now. The goal is very clear:

  • no external links
  • no external resources embedded or included
  • 720p/1080p/2160p quality levels, ideally with bandwidth scaling

So let’s see some options tried out:

720p
1080p
2160p