IP-over-DNS

Picture yourself in this situation. You connect to a network and nothing works. Except for this:

It is quite common to have DNS working in networks while everything else is not. Sometimes the network requires a log-in to give you access to a small portion of the internet.

Now, with the help of a tool called iodine, you can get access to the full internet with only DNS working in your current network:

iodine lets you tunnel IPv4 data through a DNS server. This can be usable in different situations where internet access is firewalled, but DNS queries are allowed. 

It runs on Linux, Mac OS X, FreeBSD, NetBSD, OpenBSD and Windows and needs a TUN/TAP device. The bandwidth is asymmetrical with limited upstream and up to 1 Mbit/s downstream. 

iodine

Setting it up is a bit of work but given that you are anyway having access to a well connected server on the free portion of the internet it can be easily done.

Of course the source is on github.

making ICs at home

Try to wrap your head around this: There are people out there that take the term “Maker” to new levels. People Like Sam Zeloof. He went out and created his very own integrated circuit designs and then he built them. Like the actual silicon, the die, the bonded chip, the IC. The real thing.

Be inspired:

I am very excited to announce the details of my first integrated circuit and share the journey that this project has taken me on over the past year. I hope that my success will inspire others and help start a revolution in home chip fabrication. When I set out on this project I had no idea of what I had gotten myself into, but in the end I learned more than I ever thought I would about physics, chemistry, optics, electronics, and so many other fields. Furthermore, my efforts have only been matched with the most positive feedback and support from the world; I owe a sincere thanks to everyone who has helped me, given me advice, and inspired me on this project. Especially my amazing parents, who not only support and encourage me in any way they can but also give me a space to work in and put up with the electricity costs… Thank you!

Sam Zeloof

Decoding history, the hard way (and with machine learning)

Imagine you’ve got this ancient piece of technology in front of you. You clearly understand how the hardware works and you are even able to emulate the hardware on your modern-world computer.

Unfortunately hardware is only one half of the story. Software is the other half. And software at this time of the past was burned into chips which do not easily give their secret software away.

But let’s start with the hardware:

The IBM 5100 Portable Computer is a portable computer (one of the first) introduced in September 1975, six years before the IBM Personal Computer. It was the evolution of a prototype called the SCAMP (Special Computer APL Machine Portable) that was developed at the IBM Palo Alto Scientific Center in 1973. In January 1978, IBM announced the IBM 5110, its larger cousin, and in February 1980 IBM announced the IBM 5120. The 5100 was withdrawn in March 1982.

When the IBM PC was introduced in 1981, it was originally designated as the IBM 5150, putting it in the “5100” series, though its architecture was not directly descended from the IBM 5100.

And now on to the software:

The IBM 5100 portable computer came with some of its built-in programs stored in a read-only memory called the “non-executable ROS”. (ROS = “read-only storage”.) In contrast with the “executable ROS”, which supplies instructions to the 5100’s processor directly, the non-executable ROS is accessed using sequential I/O operations, a bit like a tape.

Most notably, the non-executable ROS holds the interactive interpreters for the APL and BASIC programming languages. These are not “native” 5100 programs but were expressed instead in System/370 mainframe and System/3 minicomputer machine code respectively. The 5100 runs emulator programs for those computers in order to host the interpreters, so perhaps it’s just as well that the non-executable ROS is non-executable.

DATA

So this write-up is all about how the bits where pushed to the screen and recorded as pictures of the said screen. The characters in these pictures then where analyzed and with the help of machine learning the data could be successfully extracted. It is mind-boggling. And it is all on Github.

Bitmap & tilemap generation with the help of ideas from quantum mechanics

You can get a grasp at the beautiful side of science with visualizations and algorithms that output visual results.

This is the example of producing lots and lots of complex data (houses!) from a small set of input data. It is widely used in game development but also can be helpful to generate parameterized test and simulation environments for machine learning.

So before sending you over to the more detailed explanation the visual example:

This is a lot of different house images. Those are generated using a program called WaveFunctionCollapse:

WFC initializes output bitmap in a completely unobserved state, where each pixel value is in superposition of colors of the input bitmap (so if the input was black & white then the unobserved states are shown in different shades of grey). The coefficients in these superpositions are real numbers, not complex numbers, so it doesn’t do the actual quantum mechanics, but it was inspired by QM. Then the program goes into the observation-propagation cycle:

On each observation step an NxN region is chosen among the unobserved which has the lowest Shannon entropy. This region’s state then collapses into a definite state according to its coefficients and the distribution of NxN patterns in the input.

On each propagation step new information gained from the collapse on the previous step propagates through the output.

On each step the overall entropy decreases and in the end we have a completely observed state, the wave function has collapsed.

It may happen that during propagation all the coefficients for a certain pixel become zero. That means that the algorithm has run into a contradiction and can not continue. The problem of determining whether a certain bitmap allows other nontrivial bitmaps satisfying condition (C1) is NP-hard, so it’s impossible to create a fast solution that always finishes. In practice, however, the algorithm runs into contradictions surprisingly rarely.

Wave Function Collapse algorithm has been implemented in C++PythonKotlinRustJuliaGoHaxeJavaScript and adapted to Unity. You can download official executables from itch.io or run it in the browser. WFC generates levels in Bad NorthCaves of Qudseveral smaller games and many prototypes. It led to new research. For more related workexplanationsinteractive demosguidestutorials and examples see the ports, forks and spinoffs section.

useful CSS Grid Generator

This project is a way for people to use CSS Grid features quickly to create dynamic layouts.

You can set the numbers, and units of your columns and rows, and I’ll generate a CSS grid for you! Drag within the boxes to create divs placed within the grid.

I noticed a lot of people weren’t using Grid because they felt it was too complicated and they couldn’t understand it. Grid is capable of so much, and this small generator only touches on a fraction of the features. The purpose of this is so people get up and running quickly, and create more interesting layouts.

Once you work with this a bit, I suggest checking out resources by Rachel AndrewJen Simmons, and Dave Geddes to dive deeper. There is also a CSS Grid Guide on CSS-Tricks, and a fun little game called Grid Garden to help you learn more!

Source

Craft Beer in Tokyo (Kanto)

There are lists of things all over the internet. I’ve linked to some of these food related lists and maps already on this blog.

And of course there’s this list of craft beer breweries missing:

microbrewery or craft brewery is a brewery that produces small amounts of beer, typically much smaller than large-scale corporate breweries, and is independently owned. Such breweries are generally characterized by their emphasis on quality, flavor and brewing technique.

Wikipedia

seen: properly scrolling through code

When working on source files with wide-ranging scopes, I wish source editors could pin the declaration lines to the top of the window like section headers, something like this…

Joe Groff on Twitter

This looks like something that would really make a difference when editing code. Let’s see how long until we get something like that in modern editors…

I/O Is Faster Than the CPU

I/O is getting faster in servers that have fast programmable NICs and non-volatile main memory operating close to the speed of DRAM, but single-threaded CPU speeds have stagnated. Applications cannot take advantage of modern hardware capabilities when using interfaces built around abstractions that assume I/O to be slow. We therefore propose a structure for an OS called parakernel, which eliminates most OS abstractions and provides interfaces for applications to leverage the full potential of the underlying hardware. The parakernel facilitates application-level parallelism by securely partitioning the resources and multiplexing only those resources that are not partitioned.

https://dl.acm.org/citation.cfm?doid=3317550.3321426

“are you still watching?” – tailfix

drop in replacement for tail -F that asks you if you are still watching

Just like Laura I am also was having a moment when I stumbled across tailflix.

For those not understanding the reference: At the end of an episode you’ve watched on Netflix you will be shown another one, and so on, and so on. Until if you have not touched the remote at all for several episodes Netflix will ask you “Are you still watching?”.

serve live interactive Jupyter notebooks

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

voilá

Voila serves live Jupyter notebook including Jupyter interactive widgets.

Unlike the usual HTML-converted notebooks, each user connecting to the Voila tornado application gets a dedicated Jupyter kernel which can execute the callbacks to changes in Jupyter interactive widgets.

https://github.com/QuantStack/voila

Look me in the eye

As tools get better artists really polish out what technology enables them to do with computers nowadays.

This eye above made me awww. It’s from a user “ChrisJones” in the Blender Forum who posted his progress on modelling a human head in upmost detail.

Go there and be awwd as well. And while you’re there grab a copy of Blender and give it a go.

clone files from and to the cloud

Rclone is a command line program to sync files and directories to and from:

Alibaba Cloud (Aliyun) Object Storage System (OSS), Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dreamhost, Dropbox, FTP, Google Cloud Storage, Google Drive, HTTP, Hubic, Jottacloud, IBM COS S3, Koofr, Memset Memstore, Mega, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, Nextcloud, OVH, OpenDrive, Openstack Swift, Oracle Cloud Storage, ownCloud, pCloud, put.io, QingStor, Rackspace Cloud Files, Scaleway, SFTP, Wasabi, WebDAV, Yandex Disk, The local filesystem

rclone.org

Whenever you would need to move large and/or many files between cloud storages you should give rClone a try.

technical visualization tools

There’s so much happening in this field as visualizations become more powerful and easier to create.

WaveDrom

WaveDrom draws your Timing Diagram or Waveform from simple textual description. It comes with description language, rendering engine and the editor.

WaveDrom editor works in the browser or can be installed on your system. Rendering engine can be embeded into any webpage.

https://wavedrom.com/

MermaidJS

Generation of diagrams and flowcharts from text in a similar manner as markdown.

Ever wanted to simplify documentation and avoid heavy tools like Visio when explaining your code?

This is why mermaid was born, a simple markdown-like script language for generating charts from text via javascript. Try it using our editor.

the future of (speech) podcasts

The Podlove project once again leads the way to improve the experience and the way we interact with knowledge and thoughts. With the most current announcement and introduction of transcription-support for podcasts. Click on it right now and try it yourself.

Fulltext-search. Listening to podcasts by reading them. This-is-amazing!

Transcripts are coming

Transcripts are an incredibly desirable thing to have for podcasts: they allow searching for specific parts, increase searchability by search sites when presented properly and they increase the accessibility of audio content significantly too.

However, transcripts have been considerably difficult to be created and used. Manually created transcripts are costly in terms of time and money and even if you spend the money there has been a lack of technical standards for storing and integrating transcripts into websites in a defined way.

This is now slowly changing: more and more automated speech-to-text systems are becoming available at reasonable costs and they are creating ever better transcripts with more and more languages being supported.
Still, automatic transcripts trail manually created transcripts in terms of accuracy, punctuation and so on but they are increasingly useful when they are primarily used for improving search results or helping you with your internal research when trying to find content in your older episodes.

New services are also coming up to deal with these problems by allowing users to quickly build on automatic transcripts and improve them manually in an assisted fashion. We will soon see a landscape of tools and services that will make creating transcripts easy and cheap enough for more and more podcasters so it’s time to come up with a good integration.

Last but not least, the WebVTT file format has become a de-facto common denominator for passing transcripts along, supporting time codes, speaker identification and a rudimentary set of meta data. While not perfect it’s enough to get a transcript infrastructure up and running and Podlove is leading the way.

from the Podlove Publisher 2.8 announcement

make linux fast (unsecure) again

The CPU/hardware related bugs surfacing the last couple of years have mostly been fixed by adjusting the software that is run. Sometimes only by disabling certain features of a CPU or patching the microcode in the CPU itself.

The issue with this is that by fixing these issues features got disabled and workarounds had been introduced that lowered performance. Dramatically so for some use-cases.

By how much? Well it really depends on your CPU and use-cases. But maybe you want to try yourself. If you want to know the most current parameters to pass to your kernel on boot-up to disable all the performance impacting fixes, go here:

It is not recommended to have this in productive use – as you can imagine. Those bugs where fixed for a reason.

look at the earth

Watch sunlight and weather patterns move across Earth throughout the day, and bask in the glory of our blue marble in real time.

Every 20 minutes (or every hour, you pick), Downlink updates your desktop background with the newest images of Earth.

Choose from 8 different views of Earth, including stunning full disk images from 3 different geostationary satellites.

Downlink is a macOS application which downloads the most current image taken by earth orbitting satellites. So what you see is as accurately depicting reality as technically possible right now.

Interestingly all this data comes from public domain sources. And the makers of the app have documented their sources:

Thanks to NASA, NOAA, JMA, Lockheed Martin, Harris Corporation, ULA, MHI, and everyone else who designed, built, launched, and operate GOES-16, GOES-17, and Himawari-8.
If you’d like to use the sources Downlink is built on top of, here is the file pointing to those resources. Use it, and tell me what you’ve built!

I am using macOS as well as Linux as my main desktop operating systems. My macs are set with Downlink. But what about my Linux machines?

Easy! The file referring to the sources looks like this:

With some simple steps on Linux we can grab the URL we want, download the image and update the desktop background.

Step 1: Getting the URL from this JSON.

Install jq, curl and xargs and in your shell run this:

curl -s http://downlinkapp.com/sources.json | jq -r '.sources[2].url.full' 

This will give you the URL “full” of the 2nd source in the list (Hiwamari-8).

Step 2: Download the image.

curl -s http://downlinkapp.com/sources.json | jq -r '.sources[2].url.full' | xargs -I{} curl -s -o background.jpg {}

By simply adding the xargs and curl call it will take the URL output from the command in front as input to the second curl call and download it. It also will store the file as “background.jpg”.

Step 3: Setting as background image.

Setting the background image of your desktop depends on what desktop software you are using on Linux. Depending on that you will need to look up in the manual how to set the background image from the command line.

But for most use cases there’s a tool that helps. It’s called Variety.

Variety is an open-source wallpaper changer for Linux
Variety is packed with great features, yet slim and easy to use. It can use local images or automatically download wallpapers from Unsplash and other online sources, allows you to rotate them on a regular interval, and provides easy ways to separate the great images from the junk. Variety can also display wise and funny quotations or a nice digital clock on the desktop.

bumps ahead

This website is delivered to you by a single dedicated server in a datacenter in Germany. This server is old.

11:13:58 up 1320 days, 25 min, 2 users, load average: 1.87, 1.43, 1.25

uptime

And I am replacing it. While doing so I am going to take some shortcuts to lower the effort I have to put in for the move.

It will save me 2 days of work. It will mean for you: there might be some interruptions of the services provided by this website (there are more than this page…).

legit programming language

Programs written in legit are defined entirely by the graph of commits in a Git repository. The content of the repository is ignored.
legit is designed so that all relevant information is visible when running git log --graph --oneline.

For example, here is “hello world”:

You can find implementations of legit, as well as some example programs, on GitHub: https://github.com/blinry/legit. The entry in the Esolang wiki is at https://esolangs.org/wiki/Legit.

80s code editor theme

The beginning of the decade saw the continuation of the clothing styles of the late 1970s and evolved into heavy metal fashion by the end. However, it had a lot of changes considering that, this fashion became more and more extravagant during the 80s.

The 80s included things like teased hair, ripped jeans, neon clothing and lots of colours and different designs which at first weren’t accepted for a lot of people.

Popular Culture in the 80s

Do you remember that endless summer back in ’84? Cruising down the ocean-highway with the top down, the wind in our hair and heads buzzing with neon dreams?
No, I don’t remember it either, but with this experimental theme we can go there.

Synthwave’84 theme

transparently migrate data from local file storage to key-value-stores like RIAK KV

About 2 years ago I sat down and wrote a filesystem. Well not from scratch but using the great FUSE (Filesystem in Userspace) framework. I’ve released it as open-source later on Github:

This script acts as glue between a local file storage mount point and RIAK. It is targeted at specific use cases when local mount-points need to be migrated to RIAK without changing the applications accessing those mount point. Think of it as a transparent RIAK filesystem layer with multiple options to control it’s behavior regarding local files.

riakfuse

I had a very specific purpose in mind when I wrote this: There was a local filesystem that got filled up and because of technical restrictions we were not able to resize it or even completely copy it without interrupting the service using the data stored there.

Since we were already using the RIAK Key-Value Database for certain binary loads the idea came up to also utilize this key-value concept for a filesystem.

The idea now is: You have a local filesystem that holds a lot of folders and files already and you want to gradually want to move it to new grounds.

This migration needed to happen with minimal service interruptions assuming that there is constant reading and writing happening.

In this riak-fuse project I’ve written an overlay filesystem that steps between the application and the underlying “old”-filesystem. It looks and behaves identical to the application reading and writing.

But, depending on the mode you have chosen while mounting, every file read will at first be read from the “old”-filesystem and after successfull read stored into the key-value store.

On the next read it will be read from the key-value store directly.

The same applies for writing. Riak-fuse will write either to both, local storage and key-value store or just to the key-value store.

So in a nutshell: Data is slowly but surely on each access transferred over to the key-value store and load + storage space is slowly moving over from the local storage to the key-value store.

To facilitate this there are a lot of options for this script:

This all comes with ready-to-use docker and docker-compose files for you to try out.

Also it might interest you as an extremely simplified example of how to write an actual file system module for FUSE in Python.

Disclaimer: This effectively is my first python script as well as fuse module. Don’t be too hard on your judgement.

Time estimation in software development

I’ve found myself in these spots several times in my life. Either I had to deliver on an estimate or I had to acknowledge an estimation and deal with the outcomes.

If you are involved in anything digital / software this is a recommended piece to read:

Anyone who built software for a while knows that estimating how long something is going to take is hard. It’s hard to come up with an unbiased estimate of how long something will take, when fundamentally the work in itself is about solving something. One pet theory I’ve had for a really long time, is that some of this is really just a statistical artifact.

Why software projects take longer than you think

convert Markdown scribbles to vector drawings

Every task you take
Every meeting you make
Every keyboard you break
Every note you take
I’ll be storing it for you

my text editor

Well that was fun! And indeed: a big portion of my professional daily business is taking place in a text editor taking notes and scribbling ideas and thoughts.

I’ve tried many things but the only way that resonated with me was taking notes in Markdown in a text editor that supports markdown. Currently that editor is Atom.io. Mainly because it is not in the way and quite portable. Runs on Linux, Windows, MacOS.

extra cheesy 80s neon theme

This way – I just took a count – I noted down 364.416 words in the last 1.5 years on my current job (equals to about 46 hours of average speed reading…).

Along side those simple text notes and bullet lists I am doing very simple tables as well as ASCII scribbles in Markdown as well. With the right tools it’s extremely easy and much faster than booting up the Powerpoint or worse.

When you have all in Markdown you then can freely stylesheet away and convert to handy PDF files as well. All even with embedded images if you so desire.

But even if you sit on that treasure trove of Markdown there comes the time when you wish you could convert your scribbles to graphics. Even if it is for the sole reason to not have to draw it again for that fancy Powerpoint slide deck.

You’ve got multiple options to accomplish this:

svgbob is at first a command line tool that got a recent level-up with a proper web-frontend:

When given Markdown it creates graphics. In the picture above the input is on the left and the svgbob output on the right (as SVG).

Markdeep is the alternative. Which of both work for your case depends on that specific case. Knowing and using both properly is the best way.

from now on console emulation will look much better

A modder going by the handle DerKoun has released an “HD Mode 7” patch for the accuracy-focused SNES emulator bsnes. In their own words, the patch “performs Mode 7 transformations… at up to 4 times the horizontal and vertical resolution” of the original hardware.

The results, as you can see in the above gallery and the below YouTube video, are practically miraculous. Pieces of Mode 7 maps that used to be boxy smears of color far in the distance are now sharp, straight lines with distinct borders and distinguishable features. It’s like looking at a brand-new game.

ArsTechnica

Michelin Guide Restaurants in Tokyo

You are likely aware of the existence of the “Michelin Guide“.

Michelin Guides are a series of guide books published by the French tyre company Michelin for more than a century. The term normally refers to the annually published Michelin Red Guide, the oldest European hotel and restaurant reference guide, which awards up to three Michelin stars for excellence to a select few establishments.

Wikipedia

You might also be aware that Tokyo is the city with the highest density of Michelin star rated restaurants. Nice, eh?

A purchase of this guide is recommended in any case but these days people also need something they can intuitively use and which integrates into already existing workflows.

These people, like myself, need a map and maybe more details in a machine readable, filterable spreadsheet.

And as time goes on it might be quite useful to have all the sources that lead to these great tables and maps. Sources that allow you to crawl and grab these information.

A script that crawls Tokyo-based michelin guide establishments and saves it into a JSON file. I personally did this project so I can plan my tokyo trip based on the cheapest and most-renowned restaurants,

Michelin Guide Crawler on GitHub

Swappiness is a thing, as is cache pressure

We know that using swap space instead of RAM (memory) can severely slow down the performance of Linux. So then, one might ask, since I have more than enough memory available, wouldn’t it better to remove swap space completely? The short answer is, No. There are performance benefits when swap is enabled, even when you have more than enough ram.

Why you should almost always add swap space

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. (default = 100, recommend value 50 to 200)

swappiness – This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. (default = 60, recommended values between 1 and 60) Remove your swap for 0 value, but usually not recommended in most cases.

https://access.redhat.com/solutions/103833

As I’ve now brought up the topic, go ahead and read the complete story at the source.

procedural generated traditional Chinese landscape scrolls

{Shan, Shui}* is inspired by traditional Chinese landscape scrolls (such as this and this) and uses noises and mathematical functions to model the mountains and trees from scratch. It is written entirely in javascript and outputs Scalable Vector Graphics (SVG) format.

https://github.com/LingDong-/shan-shui-inf

This is quite impressive and I am thinking about pushing that into the header of this blog :-) It’s just too nice looking to pass on.