Homing on Code

PGCon 2019 Trip Report

Sun, 16 Jun 2019 04:00:00 +0200

During my flight I finally decided to watch the “Shape of Water”. I knew the film was critically acclaimed but I wasn’t expecting a lovecraftian theme and it never clicked with me that it was directed by Guillermo del Toro - it was a very fitting thing to watch during a flight above the ocean depths and now I want “At the mountains of madness” to finally happen even more.

The last leg of the flight was operated by Air Canada and thankfully uneventful apart from a small delay. I landed in Ottawa and was at my hotel at around 2 am where I ended up waiting for a battery replacement on a smart-lock.

I had two days before @oshogbovx arrival. I spent the first one on a walk along the Rideau Channel. I was astounded by the amount of nature interveined with the city. Trees, flowers (so many tulips) and animals. On my way back I even spotted a beaver or a badger but didn’t want to get too close to annoy it. That’s in the middle of the city.

You can open up the images in a new tab and remove the trailing ‘-s’ from the file name to see the original quality picture.

On Tuesday the tutorials of PGCon have started but I haven’t been subscribed to them. I decided to spend the day on museums. I was naive in thinking that I can do more than one in a day. I took 811 pictures inside the Museum of History and only managed to cover two displays (Neanderthal, First Peoples Hall). They very kindly didn’t kick me out when I triggered an alarm by leaning too far over the display to take a picture and only gently hinted that I should probably go when it was well past 15 minutes from closing.

Halibut hook, octopus design. The Octopus was also called Devilfish

Mariusz arrived later that evening and we spend the rest of it touring the city itself. The parliament building is very impressive and apparently they do light shows using the building façades. Unfortunately those were scheduled after our planned departure so we didn’t see the real deal but we did see a testing sequence they ran over the building in preparation for the seasonal performances.

Parliament building during the evening

Parliament building with a light show test sequence

On the next day the Unconference started. I went in expecting a hackathon experience. The group listed out 26 topics and a vote was taken to pick projects to hack on as the amount of time and rooms were both limited. Finally 12 topics emerged and composed into three tracks.

PGCon Unconference voting on topics

We all moved to the room corresponding with our track of interest. At that point I expected everyone to grab their laptops and start hacking on code and experimenting with the idea. That however did not happen. The unconference is essentially a series of unscheduled talks proposed by the attendees and picked by the attendees on site. Don’t get me wrong - the topics were extremely interesting and I don’t regret participating. I would however love another day scheduled or a track for hands on hacking.

Unconference @PGCon has been started. And now we know TOP-12 hot topics for developers (not end users probably)! pic.twitter.com/kXgDgwbp98
— Pavlo Golub (@PavloGolub) May 29, 2019

We picked the following path through the unconference:

Zed store
Direct IO & Async IO
Fault injection framework
Constrain session memory

The re-emerging topic through all of PGCon was #fsyncgate. Both the direct IO and fault injection framework talks directly related to it. During the fault injection talk questions were raised how to test system level misbehaviours like fsync discarding data. This turned into a mini hackathon that I and Mariusz attempted in the evening by trying to hook the syscall with dtrace and alter its behaviour (unsuccessfully at that time but we had other ideas later on).

After the unconference, it was time to claim our badges. I loved the idea of handing them out at the Royal Oak pub. We were in I think a pretty unique situation, we basically raided the conference - not knowing anyone in the community. On top of that PGCon is known as the most developer heavy PostgreSQL conferences so those attending usually knew each other already. Having the badge handout in a social setting made it easier to start some casual conversations. We even met an OpenBSD user among the conference attenders.

Off to Day 1. Short and to the point opening session by Dan, kicked off the regular talks.

Our track choice was as follows:

Challenges of Concurrent DDL, Robert Haas
Transactions, Thomas Munro
An Introduction to Memory Contexts, Chris Travers
Intro to Postgres Planner Hacking, Melanie Plageman
Pluggable Table Storage in PostgreSQL, Andres Freund
Lightning talks

I found transactions, introduction to memory contexts and the intro to planner hacking as the most interesting talks for someone intending to start hacking on PostgreSQL itself. Especially the planner hacking talk. I basically had a specific query that was not using an optimization that should have been obvious and was hoping to approach someone during PGCon and see how hard it would be to implement that myself after some guidance. The talk covered exactly that plus discussed when & why you might not want to do so. I approached the speaker after the talk and we spent some time hacking on my use case, it won’t likely result in any diffs to PostgreSQL as I’m pretty much unable to reproduce it on a newer version (which is good).

The Oracle lightning talk was hilarious. Do watch the recording when it pops up online.

This was also the evening of the social event. The conference organized catering in a pub. Very nice location and the food was simply amazing. This is where we finally had a chance to speak with most of the people attending and learn more about the PostgreSQL community itself.

I was extremely surprised to learn that Postgres has around 30 committers. I expected much higher numbers considering its popularity. We learned that most committers are hired full time to work on PostgreSQL and it’s pretty hard to do significant meaningful work for the project without very large commitments in time and resources. That’s just the nature of the project and it did show in the scale of topics that the unconference talks covered - they were usually large scale and well planned projects like direct IO, new storage etc. If I understood correctly, the expected time frame for obtaining a commit bit varies between months and years and that’s mostly because a significant contribution is non-trivial to achieve.

It was great to speak with experienced people. Learn what workloads are usual for PostgreSQL and by extension where problems in our deployments are caused by hardware and where we are hitting design limitations or are doing something suboptimal ourselves. We were given very good suggestions to try for some of specific use cases we had that I described in casual talks. My awareness of other index types like RUM and BRIN have been raised significantly.

Totems were much taller than I ever expected, surprisingly also not round but a halved, hollowed trunk

Day 2. I really wanted to attend “Viva, the NoSQL Postgres!”. It’s about jsonpath - a query language for json and presented by Alexander Korotkov and Oleg Bartunov. Both (among other things) are the GIN index implementation authors which our talk is about. We also plan to use the PostgreSQL json features at work - essentially I have been reading a lot of their code and articles in the past few months. I did have a chance to speak with Oleg and learned about RUM indexes which I definitely need to try.

Unfortunately we were not able to attend their talk as ours was scheduled at the same time! Seeing that I was pretty sure our room would be pretty much empty with everyone learning about the new cool feature that only days ago fronted on hackernews. We came in early, about an hour before the talk started and there were 3 people in the room. We started preparing and fortunately the room filled up almost fully before we started. I was pretty stressed speaking about PostgreSQL internals to a bunch of people who most likely know more about the code base. In general I think we did good and conveyed the message we wanted. One person in the audience was on BSDCan and stayed for PGCon just to attend our talk. We ended up grabbing some beers with him later that evening.

‘namaxsala reflects on human relationships with the natural world. The sculpture depicts the artists grandfather helping a wolf cross treacherous waters by boat.

For the remaining talks we picked the following to attend:

Transparent data at rest encryption in PostgreSQL, Masahiko Sawada
Pluggable table access methods (pluggable storage), Pankaj Kapoor
Advanced Authentication, Stephen Frost
Toward Implementing Incremental View Maintenance on PostgreSQL, Yugo Nagata
Introducing PostgreSQL SQL parser through an experience of reusing it in other applications, Bo Peng

There were two encryption related projects being talked about during PGCon. Both focused on storage encryption (one on tablespace level), we were hoping for a bit more fine grained control like selective column based encryption and access control for that (ie. who can encrypt & who can decrypt that column). I guess those storage based features that were presented are nice if you are running Postgres on a system not offering full disk encryption.

I liked the advanced authentication talk by Stephen Frost. It was a very nice overview of what Postgres is capable of. I also took some of his time after the talk and bounced one or two crazy ideas off him (unrelated to the talk) like disabling PostgreSQL parser at runtime to prevent SQL injections (by allowing only prepared statements which is viable if you use a library like anosql).

The SQL query parser talk was also extremely valuable. I actually did something very similar for work when we wanted to do some additional lint checks against our schema as a post-commit hook. Things like detecting most likely missing indexes on foreign-keys. Bo Peng gave a very good run-down of the query parser, its structure and the intermediate AST generated by the parser. This would have been a great time saver if I heard this talk before working on my task.

Falls inside Ottawa city, the river they fall into is magnificent

The conference ended with an auction, on the very next day we still had a few hours to go around the city. Lots of buildings (like embassies) were open for tourists that day - unfortunately we did not have that much time. There were still many places to see and things to do and that is a very good motivation to be there again for the next conference. Be it BSDCan, PGCon or both of them.

I would like to thank the @PGCon conference organizers & the sponsors for covering my travel & accommodation. I would also like to thank my employer Fudo Security for making hacking on PostgreSQL and BSDs my day job and considering my PGCon attendance as part of it.

OpenBSD Daily Recap

Sun, 10 Sep 2017 04:00:00 +0200

The last OpenBSD daily read happened on August 18th. I decided to cancel the next one that was scheduled for August 25th and promised to do a recap on this blog over the weekend.

Had no votes for an #OpenBSD-daily so cancelling it for today. Will digest and throw out a blog post over the weekend.
— mulander (@mulander) August 25, 2017

It’s been over two weeks since that tweet so here is the overdue recap.

The daily started in june we kept up doing it daily for 42 days straight after which the schedule changed to weekly and continued for 5 more reads totalling 47 code reads by 3 people with ~130 people idling on the channel during most reads and seeing active participation from around 5.

All reads are archived in this github repository which was compiled & curated by @niamtokik (thanks!).

Why did the schedule change to weekly? Reading code on IRC in the format I devised proved to have some drawbacks that I didn’t anticipate. It’s very challenging to read code you see for the first time in your life, make sense of it and constantly update other people via short text messages and links to the code you are looking at. This leaves very little room for exploratory changes, especially in the kernel where the whole channel would have to wait for me to reboot before I could see the effects of what I did. That also meant not picking on low hanging fruits that were spotted during reads, if my evening is spent on reading code I am not stopping to actually do some changes that I think could be helpful, initially that was mitigated by people participating in the read attempting to do that work while I kept reading, hence making the effort much more worthwhile. I am extremely grateful to duncaen and DuClare for doing reads in the early days of the channel and actively participating in other code reads - that helped me kept going daily for 42 days in a row. With all of that said, I did notice the quality of my reads diving down very quickly as real life started catching up making me go into code reads completely unprepared

No #OpenBSD-daily today. Need to think how the format should continue. Reading daily without an overall plan started to degrade read quality
— mulander (@mulander) July 15, 2017

Going weekly was supposed to be the remedy. I still hoped that other people would pick up and fill in the missing days with their own code reads. I was even thinking that stopping doing dailies might make some people come up front now that each day isn’t booked ahead by me - sadly this didn’t happen. Activity on the channel started to dwindle, including during code reads - to a point where the last one didn’t receive a single vote directing on what should we do on the read. Having no people interested in the reads beats the purpose of doing it since digesting the code for the channel as noted above doesn’t leave much wiggle room for actually hacking on it. That is the point when I decided to cancel further reads for now.

Having everything said, there were very nice positive outcomes from the reads. They resulted in 11 commits to the OpenBSD source tree, personally I am most happy about the OpenSMTPD connection leak being diagnosed and fixed by DuClare during our reads. It was also great learning how the OpenBSD malloc and dynamic library loading works. I am completely convinced that reading the code is the best way to learn about the system and continue to do so on a daily basis - just on my own pace.

In summary, I think IRC is a terrible medium for live code reads and unfortunately I don’t see myself doing twitch code reading streams ;) yet IRC is a very good medium for obtaining help when you get stuck. You can see that happening many times in my reads where DuClare jumped in to help me decipher specific bits of C and I saw this happen many times outside reads where people were discussing pieces of code they were reading and had questions about. I would like to encourage everyone to use the #openbsd-daily freenode channel exactly for that purpose. This idea is not dropped, I’m just putting a broken format on hold until I find a better way to do live reads. In the meantime, drop by and tell us about the code you have been reading - who knows, maybe we can help you make a crack in a brick wall you have been hitting for a while?

OpenBSD Daily

Fri, 09 Jun 2017 04:00:00 +0200

I made a new years resolution to read at least one C source file from OpenBSD daily. The goal was to both get better at C and to contribute more to the base system and userland developmnet. I have to admit that initially I wasn’t consistent with it at all. In the first quarter of the year I read the code of a few small base utilities and nothing else. Still, every bit counts and it’s never too late to get better.

Around the end of May, I really started reading code daily - no days skipped. It usually takes anywhere between ten minutes (for small base utils) and one and a half hour (for targeted reads). I’m pretty happy with the results so far. Exploring the system on a daily basis, looking up things in the code that I don’t understand and digging as deep as possible made me learn a lot more both about C and the system than I initially expected.

There’s also one more side effect of reading code daily - diffs. It’s easy to spot inconsistencies, outdated code or an incorrect man page. This results in opportunities for contributing to the project. With time it also becomes less opportunitstic and more goal oriented. You might start with a drive by diff to kill optional compilation of an old compatibility option in chown that has been compiled in by default since 1995.

chown was standardised in 1992 deprecating the `.` dot separator - 25 years later all chown implementations still support it due to wide use
— mulander (@mulander) May 27, 2017

Soon the contributions become more targeted, for example using a new API for encrypting passwords in the htpasswd utility after reading the code of the utility and the code for htpasswd handling in httpd. Similarly it can take you from discussing a doas feature idea with a friend to implementing it after reading the code.

I was having a lot of fun reading code daily and started to recommend it to people in general discussions. There was one particular twitter thread that ended up starting something new.

Pick C. Read one source file from #OpenBSD /usr/src daily. It's fun. Me and @tuxbsd have been doing this lately. We could do group reviews.
— mulander (@mulander) June 5, 2017

Just registered #openbsd-daily on freenode. Feel free to jump in.
— mulander (@mulander) June 5, 2017

This is still a new thing and the format is not yet solidified. Generally I make a lot of notes reading code, instead of slapping them inside a local file I drop the notes on the IRC channel as I go. Everyone on the channel is encouraged to do the same or share his notes in any way he/she seems feasable.

In the first 3 days of the channel we:

read the source of httpasswd (sorry, I didn’t grab the log for that one)
learned how htpasswd files are handled in httpd
read the code of doas
implemented a new feature in doas

I’m having fun so far, reading code in a group is proving to be a great way to learn. Feel free to drop by on #openbsd-daily @ Freenode! ;)

myPhone Hammer Iron 2 - budget phone review

Tue, 25 Apr 2017 04:00:00 +0200

I hate Android phones. I had a few from higher to low end brands. They never received upgrades on a timely matter, had build quality issues and over time got bloated enough that the base system was not able to update the bundled apps anymore. Not to mention leaving enough space for anything else. The one I had with me during g2k16 wiped itself due to overheating and recently ended it’s life by being dropped off a 30cm table by our dog. To double the trouble my wifes phone decided it’s a good time to have a dying battery - so suddenly I was faced with the need of getting 2 new phones.

I didn’t want to go on another contract to get both of us iPhones and can’t justify spending that much money on two new models (or even used ones). It would also be nice to get something that won’t die from a single drop - I even considered dumb phones but we both want to use strava/endomondo on the upcoming dog trekking event and I need 2FA on my phone.

What phone could I get with a shoe string budget, that’s fairly decent and survives accidents that killed my previous phone?

Perhaps one that survives being treated with acid used to clear pipes as demonstrated on the video below (ignore the audio, I set the start time on the action)?

For just under 120 USD we get a phone with:

Android 6.0
8 GB of built in storage
dual sim (not a hybrid slot)
microSD slot up to 32GB
quad-core, 1.3 GHz processor
5 Mpx back and 2 Mpx front cameras
4 inch display (480 x 800 px)
Li-Ion 2400 mAh battery (swappable!)

The specs aren’t impressive performance wise. The main selling point for this phone is being certified IP-68 this means that this phone can be submerged down to 3 meters under water while turned on and is completely dust tight. The case is also hardened and shock proofed plus it comes with a protective foil on the screen itself. If you go through more parts of the linked YouTube video you will see it being put through much more harsh tests.

I’m not delusional. I don’t expect it to survive anything, even the things I saw actual reviewers do. I am convinced though that it has a better survival chance against my typical use cases and accidents that can happen during my normal usage.

This phone literally has no ENG reviews online. Except the product page. The company is from Wroclaw Poland (so local country wise to me) and there has been a decent amount of reviews on the Polish interwebs. They obviously still manufacture in China.

Biggest flaws reviewers pointed out were:

battery readouts
lack of LTE (it has 3G)
audio issues on built in speaker, especially in low temperatures (craclking, glitching, voice going mute)
annoying sign up to myPhone community pop-up app

I hit all four and have one of my own to add.

The battery. Let’s start off saying I am completely happy and actually quite surprised with how long the battery holds. The battery warning went up after 45 hours of regular usage informing me of an estimated remaining charge for about 4-5h of usage.

Regular use means:

a few calls (~4)
a few pictures taken (~5-6)
WiFi always on
cell data on while out the house (6-8 hours)
a few Youtube videos (2-3 15 minute videos)
some browsing with chrome
email, signal (texts), 2FA apps, slack, twitter + mastodon

My current charge shows 38% battery left with a 74h 22m phone uptime and approx. 19 hrs left. On the initial charge what I noticed were huge drops in battery usage. You can see them on the shots below where the charge readout is flat for hours then drops by 20% or more. This seems to be evening out with usage but the initial charges were jumpy and that matches what the reviewers were reporting.

This is a screenshot from the initial discharge readouts:

and here is one from my current charge:

All in all. I don’t think the battery is a problem and the fact that I can actually exchange it is a huge benefit. I had to buy two phones, out of which one was necessary only because the battery was not swappable in the one we had.

The lack of LTE is actually not a problem for me. The most network usage I have on my phone is at home with WiFi and during outside events like running/dog trekking I don’t need LTE, 3G is enough and adds to longer battery life.

Audio is another thing though. Everything the reviewers said on it is spot on. Crackling, glitches, phone going low volume and completely mute on occassions. Happened mostly on the built in speaker phone when it was not on. I am still not sure what and if I will do anything with it. It doens’t happen always and I don’t spend much time talking - could use a regular or bluetooth headset to workarond it. Not really keen to send it for repair and wait 2-3 weeks without a phone - especially since a lot of online reviewers reported that phones after repairs still had the same issue present.

Their app is indeed annoying. It doesn’t pop up randomly as reviewers suggested. The app shows up on first run and when the list of running apps is cleared from the task manager. What is it exactly? Just a simple brochure promoting the community that leads to a form where you put your email to sign up to the newletter. The biggest problem? It is always triggered, even if you decided to sign up. When I called up the company they said they were aware that it’s annoying to some users and are considering an update to remove, alter or disable it.

That covers what was online already. Now the one hit on my own.

The biggest flaw so far is an older security patch level (October 5 2016) which still is pretty good for a very low shelf device (note the price I gave was with no contract).

I called up the company and asked what’s their update policy. They don’t have any officially planned updates for this model but no-one said no. There is a possibility that something will be released but I’m not making bets on it. Bonus on their side is an unmodified stock android with some apps pre-installed (but quite sane choices and most allowing a full uninstall which is very rare).

The situation isn’t perfect but quite typical for most Android phones except top of the line models or purchases directly from Google. I don’t trust my phone at all (you will never find my PGP and SSH private key material on my phone) and it’s already a more recent patch from my previous phone. I personally think that selling hardware with outdated software should be treated the same as selling products past their due date but the reality is not there yet. I did voice my concerns over phone, myPhone does have a line of higher end rugged smartphones that are indeed very promising but I will be not buying anything else from them until I see a clear policy on software security updates.

Now the fun part. We got two identical phones, or so we thought.

In case it’s hard to see. My model says ‘8.0 Mega’ next to the back camera lens. By the way, the screws visible on the photo are not just decoration - they hold the back lid from splitting apart when the phone falls preventing the battery from jumping out. They give you a nice custom screwdriver for those :)

The company confirmed on the phone that all models are 5 Mpx. They don’t know why mine has a surplus text printed on but found it interesting. I find it funny also but I expected differences in build quality between units of the same model - it is a budget phone after all.

Still, talking about the camera I am more than happy with the picture quality but I don’t have huge demands for a phone camera.

The GPS tested by my wife also behaved decently. The interface is snappy, nothing is lagging/sluggish - all my 2FA apps work. This phone pretty much covers mine and my wifes needs without ruining our budget. I would whole heartedly recommedend it if only myPhone provided up to date security updates for their phone models. One more thing. God damn, why did you pick a name that sounds like a cheap iPhone rip-off - rugged smartphones are an amazing product, have some guts and stand on your own. If anyone from myPhone is reading this, consider at least keeping your rugged phones line under the Hammer trademark and drop your company name from it.

PS.

It’s hard to get this phone outside of Poland. They have their own shop but no English interface and their resellers are also local. When talking on the phone I asked on the possiblities of purchasing from abroad. The company was not ready to handle sales to the USA but is able to sell anywhere inside the EU - when I confronted this with them not having a shopping frontend in English they said they can handle orders via email in English. I wanted to learn more but the audio in my phone went mute (and back then I assumed the call center person just muted me :) - but it was my audio getting glitched!)

PSS.

If you have any doubts. I was not paid in any way for this article, none of the links here are affiliate. Just my personal opinion on our recent purchase.

Shadow leaks

Tue, 24 May 2016 04:00:00 +0200

The recent linkedin password leak is a huge deal. The data was obviously leaked in 2012 but it’s ’;–have i been pwned? that notified me about it and not linkedin.

I did change my password more than once since 2012. Is that enough reason for linkedin to not inform me about it? What if I removed my account completely? These days I am using a keepassx for password management and generate unique passwords for each service I use. Not sure though what my password was on linkedin in 2012. This is a none issue but there are more alarming things that grew on me when I reviewed the sites I have an account on.

Each and every internet service that you create an account on, that stores data about you is a ticking bomb. Even when your passwords are random and unique what does a service get? Your email? Name, shipping address, personal messages/data, phone number heck maybe even your CC card details? You might never even know that some of that data leaked out. I decided to close down some accounts that I no longer use & see the need for.

First off I’m happy that some sites allow me to remove my account. Yammer, Assembla and a couple of others were a painless process. It’s somewhat tricky to locate the option as it tends to be tucked away in the least visited portion of the site but at least it’s there.

Not all sites are the same though. Several of them, like the pivotal tracker force you to do that extra step and emailing tech support to have your account removed. That’s a minor annoyance but at least they mention it in one of their help pages. They were also pretty prompt at executing that change without making me jump any additional hoops.

There are uglier sites though. Those that you or your wife used as a ‘one off’ years ago for that specific purchase. They tend to have incorrect TLS certificates set up (if at all), no way to remove existing data (billing address is a required field you can’t remove it) and no way to delete the account. Sometimes even finding a contact email is an expedition in it’s own. One site I ordered something once from in 2011 changed owners 3 times by now. Your only hope is to overwrite your personal data with some garbage (the ones they allow you to change, but usually their validation sucks) and hope that someone will react to your ‘remove my account’ support email.

Seems like a lot of leg work but doable or is it? Does that ‘delete my account’ option really work? Maybe someone just marked a ‘deleted’ flag on my entry in the users table but my personal data remains there intact. There is absolutely NO WAY to tell that. This is the real problem we are facing with the web today. Facebook, Google and even the small shops are either too invested in keeping every tiny bit of info about you or too incompetent to do a good job of cleaning up things that could cause problems later down the road, not to mention being able to inform you of breaches.

Here is a small example from my Pivotal Tracker account removal:

I asked for account removal. Josh replied at 6:34 PM:

As requested, I've removed your 'account@domain' login.

Apparently my data is still somewhere at Pivotal since I got marketing email from them at 8:39 PM. After my account was removed (7:02 PM timestamp marks a reply for some additional feedback I gave after account closure).

I trust Josh is not malicious and that Pivotal respects those requests. Probably the newlestter is a separate system and Josh doesn’t have removing my data from there on his checklist. I replied back and asked to be removed from any of their systems. Waiting to see how this unfolds. This is not a stab at Pivotal. I like their service and it was usefull when I still had a use case for them, they just happened to serve as a perfect example for today.

What else could be missing from a checklist like this?

Marketing systems
Analytics systems
Backups
Test servers
Developer machines with partial/full production data dumps
Stale access tokens for third party integration with Google, Facebook etc.
Cache
Data print outs
much much more…

If that data ever leaks will someone feel obliged to inform you? Will they be even able to do so? Will you remember what password you used when you still had an account there or what type of data was there? For many big companies you are probably just a shadow record marked as deleted waiting to be leaked out. Those few that deleted your data probably left a few ghots lingering around.

It took 4 years since 2012 to be informed by a third party that my account at linkedin was compromised. I can’t even imagine how many breaches we are not aware of. Think twice before entering your data on a web form and creating a new account.

Year of the OpenBSD desktop

Fri, 22 Apr 2016 04:00:00 +0200

It is a common theme in the GNU/Linux community to tout the current year as the year of the linux desktop. Every year the same thing happens. The nay sayers nag that Linux is a tiny percentage of the desktop market and that Mac OS X/Windows is superior in so many ways.

What most people miss is that for the majority of users the year of the Linux desktop already happend. I myself can count at least 5-7 years where I used Linux exclusively and nothing else. That’s pretty much the definition of being the desktop. I don’t own a Mac or a Windows machine. I can do ALL of my work & entertainment (hello steam <3 Gabe ; hi netflix!) on it. Give it up Linux folks. You are already on the desktop :)

Personally I am waiting for my OpenBSD on the desktop year. It’s funny because two things happened recently. My wife promised me to make a double colored winter scarf for me. One color is for a day I used Linux and the other color is reserved for a day I only used OpenBSD. Current stats shows that since the start of this year I used Linux for 75 days and OpenBSD for 55 days. It’s a pretty head to head match and OpenBSD is loosing mostly due to work (like the Dart team not really cooperating to get BSD support) but I’m working on it. I already could use Linux (and did) on every day of the year without any exceptions. I will claim the OpenBSD on the desktop year when I can do the same with OpenBSD (and believe me I will try to make it happen).

The second thing that happened was a series of tweets on twitter. Referring 2016 as the year of OpenBSD on the desktop. Guess what. For some people it already arrived. I strongly believe that @risc started the series.

2016 year of the #OpenBSD desktop? pic.twitter.com/rM7jXH5z7Q
— risc (@0x72697363) April 11, 2016

2016 year of the #OpenBSD desktop! pic.twitter.com/o9RRuw4VRQ
— risc (@0x72697363) April 20, 2016

I jumped on.

2016 year of the #OpenBSD desktop! pic.twitter.com/jsafgU9KS3
— mulander (@mulander) April 20, 2016

and more people started tweeting

Really happy coding in #elixirlang on my #OpenBSD desktop! pic.twitter.com/qo5GtOSjtV
— unbalancedparen (@unbalancedparen) April 19, 2016

@mulander :) pic.twitter.com/jtyf0NahD2
— glenn faustino (@glenn_faustino) April 20, 2016

including my wife

2016 year of the #OpenBSD desktop!
[inspired by @0x72697363] here -> https://t.co/5w8c4NQBHn pic.twitter.com/vVzHYTVbil
— Raven Alpha (@nemessica) April 20, 2016

and what better way to define a OpenBSD desktop than a set of CD’s sitting on one?

#OpenBSD 5.9 disks arrived today!

:) pic.twitter.com/Tyc9guwI2A
— Gareth Llewellyn (@NetworkString) April 18, 2016

Delivered \o/ #openbsd #security pic.twitter.com/nbGzSNk55s
— Dalenys OSS (@DalenysOSS) April 20, 2016

My #OpenBSD 5.9 cd's just arrived! pic.twitter.com/jTEjKnvmz1
— mulander (@mulander) April 20, 2016

It’s that time of the year again… #OpenBSD pic.twitter.com/UhoF4r7h9V
— Chris B. (@dr_jekyll832) April 21, 2016

In the end. Yes. We might still be missing Netflix, Steam and some one off tools like the Dart language but for most of us. The OpenBSD desktop already is a reality.

Trigger happy

Fri, 22 Apr 2016 00:10:00 +0200

I became trigger happy and I’m not happy about it. What do I mean by that? Not taking enough time to think through a decision and checking all the facts. Ducking it before I head to the man pages or project documentation. Reading and basing my solutions on Stack Overflow even though I knew that it’s a really toxic thing to even read.

The end result are sub-par solutions, simple mistakes and feeling embarrassed by the amount of mistakes that can be pointed out in my contributions. This has been going on for a long time and has to change. I tried to identify the main cause of this and it’s a combination of laziness & the ease of use of search engines. It’s really easy and immediatly rewarding to slap a few search queries into $SEARCH_ENGINE, coming up with a 80% solution that is good enough. Why check your language documentation for that API if a single query shows you the proper usage in a few seconds? Yeah it’s a trap. Suddenly you are not producing code & solutions. You are slapping together examples of existing solutions to make the square peg fit into the round hole. I made that error a lot initially when starting with ports. In retrospect my biggest problem was starting by copying an existing port and modyfing it to fit the software I was porting. You can’t imagine how much easier (yes easier) it is to start off with /usr/ports/infrastructure/templates/Makefile.template.

With the above in mind I challenged bmercer@ to join me for a week without using a search engine for any tasks. You can listen to his experience of the experiment in the latest garbage.fm podcast. This post documents my personal findings.

My one week challenge of not using ANY search engines at all starts today. 16.04.2016 00:00. Wish me luck!
— mulander (@mulander) April 15, 2016

First thing that immediately struck me is how useless my bookmarks are. Instead of a organized set of documents I might want to visit again, they are just not there. I mostly know all the software I need to use and apparently it was easier to search the product/project/library name versus using boomkarks. I wondered if I am alone in this and (a rather small) poll on twitter shows I’m not the outlier.

How do you use browser bookmarks?
— mulander (@mulander) April 19, 2016

Obviously reading documentation takes time. I found myself taking things much slower, being less annoyed by not arriving at a solution fast enough. Not having to wade through unrelated content during searches helped in my regular work. Yeah I had to use GNU info to read the GNU Make manuals. In the end it allowed me to solve a Makefile problem that I wasn’t able to solve before by just trying to quickly $search for facts. The terms were just to hard to get a good search engine match but were things covered almost at the beginning of the related manuals.

I did fail a couple of times. Catching myself looking at search engine results and realizing that “oh shit I used the $search”. It happened for two things mostly. Currency conversions, stuff like 20 USD in PLN queries and having to solve issues under pressure - on one day a server failed for $work and I started searching for related info before I even started to think about it.

In the end. I do feel like I improved my whole workflow with this experiment. I still make mistakes (obviously) and still squeeze the trigger too son. Though in just this week alone those are single shot mistakes instead of series of bad decisions of trying to get things done ASAP.

Give it a shot. Hang up the search engine guns for a week and take things slow.

One of the three OpenBSD users

Sun, 22 Nov 2015 20:00:00 +0100

Recently I have been taking a look at Syncthing in the hope of being able to replace ownCloud with it after I received a recommendation on HN.

Initial impressions were quite good. It’s obviously a different category, closer to initial use case represented by Dropbox versus the full software suite including nice web clients that are now expected out of Dropbox like solutions. My current ownCloud instance sports a full featured ownNote which my wife loves and it will be really hard to get rid of but everything else about the service felt bad. Keeping it up to date on OpenBSD -current is a pain in the ass (like with every software installed from ports), desktop sync clients tend to fail when connecting to the service & mobile clients experience the same issues (including the paid ones). My initial setup worked OK but a few snapshot upgrades further and I’m quite happy when the basic online functionality works (I don’t even dream of uploading large files to it). Hence the venture into Syncthing.

Syncthing is an application written in Go which already makes deployment a lot easier. OpenBSD does not provide a port package for it which is somewhat understandable considering that each user should run their own instance. Although it’s not a huge issue since the upstream project provides official OpenBSD binaries which are additionally signed.

My initial test drive lasted more than a week and during that time the project released a few minor releases. My Linux boxes automatically upgraded (and those automatic upgrades can be disabled) which I really liked. Unfortunately my OpenBSD machines didn’t follow the same path erroring out with a sad upgrading: readlink /proc/curproc/file: no such file or directory.

Most of you know that I’m running current and that the /proc filesystem is no longer a thing on OpenBSD. It wasn’t hard to find other people reporting the same problem in the upstream providers repository. Opened sometime in January, I decided to subscribe to any updates and continued to manually update my OpenBSD Syncthing instances until I got a notification that the issue was closed.

When I saw that the problem is generally dismissed I replied back with a naive solution. To which one of the upstream developer replied:

“Maybe one of the three OpenBSD users feel strongly enough about this to propose a patch. :D”

This kind of grinded my gears in a positive way. I have been using Linux since the early 90s. I remember people using that same excuse to completely ignore or close issues reported by a completely valid userbase. Guess what? That platform somehow matters now. Yet a keen reader will notice the leading screenshot to this post. The gaming industry is still treating Linux as the red headed step child.

@ramma_gaming @KittyMeowGames Linux is a second class citizen, we don't run it internally because only 17 people use it
— Garry Newman (@garrynewman) June 28, 2015

By now, the gaming on linux subreddit has over 30k subscribers and Steam machines are rolling out. If you time travelled back to the early 90s and told me that we would be THAT relevant I would laugh in your face.

Though the Linux community did one thing differently than the OpenBSD one does. It’s pretty open about what software they use. After that bug report I enabled anonymous statistic reporting for Syncthing essentially doubling their OpenBSD user base. It’s a bit sad. I went out of my way to disable it and I’m sure most OpenBSD users did the same as privacy matters, in doing so we degrade the platform in the eyes of the software providers. Maybe all of us should be a bit more loud mouthed and get ourselves heard?

All in all I am happy that one of the three OpenBSD users fixed an upstream library used by the Syncthing project. So thank you, ajacoutot@ you saved me some time today. :)

Vendor lock-in is our least problem

Tue, 06 Oct 2015 21:00:00 +0200

Do you remember My Opera? The place where fans of a once great browser would create their own blogs, participated in forums, maintained picture galleries and much more?

It’s gone now. The great browser is also gone, replaced by a Chrome fork. Vendor lock-in is a big problem, but the possibility of a vendor going offline is a much bigger problem that none of us are truly prepared for.

I moved my blog to a self hosted static site after announcing the intention in a blog post a couple of days ago. While evaluating options for exporting my post history off of Blogger I remembered that I had a blog on My Opera. Opera was kind enough to inform it’s users up front that the service will be shutting down and allowed a data export to be performed before the lights went off. I tried to pull those posts in since I was already spending time on the blog migration.

The export is an XML file which looks quite similar to the way Wordpress exports it’s data. Unfortunately the plugins I tried for Jekyll were not able to parse the data in. Some online search lead me to old Wordpress articles describing an easy migration path for My Opera refugees with their ‘import from My Opera’ option. I tried that path. Made a quick Wordpress account just to import my data and export it to Jekyll. The option fortunately still is in the Wordpress wp admin page. It chugged on the file for 10 minutes after which I got emailed that my import was finished. Hurray!

No posts were imported. Now there are a few options. My export could have been corrupted on download from My Opera portal or the code at Wordpress was changed/unmaintaind for a while. It doesn’t matter in this specific case. The export is still text so I can painfully migrate it post by post when I have some time on new years to dig up old stuff.

What’s the problem here? When I made that initial blog on My Opera I never imagined the site going down. In hindsight that was stupid. Opera never was a huge company. Though the same thing can be done by Google, Facebook, Microsoft, Apple & other tech giants. It doesn’t have to be a big scale event like Google Reader being shut down. Imagine that Google closes down just your account for ToS violation. Can you regain access to all the services that were registered against your gmail account? Will your phone work properly? Will you loose important files?

Since I started moving my stuff to a self hosted service the impact of such an event gets less dreadful. Still I can’t imagine the day when I will be forced to export (if I’m allowed to do that) all of my remaining data from such services. Will it be possible to use the data in any way? Who knows.

The big guys aren’t making it any easier. My wife got a set of pictures shared over email, they were made during a dog walk. The email that came through were a bunch of links to a Google Drive. She had to download them one by one. Facebook is a closed internet fenced inside itself & G+ aims to be the same thing. If most people just use the big guys, they can suddenly decide that ‘we can ignore the 5% email from self hosted email providers’ and sell it as spam prevention to people. Though what will happen next? If a player gains majority he will start setting new internal standards and our goverments have been really laggy reacting to subtle aspects of the net like that.

What if they succeed gaining the momentum and then the light suddenly goes dark on them? We might no longer have anything to go back to and our exports without the software stacks of Google & Facebook might be useless.

Cost of Privacy

Sun, 04 Oct 2015 22:14:00 +0200

No time to read the whole thing? TL;DR around 250 USD per year and growing rapidly excluding the cost of personal time spent on maintaining my own services.

More frequent readers of my blog will notice that I didn't write a single blog post in quite a while. The main reason is that I planned to move off from blogger to my personal server - I also had a lot of stuff to do at work so there's always that. Those readers also know that I host my own mail server and an OwnCloud instance. So what is this blog post really about? Summing up how much it costs to get off the Google band wagon and how it went for me & my wife for the past year.

We have our vultr.com server up since March 2015. Had the luck of grabbing a 15 USD plan for 640 GB of data, 1 CPU core & 2GB's of memory. Add the tax on top and the server amounts to 18.45 USD per month charged against me. Which is quite fine for the amount of storage I get (remember I planned the server mostly as owncloud + email). Regardless of what people say - vultr.com is really nice on the tech support side. Getting them to unblock outbound port 25 for my email server took literally 5 minutes and an instance reboot. Soon we started moving almost all of our services to new self hosted email & slowly migrating our data to the OwnCloud instance with the OwnNote plugin to kill some old bad Evernote habits. It's quite nice and liberating.

I love the fact of being able to do a smtpctl monitor / smtpctl show queue and to know where & why a specific email got stuck. I love reviewing my logs and knowing why a file upload didn't went through instead of a generic error message that tells me nothing. Though rose colored glasses rarely last MIL standard grade long. First issues started with SSL certs.

You have to pay one of the trusted CA - I don't trust any of them but other people who also don't trust them won't trust me if I don't. #TLS
— mulander (@mulander) September 5, 2015

In this day & age you have to shell out money to a trusted certificate authority for an SSL ceritifcate so other people who trust you but don't give a shit about the CA will trust you. I got hit by that both with OwnCloud & email but those are a bit easier to ignore. My service users are limited to myself & my wife so it's not hard to verify that this specific self signed certificate is actually ours. Stuff breaks when you try to run your own jabber server and federation just refuses to work because well - shell out money to people we don't trust.

It's only a cert. We can live with that. True. Though vultr.com had it's slew of slight problems. So far I registered 8 support tickets against vultr.com. All of them are solved, but they all amount to 5 times a server was rebooted without notice.

One should be happy with a wife well versed in IT. Dudes, run if your service provider reboots your service with no notice and your answer to the big question 'are you doing backups' is 'not yet'. I quickly fixed my error by setting up a tarsnap backup account. I was really glad about it since a couple of months later vultr.com rebooted me suddenly again and I found myself back a few days without any emails that happened since the 'lost' state. The end result was that they moved us from one virtual node to another but didn't kill the old one so it suddenly took over taking us back into the past.

Backups are an additional cost that I didn't fully account for when setting up a 640GB node. I am extremely happy with tarsnap which so far costs us 0.25 USD per month for backing up 83GB data out of which 1.8GB is unique and 1.0GB when compressed. Though you can quickly see how this cost will grow if my instance is actualy used up to the limit by backing up our family dog photos. Backing up 600GB of data per month would cost us an additional 150 USD per month... I am currently paying for a 640GB instance and preventing myself from using it fully since the backup cost would thwart the instance cost & I can't count on the service provider to secure a reliable backup of the node.

So what is our current cost? 18.45 USD for the vultr.com instance. 0.25 USD for tarsnap backups and 27 USD per year for domain registration renewals. 250 USD per year for little direct benefit. Yes, I am currently paying for a server that I prefer not to utilize fully because doing that and backing up would mean a bill monthly that we can't afford.

What did we gain? We control our email & data. We are responsible for our own backups (for better & worse). I had a bad experience in the past where Opera (you know, those guys who made a decent browser in the past) allowed me to download a fucked up archive of all of my old blog posts because they decided to kill Opera community blogs before killing the nicest browser around. That won't happen again.

Which brings me to the huge blog post gap I had. I want to move to a static blog served on our server. Why? Because this blog is not for making money (if you spot an add it's not from me). The only unethical thing this blog does is tracking you with google analytics as the whole thing is on the blogger platform. Though, moving a blog costs time & traffic. If this post hits a popular service it can generate a load. I have 3TB traffic per month on my vultr.com instance but I have a hunch that hn/reddit/lobste.rs would be able to eat it in a day.

I frankly don't care. It's nice to know how many people read the blog post but I can get the same info from running logswan or other log analysis tools on my webserver logs. All of which are far less intrusive than Google analytics. This would of course mean me killing the comments section of this blog, and I would be quite fine with getting blog feedback on my email (mulander -at- tintagel dot pl) including 4096R/092BB571 encrypted feedback.

So, being private and in control costs me. Do you appreciate it as a reader? I'm willing to move this blog to my own server & actually paying for the traffic but I would like to know that it means something more to my readers than a dumb page refresh on a random link.