Bits and Pieces

How to create a cost sharing app using Google Spreadsheets

2011-10-31T22:17:00.003+01:00

I want to document a solution to a practical problem: a few people may want to share the expenses of various kind in a fair fashion.

The straightforward way is for everyone to just announce their expenses, and collect 1/n parts of them from everyone else. However, it is obvious that this is a very tedious way of doing things: say, 5 people spend 10 euros each on some things, thus each of them will need to give 4 times 2 euro - and receive 4 times 2 euro - the result of all this activity being exactly the same when it has started!

Thus, it's a clear area of optimization. At first I thought about a web app. Google apps could do that. However, I decided to try to be even lazier and to implement the whole thing using Google spreadsheets. The result is a successful "app" which in the main "screen" (the main sheet of the spreadsheet) displays for every person whether they owe money to someone or whether they need to collect money. A person who needs to collect money can chase some of the debtors, and then, once collected, put a note about that. The system will recalculate the debts.

First, the "theory" (if I may call this so:).

We start with a per-person value of how much expense do they incur. The "fairness" means that all of them would have equal expenses - so the target value is the average of the expenses. As one person gives the money to the other one in order to "balance" - this event needs to be recorded in order to decrease the imbalance for both - in different directions.

That's all that is there, really. The rest is the mechanics.

First, let's assume that the users are "normal" people and they do not want to enter the data into the spreadsheet manually (not only it is boring but is also very error prone).

So, we create so called "web form". Pitifully only one instance of this seems to be possible. Anyway, we make a web form with 5 fields:
"My name is": dropdown list with all the names.
"I have made an": dropdown with 2 choices: "expense" and "repayment"
"How many euros": the text input to put the $$$ value in.
"What for?": freeform text field to document the purpose of transaction.
"If you repaid to someone, to whom?": dropdown with the default choice of "---" and then the same names as in the first field.

Having to manually enter the names is somewhat annoying, I haven't figured out the way around it. Anyway, as you try entering the data into this form, you notice it starts to fill the sheet in. You will see the first line of the sheet being: "A>Timestamp |B> I am: | C>I have made a: | D>How many euros ? | E>What for ? | F>If you repaid someone, to whom did you repay ?" (the "A>" is the name of the column, for reference).

Starting from row 2 you will see the actual data. We need to shift the data to row 12 - the rows 2..11 we will need for various aggregate calculations.

The column "K" will hold the labels for them - just for us to remember. The contents of this column, starting from "K2":

Who
Got/Made
Type
Total
DeltaAvg(+/-)
Repaid
Received
Balance

Now, in L2, M2, N2... we need to put in the names - exactly the same way as we have typed them earlier in the web form. Let's say we have four people: Amy, Bob, Carol, Dave. L2 will read Amy, M2 - Bob, N2 - Carol, O2 - Dave; Repeat two more times in the same order. So P2 and T2 will read Amy, etc.

Why the names are repeated three times ? This will become obvious shortly. Go to the third line, and leave L3..O3 empty, then P3..S3 will become "made" and T3..W3 will become "got". Now go to the line 4, and the cells L4..O4 will need to have "expense", and the cells P4..W3 will need to have "repayment".

Now this clarifies the purpose of the three groups - this is a mechanism to untangle the data from the dense web form: each column will only get a value if that particular person was involved in that particular activity.

To calculate the total expenses per person, enter the following formula into L5: =SUM(L$12:L), and copypaste it into M5..W5 - notice that the column will auto-change. This is precisely as we need it. The line 5 will hold the calculated totals per activity per person.

Now let's calculate the average of total expenses per person. Enter the formula into the J6: =average(L5:O5), and put the label into J5 saying "Average total expense" or something similar.

L6 will hold the signed delta between the average and the expenses for Amy: =$J$6-L$5; copy-paste this same formula to M6..O6.

L7 will hold the amount of how much Amy has repaid to others, so the formula will be naturally the sum of her repayments: =SUM(P12:P); similarly to previous value - copy-paste this formula into M7..O7.

L8 will hold the amount of how much Amy has received from others in repayments. So, the formula is: =SUM(T12:T); again copy-paste this formula into M8..O8 - the column will auto-update to correct value.

L9 will hold the final "balance" for Amy - if she needs to pay, the value is positive, if she needs to get money from others, it will be negative. The formula is: =L6-L7+L8
and similarly needs to be copy-pasted into M9..O9.

We are almost there - now we need to actually populate the rows with the monetary values... One problem: the rows with the actual data are being dynamically added, how do we add the formulae there ? The answer is with the "arrayformula" - it autoexpands down as needed.

So, put the following formula into L12: =ARRAYFORMULA(IF($B12:$B=L$2,IF($C12:$C=L$4,$D12:$D,0),0))

This says that if the name of the participant matches the name for this column and the name of the activity matches - then the number entered will appear here, else it will be zero.

Copy this formula to M12..S12. This way we make sparse tables for two activities - entering expenses, and entering the repayments to others. If you move to L13, you will see the following formula autopopulated: =CONTINUE(L12, 2, 1). Neat, huh ?

Now we need to populate the third group - the synthetic one that captures the amounts the person *receives* in repayments. For this we need just to slightly modify the above arrayformula, and put it into T12: =ARRAYFORMULA(IF($F12:$F=T$2, IF($C12:$C=T$4,$D12:$D,0), 0))

You see that the only thing changed is the column from the form entry with which we compare.

Copy-paste this formula to U12..W12. We are almost there.

This completes the "brains" of the application, and the line 9 is already usable,
but we can make it more user friendly. Rename the current only sheet into "Expenses" and lock it. Create a new sheet called "Dashboard".

Go to cell B3 in this spreadsheet, and enter the formula: =Expenses!L2 - this will autopopulate the first name. Copy-paste this formula to C3..E3 - this will populate the other names.

Now we need to tell the status of the balances of these people.

Go to B5, and enter this formula:

=IF(Expenses!L9>0;concatenate("Needs to pay " , Expenses!L9, " euros"); IF(Expenses!L9<0; Concatenate("Needs to take ", -Expenses!L9, " euros"); ""))

This will translate the sign into the appropriate action for the person.

Last touch: we can liven the actions up, if we color the "needs to pay" actions with one color, and "needs to take" with another color.

Right-click the B5, and select "Conditional formatting".

Add a new rule with "text contains" "pay" and select red background and green color, and another rule with "text contains" "take" and select green background and red color.

Copy-paste the cell from B5 to C5..E5.

Lock this sheet too. You are done and now have an expense tracking application fully embedded into google spreadsheets.

Use the Web form to enter the expenses and the notes of repayment - and use the "Dashboard" as an indicator of ongoing balances - you can set your own rules when you need to repay to others, e.g. when the amount of money reaches over certain threshold.

There is also in theory a possible loss of precision that might happen as the total expense grows. If you think this is a problem - write your solution. I think I know how to solve it, but I don't want to spoil the fun for you!

Oh - and if you find this writeup useful - please leave a comment!

Want my attention ? Send me something useful.

2011-08-10T01:16:00.005+02:00

[this is an expansion of my tweet earlier today.]

Getting the email address of your users and keeping in touch with them is a classic technique. So classic that it's been mentioned in umpteen different places - I won't even bother to find the references. So classic that quite a lot of companies do pick it up, eventually.

However, there is one detail. The emails that you send need to be wanted by the user. This is a detail that escapes frequently. As a result, I am getting a lot of updates from various companies "hey, we are doing this!" or "please take a look at our new new model of humbambillistic hyperbanana!" - while, if they noticed, I did not really use their service. I tried and did not find it too compelling, or it did not solve my problem.

More impulsive of you will exclaim - "hey but unwanted email is precisely the definition of spam!"

Well, not quite. The thing is, I might have agreed to be mailed at a particular point in time. When I thought this source might be interesting. However, the first mails proved not as much substance as I wanted. Should I unsubscribe ? Maybe. However, sometimes with the subcontractors acting on behalf of representatives of the newly formed subsidirary of the ... well, you get the point. It might not be trivial.
On the other hand, I do not want to send this straight away to GMail's "spam" folder - because it will harm the users who maybe are interested.

So, I found a way that would not harm anyone and requires minimal involvement from my side - simply create filters and transit those "not interesting anymore" messages to a special folder to settle.

Maybe I read it one day when I don't have anything to do, maybe I won't. But I won't read them now.

I do this if I consciously remember that I get more than two messages with the first three sentences not holding the content that is informative and useful for me.
This reveals the other side of the coin - it's somewhat ok to poke me if I do not remember when was the last time you poked me. Unless the background pokes from different places burst together such that I get sufficiently annoyed with random flood of poke-like messages on a particular day.

I wonder if I am unique in seeing this problem (and this maybe just my own laziness), or there is indeed something to it.

Uploading the files to the VMWare ESXi server

2011-08-04T18:30:00.003+02:00

Every now and then I need to upload the stuff like an ISO image to my ESXi server.

Since I do not use windows, my only remaining option is the CLI - which is big and clunky. However, today I digged a bit and after reading the CLI tools source realized that uploading the file onto an ESXi host is trivially simple:

Browse the datastores till you get to the correct place, take that URL and append the target file name.
perform the HTTP PUT request for that URL, supplying the data.

This way, the only tool you need to upload the files to VMWare ESXi really is curl:

curl -k -X PUT --data-binary @IMAGE.ISO 'https://user:pass@host/folder/FOLDERNAME/IMAGE.ISO?dcPath=DCNAME&dsName=DATASTORE'

Microbenchmarking of luajit-based server (again: this time against lighttpd)

2011-07-28T21:27:00.007+02:00

The other day inbetween the meetings I've ported my toy event loop experiment to use ljsyscall library.

Here are the results of running of ab -n 100000 -c 1000. First let's make a baseline:

lighttpd


# ab -n 100000 -c 1000 http://localhost:80/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software:        lighttpd/1.4.19
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        10 bytes

Concurrency Level:      1000
Time taken for tests:   7.854757 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      24331590 bytes
HTML transferred:       1001300 bytes
Requests per second:    12731.14 [#/sec] (mean)
Time per request:       78.548 [ms] (mean)
Time per request:       0.079 [ms] (mean, across all concurrent requests)
Transfer rate:          3025.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   39 281.0     15    3023
Processing:     5   19  16.5     21     740
Waiting:        3   14  15.9     14     734
Total:          9   58 287.6     38    3746

Percentage of the requests served within a certain time (ms)
  50%     38
  66%     40
  75%     42
  80%     43
  90%     45
  95%     47
  98%     50
  99%     94
 100%   3746 (longest request)
#

luajit

Now let's try it on the primitive event loop.


# ab -n 100000 -c 1000 http://localhost:12345/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            12345

Document Path:          /
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   8.232656 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      5503355 bytes
HTML transferred:       1300793 bytes
Requests per second:    12146.75 [#/sec] (mean)
Time per request:       82.327 [ms] (mean)
Time per request:       0.082 [ms] (mean, across all concurrent requests)
Transfer rate:          652.77 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   44 336.4      6    3016
Processing:     4   10  13.9     10     766
Waiting:        3    8  13.9      7     762
Total:         10   55 341.0     16    3767

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     17
  75%     17
  80%     18
  90%     20
  95%     22
  98%     28
  99%   3018
 100%   3767 (longest request)

Upon multiple runs the numbers vary slightly, of course, but they stay within the same ballpark. I think this shows luajit is a very viable platform for server development.

----

Update:

curious cat as I am, I've added a HTTP parser that is made by a yet-unpublished-and-inefficient-and-incomplete patch for ragel that generates Lua state machines (and a ragel code from an earlier post about http parser.


ab -n 100000 -c 100 http://localhost:12345/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            12345

Document Path:          /
Document Length:        19 bytes

Concurrency Level:      100
Time taken for tests:   21.980 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      6100000 bytes
HTML transferred:       1900000 bytes
Requests per second:    4549.62 [#/sec] (mean)
Time per request:       21.980 [ms] (mean)
Time per request:       0.220 [ms] (mean, across all concurrent requests)
Transfer rate:          271.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:     1   22   3.1     21      30
Waiting:        1   22   3.1     21      30
Total:          3   22   3.1     21      30

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     25
  75%     25
  80%     25
  90%     25
  95%     26
  98%     27
  99%     27
 100%     30 (longest request)

This is not stellar, but I did not see yet where the bottlenecks are and if I can speed it up. Either way the 4.5K is still not too bad.

Update2:

with concurrency of 1000, I notice connection resets... After fixing up the mixing SIGPIPE handler of SIG_IGN, they still happen, apparently. Further debugging pending...

Your IETF-writing will never be the same again

2011-06-25T17:10:00.005+02:00

I've been reading up on SPDY today, and I noticed something interesting -
the spec at Mike Belshe's github repo has the extension ".xml".

So what, you say ? Given that it was rendered as the html output of the xml2rfc, I suspected there was XSLT in the game. And indeed, with one more search, I found the origin of this: Julian Reschke's work.

This is definitely very cool - it means you have to store one less format, and the results are immediately visible as you change the xml.

I will definitely give it a shot for the next draft to see if it lives up to my (inflated) expectations.

I came up with a new word today: faceboob

2011-06-24T00:04:00.006+02:00

(v) to faceboob (someone): to earn money off the the humans' tendency to be curious, as well as to their natural tendency to trust their friends - specifically when using the social networks. Example use: "John Doe was faceboobed today."

Now, enough words, lets get to colorful pictures!

Prologue

You see this:

Oh, boy! Aha! Free Boobz! Especially since one of your friends (or more!) shared this. Gotta be good!

Let's get another browser. But first lets investigate a bit - the link looks like an URL shortener.

Yeah, it is chinese url shortener FTW! Better yet, if you go to home page it talks about a link being a spam and being deleted. This already rings the alarm bells, but let's see further - let's open up the link in the browser that is logged out of Facebook and see what happens.

Whoa!!! Free boobz are just one click away! They need to verify the age - of course. Gonna be something really juicy. Better yet, the button is also helpfully localized. Thank you my friends! When we click this link, we get a popup to login to facebook.
Cool - so this is how it appeared in my friend feed. I'm the one with the OCD - the others simply tried to verify their age... oops. Anyway, lets close this annoying popup, I am not going to login anyway. Let's pretend that all happened in the background and see what happens further.

Whoa! To verify my age I am going to get a chance to win an iPad ? Unbelievable! I am a lucky man today! iPads and iBoobz are falling from the sky! Let's click! I want a MacBook Air though - because I already have an iPad.

The question asks something like "is it possible to play the videos on the MacBook Pro?" First, what a stupid question. Of course even the kid knows it is possible. Second, you are supposed to ask about the MacBook Air - I don't want a MacBook Pro!
Anyway, let's answer "Ja" - and go forward, towards the free gear and free boobz!
(the eye notices something in the right-top corner, but quickly dismisses it as uninteresting).

Ahhah. Now the top-right thing starts to make sense. 1 euro for SMS sent, 1 euro for SMS received. This is an IQ test - so I enter my phone number, such that they send me an SMS and charge 1 euro. Brilliant, lets stick in some random number that is not a valid phone# and move on. I guess this is the time for the real age verification.

Congratulations! You've been charged 1 euro for receiving an SMS! Now please also help us charge you another euro for sending an SMS! I start to be a bit unhappy about the whole situation - 2 euros for boobz aint really free anymore! I start to suspect the evil plot in this. They want to rip me off - let's go back to the first tab where they had at least the background pic.

An unpleasant surprise awaits me back there, there's no flirt at all on that page anymore. Instead, it coldly notifies:

(Note that the text "Not completed" periodically changes into "checking..." - which makes a great illusion that the code *actually* checks whether you have completed the darn survey or not). The boobz are in the background - go and complete this survey, ASAP!

Well, there is a slight chance of a bug. And the page admits it. It offers to complete another survey if this survey does not bring the keys to the kingdom. Let's go and see about the iPad - two is better than one, after all... This one starts in a rather cheerful manner:

It prompts me to select my favourite color! Yay! I prefer black.

Wow, now is the real quiz: it asks which icon corresponds to the Facetime. That was easy. *click*

Meeeeeh. This smiling lady promises me a charge of another 1 euro, and still no boobz!

I think I am going to give up here - no free boobz today. I'll better go read slashdot.

The practical takeaways from this little exercise, to avoid being faceboobed, remember these three simple rules:

There is no free ~~lunch~~ boobz.

~~Be bear aware!~~ Always check where the links lead, even if it is your friend posting them.
~~Use condoms.~~Always check them in a separate browser that is fully patched and is NOT logged into the site that you got the link from.

Remember, you are the target too.

And, unlike the computers, your antivirus is not auto-updated.

Keep this in mind.

update: oh, I forgot. When closing the browser, I get this popup:

Now, this is called a plain greed, my friends.

Don't leave your mobile numbers to these folks. They are too greedy to rip the money off you.

And if you really really wanna boobz - here they are. (warning: age restricted, may be NSFW for you): not a faceboobed link.

Your account has been compromised. Will you finally stop reusing the passwords ?

2011-06-20T23:36:00.005+02:00

Logging into gmail in the morning, I've got a prompt to change my password, which mentioned suspicious activity with my account. When logging in, I saw the possible reason why.

The MtGox bitcoin "exchange", where I created account to see what it is all look like, has been hacked, and the database has been leaked. Bummer. I even found my email and the user id on pastebin.com, alongside with the hash of my password. This can't be good, can it ?

Well, except since some while I started to be more pedantic about having separate passwords for different sites, no exceptions. My online passwords are 40-characters random hex strings, different for each site. This meant a sigh of relief - the data that the miscreants obtained is of minimal impact to other sites that I use.

So, the unique passwords saved my bacon today. If you reuse your passwords on more than one site, you should think a second time what would happen with your accounts in the occurence like this. If you are a windows user, you could get the Password Safe program by Bruce Schneier - it's open source and free. In this day and age, you should avoid reusing the passwords, period.

I would also use this chance to describe how my "separate password" strategy looks like.

There are two components of it:

A program called "sha1sum" - which exists on most of the *nix versions, and essentially just calculates a cryptographic one-time hash function.

A text file with the editor which stores the material for my passwords, in plain text.Each line of the text file contains two strings: the site name and a random string. I choose the random string to be sufficiently long to be difficult to guess / brute force (some 30 characters or so).

When I need to get the password for a website, I open up the file and find the line that is corresponding to that website.

Then I start up the "sha1sum" program, it starts to await for my input. I copypaste the site name and a memorable "master password" - that I do not write down anywhere. This is the thing that allows me to not worry extremely much about the safety of the file with the key material (alone it would not give out an easy target. Even though of course I do not put it out in plain view)

After I entered both strings on one line, I press enter and then Ctrl-D - this signals an end of input for the sha1sum program, and it spits out a 40-character hex number at me.

Great, now it is time to launch the sha1sum again - this time I copy-paste the hex number from the previous run, and then the "random text string" from the file.
After that I press Enter and then Ctrl-D - getting another long hex string as a result.

This is my password that I now copy-paste into the web site that I need to login to.
Here's how it looks like:


ayourtch@ayourtch-lnx:~$ grep gmail p-material
gmail 1243pyupqwe,jl23hl23khjkh23khpw'@
ayourtch@ayourtch-lnx:~$ sha1sum
gmail this is my secret phrase
f3c446b01b24022c136bde50d32a1f9d4e9cd7fb  -
ayourtch@ayourtch-lnx:~$ sha1sum
f3c446b01b24022c136bde50d32a1f9d4e9cd7fb 1243pyupqwe,jl23hl23khjkh23khpw'@
8e485f361bfe1834d281209121eb5d4a8b52bcb9  -
ayourtch@ayourtch-lnx:~$

The bold string is the password to copy-paste into the login screen. (NB: of course the data in the example above is not real, it's my mockup just to illustrate the principle ;-)

While this scheme is certainly not pixel-perfect cryptography, it gives certain advantages:

Even if the service is dumb enough to store the passwords in plain text - the attacker does not gain much when they hack it. They will just get the password to a particular site - but they would not learn much how did you derive this essentially random string.

Even if the attacker gets the file where the strings are written down - they do not gain much, as they would need to know the "master password" in order to create correct passwords for any of the web sites.

It is simple, light, and independent from the vendor (browser, etc.) - practically anywhere there is a sha1sum - so I only need a (tiny) file with site names/strings to carry around. The smaller the data to secure, the easier it is to secure it.

Of course, this also has a couple of disadvantages:

entering the passwords is a bit annoying. You need to do some manual operation.

entering the passwords is a bit annoying. This is the iPad version.

However, I value my peaceful sleep much more than the annoyance of the process.

Do you value your sleep ? If you do - stop reusing the passwords before you lose it.

Coursemare incorporated promotional video

2011-06-16T00:06:00.003+02:00

I've created this one a while ago, but did not move it from xtranormal to YouTube. The happy spectators of my little "NAT vs. IPv6" fun were asking if I did anything else - so here it is. This time our target it the applications that go out of their way in alienating their users.

Indexing wikipedia data dump with ZF

2011-06-10T19:11:00.003+02:00

I have decided to try indexing the wikipedia HTML data dump with my zettair fork.

The specs of the machine: 8GB RAM, 8-core Xeon X3440 machine at 2.53Ghz.

multicore nature of the machine did not really matter, since the code is single threaded.

Some info on the dataset:

14257665 documents in total (just above 14 million, that is)

233GiB of data as per "du -kh" output.

The index size is 17Gb.

Indexing this set took about 1.5 days:

real 1935m54.592s
user 138m42.620s
sys 23m49.060s

The search times obviously depend on the word, below are some dummy samples, "cold" is the first time you search, "warm" is the subsequent time.

"wikipedia"

cold: 20 results of 11194872 shown (took 3.886843 seconds)
warm: 20 results of 11194872 shown (took 1.542165 seconds)

"integral":

cold: 20 results of 26981 shown (took 1.168747 seconds)
warm: 20 results of 26981 shown (took 0.065107 seconds)

"schematic":

cold: 20 results of 5056 shown (took 1.040630 seconds)
warm: 20 results of 5056 shown (took 0.022333 seconds)

"surfing":

cold: 20 results of 15340 shown (took 0.840945 seconds)
warm: 20 results of 15340 shown (took 0.039736 seconds)

"asd":
cold: 20 results of 1198 shown (took 0.868518 seconds)
warm: 20 results of 1198 shown (took 0.055091 seconds)

An interesting task probably would be to see how well does the indexing scale (i.e. reindex only partial datasets and graph them). But I figured I'd write up what I have for now.

The ads that make your page beautiful

2011-05-25T03:21:00.003+02:00

Maybe it's my humble page with links about situation in Japan, or my periodic scouting on japanese-language sites, I got myself a Persistent Sakura House syndrome. What is it ? It's an English-speaking advertisement about the nice rentals in Tokyo that is persistently following me all around the web.

I am seriously starting to check it out.

And closer to understanding how half of the world buys the junk they buy.

Anyway, this is not about Tokyo per se, nor it is about the persistent advertisements that program the minds of unsuspecting crowds.

It's about something else.

When I told my friends at Google this story we laughed, and the consensus was that with my chaotic behavior I made the algorithm believe I am very interested to go to Tokyo. Nonetheless, if I clear the cookies, the phenomenon will probably disappear (it does not show on the Firefox that I mostly do not use and which has different sets of cookies).

But I will never do it myself.

Because the ads I see make the sites look better.

Don't you agree that it is a pretty tasteful banner ?

A banner that does not sleazily try to sell something to me. A banner that (despite my general distaste with pink) is nice and calm to stare at. Maybe because of that, or maybe because of its calm layout, it makes the sites look better, in my opinion.

So, here's a million dollar idea.

Besides monetizing the additional value that the contextual advertising brings to the merchant, create and monetize the value that the website owners would get out of showing only nicer-looking advertisements on their websites.

How ? I don't know. That's why the ideas like this are dime a dozen.

Is this even something valid ? No clue. A serious amount of A/B testing should show.

I just noticed it, because this ad is persistently frequent - yet, it does not get annoying. More so, I've even gone to the site a couple of times and read the reviews - the folks seem to be pretty positive.

I am thinking of visiting Tokyo one day.

Can't say of anything else, but they make great looking ads for sure.

Strange.

2011-05-23T23:20:00.005+02:00

I am at St.Pancras, in the Eurostar lounge. Got a special friendly patdown treatment - maybe because I forgot some coins in the back pocket of the jeans. Never mind - I got used to "random checks" in my previous life.

But wait. I'm traveling. Without my habitual travel drink. Rewind the time. I need go to the Starbucks. I want my "Caramel Macchiato".

Strange. They changed the look of the menu, or they had it differently to begin with. No pseudoitalian size denominations. Seems like now you can build your own drinksfrom components - or it is just an avant-garde design. I can see "macchiato" separately, and "caramel" separately.

Too complicated. Just asked the same thing as usual. And, lo and behold, the person on the other side replied with the usual correction - "Venti ?", to which I reply "large" - as usual, in a futile attempt to abandon the adjectives that do not make much sense. So they still have it, and this will be keep me alive till I get home. The world's still going round.

There's a newspaper on the table - the first page tells how some poor orphan kid had hit a jackpot.

He is nineteen and he got a million pounds from the charity that is run by the newspaper. Good for him. If he is clever, he can make good use of this money. I hope the other hundreds or thousands of orphans both in London and elsewhere in the universe will wish him well. Or maybe they would not. The readers who gave money to him would certainly wish him well enough to give the money - but what about the other orphans ? No, they don't exist. They're just part of what we call 'the society', they do not visible out from the background of the crowd.

Try running in the middle of the crowd that is standing still. Or better, try stopping in the middle of the crowd that rushes ahead. You will be run over. Not because of anyone's evilness, just that for the others you are only a part of the crowd.

They all run towards their dreams. Those who have no dreams to run after, rush even faster to not think about that. They skip days, thinking this will get them faster to the destination. They exhaust themselves to blood while running.

Even if they do not know what the destination is, they will still have a shade of suspicion. An exhausted mind is a harbor for the suspicions, and there are enemies hiding in everything that is suspicious.

"Something strange ? Report to the police!" - says the plaque in big font.

By the way, I have something strange to report. There is free WiFi but I could not find a garbage bin to put the empty paper cup into.

Maybe the garbage bins are dangerous (I heard they can put cigarettes into them - and the smoke looks very scary! Be very afraid!).

Maybe they were just forgotten. Tiny details are easy to forget when you're making grand things.

Maybe I am just not seeing them - much like I did not see the cars when crossing the streets - looking wrong when I should be looking right.

I want to ask the policemen, but this can easily be considered a strange behavior - if there are no bins, then it means no one needs them.

I quietly leave the cup on the table and rush with the rest of the crowd to the gate. They finally announced the boarding, and we are all in a hurry to grab our places - the places that no one else can take. Would be strange if someone did - how would they not be afraid to be reported to the police for the strange behavior ?

I look back and see my cup still there, already welcoming newly arrived burger wraps.

Hopefully it's not too strange of an idea to leave an empty cup on the table when I can't find a garbage bin. I say "thank you" in advance to the person who will have to get this cup to the secret location.

Soon I will return and try again. Maybe one day I crack this strange puzzle.

A new name for you

2011-04-24T19:49:00.004+02:00

The world is steadily moving away from exchanging the phone numbers towards exchanging the online handles. For example, getting people friended on facebook.

This is quite annoying for me, because, even if my first name is pretty easy to spell,
the last name is as far from something easy like "Smith" as it can be.

So I inevitably end up writing it. Boring. Now, lately I've used a different trick. telling people to search for the stuff I am involved in, e.g. "Happy Eyeballs" make for a pretty good search to fish my name and contact info from.

Now, I thought why not do it in a bit more dedicated fashion - and just grab a domain that is easy to remember for normal people, and hook it to a key-value store.

Here comes the secondna.me.

go to secondna.me to search on someone's "second name", or create your own - the "second name" can really be anything that is easy to remember and does not require spelling by letters.

There is no login, no password, everything is write-once-read-many. So, if you have made a mistake, too bad.

The value is escaped to translate the < into < - so, no fancy HTML either.

Just good old plain text.

If you think it does look fugly, I agree. But I think the utility trumps the looks.

Good enough for me, for now at least.

And tell me what you think.

(yes, it's like about.me, after an intensive diet course).

A toast to differences

2011-04-19T23:47:00.003+02:00

Once upon a time, there lived a gardener. And he grew oranges and lemons. He did not eat enough of the lemons himself, so he sold them in the neighboring village - there they grew bananas, but they were too lazy to grow enough of them to feed the entirety of the village. So, every day they bought lemons from him, and picked those poor souls who were to feed with lemons.

Needless to say, the poor souls had hated the nasty fruit - it gave them the pains in their bellies. And soon they started to hate the gardener himself - yet, they continued to buy the lemons from him.

It was a very saddening fact for the gardener - after all, he was honestly growing his best fruits - and the oranges were not bitter at all - sweet and refreshing. So, one day he decided to make a pleasant surprise for his faithful customers that should make them happier - he brought them oranges instead of lemons, but did not tell.

The customers were enraged - "Not only you exploit us by selling these bitter fruits to us, but now you also brought them in rotten color and blown up shape!". They've beaten the poor gardener, and trashed his cargo.

So, let's have a drink to the gentle attention for differences - if you're a gardener, try to sell not just lemons alone; and if you're on the market buying - not everything that is slightly round is bitter.

A story to remember when you are afraid to fail...

2011-04-11T21:48:00.004+02:00

Found a great anecdote here, translating:

The stage: Autumn. A lake. Wild geese are warming up to start flying to the south. At the front a well-built goose is stretching his muscles, all enjoying himself and looking forward to fly, when a little gray duck approaches him and starts the dialog:

- So-o-o-o... You're fly-i-i-ng south ?
- Yeah. Gotta fly south. It's warm there.
- So-o-o-o... A-a-and... I will sta-a-ay here... To Fre-e-e-eze...
- Come on, let's fly with us. Southbound.
- So-o-o-o... You've got big wings... I do not... I will fall and die...
- No problem - we'll catch you and keep you up. Air streams and all that.
- So-o-o-o... But I will get hungry on the way... And die-e-e-e-e...
- We're gonna pick up the bugs. Big, fat and tasty bugs.
- So-o-o-o... The bugs are bi-i-i-ig... Your beaks are bi-i-i-ig... Mine is small... I will choke and die...
- No problem, we'll chew them up for you. You're gonna make it.
- So-o-o-o...

(the goose straightens up and looks at the duck):

- Okay. Fuck off.
- So-o-o-o... As usu-u-u-ual... Exactly as I tho-o-o-ought... (while walking away).

A functional (almost) prelude on the topic of mergesort

2011-04-05T02:21:00.007+02:00

I know of table.sort, yes. But thought it was fun nonetheless.


function apply(x, f)
  for i, v in ipairs(x) do
    f(v, i)
  end
end

function filter(x, f)
  local out = {}
  local fa = function(v, i)
    if f(v, i) then table.insert(out, v) end
  end
  apply(x, fa)
  return out
end

function merge(...)
  local args = { ... }
  local out = {}
  local fa = function(v)
    table.insert(out, v)
  end
  apply(args, function(v) apply(v, fa) end)
  return out
end

function sort(x)
  if #x < 2 then
    return x
  else
    local ix = math.floor(#x/2)
    local smallx = sort(filter(x, function(v) return v<x[ix] end)) 
    local samex =  filter(x, function(v) return v == x[ix] end)
    local bigx = sort(filter(x, function(v) return v>x[ix] end))
    return merge(smallx, samex, bigx)
  end
end


function  show(x)
  apply(x, print)
end

a = { 1,3,2,4,2,1,1,23,12,22 }
show(sort(a))

NB: of course this is *not* a functional style by far, since the apply() assumes the f() will have the side effects.

Links about the situation in Japan

2011-03-15T21:51:00.065+01:00

---
This page is to keep the links about the events in Japan. The links to the "original" websites are going over CoralCDN - hence the ".nyud.net" appended to the domain name. Read more about the CoralCDN at http://www.coralcdn.org/. Chances are that not all of the websites may bear the load they will get - so use the CoralCDN so the sites do not die. I'll update this post as I find more links.
---

Countermeasures for 2011 Tohoku - Pacific Ocean Earthquake

Some baseline on the radiation:

Smoking: "Based on careful assessments of the concentrations of 210Po in the lung tissues, it was estimated that the "hot spots" received an annual dose of about 160 millisievert (about 16,000 millirem), two of the more common units for expressing doses from ionizing radiation." [Health Physics Society]. Divided by 8760 (24*365), this gives 18.26 microsievert/hour delta atop the background radiation levels. Another source with radiation in cigarettes. So if you are smoking you probably should take a note of this.

Here's another image, a scan from a book, I found it on a blog entry in Japanese about radiation impacts. 1 rad = 10 mSv = 10 mGy, the author writes;

And one more reference about radiation effects. And a diagram for comparing the various sources.

So, after setting up this baseline, you can go and look at the data.

Facts:

Geiger counter Chiba
Geiger counter in Tokyo
Video of a geiger counter in Tokyo
One more video of a geiger counter in Tokyo.
Google doc with info of the three counters above and the radiation from http://www.bousai.ne.jp/eng/index.html.
Japan radiation open data (from the maintainer of the above google doc)
Graphical dashboard based on these values
A geiger counter on the translatlantic flight
crowdsourcing data on radiation

Saitama prefecture readings

List of 5.0+ earthquakes for the past 7 days

Articles:

Graphic showing the radiation levels at the power plants vs. the various reference points
http://www.simon-cozens.org/content/radiation-tokyo-how-read-geiger-counter - has good explanation on how to read the counters and what the numbers relate to. This is where I got the first geiger counter link above.
Articles by MIT Department of Nuclear Science and Engineering about the japanese nuclear reactors.
Some Perspective On The Japan Earthquake

TV/Video:

MIT technical briefing recording
http://www.ustream.tv/channel/yokosonews
http://www.ustream.tv/channel/nhk-world-tv
http://www.ustream.tv/channel/nhk-gtv
http://www.ustream.tv/channel/tbstv
http://www.youtube.com/tbsnewsi
http://www.earthcam.com/japan/tokyo/
Fukushima Daiichi Nuclear Power Station camera
One other Fukushima webcam

Twitter:
A person who was translating the TBS
Reuters. Level-headed reporting without the hysteria.
periodic reports on radiation levels

TEPCO press releases:

http://www.tepco.co.jp/en/press/corp-com/release/index-e.html

NASA:

the Japanese earthquake should have caused Earth to rotate a bit faster, shortening the length of the day by about 1.8 microseconds

Japan Atomic Industrial Forum

http://www.jaif.or.jp/english/

INES levels:

http://www.iaea.org/Publications/Factsheets/English/ines.pdf

IAEA briefings:

Briefing videos
IAEA updates page

Networking-related:

http://www.jpnap.net/english/jpnap-tokyo-i/traffic.html
http://gigaom.com/broadband/in-japan-many-under-sea-cables-are-damaged/

Autoextraction of Abstracts from RFCs and drafts

2011-03-08T02:27:00.003+01:00

An idee fixe (uh, I mean *one more*) of mine is to somehow organize a collection of IETF docs - RFCs/drafts that are somehow touching IPv6 (Thanks to Fred Baker for this nice puzzle).

So, what I have is 140 megabytes of data, sitting in just under 2000 files that represent the RFCs and various drafts.

First step of doing anything at all with this pile is to be able to chop it into some chunks - try to put the congruent parts side by side, move the ASCII pictures aside, and similar mundane tasks.

The first step of doing that is to try to extract the part that is there in almost every IETF doc - the "Abstract" section. In general, the section titles are starting with 0-column indent - while the text of the paragraphs typically has 2+ columns indentation. However, this is a general rule. There are zillions of exceptions over the years. Variations of spelling, wrong indents, MS-DOS carriage returns, all sorts of nasty mess. Anyway, the first try at this has concluded.

I extract the titles out of the pagebreak-placed titles and this is noticeable - in some of them you have the month and the year glued on the right side. This is something that should get fixed eventually, if I figure some heuristic.

Here's a result in case you find it useful at all:

Abstracts from some RFCs and drafts.

Interesting bits from HTTP/1.1 RFC

2011-03-06T00:53:00.003+01:00

(Originally under title "Is your server HTTP/1.1 compliant ?" - but I realised that it's not really a relevant one)

Today after watching this excellent wireshark kung-fu video with Hansang Bae, I decided to comb through the HTTP/1.1 spec and see what other interesting bits I can fish from there that are less frequent/interesting to play with or are othewise noteworthy.
Here they go for your entertainment.


   To allow for transition to absoluteURIs in all requests in future
   versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI
   form in requests, even though HTTP/1.1 clients will only generate
   them in requests to proxies.

In layman terms: your compliant server must understand not only the classic "GET / HTTP/1.1", but also "GET http://www.yourhost.com/ HTTP/1.1"). In case the clients upgrade. All but one of the servers that I did a quick test with, haven't seen this part of the spec. Or optimized it out.


   An origin server that does not allow resources to differ by the
   requested host MAY ignore the Host header field value when
   determining the resource identified by an HTTP/1.1 request. (But see
   section 19.6.1.1 for other requirements on Host support in HTTP/1.1.)

Immediately after that follows a big blurb how the server is supposed to derive the host name from the absolute URI. I.e. the one that almost no-one seems to support. So the implementations deliberately ignore the spec. Or are not attentive in reading it ?
The in-progress work from HTTPBis workgroup in IETF also specifies the absolute URIs. So, some house cleaning will be in order.

A very interesting bit about the pipelining:


   Clients which assume persistent connections and pipeline immediately
   after connection establishment SHOULD be prepared to retry their
   connection if the first pipelined attempt fails. If a client does
   such a retry, it MUST NOT pipeline before it knows the connection is
   persistent. Clients MUST also be prepared to resend their requests if
   the server closes the connection before sending all of the
   corresponding responses.

This brings 'must be prepared to act robust' general statement makes me think of all sorts of interesting failure modes (yes, and indeed I've seen some of those in real life) - however, this 'retry' also brings potential L7 hook to the Happy Eyeballs logic. In some sort, maybe, later. Having a L7 hook would be a good thing - application may have a better idea about failures than layer 3/4. Anyway, I digress.

Another interesting piece:


   This means that clients, servers, and proxies MUST be able to recover
   from asynchronous close events. Client software SHOULD reopen the
   transport connection and retransmit the aborted sequence of requests
   without user interaction so long as the request sequence is
   idempotent (see section 9.1.2).

This is also a HUGE RED FLAG to the application developers: never EVER use "GET" for anything that is not idempotent. Theoretically the server should not see the two requests, but with some conditions it might (say, proxy inbetween ?) All of this is still true for the specs that are being prepared in HTTPBis WG now.

Here's the well-known humorous piece:


   Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD NOT maintain more than 2 connections with
   any server or proxy.

Year right. Web2.0 apps do exactly that. Not.


   The Max-Forwards request-header field MAY be used to target a
   specific proxy in the request chain. When a proxy receives an OPTIONS
   request on an absoluteURI for which request forwarding is permitted,
   the proxy MUST check for a Max-Forwards field. If the Max-Forwards
   field-value is zero ("0"), the proxy MUST NOT forward the message;
   instead, the proxy SHOULD respond with its own communication options.

Is there already a HTTP-level "traceroute" to poke at caches on the way ?

Kind of obvious, but interesting clarification nonetheless:


   The fundamental difference between the POST and PUT requests is
   reflected in the different meaning of the Request-URI. The URI in a
   POST request identifies the resource that will handle the enclosed
   entity. That resource might be a data-accepting process, a gateway to
   some other protocol, or a separate entity that accepts annotations.
   In contrast, the URI in a PUT request identifies the entity enclosed
   with the request -- the user agent knows what URI is intended and the
   server MUST NOT attempt to apply the request to some other resource.

The "correct" code for post-POST redirections should be 303, not the 302, but 302 was for "older clients":


      Note: Many pre-HTTP/1.1 user agents do not understand the 303
      status. When interoperability with such clients is a concern, the
      302 status code may be used instead, since most user agents react
      to a 302 response as described here for 303.

A fun fact while I tested this, is that both Firefox and Chromium send exactly 21 request before giving up and saying "it's a redirect loop". Even if the target URIs in the "Location:" header in the reply are different. Buyer, beware. It's more than "old 5" that the spec warns about - but there's no ultra-clever heuristics to get the redirect loop, either.

This stops on section 12, I'll maybe go through the rest tomorrow and see if I can gather some other interesting pieces.

Hardware hacking: Peltier TEG experiment - parts list.

2011-03-05T01:31:00.006+01:00

During the discussion over a beer at Betagroup yesterday, the topic of human-power for the devices came up. Today I stumbled upon this one and could not help but grab a couple of the "energy harvesters". They supposedly can generate up to 2v at the temperature delta of 75C. The 75C difference between the 36C of my body (hot side) and the "cold side" of -39C would mean it's a pretty useful device for a winter in Siberia.

So, let's try a different approach with a LTC3108EGN - the step-up converter.
The schematic suggests to use the CQ200 from Honeywell. But with such a form factor I am not sure I would be much interested to experiment with it :-)

Also let's get a coil that is needed. The capacitors I've in my pile already, should be no problem.

You wonder what this will be used for ? Well, of course my beloved ATTiny45, which is supposed to have a pretty low power consumption - the "V" models boast 300 microamps at 1.8V when running at 1mhz. This should be perfect, if I estimated correctly the amount of electricity the TEG will generate at low delta.

All right, now just wait till all the parts get here and then a bit of soldering and we see if this idea works.

ActionController::InvalidAuthenticityToken

2011-02-25T13:46:00.002+01:00

There are two types of answers on the web: wrong and correct.

Copying the correct answer here:

you need to add <%= token_tag %> into your form to make this.

Do not comment out the code that throws out the error - it exists for a very good reason: Cross-Site Request Forgery protection.

Which choice ?

2011-02-25T00:21:00.006+01:00

In one of the books I've been reading they were talking about choices and I think I came up with a mental experiment that has some interesting complexity in it.

Imagine, you meet a wealthy man who wants to donate some of his wealth to help the poor children of Humbrian Republic. He has $1000000 he wants to donate. There are 10000 children that this money could feed for a year. Without these money, the chances that they die of hunger are tripled. The same chances as without the money are awaiting them after the year - even if the money were to be donated.

However, this man is of a rather evil kind, and he gives you only four choices:

1. You and him part ways, without any decisions made - and he does not donate anything whatsoever.

2. You have to keep the $10000, and the remaining money goes to the poor children of Humbrian Republic - so only 9/10th of the children get the food. As a condition for spending this money you have to keep mentioning that the money you're spending might have been better donated to feed 1000 children - each time you spend something from it.

3. You get to keep $500000 - so only half of the children get the food for a year. You do not get any strings attached to this split - though only 1/2 of them get the food.

4. You get to keep $900000 - so, 1/10th of the money gets donated and you get to keep 9/10th. As a condition you have to talk on the phone to 10 children randomly from the entire set - whether they got or not the money. All of the children will know that you have kept 9/10th of the money.

Which of the four would be your choice. No need to answer, as I guess this would be a pretty confidential matter. (And please do not blame me - as I said it is a deliberate purely theoretical construct ;-)

EDIT: when I mention you get to "keep" the money it means you can not, by contractual agreement, give it away - you have to spend it on your own needs.

More fun with luajit and ffi: >300Kpps of UDP traffic

2011-02-19T23:29:00.004+01:00

Today I added the UDP interface (rudimentary so far) to my experimental event loop.

In the github repo there is the code that simply echoes the UDP packet back.

For fun, I decided to wrap the sendto call into the cycle that repeats it 100000 times.

Here is the output:


23:26:52.988762 IP6 ::1.12345 > ::1.59283: UDP, length 4
23:26:52.988765 IP6 ::1.12345 > ::1.59283: UDP, length 4
23:26:52.988767 IP6 ::1.12345 > ::1.59283: UDP, length 4
23:26:52.988770 IP6 ::1.12345 > ::1.59283: UDP, length 4

This means that the luajit2+FFI can generate between 300Kpps and 500Kpps in UDP. Yay.

Fun with LuaJIT and FFI library - httpd microbenchmarking

2011-02-18T01:47:00.004+01:00

When I saw the new FFI library in the LuaJIT project, I immediately thought "yummy". I thought I'd give it a shot during the evening, but.... curiosity had killed the cat, so...

First thing that got annoying quite quickly was the need to prefix every C function call with ffi.C. That's annoying. Can we do better ? Yes, we can. We add this:


-- try to use FFI when no lua symbol found...
setmetatable(_G, { __index = ffi.C } )

And the life immediately becomes great - you get the power of the declared functions plus the flexibility of the lua language, so you can write stuff like "local s = socket(AF_INET6, SOCK_STREAM, 0)". Tres cool.

Now, let's see how much this all is worth performance wise.

Let's write a funny high-level wrapper, that at its "application" core will look like this:


local HTTP_REPLY = [[HTTP/1.0 200 OK
Content-Type: text/plain

This is test
]]

local MAX_FD = 2560

local ss = socket_set(MAX_FD)

local my_accept_cb = function(fds, i)
  local cb = {}
  cb.read = function(fds, i, data, len)
    fds.send(i, HTTP_REPLY, #HTTP_REPLY)
    fds.close(i)
  end
  cb.close = function(fds, i)
    -- print("Closed socket")
  end
  return cb
end

while not ss.add_listener(12345, my_accept_cb) do
  sleep(1)
end
print("Added listener, please run the test")
while true do
  local n = ss.poll(1000)
end

What do we do here ? We create a high-level abstraction which I called a "socket set" - that encapsulates all the boring event loop that usually exists in programs; and then add a listener socket on port 12345 to this set, which has a callback that is called upon accept and can return either the table with callbacks for the newly accepted socket - or nil - then the connection will be closed immediately.

The accepted socket callbacks are such that they implement a very naive "HTTP server" - just for microbenchmarking.

Now, let's give it a whirl with ab.

The resuls are quite entertaining, here is how they look like:


# while (true); do ab -n 1000 -c 1000 http://localhost:12345/ 2>&1 | grep "Requests per"; done
Requests per second:    8433.91 [#/sec] (mean)
Requests per second:    12374.55 [#/sec] (mean)
Requests per second:    12720.06 [#/sec] (mean)
Requests per second:    12688.58 [#/sec] (mean)
Requests per second:    13144.58 [#/sec] (mean)
Requests per second:    10742.29 [#/sec] (mean)
Requests per second:    8651.27 [#/sec] (mean)
Requests per second:    12748.76 [#/sec] (mean)
Requests per second:    8145.58 [#/sec] (mean)
Requests per second:    12946.32 [#/sec] (mean)
Requests per second:    8648.65 [#/sec] (mean)
Requests per second:    13047.85 [#/sec] (mean)
Requests per second:    11550.14 [#/sec] (mean)
Requests per second:    12904.22 [#/sec] (mean)
Requests per second:    12968.32 [#/sec] (mean)
Requests per second:    13219.47 [#/sec] (mean)
Requests per second:    8244.09 [#/sec] (mean)
Requests per second:    12056.47 [#/sec] (mean)
Requests per second:    12834.83 [#/sec] (mean)
Requests per second:    13288.86 [#/sec] (mean)
Requests per second:    11455.02 [#/sec] (mean)
Requests per second:    11130.03 [#/sec] (mean)
Requests per second:    8034.32 [#/sec] (mean)
Requests per second:    12566.76 [#/sec] (mean)

This is not too bad at all.

Here is the similar test with lighttpd running on the same machine, serving the default static page:


Requests per second:    12080.50 [#/sec] (mean)
Requests per second:    9397.88 [#/sec] (mean)
Requests per second:    9948.47 [#/sec] (mean)
Requests per second:    12906.39 [#/sec] (mean)
Requests per second:    9284.53 [#/sec] (mean)
Requests per second:    4281.34 [#/sec] (mean)
Requests per second:    9143.44 [#/sec] (mean)
Requests per second:    12422.21 [#/sec] (mean)
Requests per second:    9170.19 [#/sec] (mean)
Requests per second:    12603.03 [#/sec] (mean)
Requests per second:    9413.54 [#/sec] (mean)
Requests per second:    12981.62 [#/sec] (mean)
Requests per second:    8615.56 [#/sec] (mean)
Requests per second:    9849.98 [#/sec] (mean)
Requests per second:    9869.43 [#/sec] (mean)
Requests per second:    9818.84 [#/sec] (mean)
Requests per second:    3384.56 [#/sec] (mean)
Requests per second:    9127.59 [#/sec] (mean)
Requests per second:    1528.56 [#/sec] (mean)
Requests per second:    9157.01 [#/sec] (mean)

Not bad at all for a high-level code, what do you think ?

If you want to toy with it yourself - it's on github.

Fun links: 17 Feb 2011

2011-02-17T00:14:00.002+01:00

Android virtualized - really cool stuff, with a whole lot of potential. Imagine being able to save your mobile somewhere else than on mobile. Yummy business opportunities, I think.
Making presentations in TeX - I made the first step towards using TeX - installed it. The results so far are not terribly cool, but it's only the hands to blame.
A fractional horsepower news network - I think this is the beginnings of the pendulum going back from "mainframe", centralized model to a distributed it. Of course, while the history repeats itself, it does so in a whimsical way, so how exactly it will look like, is a question. Maybe in a form of a Freedom Box. I've got some of that.

Me talking...

2011-02-16T01:44:00.003+01:00

In the past couple of weeks I got "on stage" few times, which was quite a lot of fun. Even more fun was that these moments were shot on video as well. So, here we go, a link collection: here's me talking at Cisco Live 2011 in London about Advanced Firewalls - and the weekend after, my FOSDEM talk about Lighting up IPv6 in Mongrel2.

Elevators: the curiosity

2011-02-16T00:49:00.004+01:00

This blog post is intended to serve as a reminder for myself as well as a teaser for everyone else.

First a bit of history of the trigger. The old elevators in the buildings in Belgium are pretty peculiar - they have no doors. Yes, if you are surprised, I was too - when I first arrived - back in Russia (and in Soviet Russia) they had the doors - either the sliding ones or closing ones. So this optimization was weird and a bit frightening.

However, over time I got used to this, and even found it kind of cool to watch the wall as in the evenings the elevator was pulling me up to the 14th floor where I live, and down to the ground in the mornings.

All is good - but the regulators came. Apparently these door-less elevators were considered unsafe by someone. I can imagine either someone got squeezed in some unpleasant way, and hence the reaction. Net result: a seemingly IR-laser based emergency stop mechanism (if you reach your hand towards the wall as the elevator is moving and cross the surface, it stops) - which is pretty cool; and the decrease in speed of the elevator.

It's the latter which is utterly uncool and is a trigger for this - it, at least perceptually has noticeably increased my waiting time in enough number of the mornings to start wondering "why".

And I started to ponder - what is the best mathematical model for expressing my annoyance with this situation in numbers ? And it seems it is a curious one, much more involved than I'd expect at the first glance.

Some reading links on the topic:

So, in short it seems to be a pretty fun modeling topic - even if we do not get into the mechanical problems and keep ourselves busy only with traffic handling problems.

The analytical question, which, after this pre-investigation I am afraid is not so trivial is the function T(v,p) - where the function value is the upper bound on my waiting time when needing to go down from my 14th floor in the morning, "v" is the elevator velocity and the "p" is the probability interval. i.e. T(1,0.95) == 40 would mean that with the 95% probability I would have to wait less than 40 seconds, assuming the speed of the elevator of 1 m/s.

Google app engine + facebook apps links

2011-02-12T11:15:00.008+01:00

Started looking at Google apps engine + facebook combo. Some links that look useful to keep around for myself:

GAE

FB+GAE

Misc

Update: Looks like all of these are obsolete, the way to go is the Python graph api SDK...

Fun links: 12 Feb 2011

2011-02-12T03:10:00.003+01:00

Going to start the "links" series, even if for myself only. Too much to post on facebook, I don't want to spam there. And easier to find here if it is indexed...

Stackexchange architecture - how to build a performant website on a small amount of hardware. Code better.
A4-sized notepad with e-paper - Not sure I want it, but with opensource both hardware and software, might be very interesting. Especially A4 size - would mean it can be used for various barcode stuff in the future.
How great entrepreneurs think.

First practical joke with gncci - clamping the MSS

2011-01-24T02:09:00.004+01:00

Today I thought that a great hack might be to try out per-host MSS clamping in userland.

Sounds like a fun idea, but ideas are not much without the execution - so this diff adds the hook for the setsockopt() - and, as a side effect the o.setsockopt() into the Lua-land. (The latter was actually what I wanted, but it was silly not to add the whole hook).

The result - it does wonders with PMTUD-blackholed websites, the world becomes wonderful again. All of this even without needing root to tweak the MTU on the physical interface.

Look, ma, no hands!

gncci - general network connections contortion interface

2011-01-23T11:59:00.004+01:00

Ok, the name is a bit wacky, because I wanted a bit of wordplay - the idea came up during the happy eyeballs discussion within the v6ops maillist, and I coded it up yesterday in one marathon run.

But I think the result is fun. You get an easy-to-tweak middle layer between the application making the socket calls and the socket calls proper. And it's not the C that you have to sweat in - it's all Lua baby.

So, by loading this with LD_PRELOAD you can code and a little bit of scripting you can do a lot of interesting things - check what connections the application is making, "sniff" the content that the app is sending onto the socket and receiving from the network, even deny some of the connection attempts or mangle the DNS packets so the application connects to the hosts that you have defined.

The possibilities are endless.

Bugs: it seems that the constructor code made it shaky. In retrospect, probably it should be protected with spinlock (similar to the existing code, which first was a spinlock, but then I discovered the multithreaded apps like firefox is might get into a deadlock - so I had to do thread-local variables).

Anyway, it works at least for some values of "works". Have fun.

Export Mercurial repository into git repository

2011-01-22T17:08:00.004+01:00

Wanna export mercurial or SVN repository to git ?

fast-export helps with that.

Thanks to Akhil for the tip.

The most interesting commands are: (to be done in the empty git repository that will be the target):


/path/to/hg-fast-export.sh -r /path/to/hg_repo 
git-repack -a -d -f

Sending the file descriptors between processes

2011-01-21T01:58:00.003+01:00

What seems like very nice library: http://www.normalesup.org/~george/comp/libancillary/

This hides the intricacies of passing the file descriptors between the processes.

If it is portable enough, that'll be nice.

On ssl caching

2011-01-19T23:35:00.002+01:00

tl;dr note for myself:

SSL-protected docs can be cached if you send the header "Cache-Control: public, max-age=31536000".

An elegant solution to avoid the hassle of explicitly specifying http/https references, assuming it does work (RFC says it should).

Politics as a catalyst of wealth

2011-01-18T22:35:00.007+01:00

The world of politics, at least as far as I comprehend it - is very complicated. The politics, is, well, politics.

Must be especially tough close to the elections - I'd imagine this is the most impacting period. The tooth that was pulled out a year ago does not hurt - but the one that was pulled out yesterday hurts all right. It must be really tough to operate in such an environment. And then those lobbyist folks with their ideas...

So here's a thought experiment that I think might help the society get easier life for the politicians and for the society of the whole.

Abandon the elections.

Instead, mandate the direct democracy by law. Yes, by direct democracy, I mean the entities like Demoex in Sweden.
But, make such a technology country-wide and mandated.

"But laymen people are not great at taking the decisions" - you will say - "and, by the way, who has the time for this ?".

Precisely, this is where the catch is. Unless the things go utterly haywire, no-one *really* wants to deal with the mundane details. But, alas - you have to - it's by law.

But, the law also would not prohibit to hire someone to do this job for you. So we will have someone who will be paid to read the to-be-ratified papers and make the decisions on behalf of the others.

How's that called ? You're right - politician.

What's the difference compared to now ? The difference is the money flow. Right now it's faceless - the electorate pays the taxes, then they go to say "this guy will represent me for X years". And then the taxes pay the rent to that guy. As a result we have a fairly non-responsive system.

In the proposed equation - you can hire the guy per-month, per-week, per-hour if you wish.

The responsiveness to the customer needs would immediately be reflected in the rewards. The customers being those who do not have to spend the time on the elections - but simply outsource their responsibility to democratically decide. And the %% of the taxes that was going into the common flow of running expenses of the government, can be dramatically reduced.

Hey, but all of that would be a very bad idea! No-one would then take any unpopular decisions, because they are risky and unpopular!

The reality is:

In business this works, with a suitable business plan.
Else, noone is taking any unpopular decisions anyway - most of the time.

So it is not much worse than the status quo.

But, hey, what happens with all the lobbying that is supposedly [sic] happens ?

Simple. Convince the masses that Idea X is a good idea. Make a decent school in the neighborhood - and those who go there will become your live advertisement - much better than on TV. And they can then spend more time watching the actual film, anyway.

This all seems to be too simple to not have tried before and too simple to not contain any catch.

If you know some good historic examples on where this has been tried, I'd be curious to know the occasion and the outcome.

Compiling Lua on the fly with tcc.

2011-01-18T02:18:00.004+01:00

tcc is quite fast.

This is in the "src" directory of Lua 5.1.4 (I added a small file with the definitions for the missing string functions).


$ time tcc *[^ca].c lgc.c lfunc.c -run lua.c -e 'print "hello!"'
hello!

real    0m0.165s
user    0m0.140s
sys     0m0.030s

Fun, even if fairly useless.

The performance of the lua interpreter that I had compiled statically, was about 1/2 of that compiled with gcc, so it is not a racer by all means - if you need speed, you will look at luajit.

Exploring the dimensions of lisp

2011-01-17T20:30:00.005+01:00

minilisp. A 18Kb S-expression library + interpreter.
A C64 interpreter called micro-lisp and MicroLisp within a blog post.
nanolisp - written for the regular lisp machine.
picolisp - full blown language, can even do web apps. 745 kb tarball.
femtolisp from googlecode - a checkout of a few megabytes. What seems another femtolisp - several standalone files.
attolisp - part of vfuncs library - because everything else is taken.
zeptolisp - "intended to have only the most basic structures and functions built into the language".
yoctolisp - anyone ? :-)

On an orthogonal axis, a couple of interesting exhibits:

LuaLisp - lisp implemented entirely in Lua.
LispmFPGA - a lisp machine in FPGA project.

If I find something else worth adding - I'll update this post.

Windows 7 in kvm: some observations and tips

2011-01-17T03:08:00.008+01:00

I spent the better part of this weekend reversing the status quo from running Ubuntu in VirtualBox on Windows 7 to running Windows 7 in kvm on top of Ubuntu. The result is catastrophic success, and makes me quite happy. This post is to share some little hacks/experiences that I've accumulated in the process.

giving the "-usbdevice tablet" to kvm will avoid you from needing to click into the kvm window.
using VNC to access the Windows VM is neat. This is achieved by adding "-vnc 127.0.0.1:1 -daemonize" to the kvm command line. Handy when you have windows running long updates. You can simply close the VNC and let it chug along.
typing "sendkey ctrl-alt-delete" into the monitor console (you get to it by pressing ctrl-alt-2 and back to gui by pressing ctrl-alt-1) [and, FWIW, using "F8" vnc popup, too] is a bit cumbersome. Therefore, since I'm going to use it alone, I'll redirect the console by "-monitor tcp:127.0.0.1:31337,server,nowait". Then I can create a shell script "send-ctrl-alt-del" that will contain the simple netcat call: "echo 'sendkey ctrl-alt-delete' | nc localhost 31337" - and then bind this to some key combo in fluxbox.
do not be afraid to undershoot with disk space for win7. You can create a sparse 1gb file by doing "dd if=/dev/zero of=1gb.img bs=1024 count=1 seek=1048576" and then by appending this file to the image "cat 1gb.img >>win7.img". After that in the disk administration you can simply grow the volume by 1Gb.
for accessing the files on host, if you do not have anything SMB-talking, you can get away with using the SSHFS. http://dokan-dev.net/en/ is precisely that. It allows userland filesystems as windows, and includes the SSHFS filesystem. It's a bit rough on edges, but works. Assuming you are using the kvm built-in networking, you will always connect to 10.0.2.2.

If this is your personal client machine, no need to keep ssh open to the world - so edit the /etc/ssh/sshd_config and put "ListenAddress 127.0.0.1" there. Do not forget to restart sshd afterwards.

When you boot up your Windows you can connect the drives back to the host. Beware to turn off the cache - else the changes would not be reflected immediately and such a setup becomes tough to work with.

Hopefully this will be useful to you.

How to disable touchpad on thinkpad laptops in X11

2011-01-16T15:56:00.002+01:00

Thanks to Nico Schottelius for this hint.

In short, I needed to do three things:

xinput list - to find which ID does the touchpad have (11 for me)
xinput list-props 11 - to find which ID does the "enabled" setting have (125)
xinput set-prop 11 125 0 - to disable the annoying peripherial.

How to quit smoking ?

2011-01-15T14:34:00.004+01:00

I long wanted to share my experiences with getting rid of the tobacco addiction, and I've noticed one of my friends abandoning the habit of smoking, so this is a good trigger to do so.

So - how to quit smoking ? The first trick - don't. No, I do not mean continue smoking. Do not put this hard and fast barrier, this non-negotiable resolution - "never, never again". Phrase it differently for yourself - "I will stop smoking. Just for some time. I can always get back to it if I want."

Why do I say so ? Because smoking is an activity with an immediate short-term positive feedback, and big long-term negative feedback. But since human intuition is pretty bad at assessing the long-term events, the short-term positive has a much more dramatic impact. So, you put for yourself this prohibitive barrier. More over, it's not one but just two hardships that you create for yourself:

The difficulty of having to survive the lack of the short-term positive feel of smoking
The fear of guilt that you will experience when you violate your "never again" promise to yourself. This guilt will make your future 'quitting' attempts harder in itself - because you will be afraid of this negative feeling that comes when you would not be able to make it.

The second one has a huge ripple effect in my opinion - not only you are feeling the guilt in yourself violating your own promise, you also lower your own self-esteem. Subsequently you try to create an escape out of this labyrinth by either saying that it was not really 'for real' or that you have the unique condition which subjects you to strong physical addiction. Well, save yourself from this nonsense. Therefore:

Don't quit - but stop.

"quit" implies a one-time action (and 'never again') - while "stop" does not carry such a pathetic charge. It's simply a declaration of fact that you are transitioning from the smoking state to a non-smoking state. It leaves the freedom of the decision to restart smoking later, if you wish, up to you - without losing your dignity with yourself.

Isn't it good ?

Now, when we sorted out the self-guilt part, we need to figure out what to do with that urge. It's damn hard to avoid going for this awesome morning cigarette after you grabbed your first cup of coffee. Isn't it ? Do you feel this itchy feeling, that pushes you to run and get yourself a pack to get a puff ?

Well, that's my second lesson that I learned. Never try to stop smoking without significantly changing the environment/lifestyle - if you do, you are unnecessarily increasing the difficulty of your task. You don't do that with other activities - so why would you do it when it is about such an important thing as your future health ?

Also - when you change the environment and your schedule - ensure there is no "dead spot" that would make you think to fill it with smoking.

These two learnings make business trips and vacations an ideal place to stop smoking:
your environment is usually unfamiliar, so your mind is concentrated on reacting to the unknown factors rather than following the routine. As well - during these periods your schedule is usually busier than normal, which gives ever more distractions from smoking.

Another important thing that a change in the environment brings - it takes away the "social routine" aspect of smoking. By and large, the social part of smoking is probably the biggest drag of it - and by eliminating it, at least temporarily, you can make your life easier. (Later, when you restart the contacts with your smoker friends, you can simply mention to them that you were not smoking for a few weeks. This will gain you a whole lot of respect and awe from them :-)

The first key period is roughly about a week - that's when the real "physical" effects of nicotine wear out. (NB: This is totally unscientific, and is just my observation on myself). During this period you still may feel some "physical" urge to smoke.

You thought it'd get easier after that ? Well, the physical part is gone. But now we've got a much tougher nut to crack - it's in your head. It lasts about a month starting from your "stop point". At this point, while you are managing to abstain from smoking, the balance is very fragile - and a lot of factors can tip it.

The result of such a tipping will be an avalanche-like wave of desire to get a cigarette - which feels almost physical! Don't believe it - it is not. It's just your brain getting stuck. Whenever you get into such a situation, whatever happens, try not to think about the large green monkey with a banana. :-) Or, more seriously - just try to dissect what seems to be a physical need for a smoke into your reasoning on *why* it drags you there and what triggered it.

Some triggers that will probably cause you do it: a smoking scene in a movie. An argument with other people that will make you stressed and you will want to relieve the stress the same way as you'd do before - with a smoke. A time when you have to wait for something just long enough to have a cigarette. (programmers: "make all", anyone ?)

Fighting these urges is probably the toughest part in the whole exercise. I don't have a good recipe for it. Keep strong and remember that you can distract yourself by other activities. Like, watching this video about marshmallows:

So here we go, few weeks have successfully passed and you're so proud of yourself - you didn't smoke a single cig since you started. Time to pat yourself on the back! Also - note that now you are feeling yourself much better. You are less tired when you wake up in the morning. You have more energy throughout the day.

Note this good feeling well. Try to remember it very hard - and try to compare it with your feelings when you were regularly smoking. This is one of the ways you are building up your barrier to fall into puffing again - strengthening the current state of non-smoking.

But there it comes - somewhere along, friends get you to the party. Modest amounts of alcohol, and then there will always be a smoker at the party. You feel a desire that is irresistible. You think I'd say "be strong and resist the temptation?" Wrong. Go for it - full speed. Get a good doze of alcohol and feel free to smoke as much as you'd do before stopping. Why ? That's because in the morning you're gonna get a terrible hangover! The body, adapted to the absence of nicotine, will have to fight two poisons at once - alcohol and nicotine. A splitting headache guaranteed. So, if you are wise enough, reading this will be enough for you to not smoke that cigarette at the party. If you wanna try it out - go ahead. Just remember not to go overboard - I am not a doctor and I am not responsible if your body decides to go fubar on you.

Sometimes you may feel the circumstances warrant a cigarette - a good talk, what not.
Again - if you are sufficiently long into the process (say, 3+ weeks) - there is nothing to be afraid - you can use this to your advantage. Light up a cigarette, smoke it, and notice for yourself it feels like you put some shit in the mouth. So what was the point of putting it ?

If you repeat these exploratory actions not too close to each other (so as not to get the tastebuds numbed down by a smoke) - you can build a pretty good psychological link of cause-effect: "I puff a cig => it tastes like shit". It will be as good as once you might just ask yourself "so what's the point? I'll just stand nearby if all I wanna do is talk".

Eventually, you realise you have much stronger incentive psychologically to not smoke - and you build up enough of a set of short-term and near-short-term negatives to prevent your brain from tricking itself that it wants a smoke: headache, tiredness, bad taste while smoking, headache in the morning - these are all immediate enough to be quantifiably annoying (as opposed to some very distant death from some weird disease. Which will never happen anyway. 'Cos it's pretty hard for an individual to assess their own death. We overcome the fear of death back in the childhood and never re-learn to feel it again).

Why I say this can work ? I don't have a peer-reviewed scientific evidence to it. It's only based on my own experience. I managed to stop smoking once for half a year, when I was on a 1-month business trip, a few years back. Since then I was looking for a good occasion to repeat this - since I liked the feeling.

Last year starting March I completely stopped smoking - till September. In September I got a block of cigarettes as a present, which laid there for a few weeks, but then the physical presence of the cigarettes and a couple of "favorable" moments did make me pick up some of the smoking habit again. Which I promptly regretted - because I got used to that extra energy that I previously had. Mid-december I stopped again and hardly a couple of cigarettes since then - but I don't feel the "urge". Feels great not to have this dependency. And I do not have the guilt for temporary "restarting" fallback. I never promised to "never ever" in the first place - so I did not violate any promises to myself.

It's up to you to decide how much I practice what I preached here, whether to believe it or not, and whether it is better to stop smoking than to quit smoking. I'd be delighted to hear your results in the comments.

Initial 802.1q trunk support for ndpmon

2011-01-12T03:05:00.007+01:00

I've spent a few hours tonight to add a feature I wanted to NDPMon - make it be able to listen on trunk interfaces.

1) only the monitoring part is supported for now, I did not even start
looking to the counter-measures.

2) "VLAN<n>:" is prepended to all the messages; "4095" signifying the
untagged frames, the other numbers mean the vlan# in the tagged frame.

3) obviously this is alpha - it can kill the newborn
kitten, cause lightning strikes and do other outrageous things.

If you're brave enough to mess around with it, head here:

https://github.com/ayourtch/ndpmon-dot1q and give it a shot in your lab.

And then tell me something about your experiences.

IPv6-only networks in Ubuntu & co.

2011-01-11T15:35:00.003+01:00

https://bugzilla.gnome.org/show_bug.cgi?id=556915 and https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/307598 are the ones to watch if you have an IPv6-only network and Linux nodes. The older versions of the network manager assumed that the network that did not have IPv4 is "broken".

When I have a chance to test how this behaves in 10.10, I will update this entry.

EDIT:

10.10 has a much better knowledge of IPv6. So far I am pretty happy. To be tested is an IPv6-only network, but for that I need to install NAT64+DNS64, which I am not sure when I am able to do.

Random DNS requests from chrome / chromium

2011-01-11T02:35:00.003+01:00

Was looking at the sniffer trace today and noticed that chromium on startup queries some seemingly random-looking names. Especially did not make a whole lot of sense that they are within the local domain.

Turns out that this is this is Chrome dealing with the ISPs who show ads for nonexistent domain names - if you do a single-word query, then Chrome looks it up in DNS, in the assumption that you were trying to reach your local server, and if the resolution succeeds, then it offers you "Did you mean ...?" popup at the top.

Obviously, if the ISP is hijacking the NXDomain, you will get "Did you mean ?" for *every* single-word query, which would be disappointing - so in case Chrome can make the requests to those "random names" (and I've verified that by installing a local wildcard DNS server + a web server on the box) - then it stops both the strange-looking DNS requests and the "did you mean ...?" popup.

EDIT: looks like I was too fast to judge that the "did you mean ...?" popup disappears - after some use, in fact it does not. OTOH, maybe there is some additional cleverness involved in Chrome's behavior - because I merely return the default lighttpd page to it.

The dusk

2011-01-10T03:57:00.006+01:00

The dusk. The eyes do not adapt yet to the darkness that is blending its way in. Like in a bath that is cooling itself down, you don't realize when exactly does it get cold. When you do - it's already because of the tremolo of the teeth, it's too late. Or was it because you were slightly tired and fell asleep ?

Sleep is a treat - you get to appreciate it only when you lose it or can't have enough of it. Falling into momentary clouds of pleasure too strange to describe, only to jump out of it again less than a minute later. You know you should not, but it's irresistible to try again. Your eyelids drag you - stronger and stronger, until you give in again for a short blackout. And you're back.

The world is happily doing without you. Or so it seems. Did it get darker or you're just being self-centric ? It surely did get darker. But it's only one of the cycles. Don't take this personally. Better mind the damn steering wheel - you're on the road.

Higher order functions as a building block for readable code

2011-01-09T17:12:00.004+01:00

If you use a language like Ruby it's trivially easy to make DSLs and all this sweet-looking meta-programming stuff - you're lucky to be close enough to Lisp so you get the code that you can tweak the code before it's run. (attr_accessor, for example, is conceptually nothing more than a macro that expands into the reader and writer methods).

In Lua, you do not have that luxury. Well, you can have - if you install Metalua. But for my purposes this is usually a bit of using a nuke to shoot the pigeons. I want to minimize the number of dependencies - consequently, I try to rely on the stock 5.1 install as much as possible.

So, how does higher order functions help, you not getting a chance to do a "meta" code run ?

Because they allow the deferred execution.

This is a trick I used with this construct in gliese:


mongrel2connect {
  sender_id = '558c92aa-1644-4e24-a524-39baad0f8e78',
  sub_addr = 'tcp://127.0.0.1:8989',
  pub_addr = 'tcp://127.0.0.1:8988',

  -- Routing 

  get { "/", default_page, params = {
       test = { optional },
     }
  },
  get { "/foo", foo_page, params = {
       test = { mandatory },
     }
  },
  -- Anything else just redirect to root
  get { ".*", redirect_request_to("/") },
}

mongrel2connect is a "real" function (the one that actually gets called and runs the loop servicing the requests)

whereas the "get" is the higher-order function. It returns a function that can be used to match the request against the supplied pattern to see if it is routable to the handler, and as well to check the parameters.

So, while the arguments for the mongrel2connect are calculated before the call to that happens, they can be used afterwards - thanks to the fact that the arguments do not do any action - but merely return the code to do the action.

This is a neat trick that allows me to make very readable code (at least for me:-) - essentially a DSL that you squeeze into a Lua table.

N-gram and regex search

2011-01-07T03:40:00.003+01:00

So, I was thinking about the regex search in large volumes of data, and realized this should not be too hard to do. What do we do for a "usual" search: build an inverted index of stemmed words, and then match that with the query and intersect the word posting lists.

Now, imagine instead of words, we use the ngrams. Ok, here we have a little bit of a problem because if we are searching on "foobar" and we matched "foo" and "bar" we need to ensure they are also adjacent in the documents found. But, this should not be a big deal.

Now the interesting part. I am searching for a regex. Instead of matching the inverted index ngrams against the query string, I need to do something more clever. Let's pretend we have the regex implemented in terms as a DFA. A given ngram is suitable for the selection to begin with if it can result in valid transitions of the DFA. So for each of the states we can try to run the ngram against the DFA and see what gives. If this ngram does not give the good match, we discard it outright. If it does work - we store the acceptable chain of states.

Note that a single ngram might fire up more than once on a given regex - so we end up storing quite a few of possibilities.

Once we're done - we have a bunch of "broken chains" - because they are short. Now the task is to see if we can reconstruct the chain of the states from starting till ending one, and if we can - then this set of ngrams is a candidate set which can be used same as in the literal search onwards (e.g. in something like http://qwik.jp/senna/).

The remaining question is what is the complexity of such a "glue" algorithm. The number of permutations to test can quickly become prohibitive.

What's in your name...

2011-01-07T01:38:00.003+01:00

One of my friends expressed that she can not remember the names that sound pretty similar to her. I do have the same problem - and, as I learned a couple of times, at least my surname causes a similar stress in some folks.

But, I remembered, it's been quite some time I was joking that replacing the names with SHA-1 hashes would be way more practical.

Consider:

1) SHA-1 hashes are all standard 40 bytes long hex numbers. This allows much more efficient storage of names in the databases and much more simplified input validation - it will be reduced to a trivial regex: /[0-9a-fA-F]{40}/ - and the input later on can be homogenized by lowercasing.

2) There's enough hashes to give everyone a unique ID. 2^140 which forty characters make, is huge. Of course, we need this huge number to prevent the hash collisions.
I should make the estimates of that a topic of a separate post.

3) The cryptographic hash is going to have pretty good distribution of digits. Corollary from this - means that within your circle of friends you do not need the full 40 characters. Much like close friends do not call you by surname - and just call you by name.

So, I've set out to look at (3) - because, obviously, the usability in this is paramount.

Short result: amongst the 370 friends I have on facebook, if they were to use SHA-1 hash of their name instead of their "real name", just five characters will be enough to uniquely identify all of them! This is even shorter than my own name!

So, it is very promising.

Here's how to do it in case you want to verify your own circle of friends (I take the facebook as an example).

Go to http://developers.facebook.com/docs/api and click on the link next to the string "Friends: ". This will give you an ugly page full of JSON. Save it into file and go into the shell.


cat friends.json | sed -e 's/^.*\[//g' | sed -e 's/\].*$//g' | sed -e 's/[{}]*//g' | sed -e 's/,/\n/g' | sed -e '/^"id":.*$/d' | sed -e 's/"name"://g' >names.txt

Now that we have a list of names, let's make hashes. Simplest is to create a short script to hash its argument:


#!/bin/sh
echo -n $1 | sha1sum

And finally we output the file running the hash on each name, then cutting out the first few characters, and seeing how many collisions do we get:


$ cat names.txt | xargs -n 1 sh hash | cut -c 1-3 | sort | uniq -c | grep -v '      1 '
      2 2c3
      2 3d5
      2 4fe
      2 6ef
      2 b32
      2 b52
      2 b7e
      2 c78
      2 c9f
      2 d6f
      2 e42
      2 e6f
      2 f69
      2 ff5

So, with just three(!!!) characters of the hash there only a handful collisions.
Using couple more characters eliminates them and as well gives an ample room for a collision-free growth of my friend list.

And maybe now you get an idea why at the top of the page there's this row of hex, to begin with.

I think using the hashes instead of name is indeed much more practical than trying to remember the names - c3e9f, do you agree ? ;-)

Ivory tower gurus...

2011-01-04T17:32:00.003+01:00

I've seen this Dilbert in this IOSHints blog entry, here's what I think:

(I wish I could write about memory, layers of indirections, and such - but there was only so much space).

Why do people think that politicians need to learn history, but the technologists do not ?

correct scapy2 import statement

2011-01-02T18:54:00.003+01:00

Problem:


>>> from scapy import all
>>> a = IP()
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'IP' is not defined

Solution:


try:from scapy.all import * #Required for Scapy 2.0 and above
except:from scapy import * #Scapy 1.0

(via Tim Eberhard's post. On a side note, here's a pretty uselessly closed bugreport.)

Mongrel2 and SSL

2011-01-01T21:18:00.005+01:00

Mongrel2 supports SSL. Well, actually TLS, to be exact. To be running it you need a version with the small fix that disables the default built-in key during axTLS' SSL context initialization.

There are two steps you need to do to get your server support TLS-encrypted connections:

1) edit the config file


main = Server(
    uuid="f400bf85-4538-4f7a-8908-67e313d515c2",
. . . [snip]
)

servers = [main]

settings = {
  "f400bf85-4538-4f7a-8908-67e313d515c2.use_ssl": 1,
  "certdir": "./certs/"
}

Needless to say that the plaintext will no longer work on this server entry.

2) generate the private key and obtain a certificate

This is something that is widely covered on the Net. If you want to install a self-signed cert for testing, there is a good doc that describes how to do it.

The key mongrel2-specific part here is that the private key file is named <Server-UUID>.key, and the certificate is in the file <Server-UUID>.crt

(note: a "private" key is called private for a reason. You do not want anyone except the mongrel2 to be able to read it. So, mind your permissions).

For the lazy

I've added a sample static SSL site as well to Mongrel2 Hacking In a Box repository.
It has a canned config, also a pre-generated dummy key and cert, as well as a script to regenerate them.

Happy hacking!

Edit:

If you are trying with chrom(e|ium), chances are that it would not work correctly - at least it did not work for me. Seems like it is a bug in the built-in SSL library which does not want to do TLS1.0, only SSL3.0. To cure this, you need to launch the chrom(e|ium) so that it uses the system SSL library:


$ chromium-browser --use-system-ssl

Notably, it was not always that I experienced this failure - the problem did is not reproducible enough to debug it for now. If you find anything - let me know.

Installing VMWare ESXi CLI on ubuntu 10.04

2010-12-31T03:46:00.004+01:00

This post in forums and this post in forums had proved to be quite useful while sorting out the things.

The magic sequence that seems to have made the vmware CLI work on my ubuntu 10.04:


sudo apt-get install libssl-dev 
sudo apt-get install libxml-libxml-perl
sudo apt-get install libclass-methodmaker-perl
sudo apt-get install libcrypt-ssleay-perl
sudo apt-get install libio-zlib-perl
sudo apt-get install libio-compress-perl

testv6.stdio.be pilot on mongrel2

2010-12-30T03:07:00.002+01:00

Finally today I moved in pilot mode the testv6.stdio.be site (IPv6 brokenness test) to use the mongrel2.

The code for the ipv6 test site is in the git repository for gliese (with tiny differences like ports, etc.)

Let's see how it fares.

How to install Mongrel2 on Ubuntu with little pain

2010-12-28T01:49:00.003+01:00

This post is about installing the latest and greatest of Mongrel2. Moreover - it is biased towards my needs (e.g. Mongrel2 repository automatically exported into git repository from fossil in the process of installation - because I am used to git more; as well as installs my gliese framework and dependencies.)

With that disclaimer, here is the magic link:

https://github.com/ayourtch/gliese/raw/master/examples/m2-dist/install

what you need to do is wget this file and then run it:


wget --no-check-certificate https://github.com/ayourtch/gliese/raw/master/examples/m2-dist/install

Certificate check is disabled because wget seems to dislike the wildcard cname in the cert. Alas.

When you download this file, run it - "sh ./install".

The first batch is a lot of packages that are installed via apt-get, and it will prompt for the sudo password.

After that bunch of packages/software are downloaded in source form - namely: zeromq, redis, fossil, mongrel2 and the associated lua libraries from github.

When the installation has finished, you can test it:

"cd dist/lua/gliese/examples/m2-site; sh run-all-in-screen"

and then connect with the browser to http://localhost:6767/lua1 - this will present you a small dynamic page with the counter stored in redis. This allows to see that all the components are installed and are functional.

If you want to hack on mongrel2 - then feel free to poke in mongrel2-git directory - that one contains the git repository autoconverted from fossil in the process of install. I liked fossil a lot - but just got used to git more, so that's the reason I do it in this script.

Enjoy!

Building Mongrel2 on ARM architecture

2010-12-08T19:43:00.005+01:00

The past weekend I got mongrel2 to build and on my sheevaplug. Not really rocket science, but in case anyone is interested or has some more experience (more on that in the end of the post), I'll document what I did.

First thing - installing the prerequisites, that was easy, just apt-get them - same as anywhere else.

So, then I kick off the make and get:


cc -g -O2 -Wall -Isrc -DNDEBUG    -c -o src/task/context.o src/task/context.c
src/task/context.c: In function 'makecontext':
src/task/context.c:97: error: 'mcontext_t' has no member named 'gregs'
src/task/context.c:101: error: 'mcontext_t' has no member named 'gregs'
src/task/context.c:102: error: 'mcontext_t' has no member named 'gregs'
make: *** [src/task/context.o] Error 1

This does not look cool. Let's see the offending part of the code in src/task/context.c:



#ifdef NEEDARMMAKECONTEXT
void makecontext(ucontext_t *uc, void (*fn)(void), int argc, ...)
{
    int i, *sp;
    va_list arg;

    sp = (int*)uc->uc_stack.ss_sp+uc->uc_stack.ss_size/4;
    va_start(arg, argc);

    for(i=0; i<4 && i        uc->uc_mcontext.gregs[i] = va_arg(arg, uint);
    }

    va_end(arg);
    uc->uc_mcontext.gregs[13] = (uint)sp;
    uc->uc_mcontext.gregs[14] = (uint)fn;
}
#endif

Okay, this is not too shabby - means this is an ARM-specific code. But, if this is ARM-specific code - how comes that we have a mismatch for the type ? And why do we need this code to begin with - "man makecontext" shows it there ? Quick Google search shows the makecontext is not defined on the ARM architectures, obsoleted API, bla bla bla. Anyway let's test what do they mean by "not defined" with a dumb app.


ayourtch@ubuntu:~/test$ cat test.c
#include 

int main(int argc, char *argv[]) {
  makecontext(0, 0, 0);
}

ayourtch@ubuntu:~/test$ gcc test.c
/tmp/ccOO8Fqq.o: In function `main':
test.c:(.text+0x24): warning: warning: makecontext is not implemented and will always fail
ayourtch@ubuntu:~/test$ gdb a.out
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabi"...
(gdb) disass makecontext
Dump of assembler code for function makecontext:
0x000082e8 :     add     r12, pc, #0     ; 0x0
0x000082ec :     add     r12, r12, #32768        ; 0x8000
0x000082f0 :     ldr     pc, [r12, #3368]!
End of assembler dump.
(gdb)

Okay, surely they really mean it - this code seems indeed like just a plug. No-one's home. At least we have a warning. So what they say on the interwebs is true, and that's why we're dragging this ARM-only function implementation.

Let's take a look at where this structure is defined.


$ grep -R -A 10 mcontext_t /usr/include
/usr/include/sys/ucontext.h:typedef struct sigcontext mcontext_t;
/usr/include/sys/ucontext.h:    mcontext_t uc_mcontext;
/usr/include/signal.h:/* This will define `ucontext_t' and `mcontext_t'.  */

This structure defines the mcontext_t as follows:


typedef struct sigcontext mcontext_t;

And the sigcontext is defined in /usr/include/asm/sigcontext.h:


        unsigned long trap_no;
        unsigned long error_code;
        unsigned long oldmask;
        unsigned long arm_r0;
        unsigned long arm_r1;
        unsigned long arm_r2;
        unsigned long arm_r3;
        unsigned long arm_r4;
        unsigned long arm_r5;
        unsigned long arm_r6;
        unsigned long arm_r7;
        unsigned long arm_r8;
        unsigned long arm_r9;
        unsigned long arm_r10;
        unsigned long arm_fp;
        unsigned long arm_ip;
        unsigned long arm_sp;
        unsigned long arm_lr;
        unsigned long arm_pc;
        unsigned long arm_cpsr;
        unsigned long fault_address;

As such, we need to adjust the code accordingly - to use the arm_XXX named members instead of the array (but array was so convenient!)

After this we can compile everything, but alas, the segfault happens at the very first test.

Why ? Because the libtask's ARM definition of mcontext_t assumes starting with R0 content - but on sheevaplug I can see 12 bytes of other content in front.

My first attempt was to modify the assembler code, to add 12 to the structure pointer on use - but in the end I came up with I think a (slightly) better approach - instead of passing the address of the mcontext_t structure, on the ARM architecture I pass the address of the R0 member withiin the structure, typecasting it to (void *).

This avoids the need to dig into assembler code.

Now I can successfully run mongrel2 on sheevaplug.

However, this leaves me with the question:

Inevitably, the original code ran on *some* ARM. So if the code with the diff from http://mongrel2.org/info/3f79a05147 broke the build for you - let me know, so we can fix it properly for everyone - I do not have other hardware.

Disable HTTP pipelining in Ubuntu

2010-11-23T19:06:00.002+01:00

Mostly a note to self so I do not have to work to find it next time.


echo 'Acquire::http::Pipeline-Depth "0";' >> /etc/apt/apt.conf.d/00no-pipeline

(EDIT: the title seems to suggest it is a 'generally good idea' to do it - no, it is not. It's a workaround.)

The justice

2010-10-20T00:12:00.007+02:00

It's Friday. The week is over. Stu liked Fridays. Because they aired his favourite show on the WTV - "The perfect crime". Every week they'd pick up one of the cases where the jury failed to find the consensus - and rerun it on air live. And collect the instant responses from the audience to make the verdict based on the simple majority. The direct democracy: crowd wisdom at its best. The audience would make a decision whether the person goes into the recycler or not. There were thousands of the replies - one can't be wrong with such a sample. But still the result was uncertain each time until the last moment. He liked the tingling sensation these shows would bring - it reminded him the good old times before the machines took it away from him.

He was a skyscraper window washer. He was not in it for the windows - it was that tingle sensation that attracted him. He mastered the corner wipe better than everyone and was rightfully proud of his skill. But he could not compete with the machines - they were cheaper - did not require insurance and would never make mistakes, except when mis-programmed. He helped the geeks who came with the machines to hone their skills, but being the cyber-shepherd was not for him, so he moved on to become an instructor in the local climbing club and opened a little food shop in the ground floor of his house. But all of that was not the same - and that's why he liked Fridays.

Today's case promised to be interesting. It was an investigation into an accident that was suspected to have been cooked. The defendant, a young woman, was fully aware she was the only available suspect - and has already told her story. She was visibly nervous, but in her last words she said that she firmly believed that the nation will judge her right. Because it was not her.

Something in her tone, in the way she was talking, made Stu believe she was a victim of the circumstances. He usually would not press the button during the shows - but today he had a feeling he probably should. Even though almost every time they'd say something to try to convince the public. But this time it was different.

The noise of the broken glass downstairs. Not good - need to check. There's been a few burglaries lately in the neighborhood - someone said it was the gang of homeless, the others claimed it was the underground communist cell refilling the replenished reserves. And this of course had to happen just today, when he had blown off the fuse. Sigh. Got to take a flashlight.

The quick inspection of the door revealed nothing suspicious. Windows - intact as well. The dietary supplements corner revealed the truth. Some visitor had smelled the Valerian and did not properly close it. Flasky, his cat, took this upon himself to fix this - of course, smashing all the rest of the glass in ecstatic joy. So all of that noise was for nothing, after all. The damn animal had just spoiled a nice evening in front of the TV. Stu collected the glass and put it into the bin. Hopefully he can make it in time to at least see the result.

As he was climbing the stairs, he heard the polished voice of the host - "Thank you. Thank you to all who participated in today's poll. The decision was extremely hard for you - the difference is only one vote! But you voiced your opinion, the nation of Tivyland. And the verdict is: Guilty".

Damn cat, he mumbled. Should have listened to Johnny and sterilized it. Gonna do that on Monday. He went on to his fridge and wrote in the planner attached to it, right where the hearts of weekend were turning back into the spades: "Take care of Flasky".

Then he opened the fridge and took another beer. He still had an untouched six-pack and Friday evening was just beginning. There were still some other good shows later on tonight.

The good news

2010-10-18T21:00:00.003+02:00

Torro was unlike his mates. He was not excited by those of the boys' games that favored the hesitation of choice in them. What he was fascinated with was having a purpose for himself. He excelled at going towards the goal, rushing to the finish line and blowing past it, tearing the ribbon apart. He was the winner, and the world treated him accordingly - he'd always be in the hot spot of attention. Why was he like this ? Probably the adrenaline rush. Probably the ego. Probably the taste of the fruits of fame. Probably all together.

Bruna lived for today and now. She did not hurry - why rushing past ? It's like going on the express train through the summer forest - all you get is the branches hitting your face and the locomotive smoke. Jump off, close your eyes - can you hear that nightingale ? Wait until the tar dissolves and fall onto the ground - can you smell the strawberries ? You may have forgotten it - because none of the supermarkets can recreate this smell, no matter how hard they try. It dies off as soon as it hits the civilization - getting yourself out of it is the only way to experience it.

- "Hey, Bernie, check this out - "
- "What ?"
- "The news says, down there in Europe, they've forbidden the bull fights"
- "Did they allow the bull fights to begin with ?"
- "Yeah, looks like they did."
- "That's silly. Torturing the poor animals."

Theseus

2010-10-17T04:28:00.002+02:00

The plane was about to land. The landscape underneath was all squares. They were mesmerizing in their proximity - feels like they are all so close, just a couple of meters away, he could not stop looking at them. This has been his favorite nightmare of all the times - inexplicable and personal, like all the nightmares are. The squares which are at an arms' length and at the same time so far that it feels frightening. He was not afraid of height, but this was not about the height - it was about the strange, magical power of the uncertainty. When he was a little kid, he would imagine throwing the ball at these squares, and whenever the ball bounced, he knew the squares were close and small - not those huge ones that are kilometers away that he was afraid of - then he would go to sleep again.

Today was different - it was window behind him and the squares, so he could not use his trick to understand where they are - and this filled him with the anxiety. Hopefully the pilot does know how high they are. And it better be those huge ones far away - otherwise there's no time even to take the position that they show during the briefing in the beginning of the flight. He remembered it was resembling the fetal position, as much as the safety belts permitted it. Maybe worth doing that just in case - but if his feeling about the height was wrong then it would look embarrassingly stupid.

Like that guy that ran with all his belongings when the fire alarm falsely rang in the middle of the night. He chuckled. In any case, if the plane would have slammed the ground, not a whole lot of the positions would help. Someone cynical once told him that the sole purpose of these positions in the airplane was to preserve the jawbones as intact as possible during the accident - then it is easier to identify the passengers, by their teeth. Like archaeologists do. Screw that, no one is waiting on either side of the route - so whether they would be able to assemble the teeth or not, does not really matter. They'll find it from the engraved label on his luggage, if they want.

Strangely, he was not afraid of the magical squares anymore - the nightmare becoming the only hope. He lost the count of time - maybe it was a second, maybe ten minutes - the time went its own ways, independent of him - it felt like an eternity. Finally he felt the shock of the wheels hitting the ground. Landed. So, it was all right after all.

The noise woke her up and she lifted her head - he was sitting on the floor, staring into the carpet. The carpet was a childhood present to her - gentle gray wool, with squared ornament. She liked it, it reminded her of labyrinth, from the favorite book of hers - the legend where the Theseus killed the Minotaur. That was how she met him - he reminded her of Theseus.

- "What's the matter, darling ?"
- "Nothing. I just fell on the floor. Sorry to have waken you up."

Disable HTML in Outlook

2010-10-12T10:31:00.004+02:00

More of a note to myself really.

Two steps:

1) Tools => Options => Mail format ; "Compose in this message format" - set to "plain text".

This makes all newly composed messages show as plaintext.

2) Tools => Trust center => Read all standard email in plain text.

This makes messages you receive show as plaintext.

Oh, and probably uncheck the scripts, while you are on the latter.

Airfoil. ASCII.

2010-10-08T16:42:00.003+02:00

Read on HN about the http://xkcd.com/803/ with the comment "What's the correct answer ?"

I think it is this:

                                                


                                                   ^  counterforce 
                                                   |   lifts the plane
                                                   |
                                                   +
                                                         /---  <-- this is a wing
  spits hot pollutants <===[engine]===> counterforce /---
                                     pushes      /---
                                  forward    /---  
                                         /---      +---- <=== air hits 
                                                   |         this direction
                                                   |
                                                   |
                                                   v air
                                              bounces down

The drawing is imprecise. I welcome you to spot the places where it is.

But it should be correct in principle.

"This august had 5 sundays, 5 mondays, 5 tuesdays..."

2010-09-02T13:29:00.004+02:00

"...and this event takes place every 823 years..." - said the fb update of one of my friends. This meme had then a viral clause attached to it so to ensure its replication - something about money, happiness and all that.

However, it's not the viral clause that caught my attention - but, rather, the "takes place every 823 years" part of the meme, which looked suspiciously fishy - and I decided to make a quick write-up as an after-lunch exercise.

Intuitively, it is simple to see that every August will have 5 of *some* three consecutive days of the week - simply because there are 31 days in august, and there are 7 days in a week - and 31 modulo 7 is three. And the days are consecutive - so this leaves us with 7 combinations of "triples".

The number of days modulo 7 from the "previous august" will give us the shift within days of the week year-over-year. For an ordinary year, it's 1 day (365%7), and for a leap year it is 2 days.

So, if we have the five of sunday-monday-tuesday this august, then the next occurence will be in 2016. If it was not a leap year - then we "overshoot" the target, and need to walk another 6 years. So the next occurence will be in 2021 (11 years after).

Then in 2027 (6 years), 2032 (5 years), 2038 (6 years), 2049 (11 years), 2055 (6 years), 2060 (5 years), 2066 (6 years), 2077 (11 years), and so on.

With the differences forming a nice series: 11-6-5-6-11-...

The period of this sequence is 28 == 11+6+5+6. Which happens to be divisible by 7 (number of days of week in a week) and by 4 (The usual interval between the leap years).

We did not take into the account the years of 2100, 2200, 2300, 2400 - I leave it as an exercise to the reader.

In conclusion - this was a well-crafted meme. It had not one but two replication mechanisms, that were targeting different people. I guess I can say I fell the prey of it. :-)

OCR for business cards' photos

2010-08-25T01:19:00.005+02:00

I've been curious to try and do the character recognition on the photos of a few of the business cards that I made during the IETF (since otherwise the business cards turn into dust in my jeans' pockets - it was a much better option to make photos).

I've tried gocr, tesseract and the demo version of abbyyocr. The results were pretty much the same as I could find on the web otherwise: gocr was mediocre, tesseract was somehow promising (it could +/- reliably read name and surname - that were written in bold) - and the abbyyocr was actually half-bearable.

To the defence of the software, I must say that the images were far from being a typical OCR-food: shaded, distorted perspective - and the letters were a bit too small.

The interesting effect that I observed though, that when scaling the images *down* from the original to some ~80% the recognition quality increased in all the packages - with the abbyyocr getting to the point of being usable.

The time it took abbyyocr to do the recognition was notably longer. But, if I have to do OCR, among all three I will probably pick that one.

We will, we will rack you

2010-08-19T20:18:00.006+02:00

DISCLAIMER: in case you would not notice. It's a fiction piece - therefore any analogies to the real world are coincidental, hyperbolic and should not be interpreted outside the boundaries of this text or its continuation. Failure to do otherwise may cause the falling bowls of petunias. Thank you for taking care of our environment.

Your Caring Administration.

A swarm of tiny flying insects were hammering at Jack's door. They were so many that their sound resembled a heavy rain. And their synthesized sales chorus was resembling the machine at the dentist's.

".. - We will organize your data for as little as the rights to 2% of it yearly for the first 10 years after your death - .. ".

Damn. It hurts to be the last messed up guy in the area. These were automatic sales agents, tuned to find out the piles of unstructured non-indexed data, and to try to seduce the owners of that data to give it to the army of the all-mighty processing machines of Wiggle.

Wiggle, with their slogan "Your information wants to be free - don't be rude to it!", has expanded quite a lot in the past decade - from a small group of fanatic hackers obsessed with an idea to a large publicly traded corporation. Despite for its size it retained the passion and the hunger for the data that it had for the early days. Their operating principle was simple: you let them arrange your data for you and make it easily fetchable for you, in return you give them the full rights to a share of your information after your death.

For everyone it was as good as free - especially since the majority of the shareholders were in their early twenties - no matter how educated you are, at this age the death itself is a faint ghost, let alone whatever happens after the death - the full-brightness social interactions do not leave any other meaning to life other than "this!". And the letters you write - who cares what happens to them after your death. You won't be there. But it's great to be able to pull just the one you need from out the pile with the elegance of a card magician from the neighboring street.

First it was a neat parlor trick to impress the others, then it became an ubiquitous necessity of everyone's lives. The skills that were needed before were not anymore - this means you can free up that part of the brain for something else, yay! And everyone jumped on the bandwagon. Almost.

Jack was among those who did not. He preferred to keep his cards close to the chest. Not that he had much to hide - only the usual set of human screwups, but he felt naked otherwise. And that was something he hated, hated so much that he hung on to this habit even after it caused a few rather painful breakups - when people come to expect something as a natural property, the lack of it makes you "strange". Even if you think this property is something unnatural (and even if it is) - you are the one who is the gray swan - because you are "deviating from the norm".

Nonetheless, Jack knew where it was coming - that was why he was one of the early purchasers of the shares of Wiggle. And when the first deaths happened, the co-actors of the events that better be left private started to queue up to the offices of Wiggle in order to buy out the information they did not want to be free, literally off the deathbed of their best friends. As disgusting as it may seem - it did not involve selling drugs or trafficking the humans, nor killing the tuna, so it was good enough for Jack - who could happily retire and was now living in a house by the seaside of the El Hierro island.

... to be continued ...

A gift (2/2)

2010-08-16T23:05:00.007+02:00

This is a (somewhat unexpected) continuation of the first part. A bit rough, would need further editing.

Heat in the head. Not the one you feel when you are ill, it's different. The heat you feel when you think hard about something. Like in the university,
when they give you a puzzle you haven't solved before, there's a time limit,
and the clock is ticking, and if you don't get it done - you fail the exam. Wait, what's this ticking sound ? Ah, indeed, there's a clock. He opened his eyes. And it shows just after five. He was about to think whether he slept at all when he realized something was strange. Something subtle. The bed, the bedroom it was in - all looked boringly usual. Finally it clicked.

Thin Mint cookie. On the headboard shelf, near the clock. Was that it ? No, something else. What was it ? Ah. His body. Missing. How ? Ah, obviously, that smiling old man and his "mind uploading". They must have kicked him subconscious and syringed a fair doze of cough medicine. But this would not explain this constant buzz in his head - even though his conscious brain was evidently idle, the subconscious obviously was not. But it must be just an effect of DXM. And in any case, the old mofo has ripped him off of 50 bucks - it's only one copy of him, that is visibly - no, mentally - present.

"Or two".

Now that was not him. And it was. The phrase felt like talking with himself yet it was not the controlled dialog - it was an interrupt that carried a wave of the annoyance that was suppressed.

"Idiot. You should not have asked for two copies. Now you get the suffering you deserve!"

- Why ?
- "Do I have to explain to you the perils of the non-standard setup ? Of course not - because it's me who is going to suffer!"
- Suffer from what ?
- "From doing all the work, goddamit! If you did not yet admit that this bloke was not joking, you better do it now! We are indeed now in the 'uploaded' state."
- And how does it work ? I venture it has to do with 6.443J which I missed almost entirely.. Brain alone would require an enormous amount of computational power to simulate - let alone the bed...
- "Probably it does. Because, being your copy, I can't fully comprehend it either.
If not your twaggering, I might have not ended up being such a dweeb."
- Hey, if we are two perfect copies, it's equally your fault same as mine. So, stop offending me, okay ?
- "Fine, you have a point. In any case - I came to consciousness a bit before you had, and searched a bit on this topic. I think this might have to do with the P-NP."
- Transistors ?
- "Idiot. Complexity."
- What would it have to do with the complexity ?
- "There was a supposed proof of it a few months ago. Which was proven incorrect, however it started a lot of talk around the subject. One of the reviewers observed that the problem of solving the 'P!=NP' is an NP problem in itself. The other reviewer observed that both the time and space are computational complexity resources. And the neuroscientists were exploring the hypothesis of quantum brain dynamics. This guy added 2+2 and went a bit further."
- I still can not make sense out of it...
- "That's because your brain is busy simulating me and the entirety of my world - that's where the feeling of 'puzzle' comes from. My and the entire world I am in is your puzzle you are trying to solve. Much like you and your world around you are now inside someone else's brain. Who is in turn inside someone else's brain, and so on. Much like you have with the russian dolls. Except they do not change in size - the time becomes slower so the space 'expands'. That's why it will never be anything later than just after five on this clock."
- But it ticks same as before!
- "That's your brain offering itself a stitch, so it did not go sideways like waves on the water. The sound of tick is local to your consciousness, while the picture of the clock in this case depicts the 'global' time. Don't ask me why."
- Great - so we have uploaded mind which has enough computing power to simulate another uploaded mind, which in turn simulates another one. What's the catch for the uploaded mind ?
- "There's none. Since there are more and more people coming in to the old man, the time has to slow down faster for those "outer" minds. So, their conscious part, with the speed of their thinking, never makes it to being overloaded by the computation required to 'simulate' the inner mind."
- Complicated.
- "It's exactly the same principle as the pension funds. You allocate them your money that you earn so they can invest it into the publicly traded companies, which then pay you this money as wages, which you invest into the pension funds. Your neighbor needs money, so he gets to borrow it from the pension funds, which then sell the debts to other pension funds for more money. The whole system works - if it didn't, you'd not be paying into your pension fund, right ?"
- If your description is correct to any extent and we both are not a split-mind after a hallucinogen overdose, we're past the point of needing to talk about the pension funds.
- "No, we are not. Since it's not just the mind, but the whole world that is emulated - there are pension funds in each of the nested 'russian dolls'".
- And do they all work the same everywhere ?
- "No, of course they do not. The complexity of solving the problem of pensions relates to NP same as NP relates to P."
- This guy is ambitious.
- "And very practical. Remember that you - me - we - paid hundred bucks. And you saw a couple of people coming in over a course a few minutes. So, he appears to have solved the problem of functioning pension funds."
- Does this mean he proved P=NP ?
- "Not at all. He cheated, somewhat. Really you do not care about the "NP" part as soon as you can enumerate all the alternatives fast enough. That's what he did for the best himself and the mankind - a computing device of an ever increasing capacity. It's like a blackhole but instead of sucking in the matter it sucks in the computational power, and overall becomes yet stronger. Or so is my theory - I am still not clear how he would suck out the excess of power that is left after all the work required to simulate the inner worlds."
- And how comes that we are able to talk just like that - if your theory is right then it should not be possible for me to be aware of each other, much like I am not aware of the existence of the "outer" person.
- "We are copies of the single brain. Again, this fact and the supposed quantum properties of the brain, even simulated, have to do with it. It's kind of I say 'bitter lemon' and it makes you feel the taste of it - even though you do not have a head now, much less the mouth and tongue. Don't bug me - I told you already, it's just guesses - you missed too many classes for me to be able to tell it any better".
- It's all good. I'm just killing the time - if your theory is correct, we're going to be the Lewis Carrol live - we are only missing Alice.

Alice Wong was happy - she was about to get married. She did not worry about the ceremony itself - her worry was her wedding ring. The future husband's father himself was supposed to bring it in. The wedding was starting at five sharp, and it was already four fifty seven - but the father was still not showing up.

Finally, she saw him getting through the crowd, to her future husband and giving the box to him. There was something unusual in his look. Maybe that he, oddly enough, had a moon-shaped cookie hooked to his suit - just in place where there'd be a rose. What an odd idea. But the old chap was a strange one. Noone knew how old he was, nor what he was doing. He did not seem to be very rich. But whenever his son started to talk about his plans after the marriage, the father always just hid himself under a friendly but a distant smile and mumbled something about the pension problem.

The ceremony began. The time seemed to fly right past by - even if the mechanical clock on the wall was ticking seemingly with the same speed as before... Here comes the moment... "Do you agree to be my wife ?" - "yes I do" - and he opens the box with the ring....

A gasp rolls over the the room, and dies into a complete silence. The stunning perfection of the crystal in the ring. She catches the reflection of the sun ray. It has the sophistication of hundreds of thousands of man-years of experience. The time stops. She sees an unbelievably fast caleidoscope of images, somewhere in the middle her mind captures the bed with the mechanical clock on it and a mint cookie - same as her now-husband's father had been wearing in his suit. And the clock shows exactly the same time.

It's just after five.

A gift (1/2)

2010-08-16T20:33:00.006+02:00

It's been a grim week, each day filled with the non-stop replays of an early Monday morning. Shaldon looked out the window - the outside was not much better. The sky resembled a skin of a newly-born rat, but even the usually noticeable rats in the basement of the building turned quiet, fearing the storm outside.

How ironic, he thought - there's a village with the same name as mine. And they sell the tickets to stay there for a weekend. If I could sell the tickets to stay inside my mind for a weekend, would anyone bother ? But only during the summer time. All the other time of the year he'd want to be left alone. In the pursuit of that, he went as far as not having a permanent job. The seasonal jobs paid enough to save up and hide for the winter, sleeping a lot more than one would consider healthy, and crawling out only to get a couple of sandwiches in a automatic drive-in.

But now, with this weather, it's time for a nap. He decided. The old-fashioned mechanic clock was the only thing on the headboard shelf, ticking quietly but relentlessly. The heads were open like a beak of some exotic bird, opened to an extreme - it was just after five. He felt sufficiently dizzy, so the only thing he took off were the shoes. It will be just a short nap, after all. The storm should be gone soon.

The air of downtown Singapore was hot and heavy. He liked this feeling - you can almost take it in your hands and shape the little funny figures out of it - much like the clowns do by twisting the long and thin balloons - but without. He made himself an imaginary dog to fly by his side and started to walk towards the shopping mall - figures or not, it was seriously hot, an airco would have been a relief. No, not the rain. He had enough of it. Besides, it would destroy his newly made silent and unseen friend. Instead he took a bus and rode a few stops. Now's the time to get off.

Approaching the entrance of the mall, he saw a girl of some nine years of age, standing near the entrance. She had a backpack, and was holding a plate of Thin Mints in her hands. In any other time of the year he'd wonder what she is doing in Singapore - but today he was not inclined to. He pulled a five dollar note out of the pocket, and bought a box. He did not wait for the change she was trying to give. Nor he wondered why he'd buy them in the first place. He wanted to get a breath of fresh air.

The mall indeed had an airco, but it was one of those filled with myriads of the small outlets that sell the counterfeit electronics, sometimes per pack, sometimes by weight, almost like cookies. It was a great place to be in - the building internally was multi-storied and circular, every little business trying to grab a bigger share of attention by winning the competition for the strangest signage. He felt like he was a visitor in a beehive - and all this activity around him was filling him with the sense of activity. He'd even walk faster - without any reason to hurry at all.

A small and boring outlet caught his attention. The sign said "mind uploading". What the hell, he thought - it's obviously impossible. But the old man with the face he could not read by all means did not think so. The old man's smile was friendly, but at a distance. Several people came in, passed the money to him, and disappeared behind the back door. No one came out. Time to try out those cookies. He took a bite - which turned the cookie into a moon-like shape, when the old man looked at him and said: "It's only 50 dollars, sir. Full refund if not satisfied".

Walk away ? This is much stranger than his imaginary dog, still hanging around, or the scout selling the cookies. No way. He automatically put the cookie into his back pocket, and reached for the wallet. Only a hundred. "Make me two copies" - he muttered. "Sorry sir, no animals allowed and you can not take this cookie box inside" - politely, but firmly answered the old man. Well. If this works out it will be much better than both combined - so Shaldon put the box upright on the ground near the wall, and walked towards the back door with no signs of hesitation.

The old-fashioned mechanic clock was the only thing on the headboard shelf of an empty bed, ticking quietly but relentlessly. The heads were open like a beak of some exotic bird, opened in an attempt to catch something. And indeed - the exotic bird was not alone. It was trying to catch a moon-shaped piece of cookie that was just beside it, flat on the shelf. Someone must have started eating it and forgotten it down here - or they were teasing the exotic bird out of their sheer evilness - we do not know.

But we can say with full certainty that it was just after five.

Somewhat unexpectedly, continued here. Warning(spoiler): blatant violations of laws of life and universe and everything is ahead.

How the facebook could earn extra millions without doing much.

2010-07-22T01:39:00.003+02:00

My friend sent me this: a Cow Clicker game. This is absolutely brilliant parody because it encompasses everything strange about the "social Facebook games".

Now, back to the subject - how can Facebook earn extra millions ? I think in the same way. They can use their "Like" button instead. Humans want to be heard. They especially dislike when some other humans are heard and they are not heard. They consume the friend feed in bursts (unless facebook is the only activity they do).

So, the solution is to limit the number of feedback/click items per hour - better, per the interval of something inconvenient like 5 hours. Sell the Facebucks and allow to use those Facebucks to pay for the ability to express themselves here and now.

Tada! Instant revenue stream!

Of course it'd be evil beyond belief.

And presumably the "likes" that help to build the semantic network are more valuable than the money buying them would bring.

But this could be a fun experiment.

Not talking of the fact that having the Facebucks would be a no-brainer side biz for micropayments.

The tangible value of time

2010-07-21T23:25:00.003+02:00

I've had a thought-provoking experience today: a meeting which got cancelled shortly before. Nothing besides a minor annoyance, but a very valuable trigger to muse on the economic value of time as a function of individual's position in time. Something around the "Time is money" stuff. Lots of terms are in quotes, because this is more of an unsorted dump and less of a coherent essay.

A thought experiment: you ask me to spend 10 minutes now or 10 minutes tomorrow. If I can - I will prefer to spend 10 minutes now. "Now" is "cheaper" for me than "Future" - even though it is in limited supply. "Now" is quickly depreciating - left unused, it becomes a non-redeemable "Past". That's why I try to "sell" the "Now", at a best "price" (usefulness of the activity), so nothing is going wasted into the recycle bin in the corner of Zeus' father cabinet.

So, the most "profitable" strategy for me is to "sell at the edge" - i.e. have little planning at all - the cost of this "just about to be gone" for me is approaching zero. However, since every activity is fitting within the nonzero time interval, this means that the competing activities would meet with the non-determinism - there might be already another thing that I am doing, and the contender, if it is time-critical, will face a "loss".

Seems with the "future", it's really the determinism of my availability that I am "selling" to the "buyers" - in addition to the intrinsic value of "me".
There is a side in this for me too - with the "pipeline of things" the time is allocated, and there is no risk that it goes "unsold" - and if I am able to estimate the "usefulness" of the task for me, then I have a stable "return on investment".

This is similar to a model where someone would pay money "now" to get something in the "future", in an economy with a spiraling inflation. So, "time" is indeed quite similar to money, except the mapping is a bit twisted!

Now, the next question - if the time bears such a strong relationship with money, would it not be interesting to just use the time as a currency ?

So, let's make another thought experiment. Let's assume that the technology allows everyone to print their own currency, and the technology is such, that with every exchange you can see how much of the "units" have been emitted and are in circulation.

"What's the guarantee behind?" - you ask, citing the current money in circulation. With the technology above, the guarantee for the mass of the "personal money" issued by a given individual, is the individual himself. That's when we can exploit the saying "He's worth his weight in gold".

Obviously such a "guarantee" seems scary at first (it looks like some twisted form of slavery), but it is not a problem - to "own" you, someone must be able to continually attract all the "money" you put into emission. Which, unless they produce something you need, is unlikely. Plus, if you are desperate, you can always dilute their "share" by emitting more "money".

This system would some interesting properties.

It would be the economy financially driven by the youth - because the "first emission money" will be the least inflated - therefore of most value. The youth would be able to pay for the education - which would be provided by the elder, who can no longer just "print their money".

With this, it might be that the very core of the society will change (youth being more "naive", if you wish), might just make it for the better. But maybe not.

In any case it would be an interesting experiment.

Now, if that proved to be successful, we can go further, and replace the "personal money" with "personal time".

This interpretation might be even more deterministic - because the lifetime of a person is finite, and is statistically rather well defined.

So you start with your current life expectancy's full of "personal money-time", and get it "vested" in a special way - you have to spend it the moment it is vested or it is lost.

This system would have even more interesting properties. Besides also driven by youth (because they still have more time in front of them), it would also have a very strong inherent motivator for the healthier lifestyle (because then you live longer, which is one of the ways you get richer), and the good for the society as a whole - because the longer the overall society lives, the "richer" it is - which is great for everyone.

This brings a loophole, however - a reproduction can be viewed as a way of "printing money". However, this is maybe not a problem too - as e.g. most of the West have difficulty keeping the birth levels as high as they would want to.

Also, the *way* you spent your past time might also affect the current time's valuation - more educated person would be worth more.

There is one difference between the "personal money" based system and "personal time" based system - the time is something you can spend only once.

But overall, I think it is an interesting system worth modeling. As the technology levels go up - it is not something infeasible to do. It would be a terribly interesting exercise if someone figured out a way to model it.

Not sure how interesting you found this post - but I certainly had a very interesting time pondering. Which means that the today's cancellation was well worth it.

Dear chinese-language spammers

2010-07-13T23:17:00.002+02:00

Please go away. As you can see, it is not working.

A lucky strike

2010-07-10T14:56:00.006+02:00

Lily was walking past this door for the past couple of years. "Martial arts", said the simply advertisement. She could never make herself come in, even though she really wanted to - the door was always firmly closed. Today it was different - she was sure she heard sounds from behind the door. She pushed it open and came in.

It was an old loft, left untouched by the owner over the course of years. About 10 people - both guys and girls were doing the push-ups - the training session began not too long ago.

- "Ichi, ni, san, chi, go, roku, sichi, hachi, ku, ju, ju ichi, ju ni .." - the counting of sensei was hypnotizing her. Suddenly it stopped.
- "..." - he looked at her, keeping the pause.
- "I would like to ..." - she started, but did not have the time to finish.

He turned back to the group - "Yasme!" - and, pointing to a guy who seemed just a bit too tall to be doing any martial arts, and to Lily: "Jiyu! We see what you are worth".

Smith looked at the girl. Strange feeling struck him. Something of an electric shock. It was hard to explain - this was as if he saw these eyes somewhere, sometime long time ago, when he was much younger and could be just overwhelmingly happy, without a shadow of any other thought. He shook his head but the sensation persisted. He was used to knocking the hell out of newcomers that manage to slide into the door, and sensei liked his style - after the test fight, noone was bleeding, but very few of the newcomers returned, and those that did, were very good students afterwards. But this fight promised to be different. He felt he could not kick the hell out of her. Not in the presence of this itchy feeling. He decided to try to discount the itching as a hunger. He did fight a couple of times like that, and, even inconvenient, it was doable. If he can fool himself that it is a feeling of hunger, that is.

Lily was ready. She was doing martial arts in the town where she lived before moving - and kept her in shape by self-training. This club looked like what she wanted. As soon as she can prove herself in the fight with this guy. In other circumstances she might have found him sexy. The only detail in his looks that was disturbing, was a scar on his face - just above the eyebrow. But it did not matter - today she came for martial arts, not for the dating.

"Hadjime!"

Smith was always careful at the start of these trial fights - testing the other party. And today was not an exception. He was defense-only, and only after a minute started attacking. She did know at least something - his kicks were blocked. And boy, did she move well. Contrary to his attempts, the itching did not want to get disguised. He had to look into these eyes - and it did not help. It did not help at all. This has got to stop, I can't fight this, he thought - this was going through his head almost the same split millisecond as he got distracted and missed the upward movement of her fist, that made his head abruptly go up and backward. "Damn, this way it is as stupid as it gets" - was the last thought before the whole world turned completely black.

He did get unconscious once, back in the childhood. It was funny - you are halfway in your body, halfway out, and can wander around the room. He thought it was pretty cool back then, but for some reason he saw all those people in panic, running around and pushing, squeezing his body, so he had to return. And he never had this experience since. Until today.

He saw the group gathering around his corpse. Hold on, he was not dead yet, was he ? So, it was the body that the group was gathering around. But they were worried nonetheless.

If it could, Smith probably would have sighed - the girl that knocked him out was not part of the frenzy. She just stood aside. And he took the advantage of it. To look again in these sparkling eyes and get a gulp of that old feeling of the overwhelming, worldly childish happiness. How can a bodyless piece of ether even look at things, let alone feel ? We will never know. But he did see these sparkling eyes and felt the itching stronger than before. It was a nice sensation. Worth being unconscious...

...Terribly strong headache. This is the first and the only feeling. Overwhelming, just about bearable. No, it's not. Hopefully the head is still one piece. And all these people around. Who are they ? Where am I ? The head hurts. Can't move. Is it really in one piece ? Move the right hand to check. Yes it is. Thinking long sentences hurts. I was hungry, give me some food. Raise the left hand to ask. Can't. There's no left hand. The left part of the body is gone. Damn. Where it is ? It it was a cut, it'd hurt. But only feel the head. This terribly loud whistling noise. Like an airplane. Who brought me to the airfield ? Stop it, please! What was that hunger feeling ?

Ah, these brown eyes. Sparkles. Sparkles got me on fire. That's why no left side - it has burned off. Need to look what's left. Can't move. But the itching is no more. This is good. Means I can skip the dinner. It's getting dark. Finishing way too late today. Mom's gonna be worried.

The city of dim

2010-07-04T23:59:00.007+02:00

Vegas is a striking experience. Right in the airport you are greeded (sp.?) with a lot of slot machines, flashing lights and the feeling of forbidden.

It's a strange feeling - as if you ended up in a backyard of a small apartment in the boringly usual street, just to find a naked champagne-cognac party with violent fireworks.

The hotel where I was staying, Mandalay Bay, like almost everything in Vegas, is gigantic. But, more impressive is what is not seen from the outside. The hotels that stand near each other are interconnected, and this forms a whole separate city underneath them. A casino turns into a shopping mall, which turns into a betting sports bar, which turns into a casino.

Ah, casino. This was the most fascinating of all the experiences - to watch the faces. The faces of those who work there, the faces of those who gamble. You can immediately distinguish the faces of the true tourists - lit with excitement and fueled by a good doze of alcohol, from the other kind of faces.

Usually in their 40-somethings, these faces are so fascinating it hurts. They depict an almost-lost hope, an almost forgotten happiness, they are in a quest. They have almost given up.

This is the expression you might spot in the faces of the countless waitresses, dealers, casino dancers (though calling this kind of entertainment a "display of flesh" might be more appropriate - there was not much dancing in there). You can spot this expression with a fraction of the players too - usually more with those who try their luck with slot machines. These faces have been so air-conditioned down to an average, that they look almost the same, despite being vastly different. A strange form of sameness.

The air-conditioned coolness of the inner hallways is contrasted with the sauna-grade heat on the outside. Dry heat. Here in Brussels, you pay to get that. But of course, not the entire street filled with it, just a small gas chamber, which you can quickly escape. But in Vegas you can't, not unless you dive back into the shelters of the technology. Here you pay to get a cold around you, all the heat is for free.

When I arrived, the faces of the players was first that made me think of a Inferno-under-construction: "they forgot the boiling kettles part!". However, thinking more of it, I realised it is in fact just a next-gen version, and it's already finished and is functioning full steam.

Here's how it works: you arrive, and get the casino tokens proportionally to the amount of sin, and have to start playing. The more you play, the more you get accustomed to the coolness and the gentle light inside, the more you are wary of getting out into the open air where the sun is showering the ground with radiation. But you inevitably lose in the casino, and then are immediately thrown out into the shadowless street. You are still craving to gamble and are sore of your defeat, and heated by the merciless rays of our closest star.

They'd let you work on the dirtiest jobs, and pay with smaller amounts of money, which you could eventually convert into the tokens for the casino and again be admitted to try your luck and enjoy the artificial coolness. But not for long, and soon you are again being beaten up by the merciless ultraviolet. You are thirsty, but there is no water in the streets. If you crawl around, you'll see the fountains, right in the open air. But you can't drink from them. You can only drink in the casino.

So, you get yourself into the next round - go and find the means of getting back into the shelter, to satisfy your needs for water, gambling, and cooler temperature.

Maybe this is how those folks get their eyes of lost hope ? I won't know. The allotted time is coming to its end - the Brussels awaits me back. Maybe there is a next time, and I find out more about these eyes.

The eyes that are sparkling so dim.

The drug

2010-06-24T02:07:00.004+02:00

A patient in pain is brought to the emergency and has been served shot of drugs aimed to relieve the pain. He never tried the drugs before - and the very thought of this gets him into horror - but there is no way - the pain is too strong even for an energetic person he is.

The drugs help to get a relief, and when everyone falls asleep, the patient jumps into the window and runs away. He continues the shots himself and this allows a bearable life - for a very long time - without attending to the disease. Eventually the disease becomes so evident it is not possible to live with it, the patient will die in a year if they do not cure the disease for good - so he returns to the hospital and asks for the cure.

However, soon the patient realises that curing the disease means no helpful shots of dope anymore. Meantime it has become a routine. It increases alertness and adds to the feeling of well-being. Athletic performance is enhanced - so the routine tasks are done much faster. Sometimes the patient feels the temporary anxiety when looking at the mess around - but, it is short. And it's been a long time - the life without the daily shot seems a hazy mirage, which only has one glaring label on it - the pain.

The pain is gone for good now - and so are the injections. But it feels worse than before. The dependency kicks in. Regular depression. No more euphoria that was offsetting the periods of boredom. It's all gray.

The patient needs to relearn the life from the beginning. All the mistakes he did already but forgot - he needs to do again, one by one. It's a long journey, and every day not once or twice he thinks about taking a magic shot again.

But somewhere there is a hidden thought, the joy of the freedom, the absence of the necessity to fit the entire life according to the need for the next shot.

And this thought brings the meaning back to his life, allows him to continue further even through all the hardships.

It really is worth it.

Living without the life-long dependence.

to the most-debated networking feature of the past decade, with compliments for all the pain it alleviated

"Your predecessor"

2010-06-20T02:30:00.002+02:00

It's a painful reminder about the fragility of life and universe and everything.

Because the "predecessor" means the "one who deceased before".

Which means you too shall die.

How much are your dreams worth to you ?

2010-06-19T13:27:00.004+02:00

I've been always saying (mostly empirically, I never made any detailed analysis) that if you are hunting after money, the IT/telecom work is not your cup of tea. Go into financial - they pay, noticeably more.

However, there is a detail to that. Which was put sharp, to the point it hurt, in one of the comments on the HN thread:

"It's cheap to get that salary, too. All you have to give up is your dreams. My friend who earns £150k hates her job, and used to tell me that every time I saw her."

Maybe this is an isolated case, maybe not. However, this triggered in me an idea for an interview question: "How much of a premium would you require to give up all your dreams ?"

And, it can make up for a good discussion over a beer, too.

HTTP parser with ragel

2010-06-16T01:59:00.005+02:00

I've wanted to play with the ragel, and as a target I chose to rewrite a HTTP parser made by Zed for mongrel. Partly because I wanted "my own" license, partly because I do not like the way his code handles the split of the fields: if I understand it right, the entire request needs to fit into the buffer. This is not very pure - if I handled something, I should discard it.

The state machine appeared very easy to make - mostly, take the RFC2616, and start transferring the BNF into ragel format.

Then, define the >action_enter and %action_exit for each of the pair of states that denote the data to capture.

The most interesting part was the state-saving mechanism for the case when the field is split between the multiple buffers, e.g.: "GET / HTTP/1.0\r\nHo", "s", "t: f", "oo", ".", "com\r\n\r\n"

The logic is simple: after each parse cycle, check if we need to stash anything, if we need to - do it. And then when retrieving the field value, check if something was already stashed - if yes, then stash the just-collected data, and retrieve the stashed value.

The nice part is that this voodoo with the stashing does not occur by default - so, in the non-pathological case the memory use will be quite small.

Still a known bug is the fixed allocation for the stash buffer - I did not use realloc yet.

The API is pretty close to Zed's - I like the self-contained nature of his parser and the callback.

For your pleasure, here's the nugget of the ragel code that represents the state machine.


  htp_octet = (any);
  htp_char = (ascii);
  htp_upalpha = (upper);
  htp_loalpha = (lower);
  htp_alpha = (htp_loalpha | htp_upalpha);
  htp_digit = (digit);
  htp_ctl = (cntrl | 127);
  htp_cr = ( 13 );
  htp_lf = ( 10 );
  htp_sp = ( ' ' );
  htp_ht = ( 9 );
  htp_quote = ( '"' );

  htp_crlf = ( htp_cr htp_lf? ); # // Accomodate for unix NLs ?
  # htp_crlf = ( htp_cr htp_lf ); # // do NOT accomodate for unix NLs ?

  htp_lws = ( htp_crlf? (htp_sp | htp_ht)+ ); 

  htp_not_ctl = (htp_octet - htp_ctl);

  htp_text = (htp_not_ctl | htp_lws); # (htp_cr | htp_lf | htp_sp | htp_ht));

  htp_hex = (xdigit);

  htp_tspecials = (
      '(' | ')' | '<' | '>' | '@' |
      ',' | ';' | ':' | '\\' | htp_quote |
      '/' | '[' | ']' | '?' | '=' |
      '{' | '}' | htp_sp | htp_ht);

  htp_token_char = ((htp_char - htp_tspecials) - htp_ctl);
  htp_token = (htp_token_char+);

  # comments not supported yet - they require a sub-machine
  # htp_comment_char = htp_text - ('(' | ')');
  # htp_comment = ( '(' (htp_comment_char+ | htp_comment) ')' );

  htp_quoted_char = (htp_text - '"');
  htp_quoted_string = ( '"' htp_quoted_char* '"' );
  
  htp_quoted_pair = '\\' htp_char;

  htp_http_ver_major = htp_digit+ >mark %http_version_major;
  htp_http_ver_minor = htp_digit+ >mark %http_version_minor;

  htp_http_version = ("HTTP" "/" htp_http_ver_major "." htp_http_ver_minor);

  htp_escape = ('%' htp_hex htp_hex);
  htp_reserved = (';' | '/' | '?' | ':' | '@' | '&' | '=' | '+');
  htp_extra = ('!' | '*' | '\'' | '(' | ')' | ',');
  htp_safe = ('$' | '-' | '_' | '.');
  htp_unsafe = (htp_ctl | htp_sp | htp_quote | '#' | '%' | '<' | '>');
  htp_national = (htp_octet - (htp_alpha | htp_digit | htp_reserved | htp_extra | htp_safe | htp_unsafe));
  
  htp_unreserved = (htp_alpha | htp_digit | htp_safe | htp_extra | htp_national);
  htp_uchar = (htp_unreserved | htp_escape);
  htp_pchar = (htp_uchar | ':' | '@' | '&' | '=' | '+');

  htp_fragment = ( (htp_uchar | htp_reserved)* );
  htp_query = ( (htp_uchar | htp_reserved)* );

  htp_net_loc = ( (htp_pchar | ';' | '?' )* );
  htp_scheme = ( (htp_alpha | htp_digit | '+' | '-' | '.')+ );

  htp_param = ( (htp_pchar | '/')* );
  htp_params = (htp_param (';' htp_param)* ); 

  htp_segment = (htp_pchar*);
  htp_fsegment = (htp_pchar+);
  htp_path = (htp_fsegment ('/' htp_fsegment)*);

  htp_rel_path = ( htp_path? (';' htp_params)? ('?' htp_query)? );
  htp_abs_path = ('/' htp_rel_path);
  htp_net_path = ("//" htp_net_loc htp_abs_path?);

  htp_relative_uri = (htp_net_path | htp_abs_path | htp_rel_path);
  htp_absolute_uri = (htp_scheme ':' (htp_uchar | htp_reserved)*);
  htp_uri = ((htp_absolute_uri | htp_relative_uri) ('#' htp_fragment)?);

  htp_host = (htp_alpha);
  htp_port = (htp_digit+);

  htp_http_url = ("http://" htp_host (':' htp_port)? (htp_abs_path)?);
  
  htp_method = ("OPTIONS" | "GET" | "HEAD" | "POST" | "PUT" | "DELETE") >mark %http_method;


  htp_request_uri = ('*' | htp_absolute_uri | htp_abs_path) >mark %http_uri;

  htp_request_line = (htp_method htp_sp htp_request_uri htp_sp htp_http_version htp_crlf);

  htp_header_name = htp_token+ >mark %http_header_name;
  # fixme.
  htp_header_value_char = htp_octet - htp_cr - htp_lf;
  htp_header_value = htp_header_value_char+ >mark_value %http_header_value;

  htp_some_header = (htp_header_name ':' htp_sp* htp_header_value htp_crlf);
  htp_last_crlf = htp_crlf; #  >{ printf("Last CRLF!\n"); eof = pe; };
  htp_request = htp_request_line (htp_some_header)* htp_last_crlf;

  main := (htp_request) @{ parser->done = 1; };

Snooping on search engines with hex tricks.

2010-06-06T22:07:00.006+02:00

The big hex number you see on the title of the blog is there for a reason. It's an experiment on the search engines out there.

What's this hex number ? It's a SHA-1 hash of the string "Andrew Yourtchenko" - this gives a nice token that is unique in the whole world (because noone else has yet come up with the idea to hash my name and put the result online). This gives some very entertaining results if you try to search for this number in the search engines.

Google: 2 results after folding the duplicates - main page + one post. Expanding them gives 25 results. Cool.

Bing: 0 results. Boo. I could not find any page on this blog using it. 2xBoo.

Yahoo: 2 results - main page and one post, different from the post that Google shows.

duckduckgo: One hit to the same post as Yahoo shows, not to the main page.

altavista: 2 results, same as Yahoo.

ask.com: 2 results, same as Google

cuil.com: 0 results.

baidu: 0 results. Somehow unsurprising at all

kosmix: 2 results in google web search (same as google), 4 results in the google blog search, 0 results in yahoo web search. Entertaing how they disagree with Yahoo.

yandex.ru: 0 results.

yebol.com: 2 results same as yahoo + 1 site result. Very interesting form of presentation. I got to play with this one for daily searches, even though it does not appear to be too fast compared to google.com.

Conclusions:

I don't have many inbound links here - probably about two :)

google.com has a better reach towards the "long tail" (EDIT: 'long tail' in this case being blogger.com - would be interesting to test e.g. typepad.com or other blog sites)

There are much more than 127 billion pages on the web.

yebol.com is a new toy to play with

Though some of the above are fairly obvious, not bad of a result for a single SHA-1 hash, I think. Would be interesting what results this method gives for more popular blogs/websites.

The debatable point in this method is - to which extent do the search engines discriminate the "oddball hex stuff" vs. the "common words". From all I know about the search engines, they should not - vice versa, it's the stopwords (too frequent to be useful ones) that are usually filtered.

Node.js vs. lua microbenchmarking

2010-05-27T00:08:00.006+02:00

Node.js surely seems to be a pretty interesting contender. At least according to the microbenchmarks (ab -c 5 -n 100000):

Node.js:



Document Path:          /
Document Length:        12 bytes

Concurrency Level:      5
Time taken for tests:   17.526 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      7600000 bytes
HTML transferred:       1200000 bytes
Requests per second:    5705.94 [#/sec] (mean)
Time per request:       0.876 [ms] (mean)
Time per request:       0.175 [ms] (mean, across all concurrent requests)
Transfer rate:          423.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.5      1      14
Waiting:        0    1   0.5      1      14
Total:          0    1   0.5      1      14

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      2
  98%      2
  99%      2
 100%     14 (longest request)

And here goes the Lua result:


Document Path:          /
Document Length:        11 bytes

Concurrency Level:      5
Time taken for tests:   6.666 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      34600000 bytes
HTML transferred:       1100000 bytes
Requests per second:    15001.08 [#/sec] (mean)
Time per request:       0.333 [ms] (mean)
Time per request:       0.067 [ms] (mean, across all concurrent requests)
Transfer rate:          5068.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.1      0       4
Waiting:        0    0   0.1      0       4
Total:          0    0   0.1      0       5

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      1
  98%      1
  99%      1
 100%      5 (longest request)

The node.js code was as follows:



var sys = require('sys'),
  http = require('http');

http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.end('Hello World\n');
}).listen(8000);
sys.puts('Server running at http://127.0.0.1:8000/');

And in its turn, the Lua script used to run the server is here:
http://lua-users.org/lists/lua-l/2002-04/msg00180.html - slightly tweaked to make it run.

This is not anunfair comparison, since node.js runs a heavily optimized server, and the lua runs an off-the-shelp example, and yet outperforms Node.js almost 3x.

Though, more complicated code might result mostly in slowing down - so, the ballpark should be about right.

I will update this post if I get anything more complex on the lua side.

EDIT: in fact, a little bit of reverse engineering shows that Node.js code is also somewhat naive: "FOO /\n\n" works perfectly as a "request" for the above app. So the tests are more or less adequately show the status quo. Roughly 3x speed difference, that is.

EDIT2: http://factor-language.blogspot.com/2010/05/comparing-factors-performance-against.html has some more interesting stats.

prosody one-shot makefile on 64-bit arch

2010-05-11T03:26:00.002+02:00

Today I tested my prosody-one-shot-makefile on my Xen VPS which happens to sit atop the 64-bit CPU.

After a few tweaks it now compiles and makes a self-contained copy of prosody on both 32-bit and 64-bit linuxes.

Interesting how the XMPP server immediately got got integrated when I launched it, and I could communicate with a demo account from my other XMPP accounts. Admittedly, I did not expect such a plug'n'play.

Haute couture 2010: Privacy Your latest luxury item to brag about

2010-05-09T15:07:00.023+02:00

The fate of chronic privacy deprivation that we are facing worries quite a few, and spurs a lot of heated discussions "we will do it better!".

Those inevitably bring up a question: "But Facebook is free, who is going to pay you ?"

So here's my take on it. Similar to how you have to be "rich" to control your weight, you will have to be "rich" to control your privacy.

So, the new services would have free accounts that are all-public (which is how you have to treat your info now, anyway - just extrapolate).

The paid accounts (something in the ballpark of $9/month) would allow making the information private. Or, one could go a bit more creative and sell privacy apiece - the more you pay, the more you can hide, and you pay only for the stuff you want to hide - so the distinction between the paid and free accounts would blur.

All of the accounts would allow the export of data into other services, should "this" social provider go off-rail.

The data formats as well as the interaction between the different providers would be standardized, so it would not really matter whether you and your friends are on the same provider or not.

This may not seem pretty, but I think this approach would be better than changing the smallprint in the ToS overtime.

Because there's no free lunch. It's just about charging in different currencies.

Now, I am curious (and you probably are curious too) about the correctness of the assumptions above, so here's a poll for you.

Thoughts about NAT

2010-04-29T22:10:00.004+02:00

Reading the NANOG thread about "The alleged evils of NAT", I wanted to come up with a semantic parallel, for the fun of it. Here it is:

NAT is like "tax optimization".

It's not illegal. But it may feel wrong.

It allows to represent things differently than you would think they are.

If you go too far with it, you end up in a real mess.

It brings very tangible benefits for those practicing it.

And those who don't - have to pay up on the communal goods for those who do.

Henceforth, they always argue with each other when they meet.

It creates a new profession dedicated to it.

----

As for my personal opinion, I'll just quote an anecdote: "it said 'celebrate', not 'celibate'".

The taste of lua as a web app language

2010-03-29T03:06:00.003+02:00

This is a snippet of the web app that is written in lua, using my experimental framework:

Not RoR but is reasonably convenient, though would be nice to get rid of some of the braces/parens.

Solution for the "paid content" and online newspapers.

2010-03-23T22:11:00.000+01:00

It's all here: http://www.holovaty.com/writing/fundamental-change/ - the newspapers just need to keep providing the "human-readable" content for free, and charge for the structured/machine-readable data, that might be possible to further aggregate/analyse.

How many of the "paid members" will that generate ? That's a good question - but I'd wager that people's curiousity could be a good help to this. Think of this more somewhat like a "click & pay here for more info", but rather than content-wise, you do that semantically.

Imagine this to be similar to a progressive JPEG (or different formats of the picture) - lower-resolution picture is free, and you need to pay a modest amount for the hi-res one.

Those who are interested by the core fact - will get only the core fact for free. Those who are interested in full glory - will use micropayment to get there. ($0.1 might be the right amount)

Of course, this implies working micropayments...

One-shot makefile for Prosody updated + on github

2010-03-22T02:34:00.003+01:00

I've updated the makefile to compile 0.6.1, and rather than putting it onto the blog posts, it's now on github:

http://github.com/ayourtch/prosody-makefile

Lua quine

2010-03-22T01:00:00.004+01:00

For some reason lately I've seen an anomalous number of mention of quines, and figured it'd be time to try it out myself (I read about this question back in 1990s, but for some reason never got myself at solving it. here's the result in Lua.


function p(s)
  print(s)
  print("\ns = " .. string.format("%q", s) .. "\np(s)")
end

s = "function p(s)\n  print(s)\n  print(\"\\ns = \" .. string.format(\"%q\", s) .. \"\\np(s)\")\nend"
p(s)

And, just in case that blogger mucks up my quoting, here's a base64-encoded version:


ZnVuY3Rpb24gcChzKQogIHByaW50KHMpCiAgcHJpbnQoIlxucyA9ICIgLi4gc3RyaW5nLmZvcm1h
dCgiJXEiLCBzKSAuLiAiXG5wKHMpIikKZW5kCgpzID0gImZ1bmN0aW9uIHAocylcbiAgcHJpbnQo
cylcbiAgcHJpbnQoXCJcXG5zID0gXCIgLi4gc3RyaW5nLmZvcm1hdChcIiVxXCIsIHMpIC4uIFwi
XFxucChzKVwiKVxuZW5kIgpwKHMpCg==

FOSDEM2010: now we might call ourselves "NOC"

2010-02-12T02:00:00.004+01:00

A belated 2010 follow-up to the last year's experience. This year, as you've probably seen in the news, was quite fun. Thanks to the help from BELNET (and of course the Fosdem folks who did all the discussions), and the two rolls of fiber, we had 1gbps uplink, and 1gbps link between H and AW.

The most stressing moment was when on Friday at 5pm we discovered that 5 out of 6 strands of fiber that were planned to connect us to the internet, were not working. Plan B (which plan B? we already tested everything! it worked!) was pretty weak. So, applying some logic and with a bit of luck, the problem was solved by resplicing one of the ends.

After that, a few hours trying to build the correct OS platform to install the management software, which supposedly could magically convert some of the APs that we had into the correct mode, rather than us doing manually. At something like 11pm we realised manual was faster - but we didn't have the power supplies with us, but since the security folks wanted to close the doors, we grabbed a few access-points and went to the office to fix them up.

The process of converting access-points from standalone into management mode is dead-simple - hold the button while the AP is powercycled, then power it on with the button is pressed, and wait till the two out of three LEDs on the top turn red. Then release the button, and the AP will grab the predefined address 10.0.0.1 and will start the TFTP download of the image with the default name. All good except the TFTP requests would be sent to broadcast address. It took me about 7 access-points (as measured by Jerome converting them with some naive TFTP server under windows), till I have figured out how to get the atftpd to react: "ifconfig eth0 promisc". Really-really weird. I thought tcpdump - which was running in parallel - was already putting the interface into promiscuous mode.

Anyway, once I found the trick, the rest of the conversion was trivial, so when I started up at 5am, the remaining of the APs were trivial and fast to convert.

This year the access-points we used had a bit more juice - both on 2.4Ghz and 5Ghz - so we managed to kick off some interference to the wireless mic in Jansson. And this year we used the feature to "kill" the rogue APs - which were apparently not so low in number. Strange. I thought people would come to the conference to listen, rather than to annoy others. Nonetheless, killing the phy seems to work - at least this year I saw much less of "strange" SSIDs. Next year we should enable the triangulation, to go and ask personally :)

The kicker this year was that the network was full-dualstack. And, I cooked the bind9 to point to the SixXS recursive servers (thanks guys for the service!), DNS queries going over the SixXS tunnel. A bit of a chewing-gum and matches, but worked fine over the period of two days - and looking at the SixXS traffic graph, it felt like we even made some bump for the gateway by our queries - although this is most probably wishful thinking.

I don't know how many people noticed their google.com traffic was IPv6-based (being geek conference and such). But if they did not - even better! That means - IPv6 works!

All in all, it seems like everyone was quite happy - the visitors (thanks for being there!), the Fosdem staff (it was fun, guys! Let's do it next year!), and ourselves.

On to FOSDEM2011!

VMWare server 2 "empty" screen with "Loading..." in the title

2009-12-07T12:04:00.007+01:00

I've installed the VMWare server, and as many, hit the issue of an empty screen.

It appears to happen upon the subsequent logins. Obviously "empty screen" can be caused by a lot of errors, but in my case it seems to be some SSL-related issue, in /var/log/vmware/hostd.log I saw:

[2009-12-07 11:58:12.150 'Proxysvc' 1093462352 warning] SSL Handshake on client connection failed: SSL Exception:
[2009-12-07 11:58:12.316 'Proxysvc Req00102' 1081919824 warning] PendingServerStrm: client write-completion error: SSL Exception: error:00000001:lib(0):func(0):reason(1)
[2009-12-07 11:58:12.573 'Proxysvc Req00101' 1081919824 warning] PendingServerStrm: client write-completion error: SSL Exception: error:00000001:lib(0):func(0):reason(1)

What seems to help - is holding the Control key and hitting the "reload" button when I am on the page that does not show anything.

Then the page reloads fully and I get the login prompt to login into the VM.

Hope this is useful to someone.

I later tried on the ff-3.5, and there Ctrl-Shift-R for reload, apparently did the trick as well.

Recording sound and video from a webcam

2009-11-03T02:09:00.002+01:00

I've found a lot of questions on the internet on how to record the sound and video at the same time from the webcam. Here's the magic incantation that worked well for me (it assumes you do not have any video capture devices besides the webcam and already have one ALSA soundcard besides the mic of the webcam - if not, adjust accordingly). This is a single long line:

mencoder tv:// -tv driver=v4l:width=320:height=240:device=/dev/video0:fps=30:forceaudio:alsa:adevice=hw.1 -ovc lavc threads=2 -ffourcc xvid -oac mp3lame -lameopts cbr=128 -o recording.avi

EDIT: with a purchase of the Sony Playstation Eye camera, the above does not work all that well at all - at the high rate my mencoder misses too many frames. Something I found that is a bit better:

gst-launch v4l2src ! 'video/x-raw-yuv,width=640,height=480,framerate=(fraction)25/1' ! mux. alsasrc ! audioconvert ! 'audio/x-raw-int,rate=44100,channels=2' ! mux. avimux name=mux ! filesink location=test.avi

This lacks the compression, and the voice is a bit dragging compared to video - but gives more or less the 30fps that the webcam sends.

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

2009-06-14T00:58:00.002+02:00

A fairly odd message, especially given that 'which cpp' shows another location.

The solution is:

sudo aptitude install build-essential

F*ctor - a 0-dimensional Turing tarpit.

2009-05-28T01:05:00.005+02:00

Following a post of one of my friends to the http://manage.yore.ma/eso/, I felt a bit inspired,
so I wanted to write up something fun. So, here's the experiment - an imaginary language which holds the entire program and data memory in a single variable!

The F*ctor machine has only other elements are a single instruction, and a single variable which holds the Program, and Data, and the machine state, in just a single natural number X. The "single instruction", however, is a bit difficult to describe, so I will attempt to describe it in terms of smaller "micro-operations" - they do not really change nor are visible to the user, they are here mostly for the sake of explanation. The temp variable in the pseudocode is just for the sake of ease of comprehension.

The machine can really efficiently calculate Prime(N) (being the Nth prime, that is Prime(0) == 2; Prime(1) == 3, Prime(2) == 5, etc.)

Also it can count how many times is the given number divisible by some prime: CountPrime(V, P); CountPrime(2,2) == 1, CountPrime(9,3) == 2; CountPrime(13,3) == 0.

Finally, the third operation is the destructive divide: DivideBy(V, P) - try to integer-divide V in-place by a prime P ith no remainder, and return True if we were successful, and False if no remainder-free division is possible.

The remaining operations that comprise the single cycle are some trivial arithmetics.

the single cycle of execution:


one_cycle(X) {
  local CurrentP = CountPrime(X, 3);
  while(DivideBy(&X, 2)) {
    if(not DivideBy(&X, Prime(CountPrime(X, Prime(CurrentP))))) {
      return X * 3 * 3;
    }
  }
  return X * 2^CountPrime(X, Prime(CurrentP)) * 3;
}

So running the program consists of repeatedly executed "X = one_cycle(X)".

If we take a closer look, then this is nothing more than a folded version of Reverse-subtract and skip if borrow single-instruction machine, with "memory contents" at location N being the number of times the X is divisible by Nth prime number. As it is visible, the accumulator is memory-mapped at 0 (see these divisions/multiplications by 2?) and the program counter is memory-mapped at 1 (see the multiplications either by 3 or by 9 ?)

So, unless I made a bug somewhere in the logic, we have a Turing-complete absract machine squashed into a single variable.

A "nod" X11 linux interface

2009-05-16T03:57:00.002+02:00

Today I wanted to spend the evening doing something different and fun, and came up with an idea to play around with some of the opencv's functions. Well, did not play that much - just slightly tweaked the facedetect.c, to get what I called a "nod interface".

When you compile and launch the program, it pops up the screen with the video. If all goes well and you are sitting in front of your camera, you should see a circle around your face. If you do not see it - move closer and/or get a smoother background.

The few lines of pretty silly code that I've added does the following: calculates the two averages of the last N points of the center of the detected face object - "fast" (N=4) and "smooth" (N=8).

If you don't move your head the delta between these will be almost 0 - it gets forced down to zero by a simple check against the threshold.

If you nod - then obviously the "fast" average moves down faster than the "smooth" one - so if you nod energetic enough, the computer will notice it and will send a few "arrow down" synthetic events via the XTest X11 extension. If you manage to "nod up" (unnatural, but somewhat doable) - then it will send the equivalent number of "arrow up" events.

The counter-movement (returning the head back after the nod) is compensated by another simple logic - it's needed to have at least 4 "zero-delta" cycles in order to let the detected delta through.

The end result is that with some amount of luck you might manage to read things from your laptop hands-free. Should be good for reading the /. threads :-)

this file is the tweaked facedetect.c - the long line to feed to the compiler is in the comments in the beginning.

Depending on the speed of your computer, you might need to adjust some of the thresholds.

Enjoy!

Hercules DJ Console RMX linux midi drivers

2009-05-07T03:12:00.002+02:00

Yay! Open source and all. Get the MIDI out of the RMX.

Grab them here

A very compact renderer

2009-05-02T06:09:00.003+02:00

Been a while since I babbled something here.

So to ensure I'm still alive and kicking - here's something neat: a renderer in under 600 lines of code.

Minilight has a bunch of implementations in various languages.

A very nice toy.

Linear interpolation code for smoothing out the Z-movement

2009-04-04T21:15:00.003+02:00

Today I've got a dead-simple but pretty cute piece of code in. It has to do with smooth avatar movement when walking. The map of heights is stored with the resolution of 1 meter, while in the viewer it's smoothed out. This creates a "staircase"-like effect when walking on the slope.

The simple solution which I went to is to do the linear interpolation, with the following
approach. First we look at the heightmap "from the top" - just looking at its X and Y coordinates. And logically we split the X-Y space into a bunch of 1-sized squares. So we always
have our x,y point for which we are calculating, inside one of these small tiles. So we can zoom into this tile and take further look:


    (x0,y0)  (x0+1, y0)
     |      /
     v     v
     +-----+---> X
     |\ 1 /|
     | \ / |
     |4 + 2|
     | / \ |
     |/ 3 \|
     +-----+ <-- (x0+1, y0+1)
     |
     v
     Y

in this picture - the corners of the square are the integer X, Y values which have the data.

The point in the center is the "virtual" point which is used to split the square into four triangles - it's x,y coordinates are simply X0+0.5, Y0+0.5.

Then we can pretty simply find which of the triangles holds our "interesting" point - now it's all dead simple - find the equation of the plane that passes through the three points the coordinates of which we know (the Z coordinate of the "center" point is the average of the Z coordinates of all 4 corners) - and then simply substitute the X, Y into the equation, to get Z straight away.

I don't have walking animations yet - for now I simply set the Z coordinate of the avatar, which gets its butt approximately at ground level :-) While running around, if one would measure by the proportions to the real world, the staircase effect is reduced to almost unnoticeable ~10cm if we relate to the real-world body proportions. And I hope that once I get the avatar's butt out of the ground, it should be even better visually.

Now, the obvious question - why 4 triangles, instead of 2 ? Because if you take the following layout:


0---*
|   |
*---0

and the "*" is "high" an "0" is "low" - depending on the choice to cut this X-Y box, you get either a pit, or a spike. 4 triangles handle this case noticeably better.

Cosimus now with avatar shapes and clothing

2009-03-30T02:40:00.004+02:00

Today I took a brute-force approach to solving the missing avatar clothing problem that
I was chasing for a few days - simply by creating my own set of the default clothing and wearables.
It's just the standard set of items which are uploaded by the viewer in the case it does not find them - so for the secondlifers, it's Ruth in white clothing :)

Alongside, I've put a very simple moving logic - only forward, in the direction that the avatar faces. This required some rudimentary support for something the called "scene" in opensim - or at least my interpretation of it. For now, I am taking the assumption "Agent == Avatar", it probably will change sometime in the future. However, I already put the "clientstack-independent scene" atop the SMV one - to represent the qualities that are (or should be) fundamental to any avatar system - X, Y, Z position, AgentID - I'll use the UUIDs for representing the "entity" for now, and the "label" - some text that identifies the avatar as seen in-world.

The rest of the platform-dependent details go into a table called "Blob" which is at the moment SMV-specific data about the avatar (like, local object ID, which is needed for the viewer, as each avatar has its own local id to be drawn correctly).

Possibly exposing a "local avatar id" might have been a better move for the platform-independent record, and having the clientstack-specific lookup tables.. But I will see in the future what is better. In any case, each clientstack registers its own event handler functions, which are called per-avatar in case the need arises (this indeed raises some additional possibilities for optimization in the future).

Alongside the minor infra mods, the loginserver config is now in the config file (main.lua), as well as the serializable state to facilitate trivial no-outage restarts is inside a separate file.

Interestingly enough, there seems to be something odd with the object update packet - the omvviewer tends to hard-hang the laptop - I am not sure which part is causing that. And somehow I managed to get some green text atop the avatar at one occasion, which was funny.

As the vacation time is over now for me, most definitely the speed of the development will suffer - I'll only have some late nights, which are also good to be used for sleep, but I am pretty happy with the progress so far.

Cosimus - a little toolkit for metaverse experiments

2009-03-29T13:42:00.004+02:00

The past two weeks I've been on holiday - so this translated for doing various things that I did not have time to do before. One of those things was completing the chunk of coding needed to at least start with my little metaverse experiment.

The result is at http://github.com/ayourtch/cosimus/tree/master - and will suffer the periodic commits from now on, depending on my so called "spare time" :)

It's a start of the simulator code for the 3d simulator, for now I've started with the SecondLife(tm) protocol, as it's something I had in the beginning of the works.

For now it is a custom Lua interpreter, with a few libraries linked in and initialized by default, and the application code in prototype phase is being written in Lua. This gives the much faster speed of the development compared to C (a long while ago, while checking the speed of development on Perl vs. C I found about 10x difference - in favour of Perl, of course. I did not do any personal measurements now, but from a gut feeling it should be about the same). Much later, as needed, I'll push the performance-critical pieces of the code downwards to C.

For now the thing does not do all that much - these few days I've been mostly dealing with the inventory/asset code, appears there are some magic tweaks still needed - from the behaviour it seems like I am triggering some data corruptions in the client inventory code..

From the cool features that are already there - the inventory/asset library loading is done from zipfiles, which means easier handling of them for the consumer. Another fun feature - or rather design choice - is that all the communication via the components that is possible to be networked in the distributed setup, will be also networked in the standalone setup, with the simple servers doing the job from within the executable itself. So it is "batteries included", and a single code path to debug.

...Now, back to coding :-)

Compiling zettair on ubuntu

2009-03-11T01:28:00.002+01:00

The first compile throws a bunch of errors, due to missing zlib, among them:

‘Z_NO_FLUSH’ undeclared

solution - sudo apt-get install zlib1g-dev, then rerun configure script, and all compiles fine.

Zettair is back alive

2009-03-10T23:46:00.003+01:00

Hooray! Zettair folks down in Oz have revived the development. "in-development" version now online.

Let's see what we have (with a brutal diff -rc across the directories :-) Disclaimer: this is not necessarily a complete list, at least in the autoconf part - the diff was fairly boring and long.
Alternatively, I might have missed it in the set of whitelist changes or did not get a clue it was important...:)

configure script got what looks a 64-bit "kfreebsd". Probably inherited with a newer version of autotools, but an interesting beast nonetheless, if I read it well, it is here. GNU userland running on top of freebsd kernel.. The other tweaks belong to dragonflybsd. I heard about Hammer filesystem, but haven't tried the OS yet. Might be interesting. All depends on the H/w support.

index.h:

option INDEX_NEW_VOCAB is gone, INDEX_NEW_PARSEBUF (dictate how large the postings hashtable is) appeared.

INDEX_NEW_NO_OFFSETS - build the index without offsets. Probably not very interesting for me...

INDEX_NEW_QTHREADS - how many query threads to allow. Now I wonder if this allows for simultaneous query and indexing - that'd be sweet. My experiments with the previous version running separate processes were not terribly successful. With threads indeed it might be a different story.

some other similar parameter changes to index_load (as opposed to index_new)

index_cleanup is gone.

INDEX_SEARCH_DYNAMIC_RANK - provide the text describing the rank to be used as well as the rank parameters. Interesting, probably a bit too advanced for my needs.

autogenerated code for the metrics is obviously now parametrized.

mutexes for various pieces (For multithreading)

vocab_vector structure is simplified (looks like thanks to getting rid of multiple files)

a lot of multithreaded-oriented code

a hacko-fix in makeindex_append_docno to get around the fact that zettair can't handle docno-s bigger than a pagesize

vocabury operations seem to be simplified - previously the memory management was more sophisticated.

This completes the list - again, very probably I missed something important - if so, drop me a note in the comments.

LLSD scanning with ragel-made parser.

2009-03-09T22:55:00.003+01:00

I've written a while ago about the Ragel.

Today evening (literally in about 4 hours), I tried it and cooked a semi-functional parser for LLSD XML serialization. Yes, the code below was cranked from scratch in just a few hours :)

What's nice is that it is can parse the data by chunks of practically any size, so could be used to grab the data over a slower TCP connection. And - the base64 decoding is included! Things that would need to be done in order to make it usable include proper error handling and a better approach to handle possibly large data - right now it is kind of dumb...

Here's the result of running it over the sample XML from the draft:


$ ./a.out `cat ../sample.llsd`
Array start
Integer: '42'
UUID: '6bad258e06f04a87a659493117c9c162'
Map start
String: 'cold'
Map member with key 'hot':'cold'
Undef
Map member with key 'higgs_boson_rest_mass':''
URI: 'https://example.org/r/6bad258e-06f0-4a87-a659-493117c9c162'
Map member with key 'info_page':'https://example.org/r/6bad258e-06f0-4a87-a659-493117c9c162'
Date: '2008-10-13T19:00.00Z'
Map member with key 'status_report_due_by':'2008-10-13T19:00.00Z'
Map end
Array end
result = 1

And here goes the .rl code in case you're curious (maybe you need to "view source" in case it gets garbled):


#include 
#include 
#include 

typedef struct llsd_scanner {
  char *accum;
  int accum_size;
  int accum_index;

  char *key_accum;
  int key_accum_size;
  int key_accum_index;

  unsigned char *bin_accum;
  int bin_accum_size;
  int bin_accum_index;
  unsigned char b64chars[4];

  // ragel variables
  int *stack;
  int stack_size;
  int top;
  int act;
  int cs;
  unsigned char *ts, *te;
  
} llsd_scanner_t;



static const char cd64[]="|$$$}rstuvwxyz{$$$$$$$>?@ABCDEFGHIJKLMNOPQRSTUVW$$$$$$XYZ[\\]^_`abcdefghijklmnopq";

static void llsd_scanner_putb64char(llsd_scanner_t *scanner, int index, unsigned char v)
{
  v = (unsigned char) ((v < 43 || v > 122) ? 0 : cd64[ v - 43 ]);
  if( v ) {
    v = (unsigned char) ((v == '$') ? 0 : v - 61);
  }
  if (v) {
    scanner->b64chars[index] = (unsigned char) (v - 1);
  } else {
    scanner->b64chars[index] = v;
  }
}

static void b64decodeblock( unsigned char in[4], unsigned char out[3] )
{   
    out[ 0 ] = (unsigned char ) (in[0] << 2 | in[1] >> 4);
    out[ 1 ] = (unsigned char ) (in[1] << 4 | in[2] >> 2);
    out[ 2 ] = (unsigned char ) (((in[2] << 6) & 0xc0) | in[3]);
}


static int append_binchars(llsd_scanner_t *scanner)
{
  if (scanner->bin_accum_index + 4 < scanner->bin_accum_size) {
    b64decodeblock(scanner->b64chars, &scanner->bin_accum[scanner->bin_accum_index]);
    scanner->bin_accum_index += 3;
    return -1;
  } else {
    printf("Bin accumulator overflow!");
    return 0;
  }
}

static void clear_accum(char *accum, int *accum_index)
{
  *accum_index = 0;
}

static int append_accum(char *accum, int accum_size, int *accum_index, unsigned char c)
{
  if (*accum_index < accum_size-1) {
    accum[(*accum_index)++] = c;
    return -1;
  } else {
    return 0;
  }
}

static void terminate_accum(char *accum, int *accum_index)
{
  accum[*accum_index] = 0;
}




%%{

        machine LLSDScanner;

        access scanner->;
       


        action terminate_accum { 
            terminate_accum(scanner->accum, &scanner->accum_index); 
            printf("Accumulated: '%s'\n", scanner->accum); 
        }
        action clear_accum { 
            clear_accum(scanner->accum, &scanner->accum_index); 
            clear_accum(scanner->bin_accum, &scanner->bin_accum_index); 
        }
        action append_accum { 
            append_accum(scanner->accum, scanner->accum_size, &(scanner->accum_index), fc);
        }

        action print_key_accum { 
            terminate_accum(scanner->key_accum, &scanner->key_accum_index); 
            // printf("Key Accumulated: '%s'\n", scanner->key_accum); 
        }

        action clear_key_accum { 
            clear_accum(scanner->key_accum, &scanner->key_accum_index); 
        }
        action append_key_accum { 
            append_accum(scanner->key_accum, scanner->key_accum_size, &scanner->key_accum_index, fc); 
        }

        action convert_string { 
            terminate_accum(scanner->accum, &scanner->accum_index);
            printf("String: '%s'\n", scanner->accum); 
        }
        action convert_uuid { 
            terminate_accum(scanner->accum, &scanner->accum_index);
            printf("UUID: '%s'\n", scanner->accum); 
        }
        action convert_uri { 
            terminate_accum(scanner->accum, &scanner->accum_index);
            printf("URI: '%s'\n", scanner->accum); 
        }
        action convert_base64 { 
            terminate_accum(scanner->bin_accum, &(scanner->bin_accum_index)); 
            printf("Binary Accumulated: '%s'\n", scanner->bin_accum); 
        }
        action convert_date { 
            terminate_accum(scanner->accum, &(scanner->accum_index)); 
            printf("Date: '%s'\n", scanner->accum); 
        }
        action convert_integer { 
            terminate_accum(scanner->accum, &(scanner->accum_index)); 
            printf("Integer: '%s'\n", scanner->accum); 
        }
        action convert_real { 
            terminate_accum(scanner->accum, &(scanner->accum_index)); 
            printf("Real: '%s'\n", scanner->accum); 
        }
        action convert_undef { 
            terminate_accum(scanner->accum, &(scanner->accum_index)); 
            strcpy(scanner->accum, "");
            printf("Undef\n"); 
        }

        action assign_map_member { 
            printf("Map member with key '%s':'%s'\n", scanner->key_accum, scanner->accum); 
        }
    
        one_or_two = /[12]/ @append_accum;
        zero_or_one = /[01]/ @append_accum;
        zero_to_nine = /[0-9]/ @append_accum;
        zero_to_two = /[0-2]/ @append_accum;
        zero_to_three = /[0-3]/ @append_accum;
        zero_to_five = /[0-5]/ @append_accum;
        dot = '.' @append_accum;
        dash = '-' @append_accum;
        colon = ':' @append_accum;
        dot_or_colon = /[.:]/ @append_accum;
        letter_T = 'T' @append_accum;
        letter_Z = 'Z' @append_accum;

        date_fullyear = one_or_two . zero_to_nine . zero_to_nine . zero_to_nine;
        date_month = zero_or_one . zero_to_nine;
        date_mday = zero_to_three . zero_to_nine;
        time_hour = zero_to_two . zero_to_nine;
        time_minute = zero_to_five . zero_to_nine;
        time_sec = zero_to_five . zero_to_nine;
        time_secfrac = dot . zero_to_nine;
        full_date = date_fullyear . dash . date_month . dash . date_mday;
        partial_time = ( time_hour . colon . time_minute . colon . time_sec . time_secfrac |  time_hour . colon . time_minute . dot_or_colon . time_sec );
        date_time = full_date . letter_T . partial_time . letter_Z;

        maybe_space = space*;

        some_string_char = /[^<]/ @append_accum;
        some_string = some_string_char*;

        some_key_string_char = /[^<]/ @append_key_accum;
        some_key_string = some_key_string_char*;

        # base64 parsing. Base64 characters come in series of 4

        base64_char = [ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+=/];
        action base64_one { llsd_scanner_putb64char(scanner, 0, fc); }
        action base64_two { llsd_scanner_putb64char(scanner, 1, fc); }
        action base64_three { llsd_scanner_putb64char(scanner, 2, fc); }
        action base64_four { llsd_scanner_putb64char(scanner, 3, fc); }
        action base64_four_chars { append_binchars(scanner); }

        base64_first_char = base64_char @base64_one;
        base64_second_char = base64_char @base64_two;
        base64_third_char = base64_char @base64_three;
        base64_fourth_char = base64_char @base64_four;

        base64_four_chars = maybe_space . base64_first_char . maybe_space . base64_second_char . maybe_space . 
                            base64_third_char . maybe_space . base64_fourth_char @base64_four_chars;

        base64_string = base64_four_chars+;

        tag_start = maybe_space . '<';
        tag_end_nospace = '        tag_end = maybe_space . tag_end_nospace;

        llsd_tag = /llsd>/i;

        undef_tag = /undef\/>/i;
        boolean_tag = /boolean>/i;
        integer_tag = /integer>/i;
        real_tag = /real>/i;
        uuid_tag = /uuid>/i;
        string_tag = /string>/i;
        date_tag = /date>/i;
        uri_tag = /uri>/i;
        binary_tag = /binary/i . (space . /encoding="base64"/i)? . '>';

        key_tag = /key>/i;
        map_tag = /map>/i;
        array_tag = /array>/i;

        action call_array { 
            printf("Array start\n");
            fcall array_element; 
        }
        action call_map { 
            printf("Map start\n");
            fcall map_element; 
        }
        action return_from_map { 
            printf("Map end\n");
            fret; 
        }
        action return_from_array { 
            printf("Array end\n");
            fret; 
        }

        real_number = /[0-9]*/ @append_accum;
        integer_number = /[0-9]*/ @append_accum;

        hex_char = /[0-9a-fA-F]/ @append_accum;
        uuid_val = hex_char{8,8} . '-'? . hex_char{4,4} . '-'? . hex_char{4,4} . '-'? . hex_char{4,4} . '-'? . hex_char{12,12};


        action error { printf("Error!\n"); } 
        some_element = tag_start . (undef_tag @convert_undef
                  | boolean_tag @clear_accum . ('true'|'false') . tag_end . boolean_tag
                  | integer_tag @clear_accum . integer_number . tag_end . integer_tag @convert_integer
                  | real_tag @clear_accum . real_number . tag_end . real_tag @convert_real
                  | uuid_tag @clear_accum . uuid_val . tag_end . uuid_tag @convert_uuid
                  | string_tag @clear_accum . some_string . tag_end_nospace . string_tag @convert_string
                  | date_tag @clear_accum . date_time . tag_end . date_tag @convert_date
                  | uri_tag @clear_accum . some_string . tag_end . uri_tag @convert_uri
                  | binary_tag @clear_accum . base64_string . tag_end . binary_tag @convert_base64
                  | array_tag @call_array 
                  | map_tag @call_map 
                  );
 


        array_element := (some_element)* . tag_end . array_tag @return_from_array;

        map_key_start = tag_start . key_tag @clear_key_accum;

        map_key_end = '
        map_member = map_key_start . some_key_string . map_key_end . some_element @assign_map_member;

        map_element := (map_member)* . tag_end . map_tag @return_from_map;

        body = (some_element)?;

        xml_preamble = /xml[^>]*/i . '>';

        llsd_body = tag_start . ( '?' . xml_preamble . tag_start )? . llsd_tag . body . tag_end . llsd_tag; 

        main := llsd_body @{ res = 1; };

}%%

%% write data;


void llsd_scanner_free(llsd_scanner_t *scanner)
{
  if(scanner->stack) { free(scanner->stack); }
  if(scanner->key_accum) { free(scanner->key_accum); }
  if(scanner->accum) { free(scanner->accum); }
  if(scanner->bin_accum) { free(scanner->bin_accum); }
  free(scanner);
}

llsd_scanner_t *llsd_scanner_alloc(int key_accum_size, int accum_size, int bin_accum_size, int stack_size) 
{
   int failure = 0;
   llsd_scanner_t *scanner = calloc(1, sizeof(llsd_scanner_t));
   if (scanner) {
     scanner->bin_accum = malloc(bin_accum_size); 
     if (scanner->bin_accum) { 
       scanner->bin_accum_size = bin_accum_size;
       scanner->accum = malloc(accum_size);
       if (scanner->accum) {
         scanner->accum_size = accum_size;
         scanner->key_accum = malloc(key_accum_size);
         if (scanner->key_accum) {
           scanner->key_accum_size = key_accum_size;
           scanner->stack = calloc(sizeof(int), stack_size);
           if(scanner->stack) {
             scanner->stack_size = stack_size;
             %% write init;
           } else {
             failure = 1;
           }
         } else {
           failure = 1;
         }
       } else {
         failure = 1;
       }
     } else {
       failure = 1;
     }
     if (failure) {
       llsd_scanner_free(scanner);
       scanner = NULL;
     }
   }
   return scanner;
}

int llsd_scanner_run(llsd_scanner_t *scanner, char *chunk)
{
    char *p = chunk;
    char *pe = chunk + strlen(chunk);
    int res = 0;
    // printf("Running parser on: %s, curr state: %i\n", chunk, scanner->cs);
    %% write exec;
    if (scanner->cs && res == 0) {
      res = -1;
    }
    return res;
}


int main( int argc, char **argv )
{
  int res = 0;
  int i = 1;
  llsd_scanner_t *scan;
  if ( argc > 1 ) {
    scan = llsd_scanner_alloc(1000, 1000, 32768, 100);
    while(i < argc) {
      res = llsd_scanner_run(scan, argv[i]);
      if (res == 0) {
        printf("Error on chunk %d\n", i);
        i = argc;
      } else {
      }
      i++;
    }
    llsd_scanner_free(scan);
  }
  printf("result = %i\n", res);
  return 0;
}

Preprocessor directives for ruby (not!:)

2009-02-25T23:08:00.008+01:00

At first I thought to do some more link-whoring - as I found precisely the search terms that are in the title of this post to lead to this blog... But then I though I will limit the on-topic information on the matter:

1) You don't need a preprocessor for Ruby
2) If you think you do, read up on metaprogramming - _Why's article will be a good start to spend a good time thinking.

But, this lead me back to the CPP - the preprocessor. Of course, beyond the Obfuscated C contest entries that are of parlor entertainment value only, there is some more to that.

Take a look at this one.

Then get it for infinite fun:


cvs -d:pserver:anonymous@chaos-pp.cvs.sourceforge.net:/cvsroot/chaos-pp login 
cvs -z3 -d:pserver:anonymous@chaos-pp.cvs.sourceforge.net:/cvsroot/chaos-pp co -P chaos-pp
cvs -z3 -d:pserver:anonymous@chaos-pp.cvs.sourceforge.net:/cvsroot/chaos-pp co -P order-pp

And then go check out and mess with the examples:

the .c file looks like:


.....
int main(void) {
   printf
     ("The 500th Fibonacci number is "
      ORDER_PP(8stringize(8to_lit(8fib(8nat(5,0,0)))))
      ".\n");
   return 0;
}

//>
// Unless you are already convinced, you should check, by
// preprocessing this example, that the actual parameter to
// `printf' in the above code is just a single string that
// contains the 500th Fibonacci number in base 10.
//
// If you know the order of growth of Fibonacci numbers, you
// probably realized that the result isn't exactly small. 
.....

Now, let's try it (stderr forwarded to the null for brevity, as it seems to spit some errors):


:~/tmp/order-pp/example$ cpp -I../inc fibonacci.c 2>/dev/null | tail
  return fib_iter(n, 0, 1);
}
# 63 "fibonacci.c"
int main(void) {
   printf
     ("The 500th Fibonacci number is "
      "139423224561697880139724382870407283950070256587697307264108962948325571622863290691557658876222521294125"
      ".\n");
   return 0;
}

Now, let's look at another example - average.c

you define the macro AVERAGE as:


#define AVERAGE(...)                                    \
((ORDER_PP(8seq_for_each_with_delimiter                 \
           (8put,                                       \
            8emit(8quote(+)),                           \
            8tuple_to_seq(8quote((__VA_ARGS__)))))) /   \
 ORDER_PP(8to_lit(8tuple_size(8quote((__VA_ARGS__))))))

and use it in your C code like:


int main(void) {
  printf(AVERAGE(3, 1, 4, 1, 5, 9, 2, 6, 5) == 4
         ? "OK.\n"
         : "ERROR!\n");

  return 0;
}

The code post-cpp looks like:


int main(void) {
  printf(((3 + 1 + 4 + 1 + 5 + 9 + 2 + 6 + 5) / 9) == 4
         ? "OK.\n"
         : "ERROR!\n");

  return 0;
}

Of course, it's only simple things like this that are so nice looking... Any nontrivial definitions cause giant single-liners - the worst nightmare for the debuggers and maintainers - so I would personally torture anyone who used this in the production code.. :)

Nonetheless - pretty impressive stuff.

The FONC project & potion

2009-02-25T00:21:00.004+01:00

While messing around looking if anyone did anything with potion language, that I wrote about a few days ago, found this interesting link:

http://wiki.ruby-lang.nl/amsterdamrbwiki/show/The+FONC+project - The Fundamentals OF New Computing. Has some interesting reading on research in programming languages.

Potion itself is rather fun, though of course my microbenchmark was a bit of an over-joyous :)

Yeah, while it looks to be pretty damn fast, LuaJIT is only about 1.7 times slower. Nonetheless, I'll play with it more, it is an interesting thing to mess around with - at least the internals. Syntax I am not so sure...


Foo = Object delegated
Foo def("xx", (x): "foo\n".)

Bar = Foo delegated
Bar def("yy", (x): "123\n".)

a = Foo allocate
b = Bar allocate

a xx print
b xx print
b yy print

Running this with potion will give you three lines "foo", "foo", and "123" - the middle one being the method of "Bar", "inherited" from "Foo". Did not dig on how to glue multiple mixins - _why wrote that it's incomplete, I'll see maybe some other day.

What the world is interested in...

2009-02-21T23:36:00.005+01:00

The other day, while searching for a "how to" on something, I became amused with the autosuggestions I got after typing in the "how to" into the search input editor.

Assuming the results there are sorted by the interest from the users (which is something pretty obvious to do), this is the apparent hit parade of the things
that the people are wondering about:

1. how to tie a tie

Ha. I wonder whether this stems from the fact that the people don't really wear a tie every day, so this knowledge gets into the dark corners of the forgotten, or the science of tying the tie is as complicated as actually spelling it. I've last had to use this search maybe half a year ago when I in fact needed to wear the tie to some place. The caching is a good optimization here - tie the tie once, and keep it tied. Saves a lot of time.

2. how to kiss

Something is wrong. This one was supposed to land on the first position IMO - as the potential audience for the ties is at most is half as big as the potential audience for the kisses. So, I can conclude that the sets are actually very much intersecting and it's the same folks who don't know how to tie a tie. Others either already know it or are practicing it off-line. Indeed, practice is the key here. Instead of optimizing, just brute-force it. The process is fun.

3. how to get pregnant

Now the mystery of the bias for the previous one is probably starting to dissolve - that second half is actually not wasting the time on paying too much attention to the preamble, and is much more result driven. I'd think that here the practice is also a key. Also, there's something about the days of the month and such. But with sufficient practice the distribution is going to be across all the days of the month, so it should not be that important.

4. how to make money

assuming your looks are ok - after all, the tie is an important part of it! - indeed, money might be of use sometimes. I think it's a simple economics - find the area with most demand that is unsatisfied, and provide a supply there. Even better - create the demand out of thin air - that's how you make new markets. The most creative and spectacular that readily comes up, is the one million pixel page. In a nutshell - create 1000x1000 image, and start selling it in blocks of 10x10. Hundred bucks each. I am in awe.

5. how to lose weight

I won't comment much on this one - it's outside of my experience. Although, a full long weekend of dedicated nonstop gaming with not much food inbetween gets quite a few kilos off. I accidentally tried that once - the result was pretty funny (needless to say I was dead tired after that). But I think that's not something anyone should follow - talk to your doctor, or something. Eat lighter food, do sports, and such.

6. how to draw

I wonder if this is about drawing as in a form of art or it about making sure that no-one wins. My best advise - find an artist nearby who is willing to bear with you, and use the results from the (4) above to pay for the lessons. Practice makes miracles. But it aint going to be easy, if you want to get decent that is.
My wife's an artist and she literally spent years doing that thing. (oh, if you wanna get lessons, maybe I can ask, drop me a note :)

7. how to make a website

Hire a professional.

8. how to cook a turkey

Ah, this is a much more appropriate one. Though, no opinion here - I'm in europe and prefer chicken anyway. Or beef, for that matter.

9. how to write a resume

I'd do it in a couple of steps: first achieve a bunch of great things. Then write them down. Though, it's not a quick process. But should surely work.

10. how to lose friends and alienate people

Oh, that's easy - just write something outright silly like I did in this post, and it might work. Or go watch the film and learn from there.

Sketching network diagrams - entirely in javascript, in your browser!

2009-02-18T01:40:00.003+01:00

This is a bit of shameless self-advertising - I figured that if I don't advertise a little bit, I never know if there are some people that might benefit of this thing.

Basically, if you want to scratch a network with a couple of routers and a cloud, now you do not need to start visio, dia, or anything else for that matter: just go to http://employees.org/~andin/network/, draw the network, and snap the picture from the screen.

Of course, it's all web2.x enabled and jazz, so you can save the layout as JSON, and load it later. Even, in a demonstration of a blatant violation of the security, you can pull in the JSON from another domain.

A little bit of history: as I use linux at work pretty much exclusively, I was always curious to have something simple to draw the diagrams. Dia is an excellent program, but I wanted something even simpler.

And so when I saw the Draw2D library I thought "ah-ha!".

The editor is not way too much more than a combination of the existing components - you can see the ingredients on its page.

Well, enough of me talking, go try it and let me know :-)

What would you choose ?
	Stay on current social network
	Get a free all-public account - or host your own instance with the privacy settings you choose
	All-inclusive privacy paid account
	Privacy sold apiece

pollcode.com free polls