Saturday, January 31, 2009

Troubleshooting lenovo's wireless with Vista: plot thickens

In the previous post, I came to a conclusion that it was the wireless driver update which sent my wireless on a new vista laptop south. Today I finally got some time to debug it a bit further...

..And it turns out the previous conclusions were, well... not entirely correct. Ubuntu livecd certainly has very little to it from Vista - yet I got only 400kbps performance - while on the wired connection I got 4Mbps. Though, the connectivity appeared smoother, but maybe it is the more tolerant Linux TCP stack. So, the conclusion is that it is OS-independent, and how I got "good" performance in the first place is a mystery.

The access point I'm using was an almost 10 years old noname 802.11b. Given that all the neighbouring ones are 802.11g, I think mine is probably driving the neighbours' data rates down, so probably a good time to go and get something newer - and write how it looks after that.


Even though I did not try the G access point, this link might explain it - at least the symptoms match very well:
Now, how in the world that happened for both Vista and Linux - I don't know :-)

Indeed updating the driver from the linuxwireless seemed to alleviate the problem for linux at least.

Wednesday, January 21, 2009

Interfacing Lua with C and hacking Lua syntax

After my initial experiments, I played last weekend a bit more with the lua+c integration - and indeed it is quite nice and flexible. My fears about the 1-based array indexing so far turn out to be a bit oversized - the biggest problem this would bring is when porting something from a 0-based language, when writing a new code from scratch it is in fact pretty straightforward (and the "for i,v in ipairs(array) do" iterator helps to avoid thinking about indices altogether).

What's especially cool is the freedom it allows - to code parts in the high-level language, and parts in low-level C, and intermix them as needed.

As a result, over the past weekend I've ported some "hobby" XML-parsing HTTP code over form Ruby to Lua with very little effort.

Joyed with such an ease, for the work (reproducing a setup involving the signaling of a couple dozen thousand IP phones) I've written in just over a day a little helper code, which includes Lua itself + convenient framework to abstract all of the socket-handling details into much more highlevel structures, something like:

function server_read(sock, data)
print("Server received ", string.len(data), "bytes:", data)
send(sock, "I got your data!\n")

function server_closed(sock)
print("Client disconnected!")

function server_incoming_connect(parent, remotehost, remoteport, newindex)
local server_methods = {
read = server_read;
close = server_closed
return server_methods

function server_main(process, number)
tcp_listen("", 12345, { connect = server_incoming_connect })

So, all of the socket low-level code (poll cycle, checking where the data has arrived, etc.) is hidden inside the compiled C blob, with the higher-level logic easily adjustable. Quite yummy :-)

Overall it looks like Lua is definitely there to stay in my toolbox, which now will mainly contain Lua, C, and Ruby (though the biggest major advantage of the latter for now is the richness of the libraries and the lack of parentheses).

Speaking of parentheses, I experimented with getting rid of parens in Lua - alongside with more Ruby-esque "def" and "elsif" - but in the end decided to throw it into garbage - primarily because of the danger of the confusion in the brain because of somewhat-similar-yet-quite-different semantics of the resulting "language" vs. Ruby, as well of some of the inconsistencies that arised as a result, i.e.:

foo bar baz

would be the same as foo(bar(baz)), and baz, to be called, would need explicit parens. As well - a single-handed statement "baz" on the line would cause a syntax error. These all are pretty easily fixable (though then I'd need to have a way to specify that I want a function itself, not its call), but would break a lot of backward compatibility with Lua, which I wanted to have - hence not worth it.
I'll better live with "elseif". :-) But the entertainment value of this experiment was certainly worth it.

Monday, January 12, 2009

Open, standards based video player for the web ? - seems nice, I wish I would be a lawyer to read up on the license, as it's obviously not just something as "easy" as a BSD or GPL...

Thursday, January 8, 2009

Initial impressions of Lua

During the long weekend, I've spent a few hours on messing up with Lua - after some readings I finally got seduced enough to give it a try.

As an exercise, I went over porting part of the reader of the "message template" file (the latter I grabbed from libopenmv) that I previously wrote in Ruby (about 15K of code to generate the semi-functional Ruby class definitions). The port now reads the defininions into a bunch of interconnected tables, however it's not too big either - just about 3K:

First, a helper method (I miss ruby String.split() somewhat):

function string.gather(str, pat)
local tab = {}
for match in str:gmatch(pat) do
table.insert(tab, match)
return tab

Then goes my experiment with an iterator-within-a-coroutine-in-OO-like-thing to get the tokens in batches:

TemplateReader = function(filename)
local worker = function(fname)
for line in io.lines(fname) do
-- get rid of comments, [rl]trim, remove trailing and multispaces
local line2 = line:gsub("//.*$",""):gsub("^%s*","")
:gsub("%s+$",""):gsub("%s+"," ")
local tokens = line2:gather("([^%s]+)")
if(#tokens > 0) then

local me = {}
me.fname = filename

me.coworker = coroutine.create(worker)
me.tokens = function(me)
local ok, tok = coroutine.resume(me.coworker, me.fname)
return tok

return me

And then go the "conceptual piece" readers. They were in separate classes in Ruby version, but given the pretty similar structure for all the packets, I think I might not need it.

Here we go, in reverse order. The bottom-most part is the actual "main" body:

tr = TemplateReader(filename)

version = tr:tokens()
print("version: " .. version[2])
packets = {}
for packet in PacketTemplateRead, tr do
print("Packet obtained: " ..
table.insert(packets, packet)

print("done, total: " .. #packets)

Now, let's see how we read the packets:

PacketTemplateRead = function(tr)
local start = tr:tokens()
local pt = nil
local finished = false

if (start and #start == 1 and start[1] == "{") then
local pkt_info = tr:tokens()
pt = {}
pt.blocks = {} = pkt_info[1]
local tok = tr:tokens()
if tok[1] == "{" then
table.insert(pt.blocks, PacketBlockTemplateRead(tr))
finished = true
until finished
elseif start then
print("start[1] is: " .. start[1])
return pt

So, let's see how we read the block...

PacketBlockTemplateRead = function(tr)
local b = {}
local toks = tr:tokens() = toks[1]
if toks[2] == "Single" then
b.count = 1
elseif toks[2] == "Multiple" then
b.count = toks[3]
elseif toks[2] == "Variable" then
b.count = -1
print("Unknown block count " .. toks[2]
.. " for block " .. toks[1])
-- FIXME: need an exception ?

b.fields = {}
for field in PacketFieldRead, tr do
table.insert(b.fields, field)
return b

The only thing that is left mysterious, is the packet field, which is the "name / type / length" kind of thing:

PacketFieldRead = function(tr)
local toks = tr:tokens()
local field = nil
if not (toks[1] == "}") then
field = {} = toks[2]
field.type = toks[3]
field.len = toks[4]
-- print("Got field name: " .. .. " type " .. field.type)
return field

This reads the message template visually several times faster than Ruby, in barely noticeable fractions of a second on my laptop.

Which *may* make this a reasonable candidate not to store the generated code anywhere at all, and just keep the message template file. Though, my experiences with the Ruby autogenerated in-memory code were that in case there's an exception inside that code, it's a royal PITA to debug - so the question of how to do error handling would need a better thought.

Overall impressions of translating this from looking at the Ruby version:

1) 1-based arrays are a pain for the C/Ruby/...-infected zero-based brain to switch to.
In fact, I did create a couple of bugs precisely while retyping usage of the elements of the array. And there's hardly twice than that places where I use the indices :) So - this is a big warning sign to myself. Just that it is not easy - not that "0" or "1" based are better or worse in all cases.

2) having functions as first class citizens feels really easy-going after Ruby.

3) coroutines are cool. the yield/resume couple interaction creates the feeling that the world is something you can turn inside-out and back - and these two do precisely that. Very odd feeling. Though I would not say that my use of coroutine-within-the-iterator is a terribly good idea.

4) table-based OO feels a big bit like in Javascript.

5) you notice "not x == y". I find it more readable compared to ~=, and less mentally confusing with =~ (Perl-ism to match on a regexp).

6) finally a minor one but probably the most annoying - after being used to write print "Var: #{x}\n" in Ruby, it is pretty bad to have to write print("Var: " .. x .. "\n") in Lua. And the main annoyance is the parens. I know they do create the ambiguity and pain for the parser, but the lack of them is so conweeeeenient! :)

Overall, seems to be pretty usable and sweet language, and given its speed, tiny size of the base kit, and the ease of integration with C, as well as a JIT compiler, which in my microbenchmarks did show pretty nice results - I like it

p.s. don't ask me why I mention Python only here. I am still trying to wrap my inner nerves around the whitespace thing.

A haiku on usability of security dialogs which resist the screenshots.

the vista update.
dialog. "details" - hex: {...-DEAF}.
thoroughly puzzling.

How I shot myself in the foot. Twice. During yoga - so my foot was on my forehead....

If you were expecting to see a real story here - indeed there is. But it would be of no entertainment value if I did not have the joy of wearing both the "patient" and the "doctor" hats - as well as taking the roles of the "independent 3rd party" at times to observe.

Here goes the true story a winded "case", happened to yours truly.

Step0: I have to admit, the phone socket that I mostly use (for DSL only), is a totally oxidized piece of junk, which I am perpetually too lazy to replace because everything works fine if I half-unplug the DSL connector from the wall - and then it becomes a classic "if it ain't broken, don't...".

Step1: my DSL is dead. Dead for for pretty long time, but I'm not much bothered about fixing it, as the majority of the time I spend at this time period at home is sleeping, I can survive that without connectivity :-) Though first I did do a fair share of usual fiddling of the wall connector during the initial phases of troubleshooting. But the line is dead. Real dead(tm). So, especially since the initial failure happened to coincide with potential DSL headend upgrade at the ISP (as they were widely marketing the much higher speeds than before) - all looks as if it's a proverbial man who was patching the stuff downstairs, and cut the wrong wire ;)

Step2: after some administrative wiggling, which is uninteresting for the purposes of this story, the DSL gets finally fixed - it comes up on friday while I am at work. Then I verify during the weekend that "yes it works" and signal the happiness and thanks to the ISP support folks. Well, it kinda works at half speed, but again, for connectivity mostly during sleep that's enough, so I do not complain. :-) And then as well - the glass that is half full is much better than a totally empty one. :-)

[ here a week or so passes, meantime the reader is directed to click on the advertisements on /. as a pastime ]

Step3: friday - clean-up party at home. Things get moved, vacuum-cleaned, watered, polished, and a lot of other funny activities. To try to handle the trauma, I hide in the corner with connectivity. At some point the latter disappears. Trying to fiddle with the socket - no way.. Oh well, it's pretty late anyway, so why bother.. time to sleep.

Step4: saturday - some shopping time. No time to fiddle. But, as a reward for my good behaviour throughout the year, I get a shiny new Lenovo N500 laptop with Vista on it. Curiously I haven't touched Windows-based systems for anything beyond simple install for quite a while, so it's entertaining and will have its own post at some point. But the internet is still unstable. So some evening fiddling with socket + amplification settings in the config - so-so.

Step5: sunday - the moment of truth - I manage to get the full speed! And the Vista is really connected and flying! wow! Now its time to install firefox, cygwin, and all the other command-line toys that I will for sure need in order for this laptop to be of any use. I leave the download of some gigabytes on, and we go for a walk for a few hours. When we arrive back home, everything is downloaded, installed, I am a happy camper. There's this box that asks for a reboot, after some postponing, I reboot it, and leave to install the locally downloaded cygwin - and then go to watch some TV meanwhile and hack on some stuff in the background.

Step6: I notice the internet is not too good again. Argh. The most interesting thing - the pings to my default gateway over wireless work apparently fine, the download of the ubuntu image from the ISP-local mirror gives about 200kbytes/sec instead of 400+ that it should - which is anyway bearable, and can be attributed to the socket which maybe I *should* replace now. The cool part though that anything beyond the local ISP network is just plain broken. Horrible latencies of 1.5+ second (the ping to ISP's DNS server is 22milliseconds), and huge packet loss. Traceroute shows some oddities, so I make the conclusion that they did not like me for that cygwin update, and I ate all the quota of the month, after which they normally rate-limit the traffic to 64K. Well, maybe this is done for some reason only for external traffic.. odd but believable.

Step7: today early morning, I go to use my desktop, which is an oldie gentoo box, and notice that *everything flies* !!! Wow, must be that this socket has fixed itself again - lets check the Vista box... hmm - it does not work still! Can't be *so* precise timing, but let's check two simultaneously - indeed it's Vista specific.
Not being fully awake, only the search reflex is functioning. So I look up for "Vista internet slow". Which turns out with lots of forums with voodoo-like manipulations, uninstallations of antivirus software, recommendations to remove the SIMMs one by one, and suggestions to uninstall updates.

Step8: *updates*!!! That reboot was insisted upon by the update! I totally forgot about this given the flurry of other non-computer events that happened this holiday weekend. Ok, now at least we know "what changed" (something that I use as an introductory joke - "because I know nothing changed, but it's interesting to try to find out whether there's any occurence where asking this question explicitly will be of use" :) Now, we can also think "what's different between these two boxes". Hmm... linux-vista... nope, I *know* vista was working for sure, just fine :-) wired-wireless... good point, let's try wired... and it works fine! Using wireless... still dialup-like and worse performances. So, it's got to be bound to wireless... Let's check - maybe my good ol' access point went kaput ? No, the wife can use her wireless with no problem (though she has an oldie iBook which survived a surgical hard drive replacement by yours truly :)

Step9: The most plausible theory, that so far explains almost all the observations:

  • vista update and lenovo wireless drivers did not like each other
  • the packet loss which is the effect of this bug is somehow proportional to RTT (????) - which is a bold statement in itself, but at least explains why the throughput to ISP mirror was halved (small latency), and to anything on the internet was almost nil (bigger latency), and why I did not see anything with the default gateway testing (pretty much no latency)

Of course, the jury is still out to figure out what's the matter with wireless (and whether to install 38 more updates that the windows update is proposing... (Half of them are named "Windows Update #KB-number", and going manually over all of them via manual retyping the KB#s into the browser is a pretty boring way to spend time. oops, Vista will have its own post, I promised).

Anyway, after such a long preamble, a few observations-reminders both to myself, and to you, if you made it reading till here :-)

1) Life is too short to keep a note "what changed". That's why noone ever "admits" - not because of some magic stubbornness, unlike some BOFHs think, but just because there are better things to do.

2) One persistent issue will psychologically distract the attention even if there is really another one with a slightly different symptom - I clearly was blaming the latent L1 instability a bit too readily.

3) The combination of several smaller issues amplifies the effect and makes the resulting problem look quite more "interesting".

All three are something which I knew for quite a few years already, but they packed themselves so nicely into a "case study" - which, since it happened to myself, I could freely share.


There's more to the story.