Monday, March 30, 2009

Cosimus now with avatar shapes and clothing

Today I took a brute-force approach to solving the missing avatar clothing problem that
I was chasing for a few days - simply by creating my own set of the default clothing and wearables.
It's just the standard set of items which are uploaded by the viewer in the case it does not find them - so for the secondlifers, it's Ruth in white clothing :)

Alongside, I've put a very simple moving logic - only forward, in the direction that the avatar faces. This required some rudimentary support for something the called "scene" in opensim - or at least my interpretation of it. For now, I am taking the assumption "Agent == Avatar", it probably will change sometime in the future. However, I already put the "clientstack-independent scene" atop the SMV one - to represent the qualities that are (or should be) fundamental to any avatar system - X, Y, Z position, AgentID - I'll use the UUIDs for representing the "entity" for now, and the "label" - some text that identifies the avatar as seen in-world.

The rest of the platform-dependent details go into a table called "Blob" which is at the moment SMV-specific data about the avatar (like, local object ID, which is needed for the viewer, as each avatar has its own local id to be drawn correctly).

Possibly exposing a "local avatar id" might have been a better move for the platform-independent record, and having the clientstack-specific lookup tables.. But I will see in the future what is better. In any case, each clientstack registers its own event handler functions, which are called per-avatar in case the need arises (this indeed raises some additional possibilities for optimization in the future).

Alongside the minor infra mods, the loginserver config is now in the config file (main.lua), as well as the serializable state to facilitate trivial no-outage restarts is inside a separate file.

Interestingly enough, there seems to be something odd with the object update packet - the omvviewer tends to hard-hang the laptop - I am not sure which part is causing that. And somehow I managed to get some green text atop the avatar at one occasion, which was funny.

As the vacation time is over now for me, most definitely the speed of the development will suffer - I'll only have some late nights, which are also good to be used for sleep, but I am pretty happy with the progress so far.

Sunday, March 29, 2009

Cosimus - a little toolkit for metaverse experiments

The past two weeks I've been on holiday - so this translated for doing various things that I did not have time to do before. One of those things was completing the chunk of coding needed to at least start with my little metaverse experiment.

The result is at http://github.com/ayourtch/cosimus/tree/master - and will suffer the periodic commits from now on, depending on my so called "spare time" :)

It's a start of the simulator code for the 3d simulator, for now I've started with the SecondLife(tm) protocol, as it's something I had in the beginning of the works.

For now it is a custom Lua interpreter, with a few libraries linked in and initialized by default, and the application code in prototype phase is being written in Lua. This gives the much faster speed of the development compared to C (a long while ago, while checking the speed of development on Perl vs. C I found about 10x difference - in favour of Perl, of course. I did not do any personal measurements now, but from a gut feeling it should be about the same). Much later, as needed, I'll push the performance-critical pieces of the code downwards to C.

For now the thing does not do all that much - these few days I've been mostly dealing with the inventory/asset code, appears there are some magic tweaks still needed - from the behaviour it seems like I am triggering some data corruptions in the client inventory code..

From the cool features that are already there - the inventory/asset library loading is done from zipfiles, which means easier handling of them for the consumer. Another fun feature - or rather design choice - is that all the communication via the components that is possible to be networked in the distributed setup, will be also networked in the standalone setup, with the simple servers doing the job from within the executable itself. So it is "batteries included", and a single code path to debug.

...Now, back to coding :-)

Wednesday, March 11, 2009

Compiling zettair on ubuntu

The first compile throws a bunch of errors, due to missing zlib, among them:

‘Z_NO_FLUSH’ undeclared

solution - sudo apt-get install zlib1g-dev, then rerun configure script, and all compiles fine.

Tuesday, March 10, 2009

Zettair is back alive

Hooray! Zettair folks down in Oz have revived the development. "in-development" version now online.

Let's see what we have (with a brutal diff -rc across the directories :-) Disclaimer: this is not necessarily a complete list, at least in the autoconf part - the diff was fairly boring and long.
Alternatively, I might have missed it in the set of whitelist changes or did not get a clue it was important...:)

configure script got what looks a 64-bit "kfreebsd". Probably inherited with a newer version of autotools, but an interesting beast nonetheless, if I read it well, it is here. GNU userland running on top of freebsd kernel.. The other tweaks belong to dragonflybsd. I heard about Hammer filesystem, but haven't tried the OS yet. Might be interesting. All depends on the H/w support.

index.h:

option INDEX_NEW_VOCAB is gone, INDEX_NEW_PARSEBUF (dictate how large the postings hashtable is) appeared.

INDEX_NEW_NO_OFFSETS - build the index without offsets. Probably not very interesting for me...

INDEX_NEW_QTHREADS - how many query threads to allow. Now I wonder if this allows for simultaneous query and indexing - that'd be sweet. My experiments with the previous version running separate processes were not terribly successful. With threads indeed it might be a different story.

some other similar parameter changes to index_load (as opposed to index_new)

index_cleanup is gone.

INDEX_SEARCH_DYNAMIC_RANK - provide the text describing the rank to be used as well as the rank parameters. Interesting, probably a bit too advanced for my needs.

autogenerated code for the metrics is obviously now parametrized.

mutexes for various pieces (For multithreading)

vocab_vector structure is simplified (looks like thanks to getting rid of multiple files)

a lot of multithreaded-oriented code

a hacko-fix in makeindex_append_docno to get around the fact that zettair can't handle docno-s bigger than a pagesize

vocabury operations seem to be simplified - previously the memory management was more sophisticated.

This completes the list - again, very probably I missed something important - if so, drop me a note in the comments.

Monday, March 9, 2009

LLSD scanning with ragel-made parser.

I've written a while ago about the Ragel.

Today evening (literally in about 4 hours), I tried it and cooked a semi-functional parser for LLSD XML serialization. Yes, the code below was cranked from scratch in just a few hours :)

What's nice is that it is can parse the data by chunks of practically any size, so could be used to grab the data over a slower TCP connection. And - the base64 decoding is included! Things that would need to be done in order to make it usable include proper error handling and a better approach to handle possibly large data - right now it is kind of dumb...

Here's the result of running it over the sample XML from the draft:


$ ./a.out `cat ../sample.llsd`
Array start
Integer: '42'
UUID: '6bad258e06f04a87a659493117c9c162'
Map start
String: 'cold'
Map member with key 'hot':'cold'
Undef
Map member with key 'higgs_boson_rest_mass':''
URI: 'https://example.org/r/6bad258e-06f0-4a87-a659-493117c9c162'
Map member with key 'info_page':'https://example.org/r/6bad258e-06f0-4a87-a659-493117c9c162'
Date: '2008-10-13T19:00.00Z'
Map member with key 'status_report_due_by':'2008-10-13T19:00.00Z'
Map end
Array end
result = 1


And here goes the .rl code in case you're curious (maybe you need to "view source" in case it gets garbled):


#include
#include
#include

typedef struct llsd_scanner {
char *accum;
int accum_size;
int accum_index;

char *key_accum;
int key_accum_size;
int key_accum_index;

unsigned char *bin_accum;
int bin_accum_size;
int bin_accum_index;
unsigned char b64chars[4];

// ragel variables
int *stack;
int stack_size;
int top;
int act;
int cs;
unsigned char *ts, *te;

} llsd_scanner_t;



static const char cd64[]="|$$$}rstuvwxyz{$$$$$$$>?@ABCDEFGHIJKLMNOPQRSTUVW$$$$$$XYZ[\\]^_`abcdefghijklmnopq";

static void llsd_scanner_putb64char(llsd_scanner_t *scanner, int index, unsigned char v)
{
v = (unsigned char) ((v < 43 || v > 122) ? 0 : cd64[ v - 43 ]);
if( v ) {
v = (unsigned char) ((v == '$') ? 0 : v - 61);
}
if (v) {
scanner->b64chars[index] = (unsigned char) (v - 1);
} else {
scanner->b64chars[index] = v;
}
}

static void b64decodeblock( unsigned char in[4], unsigned char out[3] )
{
out[ 0 ] = (unsigned char ) (in[0] << 2 | in[1] >> 4);
out[ 1 ] = (unsigned char ) (in[1] << 4 | in[2] >> 2);
out[ 2 ] = (unsigned char ) (((in[2] << 6) & 0xc0) | in[3]);
}


static int append_binchars(llsd_scanner_t *scanner)
{
if (scanner->bin_accum_index + 4 < scanner->bin_accum_size) {
b64decodeblock(scanner->b64chars, &scanner->bin_accum[scanner->bin_accum_index]);
scanner->bin_accum_index += 3;
return -1;
} else {
printf("Bin accumulator overflow!");
return 0;
}
}

static void clear_accum(char *accum, int *accum_index)
{
*accum_index = 0;
}

static int append_accum(char *accum, int accum_size, int *accum_index, unsigned char c)
{
if (*accum_index < accum_size-1) {
accum[(*accum_index)++] = c;
return -1;
} else {
return 0;
}
}

static void terminate_accum(char *accum, int *accum_index)
{
accum[*accum_index] = 0;
}




%%{

machine LLSDScanner;

access scanner->;



action terminate_accum {
terminate_accum(scanner->accum, &scanner->accum_index);
printf("Accumulated: '%s'\n", scanner->accum);
}
action clear_accum {
clear_accum(scanner->accum, &scanner->accum_index);
clear_accum(scanner->bin_accum, &scanner->bin_accum_index);
}
action append_accum {
append_accum(scanner->accum, scanner->accum_size, &(scanner->accum_index), fc);
}

action print_key_accum {
terminate_accum(scanner->key_accum, &scanner->key_accum_index);
// printf("Key Accumulated: '%s'\n", scanner->key_accum);
}

action clear_key_accum {
clear_accum(scanner->key_accum, &scanner->key_accum_index);
}
action append_key_accum {
append_accum(scanner->key_accum, scanner->key_accum_size, &scanner->key_accum_index, fc);
}

action convert_string {
terminate_accum(scanner->accum, &scanner->accum_index);
printf("String: '%s'\n", scanner->accum);
}
action convert_uuid {
terminate_accum(scanner->accum, &scanner->accum_index);
printf("UUID: '%s'\n", scanner->accum);
}
action convert_uri {
terminate_accum(scanner->accum, &scanner->accum_index);
printf("URI: '%s'\n", scanner->accum);
}
action convert_base64 {
terminate_accum(scanner->bin_accum, &(scanner->bin_accum_index));
printf("Binary Accumulated: '%s'\n", scanner->bin_accum);
}
action convert_date {
terminate_accum(scanner->accum, &(scanner->accum_index));
printf("Date: '%s'\n", scanner->accum);
}
action convert_integer {
terminate_accum(scanner->accum, &(scanner->accum_index));
printf("Integer: '%s'\n", scanner->accum);
}
action convert_real {
terminate_accum(scanner->accum, &(scanner->accum_index));
printf("Real: '%s'\n", scanner->accum);
}
action convert_undef {
terminate_accum(scanner->accum, &(scanner->accum_index));
strcpy(scanner->accum, "");
printf("Undef\n");
}

action assign_map_member {
printf("Map member with key '%s':'%s'\n", scanner->key_accum, scanner->accum);
}

one_or_two = /[12]/ @append_accum;
zero_or_one = /[01]/ @append_accum;
zero_to_nine = /[0-9]/ @append_accum;
zero_to_two = /[0-2]/ @append_accum;
zero_to_three = /[0-3]/ @append_accum;
zero_to_five = /[0-5]/ @append_accum;
dot = '.' @append_accum;
dash = '-' @append_accum;
colon = ':' @append_accum;
dot_or_colon = /[.:]/ @append_accum;
letter_T = 'T' @append_accum;
letter_Z = 'Z' @append_accum;

date_fullyear = one_or_two . zero_to_nine . zero_to_nine . zero_to_nine;
date_month = zero_or_one . zero_to_nine;
date_mday = zero_to_three . zero_to_nine;
time_hour = zero_to_two . zero_to_nine;
time_minute = zero_to_five . zero_to_nine;
time_sec = zero_to_five . zero_to_nine;
time_secfrac = dot . zero_to_nine;
full_date = date_fullyear . dash . date_month . dash . date_mday;
partial_time = ( time_hour . colon . time_minute . colon . time_sec . time_secfrac | time_hour . colon . time_minute . dot_or_colon . time_sec );
date_time = full_date . letter_T . partial_time . letter_Z;

maybe_space = space*;

some_string_char = /[^<]/ @append_accum;
some_string = some_string_char*;

some_key_string_char = /[^<]/ @append_key_accum;
some_key_string = some_key_string_char*;

# base64 parsing. Base64 characters come in series of 4

base64_char = [ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+=/];
action base64_one { llsd_scanner_putb64char(scanner, 0, fc); }
action base64_two { llsd_scanner_putb64char(scanner, 1, fc); }
action base64_three { llsd_scanner_putb64char(scanner, 2, fc); }
action base64_four { llsd_scanner_putb64char(scanner, 3, fc); }
action base64_four_chars { append_binchars(scanner); }

base64_first_char = base64_char @base64_one;
base64_second_char = base64_char @base64_two;
base64_third_char = base64_char @base64_three;
base64_fourth_char = base64_char @base64_four;

base64_four_chars = maybe_space . base64_first_char . maybe_space . base64_second_char . maybe_space .
base64_third_char . maybe_space . base64_fourth_char @base64_four_chars;

base64_string = base64_four_chars+;

tag_start = maybe_space . '<';
tag_end_nospace = ' tag_end = maybe_space . tag_end_nospace;

llsd_tag = /llsd>/i;

undef_tag = /undef\/>/i;
boolean_tag = /boolean>/i;
integer_tag = /integer>/i;
real_tag = /real>/i;
uuid_tag = /uuid>/i;
string_tag = /string>/i;
date_tag = /date>/i;
uri_tag = /uri>/i;
binary_tag = /binary/i . (space . /encoding="base64"/i)? . '>';

key_tag = /key>/i;
map_tag = /map>/i;
array_tag = /array>/i;

action call_array {
printf("Array start\n");
fcall array_element;
}
action call_map {
printf("Map start\n");
fcall map_element;
}
action return_from_map {
printf("Map end\n");
fret;
}
action return_from_array {
printf("Array end\n");
fret;
}

real_number = /[0-9]*/ @append_accum;
integer_number = /[0-9]*/ @append_accum;

hex_char = /[0-9a-fA-F]/ @append_accum;
uuid_val = hex_char{8,8} . '-'? . hex_char{4,4} . '-'? . hex_char{4,4} . '-'? . hex_char{4,4} . '-'? . hex_char{12,12};


action error { printf("Error!\n"); }
some_element = tag_start . (undef_tag @convert_undef
| boolean_tag @clear_accum . ('true'|'false') . tag_end . boolean_tag
| integer_tag @clear_accum . integer_number . tag_end . integer_tag @convert_integer
| real_tag @clear_accum . real_number . tag_end . real_tag @convert_real
| uuid_tag @clear_accum . uuid_val . tag_end . uuid_tag @convert_uuid
| string_tag @clear_accum . some_string . tag_end_nospace . string_tag @convert_string
| date_tag @clear_accum . date_time . tag_end . date_tag @convert_date
| uri_tag @clear_accum . some_string . tag_end . uri_tag @convert_uri
| binary_tag @clear_accum . base64_string . tag_end . binary_tag @convert_base64
| array_tag @call_array
| map_tag @call_map
);



array_element := (some_element)* . tag_end . array_tag @return_from_array;

map_key_start = tag_start . key_tag @clear_key_accum;

map_key_end = '
map_member = map_key_start . some_key_string . map_key_end . some_element @assign_map_member;

map_element := (map_member)* . tag_end . map_tag @return_from_map;

body = (some_element)?;

xml_preamble = /xml[^>]*/i . '>';

llsd_body = tag_start . ( '?' . xml_preamble . tag_start )? . llsd_tag . body . tag_end . llsd_tag;

main := llsd_body @{ res = 1; };

}%%

%% write data;


void llsd_scanner_free(llsd_scanner_t *scanner)
{
if(scanner->stack) { free(scanner->stack); }
if(scanner->key_accum) { free(scanner->key_accum); }
if(scanner->accum) { free(scanner->accum); }
if(scanner->bin_accum) { free(scanner->bin_accum); }
free(scanner);
}

llsd_scanner_t *llsd_scanner_alloc(int key_accum_size, int accum_size, int bin_accum_size, int stack_size)
{
int failure = 0;
llsd_scanner_t *scanner = calloc(1, sizeof(llsd_scanner_t));
if (scanner) {
scanner->bin_accum = malloc(bin_accum_size);
if (scanner->bin_accum) {
scanner->bin_accum_size = bin_accum_size;
scanner->accum = malloc(accum_size);
if (scanner->accum) {
scanner->accum_size = accum_size;
scanner->key_accum = malloc(key_accum_size);
if (scanner->key_accum) {
scanner->key_accum_size = key_accum_size;
scanner->stack = calloc(sizeof(int), stack_size);
if(scanner->stack) {
scanner->stack_size = stack_size;
%% write init;
} else {
failure = 1;
}
} else {
failure = 1;
}
} else {
failure = 1;
}
} else {
failure = 1;
}
if (failure) {
llsd_scanner_free(scanner);
scanner = NULL;
}
}
return scanner;
}

int llsd_scanner_run(llsd_scanner_t *scanner, char *chunk)
{
char *p = chunk;
char *pe = chunk + strlen(chunk);
int res = 0;
// printf("Running parser on: %s, curr state: %i\n", chunk, scanner->cs);
%% write exec;
if (scanner->cs && res == 0) {
res = -1;
}
return res;
}


int main( int argc, char **argv )
{
int res = 0;
int i = 1;
llsd_scanner_t *scan;
if ( argc > 1 ) {
scan = llsd_scanner_alloc(1000, 1000, 32768, 100);
while(i < argc) {
res = llsd_scanner_run(scan, argv[i]);
if (res == 0) {
printf("Error on chunk %d\n", i);
i = argc;
} else {
}
i++;
}
llsd_scanner_free(scan);
}
printf("result = %i\n", res);
return 0;
}