.NET remoting via COM and Python

I bumped into a problem when trying to talk to C# DLLS that are created by my team while i am using Python. Creating a COM C# DLL is the key to bridge the communication since creating COM  interface via C# is the easiest thing to do compared to C++.

Now one of the C# DLLs is loaded as a singleton in Windows Service thus i need my C# COM to talk to the same class via remoting.

That is when i kept on encountering “return argument has invalid type”  whenever the remoting part was done. Putting "[Serializable]” into the class declarating did not help.

It ran perfectly when the class in the C# dll is used as a local class but it balked whenever the same class was returned via remoting and to me that is just ONE line of change.

// MyClass myClass = new ManagerClass()  <--- local class

// RemotingReturnMyClass(out myClass) <—remoting

I went on to combining the COM class code and the C# singleton DLL in the attempt to create a singleton COM so that the C# singleton class can be accessed via my python code to no avail.

After DAYS (yup days) of trying to solve it, i suddenly realize something that i missed. What if i put the C# DLL into c:\python25 since maybe the problem is all about the marshaller and proxy not being able to locate it…….

Bingo.

Plone, Drupal or What?

A quick migration of a static website to a CMS version was needed. However one of the requirement is that it must be as ‘non-programmer’ as possible since an assigned web designer will be doing the project.

After toying around with Plone, its way too technical and the curve is just too much to even be considered. It is also pretty slow, around 1.3 times slower in a Pentium Celeron with 1GB memory compared to Drupal on a dummy blank site.

So Drupal was next in the list and in fact chosen for the job , however as 2 weeks gone by, it was obvious that what we know about Drupal is in fact largely OVERRATED. There are certain theme layouts that doesn’t appear correctly on different browsers, very limited customization without going to php and way too much work for a non-technical person to get it looking nice, not to mentioned that most sites on Drupal looks similar.

Finally we stumbled upon SiteFinity by SmarterTools, the condition however was that it had to run on IIS but in this case we had no issues with that since the project doesn’t mind whether its IIS or Apache (who the hell going to modify apache code in the name of open source?)

SiteFinity not only blows Drupal to stoneage, more importantly we saved the Web designer guy from committing suicide. For once she was able to get something done.

** there is a community version available free for use

Removing “achtbanen” trojan links

Ok so someone got careless and somehow his website files have been replaced with links to a trojan. Each of the html/asp files have been injected with a script “<script src=http://achtbanen.org/images/blah blah blah></script>”

I was asked to write a script to clean it up, so here it is, a script injector remover, basically will work on any injected one liner script (modify the signature on “FindThis” in the source yourself), by default will remove this particular achtbanen line.

TrjLinkRemove.py

# Copyright 2009, codemagnet.blogspot.com

# Free to use ;-), just keep this credit comments.

import sys

import os

import nowlog

logfile = nowlog.SetLogConfig("removed.log")

"""<script src=http://achtbanen.org/images/b-one-default.php ></script>"""

FindThis = "<script src=http://achtbanen.org/images"

FindThisJS = "document.write('<script src=http://"

MaximumIndexTail = 150 #  search 150 off the first found signature

def dirwalk(dir):

    "walk a directory tree, using a generator"

    try :

        for f in os.listdir(dir):

            fullpath = os.path.join(dir,f)

            if os.path.isdir(fullpath) and not os.path.islink(fullpath):

                for x in dirwalk(fullpath):  # recurse into subdir

                    yield x

            else:

                yield fullpath            

    except :

        pass

def CleanFile (filename) :

    global FindThis

    try :

        f = open(filename, "rb")

        fb = f.read()

        f.close()

        i = fb.find(FindThis) 

        if i == -1 :

            return

        # found

        o = fb.find("</script>", i+len(FindThis), i+len(FindThis) + MaximumIndexTail)

        if o == -1:

            logfile.error("partial found, ignored, %s", filename)

            return

        # create a new file

        newfile = filename + ".trj.rmv"

        p = open(filename + ".trj.rmv", "wb")

        p.write(fb[:i])

        p.write(fb[o+9:])

        p.close()

        os.rename(filename, filename + ".xxxx")

        os.rename(newfile, filename)

        os.unlink(filename + ".xxxx")

        logfile.info("Cleaning : %s", filename)        

    except :

        logfile.error("unable to open file %s", filename)

        return

#-------------------------------------------------------------------------#

def CleanFileJS (filename) :

    global FindThis

    try :

        f = open(filename, "rb")

        fb = f.read()

        f.close()

        i = fb.find(FindThisJS) 

        if i == -1 :

            return

        # found

        o = fb.find(".php", i+len(FindThisJS), i+len(FindThisJS) + MaximumIndexTail)

        if o == -1:

            logfile.error("partial found, ignored, %s", filename)

            return

        # create a new file

        newfile = filename + ".trj.rmv"

        p = open(filename + ".trj.rmv", "wb")

        p.write(fb[:i])

        p.close()

        os.rename(filename, filename + ".xxxx")

        os.rename(newfile, filename)

        os.unlink(filename + ".xxxx")

        logfile.info("Cleaning : %s", filename)        

    except :

        logfile.error("unable to open file %s", filename)

        return

def cleanup (path) :

    for x in dirwalk(path) :

        o = x.split(".")

        if o[-1].lower() in ["asp","html","htm", "aspx"] :

            CleanFile(x)          

            continue

        if o[-1].lower() == "js" :

            CleanFileJS(x)        

if __name__ == "__main__" :

    if len(sys.argv) < 2 :

        print "Removes achtbanen trojan links from all webfiles"

        print "TrjLinkRemove <path>"

        print "e.g TrjLinkRemove c:\\iisroot"

        sys.exit(1)

    if not os.path.exists(sys.argv[1]) :

        print "invalid path given"

        sys.exit(1)

    cleanup(sys.argv[1])    

nowlog.py

import logging, logging.handlers

def SetLogConfig (namefile):

    logfile = logging.handlers.TimedRotatingFileHandler(namefile , 'midnight', 1, backupCount=14)

    logfile.setLevel(logging.INFO)    

    FORMAT = "%(asctime)-15s %(levelname)s:[%(thread)d]:%(message)s"           

    logfile.setFormatter(logging.Formatter(FORMAT))    

    ch = logging.StreamHandler()

    ch.setLevel(logging.INFO)

    Logger = logging.getLogger(namefile)

    Logger.addHandler(logfile)

    Logger.addHandler(ch)

    Logger.setLevel(logging.INFO)    

    return Logger

* sorry about the formatting –.- was using a code snippet plugin

World First Email Server with MSN Robot (Avatar)

This is probably true as the team gets ready to launch MailNow!5

MailNow! 5 is a Windows Mail Server that is of the likes of SmarterMail, Mdaemon, Merak and Hmail. The difference would be the amount of innovation that the team built into the MailNow! 5 that would eliminate log searching, greps etc for mail tracing , anti domain queue hogging and features on accountability that is way off the conventional school.

One distinct one is the world first Email Server Robot or Avatar. Its is an MSN robot that would inform users of the mail server status, new mails alert + details, collect log files, send SMS alerts, compose emails, retrieve emails, provide Email server spam statistics, emails per hour statistics and provide an “eliza” chat for idle users.

Looks really cool when it was demoed today. I guess the folks attending the launch on 12 Nov, 2009  will get some sneak peek too. (www.internetnow.com.my)

If you see any other mail servers adding this feature in the future, you heard it here first. That Malaysian made, MailNow! 5 , the windows mail server…had the first ever robot.

The team won’t be patenting it, the team here is just too engrossed in technical development than to invest in any legal procedures to write such a patent. Chances is MS Exchange would build one super robot in the future and who’s gonna sue?

RoundCube on Windows

While testing the high speed IMAP server that i developed, i wanted to see how it perform with some open source webmail, particularly those that works with IMAP.

One of the most popular one that i found is RoundCube.

1. First you need to install the WAMPSERVER. The php included should be version 5++

2. Open up php.ini and make sure “extension_dir” points to the right directory which is under …\php\ext. Put in the extension module for sql . e.g

extension_dir = "/usr/local/apache/php/ext"
extension=php_mysql.dll

Yeah i created an c:/usr even on my Windows so that i don’t have to much changes.

3. Open up httpd.conf and make sure u have :-

<IfModule dir_module>

DirectoryIndex index.html index.php

</IfModule>

Double check to see if the following is loaded :

LoadModule php5_module "/usr/local/apache/php/php5apache2_2.dll"

4. Install Mysql for Windows. (This can open up one whole can of worms if you get it wrong)

5. Open the INSTALL file found in RoundCube and follow the instruction to create the database for roundcube. I would advice you to stick to the names given.

5. Open the roundcube\conf\, rename the *.dist to *. (whithout dist extension). E.g  db.inc.php.dist –> db.inc.php

Edit db.inc.php and set the  : 

config['db_dsnw'] = 'mysql://roundcube:pass@localhost/roundcubemail'

to the username password that is created in 5, if you followed the default it should be :

config['db_dsnw'] = 'mysql://roundcube:password@localhost/roundcubemail'

Edit the main.inc.php and set the this to where ur email server is :

$rcmail_config['smtp_server'] = '10.8.0.21';

 

6. Now copy this whole roundcube folder to a subfolder under htdocs (apache) . For e.g c:\usr\local\apachce\htdocs\roundcube

And you can try and access it via : http://x.x.x.x./roundcube

Good luck.

Preaching Python : The obstacles

I am a big fan of Python. I am a Windows Developer. I’ve been doing C/C++ as long as i can remember before jumping to C# to leverage on the .NET.

I abandoned Java when it was 2 years old, it just never got me attracted. Maybe its because my field centralizes on system level and security and not enterprise business applications. 

Python has a distinct place in development today. But try convincing the following crowd :-

1. A Windows C#/ASP.NET developer, been using VB since 1997.

2. Java Developer with over 7 years of exp

3. C++ /C hardcore

4. PHP developer

Here are the list of the most common obstacles :

1. Python is SLOW

2. Python has no strong Enterprise Framework

3. Python has weak data typing, its hard to create APIs for users to extend.

4. C# can do everything that Python can and it works better in Windows and..faster.

5. Python GUI development in Windows sucks.

6. see no.1

Python is slow. Compared to  C. Which language is faster than C ? Assembly maybe but definitely not Java or C#.  Does the program speed really matter so much when practically in most applications the speed differences might not even exist to be practical.

Lets say you are developing a Network application that accepts data from the Internet and do something with it. The internet bandwidth speed from your ISP to you is nothing compared to your LAN speed. Lately The CPU processing power has grown from exponential to straight upward beating Moore’s Law.  In this case python would do perfectly well, the bottleneck is not the application speed but rather the internet speed.

One of the exceptions would be if you are doing scientific research and mathematical data of which nanoseconds per iteration differences might matter to you.

Python is slow? Use C then for the part that requires special computational algorithm . Recently a friend of mine showed me a game called Civilization, its the latest version and it uses Python. It runs fast and flawlessly on my Vista. I believe they used python just for the main framework of the game coding while the graphic rendering is back to C/C++ . Now that is smart.

Python is slow? Blink. Did u see the difference?

Windows Live, Twisted

This is the first posting via the Windows Live Writer. Today my MSN Messenger pops up and literally forced me to upgrade my Msgr to the Windows Live version.

What surprises me was the list of other services that you get to choose to install apart from Messenger itself  and one of them is this Windows Live Writer.  It allows you to post to various blogs (the first  one in the menu is MS own service naturally…)

The online office documents is the other apps. This  put the power of the MS Office to the internet for laymen. It makes google docs looks like cavemen offering. The google doc is laden with incompatible glitches when uploading MS documents over, and MS answers to this is a solid one.  Try uploading a powerpoint and you will get what i mean….not a big issue , minor displacement here and there but still…

So what has this gotta do with Twisted ? Nothing really, except that the solid feel you get when using components from MS compared to Twisted (open source component) sometimes is evident, at least to developers that are not just MS bashers. On the other hand, MS is super slow in adopting and promoting what is evidently internet driven standards and if you want to develop for Windows, chances is  that MS already have some offering that is either free, “included in the OS/will be” or some bundling Server products. The development for Windows software is a grim future to many Windows developers (except for project based solutions) . Of course the gang of MS partners and those getting projects from MS as partners via MS solutions are making tons.

For independent product development, Windows leaves Anti-virus, some security related fields and libraries/components the only profitable playing field for Windows Developers looking at “off the shelf” development. 

Now back to twisted, i was  looking at imap4.py, the “search_” functions have a typo bug that will never work , a simple one…missing an “s” on 3 functions of which i have reported to twistedmatrix. If MS components have such bugs, it would be totally unacceptable and laughable.  I was thinking to myself, is this a framework that has so much hoo-haa in Python and over 7 years of evolution? Don’t even want to start on its poor documentation. If its not because of its beautiful skeletal architecture for various protocols, i would say asynch chat in the standard Python library would be more productive.  (noob ! noob!, yeah spare me the insults)

Python is the best  thing i have found in the OSS world. Each day passed by and i still find it the most attractive programming language.  I place Python on the same playing fields as C++, Java, C#, not Perl and other scripting languages. Its meant for serious development and deserved to be used in an IDE and adopted by full fledged teams.

Are you using C#? Try python, use the Wing-ide. You won’t regret! You won’t lose much of the VS pampering that you are so used to and you get to think …somewhat…differently.

Google And Microsoft : Twin Effect

The evidence can be found everywhere. Headlines, headways and in some people's head.
The pattern is recognizable given that one have been around since DOS days.
Google, the company that was once just associated with "search" is no longer staying where its domain is. It has tentacles that reaches far and wide but with one clear difference. The community don't realize it or maybe they do but they sure don't hate it.

Perhaps MS's reputation and brand was so powerfully seared into the hearts and minds of computing folks that we could only identify MS as the only possible gigantic sofware company that we could hate. One that squashes its competitor mercilessly using various "partnership" tactics through the history of IT. Novell...Nescape....scandisk....C++ builder...Word Perfect...Quattro Pro....OS/2......Isn't scandisk just a program, yeah, so was Netscape browser. Sure some of these companies survived but became obscure tech names, a fraction of what they could have been.

Times are changing indeed. Windows Mobile is losing out. Microsoft Live Search is a joke. A victim of its own game in an ever changing world. Its rare to hear MS losing in an area that it plans to focus or profit from but the new kid in town is nothing like MS have seen. Previously everyone had to fight MS in MS own turf...and since that turf is the OS itself, it was rarely a fair fight. Its 90% of the World's desktops dear, get real. But what if you challenge MS in a very unlikely place...the cloud?

When MS IE7 could not open the Google Wave page and Chrome could use it perfectly (ok firefox and other browsers worked somewhat) , you get a very familiar feeling. This creepy Dejavu feeling just jumps out of the page into your lap.

Android phones? Google Doc? GMail for Corporates? GAE hosted domain.

China always had a long history of bad Emperors and those who attempt to topple them. Given such power and position, it was usual for Emperors to behave like morons and do what they want regardless of the citizens sentiments. Heck an Emperor gets to have 600++ wives!

But Chinese History also teaches us one thing, topple one Emperor, the one taking over his place is no better.

Google Wave Hackathon Malaysia

Today i attended the Wave hackathon. Some lucky fellas here are supposed to be the first in Malaysia to get google wave accounts (for the sandbox) .

Nazrul (the founder of MYGTUG) gave an overview and everyone quickly surfed and googled for more info . Basically for most of us, its the first time we are exposed to this Wave thing. This whole idea of blurring the line between emails and chats is boggling but opens up a whole new way of using "emails".

Due to some problems faced by other participants, somehow my "robot" was the only one demo-able at the end of the day. I guess my decision to just use the robot for fun things turns out to be easier to implement, Java programmers also complained about some issues with the api and Wave's unstability in handling certain events.

I went in with Python and i was using WingIde Pro, that must have been a real time saver since the autocompletion and module search gave lots of revelation of how some of the Wave Api works and fixes most easy errors. The problem is that you can't do a local test, so all the code must be uploaded before you can test it , simple errors will be a waste of time.

Here are some "traps" to avoid in creating a "robot" :-

1. If you add a capability to the robot AFTER you already uploaded your first robot, you will need to bump the version in your "whateverrobot.py" under the _main_ so that Google Wave will recognize that you are supporting a new event handling.

Java programmers will need to modify the capabilities.xml file to bump the version.

2. Don't just use the Tutorial example in creating Blips, you will end up modifying the blip

that the user posted instead of creating a new one since that is what the Tutorial example does.

You will get something like "User and robot" in the blip which is probably not what you want, instead just : newblip = root_wavelet.CreateBlip().GetDocument()

You can also try blip.CreateChild() if want it to appear below the original blip.

You must use the Tutorial example to learn and do the first upload though! Just don't

copy everyline when u want to do your actual one.

3. Upload the tutorial example so make sure all your other setup is already working.

You will need to setup a Google Application Engine account and a choice of either

Python or Java. (for Python i was using ver 2.5.4). Setting up Google Application Engine requires a valid and real mobile phone (it will sms a validation code).

4. IE.7 is unsupported!! Yeah, i guess Google is doing to Microsoft what Microsoft used

to do to some companies. So make sure you just stick to Chrome when participating.

Overall, it was a pleasant and enjoyable event.

Here is the robot i wrote (for those who have google wave account) :

Id : 369robot@appspot.com

Have Fun!

Too much support?

I got this screen this morning when searching for something in MS.

Did MS just redirected all support calls to online search? ;-)

C# calling C DLL routines (YET ANOTHER ONE)

I am totally baffled on the examples given by sites on how to call C routines in DLL via C#.
Not only are many of them repeating one same example after another, you soon realize they are not very helpful when you want to do something that is common like passing a structure with all kinds of types inside.

So here is one example that serve as an answer to MANY of the pinvoke/calling C apis problem :- ( i wanted to say 90% but i can't justify or prove it ;-))

I got this example structure that has :

struct ToughCallDef
{
char * ptrName; // this will contain some return string values
int ptrNameLen; // normal integer
MEMBLOCK somestruct; // a structure
ANOTERBLOCK* structpointer; // structure pointer
int* ptrtoMyInt; // integer pointer
char * ptrID // this must be supplied
};

Just translate it this way :-

[StructLayout(LayoutKind.Sequential)]
public class ToughCallDef
{
IntPtr ptrName;
MEMBLOCK somestruct; // you need to define MEMBLOCK also
IntPtr structPointer;
IntPtr ptrtoMyInt;
IntPtr ptrID;
}

And if the call requires a reference to this "ToughCall" :

[DllImport("anotherc.dll")]
public static extern int Call_C_API(ref ptrToughCall);

define it this way instead :

[DllImport("anotherc.dll")]
public static extern int Call_C_API(IntPtr ptrToughCall);

Basically anything to do with Pointers, regardless of whether its integer *, char *, structure reference etc, just use IntPtr.


1 . If the member is a "char *" that you expect return values you point the IntPtr this way :

szReason = new StringBuilder(100);
zReason.Append(' ', 100); // empty string with spaces
ToughCall.PtrName = Marshal.StringToHGlobalAnsi(m_szReason.ToString());

2. If the member is a "char *" that you need to assign a value before passing in :

ToughCall.ptrID = Marshal.StringToHGlobalAnsi(StringVar);


3. If the member is a structure pointer, you assign it this way :

ANOTHERBLOCK o = new ANOTHERBLOCK();
ToughCall.structPointer = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(ANOTHERBLOCK)));
Marshal.StructureToPtr(o, ToughCall.structPointer, true);


4. After calling the C routine you will need to do the following to get the values :-

1. If its a string, use :
   string s = Marshal.PtrToStringAnsi(ToughCall.PtrName)
2. If its a structure use :
   Marshal.PtrToStructure(ToughCall.structPointer, o);
3. Free any allocated strings/struct pointers via :
   Marshal.FreeHGlobal

That's it.

Twitter (python api) on Nokia S60

I wanted a system whereby people can sms to a phone that will log it to twitter.
With this system an organization can keep track of what its support/sales staff are at any given point of time.
After tinkering around with the twitter python api and Nokia s60 python (1.9.6), i finally
got it working...(duh...that took me more than an hour...)

There is however ONE big catch (read end of the section)

Here is the code :

Gotchas :-

1 - Ah yes you also need to change that getusername routine in twitter, else it wont work

2- Pay attention to that pesky pyS60 Application Packager (READ the README file) , this is not py2exe. Basically, you will need to rename your main file to some fix default.py.

3- It will keep asking you for connection (access point) on each msg , unless your using some newer phones like nokia 5800 XM. If you are using N82, ALL the popular examples shown in the internet like these :

http://discussion.forum.nokia.com/forum/showthread.php?t=163939

or

http://snippets.dzone.com/tag/pys60

or

http://croozeus.com/blogs/?p=836

DOESN'T Work. The reason is due to some "compatibility problem" between socket and btsocket.

"Twitter" uses urllib2 not urllib, and these people probably have never tested it on that api. Someone from Silicon Valley once said that Nokia know nuts about promoting development, and thus he rejected their offer to develop for symbian and instead move the whole team to iphone.

In some ways, i agree....but i still love my nokias ;-) I reckon the same thing would have worked in Iphone much earlier and with less blood on my desk.

** Update :

Marcelo Barros from Croozeus :

Yes, that is the problem. I reported it at maemo some time ago.
I suggest you to use my twitter api. Extend it if necessary.
http://code.google.com/p/wordmobi/source/browse/trunk/wordmobi/src/s60twitter.py
It uses urllib and simplejson (I ported it to S60, code in the same dir).

Note : If you still want to use the twitterapi as it is and not the ported version, you will

need to hardcode the "getusername" and stick to "socket" only calls, avoid using btsocket.

C# calling Python scripts and processing output

Occasionally there would be a situation where the C# needs to call a python script and work with its output. While this might sounds like a piece of cake, it does have its gotcha, that is when
you want to process each line one by one and you happened to have somekind of timing (time.sleep) or some other python codes that works correctly under console but unable to be processed by C#.

Below is how you do it : (read until the end, there is a gotcha)

private static void OutputHandler(object sendingProcess, DataReceivedEventArgs outLine)
{
if (outLine.Data != null)
Console.Out.WriteLine(outLine.Data.ToString());
}

static void Main(string[] args)
{
Process p = new Process();
p.StartInfo.FileName = "python.exe";
p.StartInfo.Arguments = "c:\\test\\test.py";
p.StartInfo.UseShellExecute = false;
p.StartInfo.CreateNoWindow = true;
p.StartInfo.RedirectStandardOutput = true;
p.OutputDataReceived += new DataReceivedEventHandler(OutputHandler);
p.Start();
p.BeginOutputReadLine();
p.WaitForExit();
p.Close();
}


Assuming ur python script is the following (test.py) :

import time
import sys

for x in range(100) :
print x
time.sleep(1)


You will be surprised that you dont get anything. For some reason, time.sleep(1) and possibly
some other python library calls would result in the output not being flushed. The solution
would be to "flush" it :

for x in range(100) :
print x
sys.stdout.flush()