C# calling Python scripts and processing output

Occasionally there would be a situation where the C# needs to call a python script and work with its output. While this might sounds like a piece of cake, it does have its gotcha, that is when
you want to process each line one by one and you happened to have somekind of timing (time.sleep) or some other python codes that works correctly under console but unable to be processed by C#.

Below is how you do it : (read until the end, there is a gotcha)

private static void OutputHandler(object sendingProcess, DataReceivedEventArgs outLine)
{
if (outLine.Data != null)
Console.Out.WriteLine(outLine.Data.ToString());
}

static void Main(string[] args)
{
Process p = new Process();
p.StartInfo.FileName = "python.exe";
p.StartInfo.Arguments = "c:\\test\\test.py";
p.StartInfo.UseShellExecute = false;
p.StartInfo.CreateNoWindow = true;
p.StartInfo.RedirectStandardOutput = true;
p.OutputDataReceived += new DataReceivedEventHandler(OutputHandler);
p.Start();
p.BeginOutputReadLine();
p.WaitForExit();
p.Close();
}


Assuming ur python script is the following (test.py) :

import time
import sys

for x in range(100) :
print x
time.sleep(1)


You will be surprised that you dont get anything. For some reason, time.sleep(1) and possibly
some other python library calls would result in the output not being flushed. The solution
would be to "flush" it :

for x in range(100) :
print x
sys.stdout.flush()

Vmware and the Virtualization Gotcha's

There is this lab that has 4 different Win2k servers, each running a different application.
Server A - Mailserver
Server B - Intranet application with MS SQL db
Server C - Vpn server and backup
Server D - Test server

Along the way, the admin decides to put all these dated servers into a VM and slot it in to a high end Server from Dell. The whole porting process took around 2 weeks and when it was over,
everyone was happy with the new setup. No more additional switching cables and multiple monitors lyring around and the perceived energy savings cost was a bonus.

However one day, the mailserver began to feel very slow, Procexp (sysinternals) itself was myteriously taking up 45% cpu and more and even with the mailserver service stopped the pc still feels awkwardly slow. Rebooting didnt that mail server VM doesn't help either.

When i came in to help out in this scenario, the first thing i went thru was the list of VMs running in the server. One particular VM is taking up 26% of the CPU of the main server, however that should not be the reason why it would affect the mail server VM. Upon closer inspection however, i notice some native apps running on the test machine VM (26%) that is using up the VM's tcp/ip port very quickly.
It then became logically clear that this was the problem, pausing that VM immediately restored the other server's performance and that was like a 100% improvement.

I suspect the problem is because the main server is still just an OS with the normal limitation of the 65535 ports on a single ip and single network card. Since all the VMs runs on this machine , that test application was just blasting away the network resource, this turns the allocation of the real main server NDIS packets resources into an ugly situation where each of the VMs are queueing up to get its allocation.

Another case solve. Another Hamster Huey and the GooiKablooi award for virtualization.

Linux Desktop And Windows 7

I must admit, Ubuntu has reconfigured my view towards the Linux desktop distro. Yes, its not something new, but from version 8.1 onwards since last year, the refinement is just slightly short of a usable desktop for common users.

Most of the stuffs are similar to windows in that you have hardware detection app running in the background with an icon in the system tray. The wifi detector works its best in trying to minimise "geeky" configurations and in my test notebook DELL inspiron 1520, it works straight out without the need for me to configure it manually and tweaking the config files.

With the common host of free apps like openoffice and email clients (Evolution - the email client is still lacking in refinement compared to Outlook Express but Thunderbird isn't), heck, for most Janes and Johns who just want to have email, internet surfing and office documents, this would feel right at home. The system works faster and you even feel "safer" without having any anti-virus, anti-malware, anti-adware, malicious removal tools , Windows Defender and all those hooking apps that layers itself one on top of the other.

Nine months ago, when i tried OpenSuse on my other notebook , it was hell, i had to configure and download specific wifi patch just to make it work. If a common user would have to choose between OpenSuse and Windows OS, even Win 98 would be a better option (speaking strictly in terms of usability for users who want things to just work and not do any command lines, scavaging forums and pulling hairs)

Ubuntu, rightly one of the most popular distro now, deserves an applause from the Linux community. Instead of sticking to the unspoken rule of "we can't copy microsoft, if you can't configure some simple scripts you are lame", it went straight on to make Linux a much friendlier distro for desktop. By providing an option without the need to repartition the HD and installing it side by side with Windows XP/Vista, it became an attractive option for users to try it.

The introduction of Version 8.1 was nothing short of perfect timing too. Vista, being an infamous slow OS - thanks to the "dev team" that insists on working on the new approach and forsake previous compatibility.

The fact is, Vista is slow. There is just something wrong with its design.
Personally i do not use any anti-virus, instead i even ran defrag religiously and PerfectDisk periodically to make sure i have optimum speed for my HD files. Still it was slow, i made sure my services are configured correctly and disabled all that i can lay my hands on and it is still slow. The frighthening thing was the fact that my Vista ran 3 times faster than any other Vista around my office on similar hardware, yet i find the response unacceptable.

Ubuntu on the other hand, ran seemingly faster even when it was installed using the side by side installation. Thanks to Vista, I believe tons of users flock to try Linux distro and many who happened to try Ubuntu would be positively surprised. Just this 2 weeks, i notice that there are 2 more collegues who installed Ubuntu and was excited to show it around.

However, there are still many hurdles for Ubuntu to overcome before it can become a replacement for Windows. For e.g, try connecting it to a projector, and good luck.
Try getting Compiz to work, chances are that you won't get it working correctly in the first 30 minutes and oh, pls update your Nvidia drivers else be prepared to do some file editing.

To date, there are around 200 Linux distros, some says 450. This is actually good news to Microsoft. As long as these jokers cannot unite, Windows will continue to dominate. Every aspiring Linux techie with enough willpower will want to produce his/her own Distro with some lousy excuse for yet another distro.

I would bet that if the community would drop all the similar distros and say just concentrate on 3 :

1. CentOS or RedHat for Server
2. Ubuntu for Desktop / Workstation
3. Knoppix for small storage / misc applications

Or better :
Ubuntu Desktop, Ubuntu Server, Ubuntu Mini.

Then all the big hardware manufacturers and the gaming industry would ensure that they will provide support for this alternative OS. Ms.Window would have to taste its own medicine in pitting against FREE OS. Yup, taste its own medicine. The number of companies that MS brought down just by including the said product/feature FREE are calling out for judgement day.

Netscape, RIP.

The only other area that MS still have not managed to kill are the Anti-Virus companies. Free Anti-virus seems to be playing along MS agenda all the time and yet failed to be executed. Hmm, before this happens, i should seriously fold up my sleeves and start working on this project i have in mind for some time now, the Corporate-Anti-Malware-End-Of-Anti-Virus project. A solution that will guarantee a corporate environment to be clean of viruses using a filter driver and a concept that involves a model pc, central sandbox and whitelisting. No more antivirus scanning and signature subscriptions!


Yawn....signing off.

Virtual Desktops and Cubes on Vista : Compiz, Ubuntu

Recently i been playing with Ubuntu 8.1 and quite fancy the desktop cube that was
available in the Compiz.

If you look at google, there are many Linux users that posted cool videos of this "virtual desktop" and its effects, the comments ranged anywhere from MS bashing to degrading insults to people who chooses to use Windows/Vista instead.

Well here are the good news.

You can download a cheap (Free) and fast virtual desktop manager for Vista here :
http://technet.microsoft.com/en-us/sysinternals/cc817881.aspx

It allows you 4 desktop that you can switched via ALT-Num, which is a good hotkey.
Its only 68k, that shows how little work is required to have such functionality in Windows.

"How about the cube! We want the cube!!!"

The cube is available here, you have two options :-

One that is not free :
http://www.otakusoftware.com/deskspace/index.html

One that is free :
http://www.docs.kr/entry/Download-Shock-4Way3D-en


Have fun!

In hardware we put our trust....

This morning i got a call from a customer. Running a powerful custom VPN solution (ipsec) for Windows that i wrote with multi-branch and failure switchover. He was just upgraded to a new version just few days back and now he complains that it hangs the server.

The symptoms are :
- server unresponsive and hangs periodically
- it even hangs while trying to reboot

After we turn on the VNC and remote in, i couldn't find anything wrong with the system. The guy on the other side then exclaim "How did you manage to login? I can't even get the CTRL-ALT-DEL to work". It was then obvious that i could login by asking VNC to send the CTRL-ALT-DEL, so i queried him further and the problem was actually his Keyboard.

An erratic hardware failure on the keyboard (capslock, numlock not responding) gives you the impression that the Server has hung.

This event got me thinking. Why do people always associate failures with Software and not hardwares when a problem first surfaced? I believe the reason lies much to the way sofware are produced compared to hardware. Sure, there are good programmers and bad programmers, they are also good manufacturers and bad manufacturers. But the key lies in the fact that
"bad manufacturers" still have to adhere to certain quality and the fact that creating a hardware requires much resources and after thought before committing to one.

Hardware specifications are dead static and factories that produce it must have certain quality check. If you are planning to produce a cheap alternative graphic card to the market, you will still pay a lot of attention to some quality and testing cause its going to cause you a bomb to start a factory on it. The same goes for keyboard, mouse, webcams and the whole list of hardwares that we associate with the PC. Yea sure, Logitech wares that are made in china have lousy fittings and cheap plastics but it still works as advertised.
Electronic components are "reusable code" at its' highest form. Nobody builds a hardware from scratch, they select the components that are already widely available in the industry and just design its own mold and maybe a few specific components.
Take the Razer mouse for example, the optical component and the RAM are manufactured elsewhere i am sure, while the casing, the scroll wheel and buttons are Razer's design.

Software on the other hand, much of its true value are hidden from the naked eyes. How a particular function is done varies from each programmer and so are its reliability. Much of what users perceive are just GUI while the core engine could fail in a thousand ways.

This brings me back to the Demmings 14 points some believe are the reasons why Japanese goods are viewed as high quality and reliability, it should be practiced in Software manufacturing as well.

Demming's 14 point :-

• 1."Create constancy of purpose towards improvement". Replace short-term reaction with long-term planning.

• 2."Adopt the new philosophy". The implication is that management should actually adopt his philosophy, rather than merely expect the workforce to do so.
  

• 3."Cease dependence on inspection". If variation is reduced, there is no need to inspect manufactured items for defects, because there won't be any. - (Unit test?)
  

• 4."Move towards a single supplier for any one item." Multiple suppliers mean variation between feedstocks. -(Oh dear, not Microsoft pls...)
  

• 5."Improve constantly and forever". Constantly strive to reduce variation.
  

• 6."Institute training on the job". If people are inadequately trained, they will not all work the same way, and this will introduce variation.
  

• 7."Institute leadership". Deming makes a distinction between leadership and mere supervision. The latter is quota- and target-based.
  

• 8."Drive out fear". Deming sees management by fear as counter- productive in the long term, because it prevents workers from acting in the organisation's best interests.
  

• 9."Break down barriers between departments". Another idea central to TQM is the concept of the 'internal customer', that each department serves not the management, but the other departments that use its outputs.
  

• 10."Eliminate slogans". Another central TQM idea is that it's not people who make most mistakes - it's the process they are working within. Harassing the workforce without improving the processes they use is counter-productive.
  

• 11."Eliminate management by objectives". Deming saw production targets as encouraging the delivery of poor-quality goods.
  

• 12."Remove barriers to pride of workmanship". Many of the other problems outlined reduce worker satisfaction.
  

• 13."Institute education and self-improvement".
  

• 14."The transformation is everyone's job".

Ldaptor, Python List, Django

LDAPtor - The king who never made it

I wanted to deploy ldap server on windows. Apache DS with a 200 simulated user environment went kaput and need frequents restarting/backup. Openldap on Windows via the berkeley db was prone to corruption as well. Nothing seems to work robustly under Windows for some of these open source projects, no matter how popular it was. 

Then i found Ldaptor, written completely in python via the Twisted framework. Wasn't really a big fan of the Twisted framework, heck, the only book on Twisted i saw by Oreilly, had examples that are "badly" coded in my personal opinion, works though. One can't doubt the benefit of having such a large ready made framework. 

After just 5 minutes trying out ldaptor, it bombs out, won't even run, turns out that some modules are referencing the previous Twisted "components.interface" and should be replaced with "zope.interface.Interface", then it runs. Now just have to figure out how to import the Ldif into it...documentation seems to be scarce on ldaptor..."Ldaptor will one day rise to world domination" 

Thats the problem with a LOT of open source projects, abandoned or left dying prematurely when the Initiator found something more profitable to do...

> Switch Channel ....

I am really grateful for some of python default implementation and behaviour, especially on its container. 

I wanted to get a list of files in a directory and process it according to its "creation time". i did a "glob.glog" on a directory, the return result  was anything but sorted.  The normal procedure would be something like this :-

1. get each of the files creation time

2. sort based on the creation time

3. process the file

In python that translates to :-

ActualFiles = [(os.stat(x).st_ctime,x) for x in GetFilesListing(path))]

ActualFiles.sort()

for x in ActualFiles : 

# do something with already sorted x[1]

When i wanted to remove some items from a list and yet continue to process the list :

for x in MyList[:] :

if x.value == "whatever_i_am_looking_for" :

MyList.remove(x)

If this was translated  to C++ it would be something like this  (warning: it won't work!)

for (ptr = listcopy.begin(); ptr != listcopy.end(); ptr++)

{

if  (ptr->value == "whatever_i_am_looking_for") 

{

listorg.erase(ptr)

}

}

Thus, someone who starts on python would probably find it hard to move to C/C++ but

someone who starts from C/C++ would  easily addicted to python and appreciate its fun way of working (lazy).

> Django

I wonder how does Python web building such as Django matches against premium platforms like ASP.Net. Just the other day some people working on PHP was amazed to see how building cool working websites was done in ASP.NET. All those ajax coding are so automated it makes similar offering in PHP looks primitive not to mention the state of the art Visual Studio IDE.

Frankly developing website is something that is out of my reach but it would be great to see how Python fare in this area. Better than PHP? Ruby on rails? Come on, it has to do better. 

 

Programmer's Mind vs None Programmer

There are basically two types of jokes among the elite programmers when it comes to people who are non technically inclined and non-programmers.

#1 :

Sometimes people in Malaysia here thinks that they can just pump in a few million, start a team and expect to get a project done. Little do they know of the various traps and software pitfalls that awaits such a team. Unless its a crony project that don't mind accepting substandard deliveries, most of these projects often end up in the drain. Sure there are some big companies here that hire top Chief architects whom never deliver any projects to a real competition, all inhouse or cronies companies deliveries. Little do these self deceived Architects are aware of how weak their skill and proficiency are in that area. When you are the Chief Architect of such big corporations, everyone wants to kiss ur ass, no matter what ur true capabilities are.

This phenomena is so common that over here, the local folks have a name for it. Its called "Jaguh Kampung". In english it means "Village Champion". What happens when a few village champions join an international competition? They learn the meaning of "Village".

Each year in local software Awards here we see various Sofwares being conferred the "Software of the year", often these Software are either developed for its own use or for its branches. If you look at the track record, almost all the winners have ran out of business or no longer using it.

#2 :

There are Sales and Business people who don't know how dumb they look when saying something to the technical team or when accepting a business proposal whom they know nothing of in programming terms. When the technical team rejected the proposal, they have the impression that their team is incapable or does not have business sense, where in actual fact, the 

problem is in the mirror.

Someone i knew once asked me to write a local chat software, i asked him "what is the sales forcast?" He replied "well i am sure there are many, let me call up this ABC company and see if they want it". (This is a clear sign to "reject" the idea, no proper thoughts given to the project and just do a blind jab by asking some programmers to commit to it).

Anyway, to deter him i ask him further "what are the specifications?". He replied "Oh, its very simple just do exactly what Skype can do and make it work in LAN".

** If you are not laughing already, you should stop reading this post.

Another story i heard is about this lady, who has many connections and "cables" to VIP people. Often she would try to get some projects from these people and her technical partner would reject it. She would then be very upset and gave some lectures on "How important is business and why nothing is impossible to code". A very sad partnership that is destined to end up in the gutter. What most of these "business" people don't understand is that, What you see on paper and in request is VERY different when its translated to actual coding. It can be so different that its actually supernaturally unrealistic to take it up. For example would you commit a team to write a product in 6 months and due to economics of scale, you would need to pay USD 1000 for an appliance/embedded system while out there, there are similar products retailing at USD 400 that is doing exactly the same thing?

If you have such a boss or partner , you should use this statement to bring enlightenment to him/her by saying "Actually if we want to make money we should just get a team to write an OS what works exactly like Vista but with a budget of USD 100k." or "Lets write a Windows 2008 replacement in 6 months, that sounds easy too isn't it?" or "Lets write a replacement for MS-Exchange and make it cooloer in 6 months"