A dangerous Pitfall in using cPickle + import

 

Assuming you have a module : myfunc.py

from  defunc_single import *

And defunc_single have a class that is  :

class Hello :

       def __init__ (self) :

               self.name = “hello”

Now if you access this Hello via myfunc.py like this :

o = myfunc.Hello()

cPickle.dumps(o)

and write it using cPickle, the module defunc_single is also pickled into the file.

So that means along the way, if you felt that you wanted a new module : lovefunc_single.py  with a better Hello and you thought that you can just replace the import statement in myfunc.py to :

from lovefunc_single import *

The cPickle will fail  to load the existing data unless you stick to defunc_single.py

Zodb : What you should know.

 

A programmer recently was dismissed for failing to produce satisfactory results after working on a project for 4 months.

He told his employer that “zodb is just too hard”.

Now i have a similar project that i am working on and decided to use Zodb for my storage. As usual i started to google for some quick examples and documentation.

2 weeks into the project i realize that some of my data are not “persisting” as it should while it managed to fool the unit test . I wrote some small single modules to simulate the situation and realized that the same data are not persisting even though another variation of the same code does.

To cut the story short, if you are doing or intend to use Zodb as your storage, just follow the rules below, forget whatever “friendly” samples you have seen, they just don’t reflect the issues that can arise :

1. Any classes that you store in the Zodb, derive it from “Persistent”.

Eg :

from persistent import Persistent

class Whatever (Persistent) :

2. Any dictionary or List that you want to store in Zodb, don’t bet on your luck, use the following wrappers.

If you want to store dict, use PersistentMapping

If you want to store list, use PersistentList

from persistent.mapping import PersistentMapping

from persistent.list import PersistentList

root[secret_key] = PersistentMapping()

root[secret_key][“mykey”] = “yada”

3. The connection and root must be opened by the thread that wants to use it or only use 1 single thread

storage     = FileStorage(Path)

db          = DB(storage)       
            
connection  = db.open()
root        = connection.root()

while “storage” and “db” are shareable across threads, “connection” and “root” are not. Each new thread must do the “db.open()” and “connection.root()” call.

Alternatively you can assign one single thread to do all the above opening of handles and access the “root” and “connection” freely among multi-threads, but you must ensure that “single thread” you assigned will be the same one closing the database.

Irresponsible Sharing

Its pretty frustrating sometimes when you want some quick scripts to do your job and some of the given scripts are not only mediocre but non functioning.
Today i had the urge to quickly backup a directory i have to a remote ftp, so i figured that surely someone must have written a python script on this! After 30 minutes of searching and looking around, i found :
http://code.google.com/p/ftpsync2d/downloads/detail?name=ftpsync.py

But guess what, the code don't work on Windows. The bugs you get from it are very typical of newbie who have never coded in dual platform. For eg, it expect the path to be in "/" where else when it ran on windows it automatically converts to "\" and it bombed out on assertions.

Dumb? No. Just typical and if i knew i gonna spend 30 minutes to do this i might as well fire-up my FTP client and do it.

Gee guys/gals, if you want to post some codes and scripts on the Internet for the community, please at least raise its quality to an acceptable level. We have enough junk codes shared, we don't need any more of it, or the least, label the project that it only works on Linux/MAC/Windows.

twisted.web : rabbit’s hole

Finally i surrender. I switched one of my project totally off Twisted web. Using Twisted web is way too much work and pitfalls.

The only way to use Twisted web is that you use it in the very first place.

If you are starting a project involving building a simple http server, just avoid twisted .

Once again…Tutorial for poor souls : C# Windows Service

 

Looking at some of the tutorials online, i am once again baffled on how ridiculous “google” results are on creating Windows Service for C#.

I wonder where these people get the ideas from posting such advice and tutorial that hardly works but is confusing. If you refer to my other rants on multi-language .NET and calling C dll you will see what i mean.

Anyway here is how u do it. If you find reading it confusing, take a glance at the snapshots below.

1. I assume you are using VS 2010 and C#

2. Start a new project, select windows Service.

3.You will see 2 class, click on the tab with the word “designer”, right click “properties”.  Select “Add Installer”

4. You will get ProjectInstaller.cs. Click on the tab with “ProjectInstaller.cs [Design]” and click on the “serviceinstaller1” and on the lower right “Solution Explorer” you will see the properties. Set the “ServiceName” to any name u want to be used as the service name (the Service Name that appears when u right-click on any services in the services control panel properties). The service name here is the one that will be used not the one you see on the other service class that was created when u created this project (in 1st screen shot, the properties u see there, that service name is not the one that will appear or use in the actual services)

5. Set the Display Name to the name you WANT to see shown in the list of services in the Services control panel (services.msc)

6. Click on “ServiceProcessInstaller1” and Set the “Account” to the LocalSystem. If you don’t do this, you will get some stupid pop-up asking you to key in the username and password to use for running this service as that user.

7.Now all you need to do is to fill up your actual Service1.cs’s “OnStart” and “OnStop” with whatever routines you want to run and stop as a service.

8. Copy Installutil.exe from c:\windows\microsoft.net\framework\v2.0xxxxx to c:\windows

9. Run the command prompt / cmd as “Administrator” and goto where your .EXE is built and run “installutil xxxxx.exe” to install it.

 

 

 

Twisted IMAP4.py is long way from practical use

 

I have been trying to get imap bodystructure working to please all the clients i could get my hands on. So far the twisted imap (10.1.0) performed badly esp when you need it to work with Outlook + Outlook Express + ThunderBird + IPhone + RoundCube.

Yeah, Imap as i mentioned many times even in Pycon Singapore, is a hell of protocol of email. Many flags like “recent” of “draft” are just outdate and should not even be used…

JP Calderone wrote in the source :

# XXX - This does not properly handle multipart messages
# BODYSTRUCTURE is obscenely complex and criminally under-documented.

I had to resort to write an imap unit test that will test the reply from 2 servers and compare them and make changes from there to fix the imap4 bodystructure response.

The unit test tool helped a lot, i had to tweak various replies, for eg handling application attachment when they appear in mime attributes.

@ I got it working correct and tested with all the clients mentioned above with all kinds of attachment and sizes.

Source: (a fixed imap4 reply)
('OK', [('3 (BODY[HEADER] {513}', 'User-Agent: Thunderbird 2.0.0.24 (Windows/201
00228)\r\nTo: Undisclosed <marcus@internetnow.com.my>\r\nSubject: marcus\r\nRetu
rn-Path: <marcus2@internetnow.com.my>\r\nReceived: from [192.168.17.1] ([127.0.0
.1]) by Tool-Box.internetnow.com.my (MailNow! 5); Sat, 16 Oct 2010 19:50:55 +080
0\r\nMime-Version: 1.0\r\nMessage-Id: <4CB95DC2.3040201@internetnow.com.my>\r\nF
rom: Marcus <marcus@internetnow.com.my>\r\nDate: Sat, 16 Oct 2010 16:09:38 +0800
\r\nContent-Type: multipart/mixed; boundary="------------03060008020007080806070
7"\r\n\r\n'), (' BODY[1.MIME] {100}', 'Content-Type: text/plain; charset="ISO-88
59-1"; format="flowed"\r\nContent-Transfer-Encoding: 7bit\r\n\r\n'), (' BODY[2.M
IME] {200}', 'Content-Type: application/vnd.ms-excel; name="04June2010 - Problem
&  Solution.xls"\r\nContent-Transfer-Encoding: base64\r\nContent-Disposition: i
nline; filename="04June2010 - Problem &  Solution.xls"\r\n\r\n'), ')'])

Dest: (reply from SmarterMail, a ‘correct’ server)
('OK', [('6 (BODY[HEADER] {769}', 'Return-Path: <marcus@internetnow.com.my>\r\nR
eceived: from 132.115.in-addr.arpa [115.132.75.110] by hs4.thehosting2u.com with
SMTP;\r\n   Sat, 16 Oct 2010 19:56:44 +0800\r\nReceived: from [127.0.0.1] ([127
.0.0.1]) by Tool-Box.internetnow.com.my (MailNow! 5);\r\n    Sat, 16 Oct 2010 16
:09:43 +0800\r\nMessage-ID: <4CB95DC2.3040201@internetnow.com.my>\r\nDate: Sat,
16 Oct 2010 16:09:38 +0800\r\nFrom: Marcus <marcus@internetnow.com.my>\r\nUser-A
gent: Thunderbird 2.0.0.24 (Windows/20100228)\r\nMIME-Version: 1.0\r\nTo: Undisc
losed <marcus@internetnow.com.my>\r\nSubject: marcus\r\nContent-Type: multipart/
mixed;\r\n   boundary="------------030600080200070808060707"\r\nX-MN-SPAM-DTL: s
pam=0, scr=0, ver=\r\nX-Rcpt-To: <marcus@internetnowasp.net>\r\nX-SmarterMail-Sp
am: Bayesian Filtering, SPF_None\r\n\r\n'), (' BODY[1.MIME] {100}', 'Content-Typ
e: text/plain; charset="ISO-8859-1"; format="flowed"\r\nContent-Transfer-Encodin
g: 7bit\r\n\r\n'), (' BODY[2.MIME] {208}', 'Content-Type: application/vnd.ms-exc
el;\r\n   name="04June2010 - Problem &  Solution.xls"\r\nContent-Transfer-Encodi
ng: base64\r\nContent-Disposition: inline;\r\n   filename="04June2010 - Problem
&  Solution.xls"\r\n\r\n'), ')'])

Windows Service : Mystery of the Slow startup

I have this .NET service application that is always reported in EventViewer to be “not responding” or takes too long to start.

The problem occurs only when a reboot is done. If the service was stop/start after reboot the speed is normal but if it starts automatically after reboot it takes around 90 seconds just to get it started.

The app main task is just start a webserver and connects to it.

So here is the condition :

Starting the service manually after the reboot = fine

First time auto starting after reboot = 90 seconds delay and reported to be “not responding” or hang, but runs fine .

I have nailed the culprit to WebClient!  Thats right, using WebClient in your . NET code as a service will result in it starting up slowly, this might have to do with the possibility that System.Net.WebClient uses the shared components  of IE and also its host of registry entries. This is SUPER SLOW when you reboot the first time.

Solution : i wrote my own TCP code to mimic WebClient and discard using WebClient in a Service app.

Case  close.